three essays on the economics of education miguel e. martinez
TRANSCRIPT
Three Essays on the Economics of Education
Miguel E. Martinez
Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy
under the Executive Committee of the Graduate School of Arts and Sciences
COLUMBIA UNIVERSITY
2017
2
Copyright 2017
Miguel E. Martinez
All Rights Reserved
3
ABSTRACT
Three Essays on the Economics of Education
Miguel E. Martinez
Essay 1: Determinants of NCLEX-RN Success Beyond the HESI Exit Exam: Performance in Nursing
Courses and Academic Readiness
Abstract: In this study, I examine whether demographics, pre-college academic readiness
measures, and performance in nursing courses improve the correct identification of students who
will pass/fail the NCLEX-RN above and beyond the HESI exit exam scores. I find that their inclusion
can improve the identification of those who will fail but not those who will pass.
Essay 2: The Impact of Remediation on NCLEX-RN Success: Positive, Neutral, or Negative?
Abstract: In this study, I use two nationally representative samples to explore the impact of
required remediation on passing the NCLEX-RN on the first attempt using regression discontinuity
design both as local randomization and as continuity at the cutoff using. As the former, I find
some evidence that remediation has a negative impact on passing the NCLEX-RN on the first
attempt. As the latter, I find limited evidence of positive treatment effects.
Essay 3: Testing a Rule of Thumb: For STEM degree attainment, More Selective is Better
Abstract: In this essay, I test the rule of thumb that, for STEM students, attending a highly
selective institution instead of a moderately selective institution improves the probability of
obtaining a STEM degree. Overall, I find that highly selective institutions have a comparative
advantage in producing STEM graduates among those already interested in STEM but not among
those initially not interested in STEM.
4
This page intentionally left blank
i
Contents List of Figures………………………………………………………………………………………………………………………………………… iii
Acknowledgements ...................................................................................................................................... iv
Chapter 1: Determinants of NCLEX-RN Success Beyond the HESI Exit Exam: Performance in Nursing
Courses and Academic Readiness ................................................................................................................. 1
1.1 Introduction ........................................................................................................................................ 1
1.2 Review of the Literature ..................................................................................................................... 4
1.3 Data ..................................................................................................................................................... 6
1.4 Organizational Context ....................................................................................................................... 7
1.5 Methodological Approaches ............................................................................................................. 11
1.6 Results ............................................................................................................................................... 14
1.6.1 Explanatory Models ................................................................................................................... 14
1.6.2 Out-of-Sample Predictive Models .............................................................................................. 18
1.6.3. Severe Error Rates ..................................................................................................................... 21
1.7 Conclusion and Further Research ..................................................................................................... 24
Chapter 2: The Impact of Remediation on NCLEX-RN Success: Positive, Neutral, or Negative? ............. 26
2.1 Introduction ...................................................................................................................................... 26
2.2 Review of the Literature ................................................................................................................... 30
2.2.1 HESI Exit Exam ............................................................................................................................ 33
2.3 Data ................................................................................................................................................... 34
2.4 Identification Strategy....................................................................................................................... 36
2.5 The Assumptions of Regression Discontinuity are Met .................................................................... 37
2.5.1 Manipulation of the Running Variable: Visual Evidence ........................................................... 38
2.5.2 Manipulation of the Running Variable: Statistical Evidence ..................................................... 39
2.5.3 Continuity of Pre-treatment Variables: Statistical Evidence ..................................................... 40
2.5.4 Covariate Balance: Statistical Evidence ..................................................................................... 40
2.6 Results ............................................................................................................................................... 42
2.6.1 Estimation of Treatment Effects: RD as Discontinuity at the Cutoff ......................................... 42
2.6.2 RD Estimates Conclusion and Discussion ................................................................................... 43
2.7 Robustness Checks ............................................................................................................................ 43
2.7.1 Logistic Regression ..................................................................................................................... 44
ii
2.7.2 Propensity Score Matching Analysis .......................................................................................... 46
2.8 Looking for Heterogenous Treatment Effects................................................................................... 47
2.8.1 Logistic Regression ..................................................................................................................... 48
2.8.2 Propensity Score Matching ........................................................................................................ 48
2.9 Conclusion and Policy Implications ................................................................................................... 49
Chapter 3: Testing a Rule of Thumb: For STEM Degree Attainment, More Selective is Better ................ 52
3.1 Introduction ...................................................................................................................................... 52
3.2 Competing Definitions of STEM ........................................................................................................ 55
3.2.1 STEM as a Career........................................................................................................................ 55
3.2.2 STEM as a College Major ............................................................................................................ 57
3.3 Review of the Literature ................................................................................................................... 58
3.3.1 Interest in STEM Careers/Majors ............................................................................................... 59
3.3.2 College Choice ............................................................................................................................ 61
3.3.3 Matching in the College Choice Process .................................................................................... 62
3.3.4 STEM Attrition and Institutional Selectivity ............................................................................... 66
3.4 Data ................................................................................................................................................... 67
3.4.1 Defining Interest in STEM Interest in ELS of 2002/06/12 .......................................................... 68
3.4.2 Defining a Highly Selective Post-Secondary Institution in ELS of 2002/06/12 .......................... 69
3.4.3 Defining a STEM Degree in ELS of 2002/06/12 .......................................................................... 69
3.5 STEM Students .................................................................................................................................. 70
3.6 Methodology ..................................................................................................................................... 72
3.7 Results ............................................................................................................................................... 75
3.7.1 Logistic Regression Analysis ....................................................................................................... 76
3.7.2 Matching Analysis ...................................................................................................................... 77
3.7.3 Summary of Results ................................................................................................................... 79
3.8 Conclusion ......................................................................................................................................... 80
Works Cited/Consulted ............................................................................................................................... 81
Appendices .................................................................................................................................................. 92
iii
List of Figures
Figure 1. Predicted Average Marginal Probabilities of Passing the NCLEX-RN by Foundations I Course
Grade........................................................................................................................................................... 17
Figure 2. Predicted Average Marginal Probabilities of Passing the NCLEX-RN by Health & Illness ICA
Course Grade. ............................................................................................................................................. 17
Figure 3. Predicted Average Marginal Probabilities of Passing the NCLEX-RN by Psych/Mental Health
Course Grade .............................................................................................................................................. 18
Figure 4. Predictive Validity of Models (1) and (3) and Current Progression Rule based on HESI Exit Exam.
.................................................................................................................................................................... 19
Figure 5. Possible Impacts of HESI exit exams scores on ability to sit for the NCLEX-RN. ......................... 33
Figure 6. Distribution of HESI Exit Exam Scores by Required Remediation Status ..................................... 39
Figure 7. Remediation Marginal Effects over the HESI Score Response Surface ....................................... 45
iv
Acknowledgements To my dad.
I would like to express my sincere gratitude to Thomas Bailey and Peter Bergman who
oversaw my work during the research proposal stage and writing of the dissertation. The
insightful comments of the defense committee greatly improved the quality of my work. I am
very thankful to Luis Huerta, Aaron Pallas, and Clyde Belfield.
The dissertation would not have been possible without the help of Rutgers School of
Nursing. I would like to thank Bill Holzemer, Jeannie Cimiotti, Mary Johansen, Edna Cadmus,
Pam de Cordoba, and John Runfeldt for their encouragement and support during the dissertation
process.
To the teachers and instructors who inspired and influenced me along the way: Mr.
Reeves, Mrs. Williams, Mrs. Schaeffer, Dr. Dorce, Dr. Haynes, Dr. Rumiano and Dr. Rivera-Batiz.
To my parents for the love and affection they showered upon me since my birth and for
fostering the desire to understand. To my siblings for their example of hard work and
perseverance in their professional careers and for their unwavering support in all my endeavors.
Most of all, to my wife who supported my academic studies every step of the way. To
my children who generously gave up their time with me so I could push the dissertation
forward.
1
Chapter 1: Determinants of NCLEX-RN Success
Beyond the HESI Exit Exam: Performance in
Nursing Courses and Academic Readiness
1.1 Introduction
First time NCLEX-RN (National Council Licensure Examination for Registered Nurses) pass
rates play a significant role in nursing programs. National accrediting bodies like the Commission
on Collegiate Nursing Education (CCNE) and the National League for Nursing Commission
for Nursing Education Accreditation (CNEA) require minimum first-time NCLEX-RN pass rates of
80 percent to maintain accreditation. These standards must be met by all pre-licensure
programs (Generic Baccalaureate, 2nd Degree Accelerated, pre-licensure Masters in Nursing
Science) and campuses (home and satellite) within an institution. Failure to meet the standard
risks the loss of accreditation and/or placement on probationary status (CCNE, 2014) (CNEA,
2015). Nursing programs must also meet the NCLEX-RN pass rate standards established by their
state boards of nursing which may be higher than those established by national accrediting
bodies.
First time NCLEX-RN pass rates have a marked signaling function. For prospective
students, high pass rates suggest a high-quality nursing program (Norton et al, 2006). Thus,
improving first time pass rates can increase the quality of applicants and enrollees which can
further improve NCLEX-RN outcomes while prolonged decreased performance can have the
opposite effect (Beeson and Kissling, 2001). For administrators within and outside of nursing
2
decanal units, first time pass rates are often perceived as an indication of the quality of
instruction (Giddens et al, 2009; Taylor et al, 2014).
Thus, nursing programs have a strong incentive to identify students who are likely to fail
the NCLEX-RN. Once identified, administrators may assign students to remediation or may
prevent them from taking the licensure exam (progression). Typically, nursing programs identify
at risk-students based solely on the results of exit exams like the HESI (Health Education Systems,
Inc.), ATI (Assessment Technologies Institute), ERI (Educational Resources Inc), or NLN (National
League of Nursing).
Eleven validation studies have found that the HESI exit exam (E2) is an accurate predictor
of passing NCLEX-RN - National Council Licensure Examination for Registered Nurses on the first
attempt (Adamson and Britt, 2009; Launcher et al, 1999; Langford and Young, 2013, Lewis, 2005;
Newman et al; 2000, Nibert and Young, 2001; Nibert et al, 2002; Young and Willson, 2012;
Zweighaft, 2013; Schreiner at el, 2014; Zweighaft, forthcoming). E2 results have been used to
gauge student readiness for successful entry into the nursing profession (Pennington et al, 2010).
Yet the sole use of the examination for progression or remediation may not be justified since it
has predictive power for NCLEX-RN success but for not NCLEX-RN failure (Spurlock, 2006).
Students scoring above 850 on the HESI exit exam may be more likely to pass the NCLEX-RN than
those who do not but the HESI exit exam is not able to identify those who score above 850 but
do not pass the NCLEX-RN.
Nationally, approximately 97 percent of students who have scored 850 and above on the
HESI have passed the NCLEX (Young & Willson, 2012; Lewis, 2006). Not meeting the HESI cut
score, however, does not entail a precipitous drop in the predicted likelihood of NCLEX success.
3
Those who achieve scores between 800 and 849 have more than a 93 percent probability of
passing the NCLEX. Students with HESI scores between 700 and 799 and those with HESI scores
between 600 and 699 have predicted probabilities of 85 percent and 69 percent, respectively
(Lewis, 2006). Within each of the ranges, HESI is not able to identify which students are most
likely to fail the NCLEX. The current use of HESI exit exam scores raises questions of equity and
efficiency. To require students, who have at least an 85% probability of passing to undergo
remediation may be a suboptimal allocation of resources. Attaching strong penalties
(withholding degree or eligibility to sit for the NCLEX) for not meeting the HESI exam cutoff may
also introduce a market inefficiency. As such, progression rules should not be based on a single
metric but on a broader assessment of student readiness to pass the NCLEX (Spurlock, 2006).
Using a rich administrative database of more than 700 students from a single nursing
school in the northeast, the first paper on the economics of education examines whether nursing
course grades, demographics, and SAT scores can improve the correct identification of students
who will pass/fail above and beyond the HESI exit exam. I do this by comparing the predictive
validity of the most common progression rule of scoring 850 on the HESI exit exam to two
statistical models: one that contains demographics, SAT scores, and nursing course performance
and one that, in addition, also contains an indicator of scoring 850 or above on the HESI exit exam
score. I find the HESI-based progression rule is best in identifying students who pass the NCLEX-
RN on the first attempt. In contrast, the model-based predictions are better at identifying
students who do not to pass it on the first attempt. Temporal variance in the predictive accuracy
of models exists. The model that includes HESI scores is better for the period after the more
4
rigorous NCLEX-RN standard was implemented (in March of 2014) and the other for the previous
period.
I also assess the degree to which students are severely misplaced as a result of the HESI-
based remediation protocol and investigate if model-based protocols could potentially reduce
such misplacement using Community Health course outcomes as a proxy for NCLEX-RN
outcomes. In comparison to HESI-based progression rule, I find that the models result in
modestly lower under-placement and over-placement rates. The different models are not,
however, equally effective at reducing the severe error rate for all student subgroups. Yet the
variability in predictions across the models can be leveraged to more accurately place students.
Establishing inter-model agreement for placement decisions eliminates over-placement rates
and lowers under-placement rates by 30 percent.
The structure of the essay is as follows. In Section 2, I review the current literature on
determinants of NCLEX-RN success. The next section describes the context in which the
outcomes were generated and profiles the students. In Section 4, I briefly detail the analytical
sample. The methodological approaches are articulated in Section 5. Results are discussed in
the following section. Finally, I summarize results and advance ideas for further research in
Section 7.
1.2 Review of the Literature The literature suggests that demographic variables are associated with NCLEX-RN
outcomes. Males and African-American students fail the NCLEX-RN at significantly higher rates
5
(Daley et al, 2003; Has et al, 2003). No difference in pass rates exist between the students based
in the main campuses and those students based in the satellite campuses (Has et al, 2003). Older
students were more likely to pass the NCLEX-RN on the first attempt (Briscoe and Anema 1999;
Beeson and Kissling, 2001; Daley et al, 2003; Fortier, 2010). English as primary language is
positively correlated with NCLEX-RN success (Arathuzik and Aber, 1998).
Studies also indicate that academic readiness and academic performance correlate with
positive NCLEX-RN outcomes. Students who passed had statistically higher verbal SAT scores
(Has et al, 2003). The number of grades below B in nursing courses is negatively correlated with
NCLEX-RN results (Beeson and Kissling, 2001). Standardized nursing end-of-course exam scores
have a statistically significant and positive correlation with passing the NCLEX-RN (Briscoe and
Anema, 1999; Daley et al, 2003). Grades in anatomy, pathophysiology, and medical surgical
courses were positively associated to passing the NCLEX-RN on the first attempt (Daley et al,
2003). Pre-nursing GPA in college, performance in college science courses, and final college GPA
are correlated with positive NCLEX-RN outcomes (Arathuzik and Aber, 1998; Sayles et al, 2003;
Tipton et al, 2007; Fortier, 2010; Bosch et al, 2011). Confidence in test taking and critical thinking
further increase the likelihood of NCLEX-RN success (Arathuzik and Aber, 1998; Giddens and
Gloeckner 2004; Santiago, 2013). In addition, family demands and negative emotions (anxiety,
anger and guilt) negatively correlate with performance on the NCLEX while an oral dependent
learning style have a positive association (Arathuzik and Aber, 1998; Sayles et al, 2003)
Overall, the literature examines the association among demographic, academic, and
contextual factors and NCLEX-RN outcomes. The analyses suggest that pre-admission measures
of academic readiness and performance are correlated with passing the NCLEX-RN on the first
6
attempt (Has et al, 2003; Sayles et al, 2003; Bosch et al 2011). Non-minority and older students
tend to have better NCLEX outcomes (Daily, 2003; Has et al, 2003; Fortier, 2010; Bosch, 2011).
Performance on nursing standardized end-of-course exams, exit exams, and nursing courses are
associated with passing the NCLEX-RN (Beeson et al, 2011; Briscoe and Anema, 1999; Fortier,
2010; Tipton et al, 2007; Santiago, 2013). Critical thinking and test-taking skills, learning style,
and non-academic time demands are also predictive of NCLEX-RN success. Although the
methodologies leveraged by the above-mentioned studies are not rigorous they examine
theoretically relevant determinants of NCLEX-RN success. My determinants of NCLEX-RN model
incorporates the various aspects taken up by the studies – demographics, pre-admissions
measures of academic performance, and nursing course and standardized test performance – to
build the most comprehensive determinants model in the literature and to test its predictive
power. In addition, my essay will be the first study on determinants that uses NCLEX-RN
outcomes from 2014 and 2015 – the first two full calendar in which the new NCLEX-RN cut-score
(0.00 logits) was implemented.
1.3 Data I use a student level database of almost 750 students who attended a four-year nursing
program who took a HESI exit exam at a university in the northeast. The database draws data
on each student from three distinct sources: a university-managed student records information
system, a nursing school controlled HESI management system, and NCLEX-RN reports generated
by a state Board of Nursing. The university-based data source contains demographic
information: race, gender, status in the Educational Opportunity Fund program (a measure of
7
poverty), SAT scores, and performance in nursing courses. The HESI management system
contains HESI exit exam scores while the Board of Nursing NCLEX-RN reports provide the
outcomes of NCLEX-RN test takers.
1.4 Organizational Context During the period covered by the analysis, the organizational context in which students
progressed through the nursing program experienced structural changes. In 2009, a new dean
took leadership of the nursing school after two years of having an interim dean. As the new
dean took the helm, a satellite campus opted to break off and become its own nursing school. In
2013, the main campus compensated for its previous loss by establishing a new satellite campus.
That same year, the university was mandated to merge with another university. Since both
institutions of higher education had nursing schools they were merged the following year. In
anticipation of the merger, both schools dedicated considerable resources to aligning curricula
and business processes. To this end, 22 joint-committees were convened. Program outcomes,
measures, and performance criteria were agreed upon for each pre-licensure and post-licensure
programs by the end of the 2015-2016 school year (Martinez et al, 2016).
At the end of calendar year 2015, the newly created nursing school was not fully
integrated. After the merger, faculty and staff of the two institutions still operated under
different statutes and unions. During the period covered by the data, the pre-licensure four-year
program remained the same since it was not a redundancy between the two institutions. There
were no changes to admissions criteria, progression rules, and/or graduation requirements.
Faculty attrition for that program was minimal (Martinez et al, 2016).
8
Before the merger, the institution that absorbed the other institution had two pre-
licensure nursing programs: a second degree program and traditional four-year program. The
former was geared for students who already had a bachelor’s degree and wanted to receive a
nursing degree in a compressed time period. Cohorts of the accelerated program increased from
15 in 2009 to 38 in 2015. Historically, these students fared better on the NCLEX-RN than those
of the four-year nursing program. As such, they were not required to take the HESI exit exam
and, consequently, were not subjected to a remediation policy. The traditional program had
much larger enrollment and had a less seasoned student body.
Students apply to the four-year program pre-licensure program during high school. For
the cohorts covered by the period of analysis, yearly applications ranged from approximately
2,500 to 3,100 for 105-125 seats. Admissions were granted to 15 percent of applicants. About
one in three took up the admissions offer. Those who opted to attend school elsewhere had,
on average, higher SAT scores than those who did not (Martinez, 2014). The vast majority of
these students attended research-intensive universities in the northeast (Martinez et al, 2016).
The nursing program is highly structured. During the first two years of study, students
take their general education courses. Unlike their counterparts in the social sciences and
humanities, they take a considerable amount of entry-level science courses: Anatomy and
Physiology I and II, Organic Biochemistry, Microbiology, and World of Chemistry Lab. They are
also required to complete Statistics and Nutrition courses. Year three of the program is
exclusively dedicated to nursing courses. During the last year, students are required to take a
general elective and a literature course in addition to their nursing courses.
9
Cohorts progress through the program synchronously. On average, about 10 percent of
students leave the program during the first year and an additional eight to nine percent during
the second year. Attrition during the third and fourth years is almost non-existent. Students
who receive a “C” or lower in a nursing course are placed on academic probation and required
to retake the course. Failure to successfully complete the course a second time triggers dismissal
from the program. The same is true of withdrawing from two nursing courses. During the last
semester, students take Community Health Nursing and Leadership and Management courses
and take the HESI exit exam. The former course focuses on the application of technical nursing
skills into a specific context while the latter emphasizes a softer set of competencies. The HESI
exit exam is taken early in the semester and students receive the scores almost instantaneously.
Those who score less than 850 on the HESI exit exam are mandated to go through remediation.
Remediation consists of reviewing nursing specific content on a one-on-one basis until they reach
an 80% score on an online HESI practice exam. The online practice exam is not
proctored/monitored and actual scores are not recorded on an official database. No students
have been reported to have not met the remediation requirements. Once the remediation
requirement is met the names of the students are sent to the state Board of Nursing so they may
be eligible to sit for the NCLEX-RN.
Differences exist between those who meet the HESI exit exam standard and those who
do not. As Table 1 points out, about 53 percent of the sample met the HESI standard and the
average difference in HESI exit exam scores between the two groups is almost 200 points. In
comparison to those who did not, those who met the standard are more likely to be white, have
higher SAT math scores, and have nursing course GPAs about 0.40 point higher (with the
10
exception of the last two nursing courses). A twenty-four percentage point difference exists in
first-time NCLEX pass rates between those who met the HESI standard and those who did not.
Table 1. Summary Statistics for Analytical Sample
NCLEX
HESI>=850 HESI<850
n=392 n=351
Demographics
EOF Status 5% 16%
African American 8% 13%
American Indian 0% 1%
Asian 25% 28%
Hispanic 10% 14%
International 1% 1%
Multiracial 4% 4%
Native 1% 4%
Other 1% 0%
Unknown 3% 6%
White 49% 30%
Male 9% 13%
Campus 1 49% 43%
Campus 2 49% 54%
Campus 3 2% 2%
Academic Readiness
SAT Verbal Score 613 656
SAT Math Score 591 564
Academic Performance
Healthcare Delivery 3.65 3.28
Pathophysiology 3.63 3.22
Health Assessment 3.67 3.24
Foundations I 3.52 3.14
Childbearing Family 3.39 2.95
Health and Illness ICA 3.56 3.09
Health and Illness AOA I 2.89 2.36
Foundations II 3.61 3.30
Pharmacotherapeutics 3.58 3.14
Psych Mental Health 3.73 3.38
Health and Illness AOA II 3.54 3.27
Leadership & Management 3.67 3.55
Community Health 3.78 3.59
HESI Score 948 750
NCLEX Pass Rate 94% 70%
11
1.5 Methodological Approaches Using an administrative dataset of more than 700 nursing students from a research
intensive university in the northeast covering calendar years 2009-2015, the first essay examines
the explanatory and predictive power of: a) pre-admission achievement, demographics, and
academic performance in nursing courses on passing the NCLEX-RN on the first attempt and
comparing them to the explanatory and predictive power of b) HESI exit exam scores on passing
the NCLEX-RN on the first attempt.
A simple logistic regression assesses the explanatory power of a). The model takes the
general form:
(1) Yi= β0 + βdΣXd + βpΣXp + βaΣXa + βyΣYeari + ei,
- where Y is passing the NCLEX-RN on the first attempt for student i, β0 is the intercept term, Xd
is a vector of student demographic variables (race, gender, and socio-economic status), Xp is a
vector of preadmission academic performance measures (verbal and quantitative SAT scores), Xa
is a vector of academic performance measures in nursing courses (Health Assessment;
Foundations of Nursing Practice I and II; the Childbearing Family; Pharmacotherapeutics; Health
and Illness of Infants, Children, and Adolescents; Health and Illness of Adults and Older Adults I
and II; Psychiatric Mental Health Nursing; Community Health Nursing; and Leadership and
Management), βy captures the cumulative impact of year fixed effects, and ei is an error term
with a mean of zero and variance of one. The betas capture the cumulative impact of their
respective vectors or variables.
A second model assesses the explanatory power of b). The model takes the general form:
12
(2) Yi= β0 + βhXh +ei,
- where Y is passing the NCLEX-RN on the first attempt for student i, β0 is the intercept term, βh
captures the association between not meeting the HESI exam cutoff and the outcome, and ei is
an error term with the usual properties.
A third explanatory model will combine both models a) and b). The model will take the
general form:
(3) Yi= β0 + βdΣXd + βpΣXp + βaΣXa + βhXh + βyΣYeari + ei
- where the terms are as previously defined.
Since nursing programs find it useful to identify students most at risk of not passing the
NCLEX-RN on the first attempt, the predictive power of the three models will be tested as follows.
For models (1) and (3), three years of data (period p) will be used to predict NCLEX-RN outcomes
at p+1. Predicted outcomes will be compared to actual outcomes to identify the best performing
model (Hansen and Timmerman, 2012).
Finally, I explore a method to ascertain the degree to which students are severely
misplaced [assigned to remediation but passed the NCLEX-RN or was not assigned to remediation
but failed the NCLEX-RN] as a result of the current NCLEX-RN remediation protocol and to
investigate if other protocols could potentially reduce such misplacement. Inspired by Clayton
and Belfield (2014), I simulate what researchers/administrators could do if they had access to the
NCLEX-RN scores rather than NCLEX-RN pass/fail designations by calculating the severe error rate
for each of the three above-mentioned models for the Community Health Nursing course. The
Community Health Nursing course is ideal for this purpose since a) it is chronologically the last
13
nursing-specific course that students take, b) it requires that all other nursing courses have been
successfully completed, c) the grade is granted after students take and received the HESI exit
exam scores and d) the distribution of grades is similar to NCLEX-RN pass/fail outcomes. About
11 percent of students in the sample earn a grade of B or lower in Community Health Nursing
which approximates the 10 to 11 percent of students who failed the NCLEX-RN on the first
attempt nationally during the period covered by the data (National Council Licensure
Examination for Registered Nurses, 2016).
Following Scott-Clayton and Belfield (2014), for individual students, the probability of
being severely misplaced is the sum of the probabilities of over-placement and under-placement.
For subgroups, it is average of the sum of those probabilities. The probability of being severely
under-placed is operationalized as:
Pr(Severally under-placed=1) = Pr(Grade of A=1)
if remediated, 0 else.
The grade of an “A” as the criterion to assess under-placement is not excessively high since the
average grade for nursing courses in the sample is barely below a 3.5 (B+). Moreover, more than
half of the students in the sample received an “A” in the course. The probability of the being
severely over-placed is:
Pr(Severally under-placed=1) = Pr(Grade of B or lower=1)
if not remediated, 0 else.
Approximately 11 percent of the sample earned a grade of B or below.
14
Unlike the other analyses, severe error rates are derived by only using the observations
of students who were not subjected to remediation and extrapolating results to those students
who were remediated since the associations between the independent variables and the
outcomes have to be estimated net of remediation effects. Failure to do so would introduce
further bias into estimates. Those who have a probability higher than .50 will be considered to
have been under-placed or over-placed, respectively. The under(over)-placement rate is the
number of (non-)remediated students predicted to be under(over)-placed divided by the total
number of (non-)remediated students included in the model. The under-placement and over-
placement rates derived from two separate models – the aforementioned model (3) and a version
of model (3) without SAT scores. The latter approximates the data availability of typical two-
year nursing programs at community colleges where SAT/ACT scores are often lacking. The
misplacement rates of the two models are compared to those of the current progression
standard based on meeting the HESI exit exam cut-score. For the progression rule, the under-
placement rate is the number of remediated students who earned an “A” in the course divided
by the total number of remediated students. The over-placement rate is the number of non-
remediated students who earned a “B” or below divided by the number of non-remediated
students.
1.6 Results
1.6.1 Explanatory Models
Nursing programs need to know what student characteristics and/or academic pre-
admission measures are associated with passing the NCLEX-RN on the first attempt so they can
15
admit students most likely to succeed and, once enrolled, can intervene if needed to. Table 2
below summarizes the point estimates for the full model for the entire period and for four-year
periods. The Foundations I course was the only nursing course found to have a statistically
significant relationship with the outcome in all the time periods covered. The Health and Illness
of Infants, Children, and Adolescents course also had a statistically significant association with
NCLEX-RN success for all but the period covering years 2012-2015. In periods 2011-2014 and
2012-2015, the Psych/Mental Health course was statistically significantly associated with the
outcome.
Table 2. Logistic regression results pooled and four-year periods
2009-2015
2009-2012
2010-2013
2011-2014
2012-2015
n=455 n=230 n=238 n=269 n=287
NCLEX Odds Ratio
Odds Ratio
Odds Ratio
Odds Ratio
Odds Ratio
Demographics EOF Status 1.15 0.99 1.30 1.03 2.55 American Indian 1.00 1.00 1.00 1.00 1.00 Asian 1.13 1.39 2.31 1.56 1.62 Hispanic 0.48 1.18 1.95 2.40 0.63 International 1.00 1.00 1.00 1.00 1.00 Multiracial 0.60 1.38 2.43 1.66 0.46 Native 1.67 0.73 3.98 2.99 4.23 Other 1.00 1.00 1.00 1.00 1.00 Unknown 0.40 1.00 1.00 0.81 0.30 White 1.08 1.63 1.62 0.98 0.90 Male 1.60 2.77 1.52 0.66 0.99 Campus 2 1.00 0.62 0.69 1.00 0.31 Campus 3 0.54 1.00 1.00 0.27 1.00
Academic Readiness SAT Verbal 1.00 1.01 1.01 1.00 1.00 SAT Math 1.00 0.99 1.00 1.00 1.00
Academic Performance HC Delivery 0.59 0.39 0.48 0.83 0.80
16
Pathophysiology 1.40 1.77 1.29 1.29 1.01 Health Assessment 1.30 0.56 0.95 2.04 1.71 Foundations I 3.21†1 3.18† 4.36† 3.24† 4.94† Childbearing Family 1.54 1.62 2.46 1.07 1.35 Health and Illness ICA 3.03† 6.88† 5.02† 4.00† 2.45 Health and Illness AOA I 2.50 1.93 2.79 3.06 2.65 Foundations II 0.64 1.20 0.45 0.46 0.42 Pharmacotherapeutics 1.10 0.80 1.03 1.45 1.24 Psych Mental Health 2.55 1.23 1.45 4.12† 4.71† Health and Illness AOA II 1.15 1.60 0.89 0.32 0.80
Intercept 0.00 0.00 0.00 0.00 0.00
The figures below depict the average marginal probabilities for the three courses with
statistically significant relationships with the outcome for the 2009 to 2013 period and the 2014
to 2015. The latter period covers the years in which the NCLEX-RN exam cut-score was raised
from -0.16 logits to 0.00 logits. For the Foundations I course, the pattern is clear (Figure 1). In
comparison to the period 2014-2015, grades during the antecedent period have more
discriminating power. The average predicted probability ranges from 69 percent (C) to 93
percent (A) with the decreasing marginal rates. In contrast, average marginal probabilities for
the subsequent period only range from 73 percent to 77 percent with increases at a monotonic
rate. The difference in discriminating power between the two periods is similar for the Health
Assessment of Infants, Children, Adolescents (Figure 2) course but it is less pronounced. The
discriminating power for the Psych/Mental Health course, in contrast, significantly increases in
the latter period (Figure 3). For the 2009-2015 period, the average marginal probability of
1 † indicates statistical significance at 95% level. See Appendix 1.8 for greater detail.
17
passing the NCLEX-RN on the first attempt is constant across the range of grades. The average
marginal probabilities in the adjacent period range from 36 percent (C) to 91 percent (A).
Figure 1. Predicted Average Marginal Probabilities of Passing the NCLEX-RN by Foundations I Course Grade.
Figure 2. Predicted Average Marginal Probabilities of Passing the NCLEX-RN by Health & Illness ICA Course Grade.
69%
77%
84%
89%
93%
73% 74% 75% 76% 77%
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
100%
C C + B B + A
Period 2009-2013 Period 2014-2015
65%
74%
81%
87%
92%
71%
74%
78%
81%85%
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
C C + B B + A
Period 2009-2013 Period 2014-2015
18
Figure 3. Predicted Average Marginal Probabilities of Passing the NCLEX-RN by Psych/Mental Health Course Grade
Since the Foundations I course is taken during the first semester after completion of all
general education courses it the allows the program to intervene early in the program to optimize
the likelihood of passing the NCLEX-RN on the first attempt. Students take the Health and Illness
of Infants, Children, and Adolescents course during the second semester of the third year and the
Psych/Mental Health course during the first semester of the fourth year giving the program a
string of markers to assess if interventions are having the desired effects or if further
interventions are needed. The temporal instability of statistical relationships between given
performance on nursing courses presents a challenge only insofar as NCLEX-RN undergoes a
structural change such as a more rigorous standard as it was established beginning in 2014.
1.6.2 Out-of-Sample Predictive Models
In discerning the best progression rule for sitting for the NCLEX-RN exam and applying it,
nursing programs have to use previous results to extrapolate into the future. In this section, I
85% 85% 85% 85% 85%
36%
53%
69%
82%
91%
30%
40%
50%
60%
70%
80%
90%
100%
C C + B B + A
Period 2009-2013 Period 2014-2015
19
use three years of data (period p) to predict NCLEX-RN outcomes at p+1 for models (1) and (3). I
then compare the predictive outcomes of the models and the current progression rule to actual
outcomes. Under the HESI-based progression rule, all students who do not score above a score
of 849 on the HESI exam exit are predicted to fail the NCLEX-RN while those who score above it
are predicted to pass it. Individuals with model-based probabilities (MBP) at or above .50 are
predicted to pass the NCLEX-RN.
Figure 4 below compares the predictive validity of models (1) and (3) and the current
progression rule that only uses HESI scores. With the exception of 2012, the current progression
rule (HESI Only) best identifies students who pass the NCLEX-RN. The model with HESI scores is
best for identifying students who will not pass the NCLEX-RN in years 2012 and 2013 while the
model without HESI scores is best for years 2014 and 2015. Overall, the results suggest a
predictive validity trade-off between identifying those students who fail and those who pass.
Figure 4. Predictive Validity of Models (1) and (3) and Current Progression Rule based on HESI Exit Exam.
20
The optimal predictive method may be the one that minimizes that tradeoff. Overall,
HESI Only is the predictive method where the gap between correctly predicted NCLEX success
and failure is biggest for all years. MBP without HESI score is the only method that accurately
predicts failures for at least half of those who fail for all four years. MBP with HESI score, on the
other hand, rightly predicts the highest percentage of failures in years 2014 and 2015. The
change in NCLEX-RN cut-score starting in March of 2014 may, in part, explain the better
performance of MBP with HESI score in the last two years. The data suggest that, going forward,
MBP with HESI score may be the most appropriate method to identify students likely to fail and
HESI only is the best way to identify students likely to pass. Using both methods may be optimal
and, when model predictions contradict each other, side on the side of caution and assume that
the student is more likely to fail.
86%
97%100%
94%100%
94%
78%
88%
80%84%
93%87%
50%
20%24%
50%
24%
45%
69%
46%
71%
50%
28%
57%
0%
20%
40%
60%
80%
100%
120%
2012 2012 2012 2013 2013 2013 2014 2014 2014 2015 2015 2015
MBP woHESI
HESI Only MBP wHESI
MBP woHESI
HESI Only MBP wHESI
MBP woHESI
HESI Only MBP wHESI
MBP woHESI
HESI Only MBP wHESI
% of predicted to pass who actually passed % of predicted to fail who actually failed
21
1.6.3. Severe Error Rates
Subjecting students to remediation who have a high probability of success without
remediation may be a sub-optimal allocation of resources and may result in negative unintended
consequences like poorer performance through moral discouragement and/or opportunity costs.
In this section, I assess the degree to which students are severely misplaced as a result of the
current NCLEX-RN remediation protocol and investigate if other protocols could potentially
reduce such misplacement using Community Health course outcomes as a proxy for NCLEX-RN
outcomes.
Model results converge. Students are much more likely to be under-placed rather than
over-placed. As Table 3 indicates, for both models, students are more likely to under-place than
to over-place by a factor of approximately five.
Table 3. Predicted Average Probabilities of Under-Placement or Over-Placement
Demographics Variables
Over-placed model (3)
Over-placed wo SAT Scores
Under-placed model (3)
Under-placed wo SAT Scores
n=224 n=300 n=156 n=245
Female 8.70% 7.62% 51.19% 43.62%
Male 11.11% 13.64% 23.10% 17.29%
EOF 40.00% 25.00% 45.77% 37.76%
non-EOF 7.95% 7.59% 48.29% 40.69%
Minority 14.63% 11.86% 47.30% 35.95%
White 2.70% 4.72% 49.01% 47.36%
All Students 8.97% 8.16% 47.80% 40.20%
Predicted probabilities mirror predicted misplacement rates. As Table 4 indicates, under-
placements are more likely by a factor of six or more. Both model-based predicted over-
placement rates are lower than the rate of the progression rule and model without SAT scores
has the lowest rate for “all students” and the majority of student subgroups. This pattern is
22
slightly different for under-placement rates. The HESI progression rule has lower predicted
under-placement rates than model (3). The model without SAT Scores also has the lowest overall
under-placement rates.
As Tables 4 and 5 show, the model without SAT scores performs best for students in
general but not for each student subgroup. The model without SAT scores minimizes under-
placement and under-placement rates for three subgroups: male, non-EOF, and minority
students. This result may be driven by the differential predictive validity of SAT scores across
student subgroups (Mattern et al, 2008; Santelices et al, 2010). Model (3) performs best for white
students for both misplacement rates. The current HESI progression rule does not outperform
the models for any specific subgroup.
The convergence in predictions across the models can be leveraged to more accurately
place students. A simple business rule where students are identified as misplaced only when the
two models agree with the progression rule decreases severe error rates. The second column
of Table 6 captures the under-placement and over-placement rates under the above-mentioned
business rule. As can be seen in the second and third columns of Table 6, this approach of
including three distinct data sources eliminates over-placements and produces a modest
improvement in under-placement rates. The final column captures the results of adding a third
model to the business rule. The third regression model acting as the fourth data source
contains only an intercept and the nursing courses found in model (3). This model specification
may prove useful in combating perceptions that personal characteristics may be used to identify
students who could potentially benefit from remediation. The expansion of the business rule to
23
include this model has marked effects on under-placement rates. It reduces the under-
placement rate by slightly more than 30 percent – from 39 percent to 27 percent. This inter-
model agreement approach proves beneficial in reducing severe misplacements.
24
Table 6. Predicted Under/Over-Placement Rates By Number of Data Sources.
Demographics Variables
Under-placed Over-placed Under-placed
# of sources = 3 # of sources = 3 # of sources = 4
n=223 n=222 n=223
Female 43.88% 0.00% 30.10% Male 7.41% 0.00% 7.41% EOF 39.53% 0.00% 25.58% non-EOF 39.44% 0.00% 27.78% Minority 38.4%* 0.00% 24.00% White 40.00% 0.00% 30.00%
All Students 39.46% 0.00% 27.35%
1.7 Conclusion and Further Research I find that demographics, SAT scores, and performance in nursing courses provide
predictive power beyond that of the HESI but only for those who fail. Using them in statistical
models increases the percentage of students correctly predicted to fail by at least 50 percent
(2014) and, as much, as 100 percent (2012, 2013, and 2015). Nursing programs may improve
their NCLEX-RN pass rates by dedicating more resources to help students predicted to fail. Using
Community Health course outcomes as a proxy for NCLEX-RN outcomes, I also demonstrate that
model-based protocols could potentially reduce the number of students assigned to remediation
who will pass it with a high score and eliminate the number of students not assigned to
remediation who will fail it.
The analyses found in this paper could be further extended to include other four-year
nursing programs, community college nursing programs, and accelerated programs as well as
additional periods in which NCLEX-RN standard underwent a change to ascertain the
generalizability of findings and to investigate the possibility of non-HESI exit exam “universal”
25
predictors of NCLEX-RN success. In addition, severe placement error rates should be calculated
using actual NCLEX-RN scores across various periods in which the passing standards changed to
investigate potential adverse impact of such changes on various student subgroups.
26
Chapter 2: The Impact of Remediation on
NCLEX-RN Success: Positive, Neutral, or
Negative?
2.1 Introduction “That makes me nervous now. Our school drops us out of the program, if we do not pass
the HESI. They only give us 1 chance. There were plenty of people last year that failed the HESI and could not proceed.…The grades that you make and your average do not matter here when you take the HESI. You can make all A's through the nursing program, but if you fail that test, you are out” by Ivana (retrieved from http://allnurses.com/general-nursing-student/failed-the-hesi-137577-page2.html).
“In my case, our school required us to pass the Hesi Exit Exam with a 900 in order to officially graduate. Fellow students who failed the test had to take 5 weeks of remediation classes... it did not help the students at all. Most of them also failed the second, even the third time around.” by Claire (retrieved from http://www.yourbestgrade.com/hesi/letter/).
The experiences described by Ivana and Claire in their respective online fora, in some way,
capture key commonalities and divergences in the undergraduate nursing program market. Both
identify high stake testing as part of their programs. One offers remediation to assist their
students achieve desired performance levels while the other offers no such help. Despite their
apparent differences, both performance management approaches have the same aim – to
maximize the likelihood that their students will pass the NCLEX-RN (national licensure exam for
nurses) on the first attempt.
First time NCLEX-RN pass rates shape nursing programs. State Boards of Nursing make
first time NCLEX-RN pass rates publicly available as part of their charge to inform consumer
choice. Ceteris paribus, programs with relatively higher pass rates attract more applicants and/or
more qualified applicants, enroll and maintain them (Beeson and Kissling, 2001; Norton et al,
27
2006). The first-time pass rate is also a discrete metric which can assist administrators
overseeing nursing programs to gauge their overall effectiveness and competitiveness (Giddens
et al, 2009; Taylor et al, 2014).
Regulations incentivize nursing programs to make remediation as effective as possible.
The Commission on Collegiate Nursing Education (CCNE) and the National
League for Nursing Commission for Nursing Education Accreditation (CNEA) require minimum
first-time NCLEX-RN pass rates to maintain accreditation. In addition, nursing programs must
also meet the NCLEX-RN pass rate standards established by their state boards of nursing which
may be higher than those established by national accrediting bodies. Prolonged inability to meet
standards could result in loss of accreditation and/or placement on probationary status (CCNE,
2014) (CNEA, 2015).
Nevertheless, nursing programs that offer remediation – like the one attended by Claire
- may not provide an advantage to its students. If the remediation is not effective then outcomes
may not be any different than those who do not offer them. The cycle of taking the HESI exit
exam (standardized exams used to determine eligibility to receive a nursing degree or sit for the
NCLEX-RN) may be extended through repeated testing but their probabilities of passing the
NCLEX-RN will not improve.
Remediation may not be effective for a variety of different reasons. The length of
remediation may be inadequate to compensate for knowledge not acquired over the course of
several years. If the established HESI exit exam performance is not met because of instructional
deficiencies then remediation conducted by the same instructors may not yield improved results.
28
If students are not motivated to acquire the knowledge they did not learn during their regular
course of study then remediation may not help increase their probabilities of NCLEX-RN success.
Benchmarking practices may render remediation ineffective. Nursing programs set a
minimum HESI exam score to identify students who may benefit from remediation before taking
the NCLEX-RN (Langford and Young, 2013; Young and Willson, 2012; Zweighaft, 2013; Schreiner
at el, 2014). Most institutions set the cutoff score at 850. About 97 percent of students who
met or surpass the 850 benchmark have passed the NCLEX (Young & Willson, 2012; Lewis, 2006).
Not reaching a score of 850 on the HESI, however, does not indicate a sharp decrease in the
predicted probabilities of passing the NCLEX. More than 93 percent of students with HESI scores
between 800 and 849 pass the NCLEX pass the NCLEX-RN on their first attempt (Lewis, 2006).
The predicted probabilities at the lower end of the HESI score distribution are considerably lower.
Eighty-five percent of students with HESI scores between 700 and 799 pass the NCLEX-RN on the
first attempt while approximately seven out of ten of those with HESI scores between 600 and
699 do so (Lewis, 2006). Hypothetically, a remediation intervention that moves students 100
points along the HESI score distribution from 699 to 799 would increase the probability of passing
the NCLEX-RN on the first attempt by somewhere in the range of 15 percentage points while the
same hypothetical intervention for students with scores of 849 would increase their probability
of passing the NCLEX-RN around three percentage points. Thus, all things being equal, the
precise point at which programs establish the benchmark may affect the magnitude of the impact
of remediation, if one does indeed exist.
Using two national samples from the seventh and eight HESI exit exam validity studies
augmented by the sample from Chapter 1, the second economics of education essay aims to be
29
assess the impact of required remediation on passing the NCLEX-RN on the first attempt through
regression discontinuity (RD). First, I find that the assumptions of RD as met. Next, I explore the
impact of required remediation using regression discontinuity design both as local randomization
and as continuity at the cutoff. As the former, I find some evidence that remediation has a
negative impact on passing the NCLEX-RN on the first attempt. As the latter, I find very limited
evidence that remediation may improve NCLEX-RN outcomes. Both sets of statistically significant
findings, however, are sensitive to bandwidth, functional assumptions and/or sample trimming.
Most of the RD evidence suggests that remediation has no impact on NCLEX-RN outcomes.
Logistic regression and propensity score matching (PSM) results do not contradict the overall
findings from the RD analysis. They suggest no association between remediation and success on
the NCLEX-RN. After running the robustness checks, I look for possible heterogeneous treatment
effects of remediation. Logistic regression results suggest that students who score below 600 on
the HESI score are more likely to pass the NCLEX-RN on the first attempt than those who do not
receive remediation. The results of PSM suggest otherwise.
Overall, I conclude that remediation most likely does not have an impact on students’
probability of passing the NCLEX-RN on the first attempt. This may be due to a ceiling effect
(Scott-Clayton and Rodriguez, 2012). Prior to remediation, remediated students just below the
modal cutoff of 850 have about a 93 percent chance of passing the NCLEX-RN – a few percentage
points above the national first-time pass rate. Measurably increasing the probability of passing
the NCLEX-RN of those students may not be possible. If remediation does have an impact, the
impact is likely different along the HESI score distribution. Students at the low end of the
distribution are most likely to benefit from it. I recommend that nursing programs lower their
30
HESI benchmarks to increase per student allocation of remediation resources to students who
are most likely to benefit from it.
The structure of the essay is as follows. In Section 2, I review the current literature on
interventions to improve the likelihood of NCLEX-RN success. I then describe the analytical
sample. The identification strategy is articulated in Section 4. Section 5 demonstrates that the
assumptions of RD are met. RD estimates are discussed in the following section. In Section 6,
RD estimates are compared to logistic regression and propensity score matching (PSM)
estimates. Section 7 explores the possibility of heterogeneous treatment effects using logistic
regression and PSM. Finally, I summarize results and advance ideas for further research in
Section 8.
2.2 Review of the Literature In recent years, strategies to help students pass the NCLEX-RN on the first attempt have
received considerable attention in the literature. Overall, most of these studies are descriptive
or correlational and have modest sample sizes. Some even lack a control group. Seldom are the
interventions covered by the studies “single treatments”. Most often, the interventions involve
multiple simultaneous or staggered treatments. Like the larger remediation literature, the
studies reviewed in this section do not leverage adequate methodologies to capture causation
between interventions and outcomes (Bailey and Alonso, 2005).
The literature suggests that remediation is associated with improved NCLEX-RN
outcomes. Bondmass, Moonie, and Kowalski (2008) find an almost 9 percent point increase in
first-time NCLEX-RN pass rates after implementation of a remediation program. Heroff (2008)
31
reports a 17 percent point increase in first-time pass rates after the introduction of an ATI exit
exam with an accompanying progression rule and remediation policy. Morton (2006) evaluates
the effect of structured remediation throughout the nursing program of study on NCLEX-RN pass
rates. Students who did not meet the benchmark score on HESI end-of-course exams and were
required to attend structured learning assistance workshops to review course specific content
had higher first-time NCLEX-RN pass rates than those who did not. Norton, Relf, Cox, Farley,
Lachat, Tucker and Murray (2006) find that the introduction of end-of-course and exit
standardized testing, a remediation course, and higher progression policy with NCLEX outcomes
increased the first-time NCLEX-RN pass rates by 11 percentage points. Schroeder finds an
eight-percentage point difference in first time pass rates between students under a new
testing/remediation policy and their predecessors (2013).
Studies on curricular interventions yield mixed results. Morris and Hancock investigate
the results of the introduction of new curriculum on HESI exit exam and NCLEX-RN results (2008).
The authors find no statistically significant difference in results between last cohort of graduates
under the old curriculum to the first cohort of graduates of the new curriculum. Frith, Sewell,
and Clark (2005) report that introduction of a capstone course and remediation in the form of
developing a study plan based on performance on the HESI exit exam increased first-time NCLEX-
RN pass rates. Lyons (2008) investigates the relationship between a problem-based NCLEX-RN
review course and NCLEX-RN outcomes. The four-month review course took place during
students last semester of study and was designed to increase their critical thinking as measured
by the ATI Critical Thinking Test. After students completed a baseline critical thinking test, the
author randomly assigned the 54 students to either the problem-based review course
32
(treatment) or the traditional lecture based (control) to assess the impact of the problem-based
methodology. Groups were comparable in the baseline critical thinking test, overall GPA, nursing
GPA, ACT scores, and age. No statistically significant difference was found between treatment
and control groups in the post critical thinking test. The treatment group, however, had higher
first-time NCLEX-RN pass rates.
The literature associates interventions aimed at improving outcomes of specific student
subgroups with positive outcomes. Sifford and McDaniel investigate the impact of remediation
and test-taking course among at-risk students identified through HESI exit exam results (2007).
The authors find a statistically significant difference between pre and post exit exam results
among course participants. Parrone, Sredl, Miller, Phillips, and Donaubaur (2008) find that a
program that identified at risk students, provided students with faculty mentoring, individualized
remediation services, and instituted progression/graduation based on HESI exit exam, improved
NCLEX-RN pass rates from 70 percent to 100 percent. Sutherland, Hamilton, and Goodman
(2005) find that the Affirming At-Risk Minorities for Success (ARMS) program improved retention
rates and performance on a capstone course and mitigated the association of minority status
with passing the NCLEX-RN.
Overall, the literature suggests that the probability of students to pass the NCLEX-RN can
be improved through various interventions. Remediation has been consistently found to
improve the probability of NCLEX-RN while curricular interventions have proved less so. The
remediation literature, however, relies on methodologies that cannot establish causation. This
study makes a contribution to the NCLEX-RN remediation literature by using, for the first time, a
33
quasi-experimental technique (regression discontinuity) to establish a causal link between
remediation and NCLEX-RN outcomes.
2.2.1 HESI Exit Exam
In practice, the HESI exit exam is used as a formative or as a summative assessment.
Figure 5 depicts the various ways nursing programs use the HESI exit exam to determine if
students will be allowed to take their licensure exam.
Figure 5. Possible Impacts of HESI exit exams scores on ability to sit for the NCLEX-RN.
Nursing programs administer the HESI exit exam during students’ last semester. Programs
with remediation may ask students who did not meet the cutoff to remediate until they are able
to reach the HESI exam exit cutoff. The number of attempts allowed to reach the HESI exit exam
cutoff may be limited. Those who reach it become eligible to sit for the NCLEX-RN while those
who do not will not get their nursing licenses. Programs with required remediation may not
require retaking of the HESI exit exam. Instead, they may ask students to correctly answer a
certain number or percentage of review questions or to meet with faculty members a certain
number of times to go over content-specific materials. Programs that offer remediation but do
not require it view the HESI exit exam as a formative assessment. In these cases, students review
their respective HESI exit exam scores by subject areas and concentrate their review efforts on
Completion of
General
Education and
Nursing Courses
HESI Exit Exam
(Cutoff)
Remediation
a) Required
b) Non-Required
redN No Remediation
c) Removal
d) Review
redN
34
the areas they deem most in need of improvement. Some programs that do not offer
remediation may also use the HESI exit exam as a way to identify what content to review while
others use it as the definitive measure of NCLEX-RN readiness. For the latter, those who do not
meet the HESI exit exam benchmark are automatically kicked out of the program. No publication
exists that reports the number of students who are not allowed to sit for the NCLEX-RN because
of failure to meet the HESI exit exam cutoff or to meet remediation requirements.
2.3 Data Not all students in the full sample are in programs that require remediation. Thirty seven
of the 141 schools do not have compulsory remediation. As such, approximately one third of
the 10,022 students in the combined sample are not required to remediate if they do not meet
their respective HESI benchmark or do not have a benchmark at all. As Table 7 points out, among
those that require remediation, the distribution of students and institutions mirror each other.
850 is the modal HESI benchmark option that covers about two thirds of institutions and students
while 900 is the second most popular choice. This is not coincidence since the maker of the HESI
recommends “that students seriously remediate any subject area category in which they
obtained a score of less than 850 …” and also maintains that the “recommended level is 900”
(2014).2
Table 7. Distribution of Schools and Students by HESI Benchmark: Required Remediation
HESI Benchmark Schools # Schools % Students Students %
700
2 RD analyses are restricted to programs with required remediation with benchmarks of 850 and 900 to minimize the possibility of cutoff levels not being exogenously determined.
35
725
750 1 1% 57 1%
800 1 1% 30 0%
825 1 1% 25 0%
850 69 66% 4,464 67%
875 1 1% 166 2%
900 15 14% 777 12%
950 1 1% 146 2%
Not Applicable 15 14% 981 15%
Total 104 100% 6,646 100%
All students in the full sample have HESI exit exam scores and NCLEX-RN outcomes, the
type of program attended (Bachelor of Nursing Science [BSN], Associate Degree in Nursing [ADN],
Practical Nursing and Diploma), remediation policy of their program, and HESI benchmark.
Approximately 725 students of those students also have demographics, SAT scores and nursing
course performance data.
Theoretically, for the subsample of almost 725 students, regression discontinuity (RD) as
a “local randomized experiment” can be leveraged to obtain unbiased estimates of treatment
effects of required remediation based on a HESI benchmark since: 1. HESI scores are anterior to
the treatment (i.e. HESI exam is taken prior to the assignment to treatment [remediation] or
control groups), 2. the HESI benchmark is exogenously determined (i.e. established prior to
assignment to treatment [remediation] or control groups, and 3. students are not able to
manipulate their score on the HESI. When the above conditions are met then the groups of
students just above it and just below should be comparable if not identical, with the only
difference between them being remediation (Trochim, 1984; Lee and Lemieux, 2010). This
situation approximates a randomized control trial (RCT) such that researchers can obtain
unbiased estimates of the local average treatment effect (LATE) of remediation. The relatively
36
small sample size, however, would only allow for the detection of a very large treatment effect.
Instead, the maximum bandwidth at which covariate balance between treatment and control
groups in the subsample will be used as the maximum bandwidth at which the entire sample may
have covariate balance. Using the full sample, treatment effects will be estimated using
bandwidths of 10 points and less.
2.4 Identification Strategy The identification strategy restricts itself to local estimates. The econometric RD model
to identify the LATE takes the following general form (Jacob et al, 2012):
Yi = β0 + β1Remediationi* HESIi + β2HESIi + εi
- where i indexes students, Y is passing the NCLEX-RN on the first attempt, β0 is the average value
of the outcome for those in the treatment group conditional on the HESI score, β1 captures the
local average treatment effect [where Remediation=1 if x<HESI exit exam cutoff score and
Remediation=0 otherwise] and allows for different slopes on either side of the cutoff, β2 captures
the relationship between the outcome and residual differences of HESI scores between the
treatment and control group, and ε is an error term with the usual properties. School fixed
effects and school type are not included in the model because of collinearity. Their inclusion
reduces the sample by almost one third. 3
Sensitivity analysis will test the stability of estimates using the full and trimmed samples.
Although it is not likely that the assignment variable was manipulated by students the McCrary
3 Although not discussed in the essay, the local treatment estimates were nevertheless calculated with school fixed effects and school type in the models. None of the estimates were statistically significant.
37
and Cattaneo, Janssen, and Ma (CJM) test will be performed to assess the presence of a
discontinuity in the density of observations at the cutoff (McCrary, 2008; Cattaneo, Janssen, and
Ma, 2016). Estimates of treatment effects will be derived by employing both local linear
regression and polynomials. Optimal bandwidth will be determined by three bandwidth
selectors: Ludwig and Miller (2007), Imbens and Kalyamanaraman (2012), and Calinco, Cattaneo,
and Titiunik (2014). Polynomials are capped at the third order to avoid noisy estimates (Gelman
and Imbens, 2014; Gelman and Zelizer, 2015).
Although not presented in the essay global effects were estimated with models that
included school fixed effects and school type using first, second, third, fourth, fifth and sixth order
polynomials with their respective interactions with the treatment term (Jacob et al, 2012). The
results are not presented because a) none of the treatment estimates were statistically
significant and b) the cross-validation band selectors for local second and third order polynomials
cover almost the entirety of the sample space – making them virtually global treatment
estimates.
2.5 The Assumptions of Regression Discontinuity are Met Heaping on either side of the cutoff may bias causal estimates of the treatment effect of
remediation based on HESI exit exam scores. If remediation is effective students who are able
to manipulate their scores just enough to meet the cutoff may upwardly bias estimates of
remediation effects since those are who manipulated the HESI score would be less ready for pass
the NCLEX-RN score than those just below the threshold who were helped by the remediation.
Conversely, if the remediation has a negative impact on NCLEX-RN performance then the
38
treatment effect estimates would be downwardly biased. Students may also have an incentive
to manipulate HESI exam scores just below the established benchmark. If not meeting the
benchmark does not carry a significant penalty (i.e. lower course grade, losing eligibility to
graduate and/or sit for the NCLEX-RN) and remediation grants access to NCLEX-RN readiness
resources not available otherwise (i.e. one-on-one tutoring and mentoring) then students may
also have an incentive to manipulate HESI exam exit scores just below the benchmark. In this
case, if remediation has a negative impact the treatment effect estimates may be upwardly
biased. If the impact of remediation is negative, treatment estimates would be biased in the
opposite direction. If remediation has no impact, the direction of the bias would upward if
manipulation lands students above the cutoff and downward if they land below it. Table 8
captures the direction of bias based on the direction of manipulation and treatment
effectiveness. Under the assumption that not meeting the benchmark carries more disincentives
than incentives, if present, the modal manipulation would be above the benchmark.
Table 8. Direction of Bias Based on Remediation Effectiveness and Direction of Manipulation Remediation Positive Remediation Negative Remediation Neutral
Manipulation Above Upward Bias Downward Bias Upward Bias Manipulation Below Downward Bias Upward Bias Downward Bias
2.5.1 Manipulation of the Running Variable: Visual Evidence
I find no visual evidence of manipulation of the running variable induced by requiring
remediation for not meeting a HESI exit exam benchmark. Figure 6 displays the distribution of
centered HESI exit exam scores in 30 point bins – one fourth of a standard deviation- by required
39
remediation policy status. 4 Both distributions are normal and have their peaks to the right of
their benchmarks (indicated by the red reference line). Most of the observations in both
distributions are also found to the right of the benchmarks. The median centered HESI score is
20.5 for the students in schools with required remediation and 40.5 for those without. This
suggests that whatever heaping exists around the cutoff may not be triggered by something other
than the required remediation policy.
Figure 6. Distribution of HESI Exit Exam Scores by Required Remediation Status
2.5.2 Manipulation of the Running Variable: Statistical Evidence
I find that required remediation does not induce manipulation of the running variable. I
use the McCrary test (2008) using a variety of a bandwidths. I start with a 5-point bandwidth and
increase it by 5 point increments until they reach the maximum of 250 points (slightly less than
two standards deviations). None of the 47 bandwidths tested yield statistically significant results.
4 Bin widths of 15 and 7.5 display the same pattern. See Appendix L.
0
.002
.004
.006
-1000 -500 0 500-1000 -500 0 500
Remediation Not Required Remediation Required
Den
sity
Centered HESI Exit Exam Score
40
See results in Appendix B. Results of the heaping using test proposed by Cattaneo, Janssen,
and Ma (CJM) also suggest no manipulation. See Appendix C for results.
2.5.3 Continuity of Pre-treatment Variables: Statistical Evidence
The full sample of schools with required remediation and a benchmark of 850 or 900 does
not have demographics, academic readiness, and nursing course performance data. More than
720 students at one of these schools with a benchmark of 850 have such data. To assess the
continuity assumption needed to derive unbiased treatment effects of remediation under the
characterization of regression discontinuity design as discontinuity at the cutoff, each
demographic and academic variable is treated as a dependent variable in a regression
discontinuity model under three functional specifications (Hahn, Todd, and van der Klaauw,
1999). If any of the variables display at discontinuity at the cutoff point then the effect of
remediation could be confounded with that of those variables. Some evidence exist that the
continuity assumption is not met for a single nursing course but the finding is highly sensitive to
functional form and does not hold under a falsification exercise. I, thus, conclude that the
continuity assumption holds. See Appendix D for results and further discussion.
2.5.4 Covariate Balance: Statistical Evidence
Covariate balance in pre-treatment variables to assess the unconfoundedness
assumption of regression discontinuity design as a localized experiment is also tested using, once
again, the subset of the full sample with pre-treatment measures (Lee and Lemieux, 2010). Table
9 captures the differences in the student characteristics, measures of pre-admissions academic
aptitude and performance in nursing courses 10 points above and below the cutoff of 850. There
are no statistically significant differences at that bandwidth and at any smaller bandwidth.
41
Statistical significant differences in performance in nursing courses begin to appear when the
bandwidth is extended beyond 10 points. For student characteristics and pre-admissions
academic aptitude measures, the bandwidth can be extended to 50 points before statistically
significant differences are detected between students subjected to remediation and those who
were not. See Appendix E for details. The findings of balance in all pre-treatment covariates
within 10 points of the benchmark are extrapolated to the larger samples to find the local average
treatment effect (LATE).
Table 9. Summary of Pre-Treatment Variables by Remediation Status: 10 Point Bandwidth
Variables No Remediation Remediation Difference
Student Characteristics
Non-White 63% 62% 1%
White 37% 38% -1%
Male 10% 5% 4%
Main Campus 48% 57% -13%
EOF 4% 5% -1%
Pre-admission Academics
Composite SAT 1110 1145 -35
SAT Verbal 549 566 -17
SAT Math 561 579 -18
Nursing Courses
Pathophysiology 3.4 3.5 -0.1
Health Assessment 3.5 3.3 0.2
Foundations I 3.4 3.1 0.3
Child Bearing Family 3.0 3.1 -0.1
Health and Illness Children 3.3 3.5 -0.2
Health and Illness Adults I 2.5 2.7 -0.2
Foundations II 3.2 3.3 -0.1
Research Process 3.2 3.3 -0.1
Pharmacotherapeutics 3.2 3.4 -0.2
Psych/Mental Health 3.7 3.4 0.2
Health and Illness Adults II 3.3 3.4 -0.1
Observations 22 21
% of treatment/control 6% 7%
**statistically significant at 95% level *** statistically significant at 99% level
42
2.6 Results In this section, RD estimates are discussed and then followed by an overall conclusion
2.6.1 Estimation of Treatment Effects: RD as Discontinuity at the Cutoff
Treatment effects are estimated using three different bandwidth estimators under first,
second, and third order polynomial specifications. All specifications used a triangular distribution
that gives more weight to observations near the cutoff (Pereillon, 2013). As Table 10 points
out, all cross-validation (CV) estimates are not statistical significant. It is the same for estimates
derived using the CCT bandwidth selector. The second order specifications of the models using
the Imbens and Kalyamanaraman (IK) selector, however, yielded negative statistically significant
results.
Table 10: Treatment Effects Coefficients By Bandwidth Selector
Bandwidth Selector CV IK CCT
Polynomial Order 1st 2nd 3rd 1st 2nd 3rd 1st 2nd 3rd Bandwidth (b) 184 462 438 108 74 91 75 111 134 Bandwidth (h) 184 462 438 79 134 77 119 162 189
Methodology β β β β β β β β β
Conventional (b) 0.01 0.01 0.02 0.03 0.03 (0.02) 0.03 0.02 0.01 Bias-corrected (h) 0.02 0.02 0.03 0.02 (0.26) †† (0.05) 0.03 0.02 0.00 Robust (h) 0.02 0.02 0.03 0.02 (0.26) † (0.05) 0.03 0.02 0.00
†statistically significant at 95% level ††statistically significant at 99% level
Statistically significant results are sensitive to trimming of the sample by five percent of
the observations at either end of the distribution. Trimming renders the bias-corrected estimate
for the second order polynomials statistically insignificant.5 Robust estimates lose their
5 Conversely, trimming the sample by 5 percent for all the model specifications covered
in Table 4 does not generate any additional statistically significant results.
43
significance when the bandwidths are decreased by 10 or 20 percent. See Appendix E for results
and discussion. “RD as a local randomized experiment” estimates also suggest no treatment
effects. They can be can be found in Appendix F.
2.6.2 RD Estimates Conclusion and Discussion
Overall, most RD estimates suggest that remediation has no impact on the probability of
passing the NCLEX-RN on the first attempt. This may be the result of a ceiling effect (Scott-
Clayton and Rodriguez, 2012). RD results that suggest an impact are sensitive to kernel functional
assumptions, sample trimming, and/or bandwidth. Resources allocated to remediation by both
nursing programs and students alike do not seem to be having their intended impact. The lack
of effectiveness may also be in part to their large percentage of students who are required to
remediate. From the nursing programs with HESI exam exit with required remediation and
benchmarks of 850 or 900, about two out of five students partake in remediation. Nursing
programs may not have adequate resources to cover such a large proportion of students who are
finishing their degrees. Also, since so many students are asked to remediate remediation may
also be regarded as a “common and normal experience”. The expectation/high probability of
having to remediate may be a disincentive to students to dedicate the necessary time to study
for the HESI exit exam.
2.7 Robustness Checks To assess the stability of the potential impact of the remediation on the likelihood of
passing the NCLEX-RN on the first attempt in the previous sections, outcomes of students subject
to remediation in institutions where remediation is required are compared to students in
44
institutions where remediation is not required over the same range of HESI exit exam scores.
The comparison is done in two ways: logistic regression analysis and propensity score matching.
For both approaches, the analytical sample is restricted to institutions with required
remediation and that have benchmarks of 850 and those institutions that do not require a
remediation and do not have a benchmark. This ensures that all the observations from the
institutions with required remediation receive the intervention and all the observations from
institutions that do not require remediation do not receive the intervention.
2.7.1 Logistic Regression
The logistic regression model takes the following form:
Yi = β0 + β1Remediationi + β2HESIi + βjInstitutioni + βdInstitutionTypei + εi
- where i indexes students, Y is the outcome, β0 is the average value of the outcome for those in
the treatment group conditional on the HESI score, β1 captures the association between
remediation and the outcome [where Remediation=1 if x<850 HESI exit exam score, attendance
at institution where remediation is required with a benchmark of 850, Remediation=0
otherwise], β2 captures the relationship between the outcome and HESI scores for both the
treatment and control group, each βj captures institutional fixed effects and each βd captures the
association between institutional type and the outcome.
The results of the logistics regression analysis converge with those of the regression
discontinuity analysis. As Table 11 points out, holding constant school fixed effects and school
type, no statistically significant association was detected between remediation and passing the
NCLEX-RN on the first attempt. The average marginal effect of treatment was also statistically
45
not significant and had a coefficient of .13. Including observations with HESI exit exam scores
above 850 and adding a dummy variable indicating attendance at an institution with required
remediation policy does not change the direction or the significance of the treatment variable.
See Appendix F.
Table 11. Logistic Regression Results (controlling for fixed school effects and school type effects)
Variable Odds Ratio Std. Err. z P>z UB LB
HESI Score 1.01 0.00 12.50 0.00 1.01 1.01
Treatment 2.34 1.90 1.05 0.30 0.48 11.47
Marginal effects over the HESI score response surface suggest the possibility of positive
treatment effects at the lower end of the HESI score distribution. Although statistically
insignificant at each 25-point interval, Chart 4 illustrates that the marginal effects of remediation
increase at a decreasing rate up to the 600-point mark. Starting at the 700-point mark, marginal
effects decrease at an almost monotonic rate. This suggests the possible existence of
heterogeneous treatment effects which will be explored in Section 2.8.
Figure 7. Remediation Marginal Effects over the HESI Score Response Surface
46
2.7.2 Propensity Score Matching Analysis
To potentially reduce bias induced by imbalance in the pre-treatment variables,
propensity score matching is leveraged to estimate treatment effects. The matching model takes
the form:
Yi = β0 + β1HESI + βdInstitutionTypei + εi
- where Yi is remediation and the rest of the terms are as previously defined. Since, in the
restricted sample, in any given school all students are either subject to remediation or not subject
to remediation, institutional effects cannot be included in the model. The matching model used
logistic regression with no replacement. Covariate balance was achieved. See Appendix E.
Results were not sensitive to symmetrical trimming of the sample by five percent. In the
outcome model, observations were restricted to the area of common support. See Appendix F
for graph of area of common support.
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
20%
400 425 450 475 500 525 550 575 600 625 650 675 700 725 750 775 800 825 850
HESI Score
47
The PSM results suggest no treatment effect. Table 12 summarizes pre and post
matching differences between the two groups. Prior to matching, students who received
remediation in schools with required remediation were statistically less likely to pass the NCLEX-
RN on the first attempt. Once matched, the difference disappears. The finding holds with
covariate adjusted PSM6 7. Overall, the PSM results converge with the logistic regression analysis
results.
Table 12: Pre and Post-Matching Differences Outcomes between Treated and non-Treated Variable Sample Treated Controls Difference S.E. T statistic
NCLEX-RN Pass Unmatched 0.80 0.83 -0.03 0.02 2.00 NCLEX-RN Pass ATT 0.79 0.83 -0.03 0.04 0.82
2.8 Looking for Heterogenous Treatment Effects8 Regression discontinuity cannot detect treatment effects far from the cutoff. In this
section, logistic regression and PSM are used to ascertain the possible existence and magnitude
of such effects.
6 Running the matched sample (in the area of common support) through an outcome model containing the same independent variables as the matching model. 7 The covariate adjusted PSM results are not the result of the violation of one or more of the three conditions needed for unbiased estimates under the usual PSM assumptions (Rubin, 2001). Roughly, the three conditions are: a) the difference in average propensity scores must be less than half a standard deviation, b) the ratio of the variances of the propensity score in the two groups must be more than one half and less than two and c) the ratio of the variances of the residuals of the explanatory variables after adjusting for the propensity score must be must more than one half and less than two (Rubin, 2001). All three conditions are met. Specifically, the results approximate the ideal for all the conditions – parity for conditions b and c and .01 for condition a. 8 Using the same sample as in Section 2.7
48
2.8.1 Logistic Regression
To test for the presence of heterogeneous treatment effects of remediation, the following
model is formulated:
Yi = β0 + β1Remediationi + β2HESIi + βjInstitutioni + βdInstitutionTypei +βbBelow600i + βiBelow600 x Remediation + εi
- where the terms are as previously defined, βb quantifies the association between having
a HESI score below 600 and the outcome and βi captures the difference between the
observations below 600 who received remediation and those that did not.
The results from Table 13 suggest that students who score below 600 on the HESI exit
exam are less likely to pass the NCLEX-RN on the first attempt (marginal effect of -.12). At the
same time, those students with below 600 on the HESI exit exam who receive remediation are
more likely to pass the NCLEX-RN than those who did not receive remediation (marginal effect of
.14).
Table 13. Logistic Regression Results (controlling for fixed school effects and school type) Variable Odds Ratio Std. Err. z P>z UB LB
Remediation 2.02 1.70 0.83 0.41 0.39 10.52 HESI Score 1.01 0.00 9.36 0.00 1.01 1.01 Below 600 x Remediation 2.49 1.18 1.93 0.05 0.98 6.29 Below 600 0.45 0.19 -1.87 0.06 0.19 1.04
2.8.2 Propensity Score Matching
Once the analytical sample is further reduced to observations with HESI scores of 600
points and below, non-covariate adjusted PSM results do not support the logistic regression
findings of heterogeneous treatment effects among those students with HESI score below 600.
Table 14 summarizes the PSM results for the subgroup at the low end of the distribution.
49
Matching achieves covariate balance. See Appendix I. Prior to matching, students who received
remediation were 20 percent points more likely to pass the NCLEX-RN on the first attempt. Once
matched, the statistically significant difference disappears.
Table 14: Pre and Post-Matching Differences Outcomes between Treated and non-Treated
Variable Sample Treated Controls Difference S.E. T statistic
NCLEX-RN Pass Unmatched 0.48 0.28 0.20 0.1 1.98
NCLEX-RN Pass ATT 0.50 0.56 -0.06 0.2 0.37
The covariate adjusted PSM result also provides evidence that does not support the regression
analysis findings of heterogeneous treatment effects among those students with HESI score
below 600.
2.9 Conclusion and Policy Implications Overall, remediation most likely does not have an impact on students’ probability of
passing the NCLEX-RN on the first attempt for most students. Evidence suggests a possible
positive treatment effect but restricted to the low end of HESI score distribution. Lowering HESI
benchmarks to increase per student allocation of remediation resources to students who are
most likely to benefit from it may improve NCLEX-RN outcomes.
State Boards of Nursing are charged with informing consumer choice. Historically, they
have fulfilled this obligation by making publicly available lists of accredited nursing programs
along with their first-time NCLEX-RN pass rates. For prospective consumers, first-time pass rates
are the primary indicator of program quality.
50
Nursing programs, however, can artificially inflate their first-time pass rates by restricting
the number of their students who will be allowed to sit for the NCLEX-RN. Exit exams have been
the primary tool to restrict access to the NCLEX-RN. In some instances, students who do not
meet exit exam cutoff are automatically removed from the program. In other cases, students
who do not reach the exit exam cutoff are required to remediate and retake the exit exam until
they meet the cutoff. If they do not meet the cutoff they cannot take the NCLEX-RN. Still in
other cases, those who do not meet the exit exam threshold are required to remediate but not
to retake the test. If students refuse to remediate or do not meet the remediation standard then
they are not allowed to sit for the NCLEX-RN.
To address the potential for manipulation of first-time NCLEX-RN pass rates,
National Council of State Boards of Nursing (NCSBN) should encourage State Boards of Nursing
to develop and make publicly available new metrics of nursing programs productive efficiency.
The first-time pass rate is easily calculated and understood. For a given program, it is the number
of students who passed the NCEX-RN on the first attempt divided by the number of students who
took it for the first time. In addition to first-time pass rates, State Boards of Nursing should
employ two simple cohort-based metrics. The first would be the number of students who passed
the NCLEX-RN on the first attempt within a year of expected graduation divided by the number
of students in a given cohort. In practice, this would treat the students who left the program or
did not take the NCLEX-RN [either because they were not allowed to take it or because they chose
not to] as having failed it. Thus, it would “punish” programs for attrition. The metric would give
the public, the average probability of passing the NCLEX-RN on the first time of a student upon
entering the program. The second metric would be the number of students who were not
51
allowed to sit for the NCLEX-RN after successfully completing their nursing courses divided by the
number of students in their cohort. This “sequester rate” would help state boards of nursing
and state nursing workforce centers identify how many students are potentially kept from the
nursing labor force, and, as such, may be contributing to nursing shortages. This metric would
be of particular interest to the public since it would indicate the percentage of students who after
having allocated considerable resources to their nursing course of study and having performed
well on nursing courses were not allowed even to sit for the NCLEX-RN.
Since exit exams have robust predictive validity for those who will pass the NCLEX-RN but
not for those who will fail, state Boards of Nursing should prohibit nursing programs from not
allowing their students to sit for the NCLEX-RN solely based on exit exam scores and/or
subsequent remediation outcomes. If exit exams are used in the decision to prohibit students
from taking the NCLEX-RN the cutoff score should not be above the score associated the
minimum pass rates established for programs to stay out of probation by state boards of nursing.
For example, in a state where the state board of nursing sets the minimum first-time pass rates
for nursing program to maintain accreditation at 80 percent, the HESI exit exam cutoff could not
exceed 650. In the analytical sample, the implementation of such a cutoff would reduce the
number of students required to undergo remediation by approximately 90 percent.
The current essay is limited by its data. The similarity of curricula across nursing programs
and increasing use of HESI end-of-course exams could facilitate the possibility of having
standardized nursing course outcome data to include in analyses. The inclusion of such data
could improve the precision of treatment estimates and provide the necessary inputs to test
covariate balance as localized random experiment.
52
Chapter 3: Testing a Rule of Thumb: For STEM
Degree Attainment, More Selective is Better
3.1 Introduction The rise of the internet and other digital technologies in the mid to late 1990’s increased
demand for employees with STEM backgrounds (Coble and Allen, 2005). During that time, the
wage gap between the college educated and the rest of the population continued to widen at an
accelerated rate as markets became increasingly liberalized (Autor and Dorm, 2009). Thus, post-
secondary education in technical fields became increasingly perceived not only as a way to stay
globally competitive but to achieve equitable social outcomes domestically (Russell and Atwater,
2005) (Ma, 2009). By 2005, more than three fourths of the fastest growing fields were in science
and technology (Coble and Allen, 2005). From 2004 to 2008, the demand for science and
engineering workers increased at double the rate of the non-STEM sectors (Bureau of Labor
Statistics, 2008).
The supply of STEM workers depends to some degree on the ability of post-secondary
institutions to keep those students already interested in STEM engaged and to generate interest
among those initially not interested. High attrition rates among college students in STEM are
well recorded and give pause to high school students considering STEM college majors and their
guidance counselors and parents (Grandy, 1998; Bonous-Hammarth, 2000; Benbow et al, 2010).
About half of STEM majors do not get their bachelor’s degree within six years (Chen, 2013). The
institutional attributes which may exert positive or negative influences on STEM degree
attainment are many and students and parents may not be able to assess the status of each factor
53
or a bundle of factors for specific institutions in their college choice sets. In this regard, a rule of
thumb may prove useful to help overcome the “dark side of choice,” (Scott-Clayton, 2012).
In this essay, I test the rule of thumb that, for STEM students, attending a highly selective
institution instead of a moderately selective institution improves the probability of obtaining a
STEM degree at the first attended institution among those interested in STEM among and among
those who are not initially interested. The potential formal mechanisms for the hypothesized
advantage of highly selective institutions cover a wide range. Frequency of faculty-student
interactions, participation in STEM peer groups, the size of the graduate school population,
participation in STEM research activities or co-op programs, peer effects and membership in
student learning communities have been found to have exerted a positive influence on STEM
persistence (Leslie et al; 1998; Springer et al, 1999; Gloria et al, 2005; Hernandez, 2005; Cole and
Espinoza, 2008; Hurtado et al, 2009; Ost, 2010; Griffith, 2010; Light and Micari; 2013). On the
other hand, attending a highly selective institution may hinder STEM degree attainment since
average institutional achievement has a negative relationship to academic self-concept in science
(Nagengast and Marsh, 2012). The challenge of isolating the impact of selective institutional
status on STEM degree attainment is controlling for self-selection. Students who opt to attend
highly selective institutions are systematically different among observable and non-observable
characteristics from those who attend other types of institutions.
Inspired by Krueger and Dale (1998), I estimate of the impact of attending a highly
selective post-secondary institution (vs. attending a moderately selective institution) on STEM
degree attainment among students interested in STEM at the beginning of their collegiate careers
through matching. To address potential omitted variable bias, I include total number of
54
applications and number of admissions to highly selective and moderately selective institutions
for each student in matching model which confers information on typically unobserved student
characteristics such as motivation. I use regression on the matched sample to more precisely
estimate treatment estimates (covariate adjusted PSM estimates). To assess the impact of the
possibility that theoretically relevant variables were not included in the models, I subject
unadjusted propensity score matching (PSM) treatment estimates for the general population to
the Mantel and Haenzel (MH) test.
Overall, I find that highly selective institutions have a comparative advantage in producing
STEM graduates among those already interested in STEM but not among those initially not
interested in STEM. Attending a highly selective post-secondary institution has a positive impact
on STEM degree attainment among those interested in STEM fields. They are 16 to 30 percentage
points more likely to earn their degree at their first attended institution than their peers at
moderately selective institutions. On the other hand, among those not interested in STEM fields,
attending a highly selective institution makes them two to three percentage points less likely to
graduate with such a degree. For females, the negative association is one percent or non-
existent.
The rest of the essay is structured as follows. First, competing definitions of STEM are
discussed. Review of the literature follows. The next section provides an overview of the data
source. The analytical sample is described in the subsequent section. The next section explains
the methodology. The penultimate section is dedicated to results and the final section provides
an overall conclusion.
55
3.2 Competing Definitions of STEM
3.2.1 STEM as a Career
What constitutes a STEM – formerly known as “SMET” - career has varied over time and
across stakeholders (Sanders, 2009) (Zinth, 2006). The most narrow STEM definition restricts
STEM employees to individuals directly involved in the physical and biological sciences,
mathematics, information and computer sciences, and electrical, chemical, civil, and mechanical
engineering. The majority of studies on the subject and the most recent legislative efforts to
influence STEM education have used this highly focused definition (Chen and Weko, 2009). For
example, in the report STEM Education: Preparing Jobs for the Future by the U.S. Congress Joint
Economic Committee, STEM careers only extend to “life sciences (except medical sciences),
physical sciences, mathematics and statistics, computing and engineering” (1). The U.S.
Department of Commerce definition of a STEM career has slightly broader parameters and
includes individuals in managerial positions in STEM fields (2012).
More inclusive definitions of STEM careers also exist. In some cases, they incorporate the
behavioral sciences in addition to the afore-mentioned fields. The National Science Foundation,
for example, groups economics, psychology, sociology and political science under its STEM
umbrella (Chen and Weko, 2009). The Organization for Economic Development and Cooperation
(OECD) goes even farther by including “manufacturing, processing, architecture and building” in
its STEM definition (van Langen and Dekkers, 2005).
Unable to reach consensus on a linguistic definition of a STEM career, researchers have
attempted to bring more clarification on the topic by applying quantitative techniques. Koonce
et al (2011) looked at Standard Occupational Classification (SOC) Codes of the Bureau of Labor
56
Statistics from years 2000 and 2010 to ascertain the likelihood that a certain code would be
categorized as a STEM discipline by “papers, conference reports, websites for programs,
government documents, and other statistical data” (4). Formally,
p= (nj-ne)/N
where the term ni is the number of definitions where the code is included, ne is the number of
definitions where the code is explicitly excluded, and N is the total number of definitions. Using
eleven sources for STEM definitions, the authors find that the disciplines most likely to be
included in those STEM definitions were as follows:
P Score Program (SOC Codes 2000)
The vast majority (684 out of 860) of SOC codes, however, were never included in any of the 11
STEM definitions. Although the authors’ results do not give a definitive answer to what
57
constitutes a STEM career, they help identify which disciplines are never or almost never
considered to fall under the STEM career umbrella. The competing definitions of a STEM career
are not merely a question of semantics. They represent a challenge to the interpretation and
comparison of empirical findings on the subject.
3.2.2 STEM as a College Major
The ambiguity about the definition of STEM also extends to college majors. The National
Center for Education Statistics (NCES) of the U.S. Department of Education produces more than
2,200 Classifications of Instructional Program (CIP) codes. NCES does not categorize any of the
CIP codes as STEM-related or non-STEM related. Federal agencies have selected different CIPs
as indicating a STEM college major. U.S. Immigration and Customs Enforcement places more
than 100 CIPs under its definition of a STEM major. These are further divided into six different
subcategories: computer and information sciences, engineering and engineering technologies,
biological and biomedical sciences, mathematics and statistics, physical sciences, and science
technologies. This demarcation of what is considered a STEM major allows the U.S. Immigration
and Customs Enforcement to offer work visas to foreign students majoring in those fields (U.S.
Department of Education, 2011). In contrast, research sponsored by the National Science
Foundation has defined STEM majors based on the amount of math and science courses required
(Hartwell, 2012). Under this definition, nursing is a STEM major.
In the same way that they attempted to refine the meaning of a STEM occupation, Koonce et
al (2011) used the afore-mentioned CIP codes to decipher which college majors were most likely
to be designated as STEM. Like their SOC counterparts, a sizable fraction of CIP codes were not
consistently included in the 40 separate STEM definitions. More than one third of them were
58
considered STEM in less than five percent of the definitions (two out of 40). A sensible approach
to starting the process of standardizing a definition of a STEM major might be to first consider
the removal of these CIP codes from the definition. Mathematics, physical and computer sciences
were the most likely to be considered STEM in this case.
P Score Program (CIP 2000 Code)
0.875 Mathematics General
0.800 Chemistry, General
0.775 Computer Science, Biology/Biological Sciences, General
0.750 Physics, General
0.725 Mathematics, Other
The authors’ intuitive analyses suggest an overall lack of consensus on a working understanding
of what should be designated as a STEM major or as a STEM career. This is further exacerbated
by an even more ambiguous link between STEM majors and STEM occupations. This is
problematic for ascertaining the effectiveness of policy interventions directed at increasing the
number of STEM workers in the U.S. via increases in the number of students in STEM disciplines.
The essay, in its review of the literature and its analyses, restricts STEM students to those directly
involved in the physical and biological sciences, mathematics, information and computer
sciences, and electrical, chemical, civil, and mechanical engineering (Chen and Weko, 2009).
3.3 Review of the Literature
59
3.3.1 Interest in STEM Careers/Majors
The STEM interest related literature is extensive. Overall, the literature reaches relative
consensus on personal traits associated with interest, commitment, pathways to and persistence
in STEM. The roles of family and schools are more nuanced and admit to a greater variety of
findings. The heterogeneity of conclusions is concomitant with the analytical samples and
modeling choices used to explore their respective hypotheses.
The literature suggests that interest in STEM is shaped and solidified after elementary
school. Eighth graders who reported interest in future science careers at age 30 had almost
double the odds of getting a life science degree and more than three times the odds of getting a
degree in a physical science (Tai et al, 2006). STEM students take more academically rigorous
courses in HS, perform better academically, come from more affluent families and are more likely
to study full-time while in college (Chen and Weko, 2009). Students who enter IT majors at
unconventional ages do not have this profile (Chen and Weko, 2009). Math achievement,
number of math courses taken and math efficacy have a positive impact on a student’s decision
to select a college major in a technical field (Ma, 2009). Perception of better-than-average
quality of instruction in math and science during high school is positively associated with the
decision to pursue a physical science college major (Leslie, McClure, and Oaxaca, 1998). Harris
Interactive (2011) finds that parents and guidance counselors were influential in students’
decision to pursue STEM majors, females were primarily motivated by intellectual challenge
while males were motivated by monetary compensation. More than 75% of students decided to
pursue STEM while in HS (Harris Interactive, 2011). STEM summer program for gifted high school
students does not increase STEM identity and STEM salience among participants (Lee, 2002).
60
Participation of women in tertiary STEM education increases as the achievement gap between
females and males in secondary school decreases (van Langen and Dekkers, 2005). Female
students attending single-sex schools are not more likely to enter male dominated college majors
(Jennifer Thompson, 2003). The same holds true for male-only schools (James and Richards,
2003). Both boys and girls in single sex schools are more likely to enter more “gender neutral”
fields but not fields historically dominated by the opposite sex (Karpiak, Buchanan, Hosey, and
Smith, 2007).
A considerable number of studies associate personal and family characteristics with
developing and maintaining STEM interest. Mathematical ability is not the only determinant of
vocational choice, even for those who opt for careers in STEM (Benbow et al, 2010). Verbal and
spatial ability also play a role. In addition, males are more likely to report STEM interest (Benbow
et al, 2010). An increase in socio-economic status is associated with a decreased likelihood of
choosing a technical field (Ma, 2009). Female students from families with lower levels of socio-
economic status are as likely to choose a technical field as their male counterparts (Ma, 2009).
Mothers with college degrees increase the likelihood of their daughters getting an advanced
degree in physical science/engineering while the opposite was true for their sons (Leslie,
McClure, and Oaxaca, 1998). Expectation of graduate school attendance also increased the
likelihood of choosing a major in physical sciences, engineering or math (Leslie, McClure, and
Oaxaca, 1998). Plans of marriage shortly after college graduation increased the likelihood of
choosing a physical science/engineering major for white females and Hispanic males (Leslie,
McClure, and Oaxaca, 1998).
61
Overall, students interested in STEM are systematically different from those who are not.
They are more academically ready than those who are not. Interest in STEM is not stable in junior
high school, high school, or college. Together this suggests that students in college could develop
interest in STEM in college and that the environment in which this interest is maintained and
fomented among these students could be different from the environment in which students
already interested find it easier to maintain theirs.
3.3.2 College Choice
College choice process has typically been conceptualized as having three distinct stages
with each stage having a set of hypothesized inputs and resulting outcomes (Hossler, 1989;
Cabrera and LaNasa, 2000). These stages are not strictly compartmentalized. They overlap and
interact with each other (Cabrera and LaNasa, 2000). The initial phase is typically referred to as
the predisposition phase which may begin as early as seventh grade. Factors external to the
student (i.e. family and school resources) dominate the input side while academic skills and
educational and career aspirations constitute the primary outputs. The outputs of the
predisposition stage, in turn, become part of the inputs of the subsequent stage: search. Taking
place during the last three years of high school, the perceived attraction of potential post-
secondary institutions of attendance is the characteristic input of the penultimate phase. The
resulting outcomes are formulating a list of potential institutions, gathering information about
them and finalizing a choice set for decision-making. (Cabrera and LaNasa, 2000). The search for
appealing post-secondary alternatives is a dynamic process in which newly discovered knowledge
about a potential institution may lead to further research into previously unconsidered
62
institutions or to the rethinking of search criteria itself (Hossler, Schmit and Vesper, 1998 ; Dawes
and Brown, 2002; Moogan and Baron, 2003). In the choice stage during the junior and senior
years of high school, perceptions of affordability and institutional attributes become the most
salient inputs while post-secondary applications, registration, and attendance are the results of
interest (Cabrera and LaNasa, 2000).
Three types of models have been primarily used to explore college choice (Hossler and
Palmer, 2007). Economic models assume that agents (i.e. students and parents) are rational and
evaluate the relevant information for the decision at hand, including personal preferences, to
arrive at a utility-maximizing decision (Hossler, Braxton, & Coopersmith, 1985; Hossler, Schmit,
& Vesper, 1999; Gemici et al, 2014; Wiswall & Zafar, 2015; Reuben et al, 2015). Studies using
economic models often explore the role of college costs and financial constraints on college
choice. Sociological models, in contrast, focus on how personal and contextual factors contribute
to the various decisions in the college choice process. Finally, mixed models split the difference,
acknowledging that decision-making takes place within textured contexts (Perna, 2006). The
matching and outcome models in this essay are mixed. They include measures of income which
may not only influence the decision to select a highly selective institution over a moderately
selective one but also STEM degree attainment. The hypothesis tested in this essay is implicitly
sociological in that it seeks to ascertain if the academic environment in which students carry out
their studies may influence STEM degree attainment.
3.3.3 Matching in the College Choice Process
63
The ideal outcome of the college choice process is fit. Fit may be best characterized as
the degree to which applicants’ overall expectations are met by an institution (Cabrera et al,
2011). The better the fit the more likely that student goals will be achieved. The expectation may
include proximity to family and friends, academic reputation/rigor, offerings of majors and/or
programs, characteristics of the student body, time-to-graduation and post-graduation market
outcomes. Match, for its part, is only concerned with a single dimension of fit - the
correspondence of a student’s academic ability to the modal or average academic profile of an
institution (Cabrera et al, 2011). In this regard, students may construct choice sets which include
“safety” (i.e. under-matched), “par” (i.e. matched) and “reach” (i.e. over-matched) schools
(Hoxby and Avery, 2012).
The literature suggests that the majority of low-income high achieving students do not
apply to selective institutions. Not applying may stem from not having a network of achievement
peers from whom to learn and emulate optimal behavior in the college choice process nor
teachers/counselors to guide and/or encourage them in that process (Hoxby and Avery, 2012).
Expanding College Opportunities (ECO) project which provided tailored information related to
the college choice process and “paper-free” application fee waivers to approximately 175
selective post-secondary institutions to high-achieving (i.e. 1300 combined Math and Verbal SAT
or 28 ACT or higher) low-income (i.e. bottom third of the income distribution) students increased
the number of applications by 19 percent and the probability of college match by slightly more
than 40 percent (Hoxby and Turner, 2013). The application patterns of low-income students
present a challenge to any matching methodology that uses income and/or number of
applications/admissions to highly selective institutions to assess the impact of attending highly
64
selective institutions. The low number of low income students who apply to these institutions
diminishes the pool of students from which matches can be drawn.
Under-matching is found over the entire academic performance distribution, but is more
pronounced in rural areas and among those from families with lower levels of income and
educational attainment. Despite decreases in under-matching over time, more than 40 percent
of students of the graduating class of 2004 were still under-matched (Smith, Pender and Howell,
2013).
The literature links institutional selectivity to better student outcomes. The gap in
graduation rates between the most selective institutions and the least selective exceeds 30
percentage points (Bowen, Chingos, and McPherson, 2009). The increase in the average time-
to-degree is driven by institutions at the lower end of the selectivity distribution that experienced
declines in overall funding eroding student support services and increased the direct cost of
attendance (Bound, Lovenheim, and Turner, 2012). To cover costs, students increased their time
dedicated to work, which in turn, diminished the availability to engage in academics. Hoekstra
(2009) finds a 20 percent wage premium for having attended the flagship university.
The cost of attendance may also contribute to lack of overall fit. Four and six-year
graduation rates increase as net price increases conditional on student SAT scores (Smith et al,
2013). Thus, in choosing a post-secondary institution, students face an inherent trade-off
between cost and quality. The cost of attendance may also be related to favorable labor market
outcomes. Students who attend institutions with higher tuitions- not those who attended more
selective institutions - earn more than their peers (Krueger and Dale, 1999). In a follow-up study,
long-term elite school earnings premium, however, was found among Hispanic and Black
65
students and those from disadvantaged backgrounds (Krueger and Dale, 2011). At the same
time, cost may also hinder access to post-secondary education. The literature converges on the
overall negative relationship between tuition cost and enrollment but the magnitude of the
relationship varies by type of admitting institution and student characteristics (Heller, 1997).
Conditional on financial aid received and student characteristics, non-Asian minority students are
more sensitive to grants and loans than their white counterparts (Heller, 1997). Some of the
differences in reactions to the cost of attendance among student subgroups may be explained by
imprecise perceptions of the cost of college attendance, late acquisition of financial aid
information and difficulty in navigating the financial aid process among lower income and
minority students (Gronsky and Jones, 2004: Heller, 2006; Dynarsky and Scott-Clayton, 2006 ).
Over-matching may not hinder student outcomes. It does not increase time-to-degree or
decrease course loads (Kurlaender and Grodsky, 2013). Others find evidence that increased
selectivity is not always better or neutral. Aggregate school performance exerts a negative
influence on student academic self-concept while individual academic achievement makes a
positive contribution to academic self-concept (Marsh and Hau, 2003). Something similar
happens with students interested in STEM careers, average institutional achievement had a
negative relationship to academic self-concept in science while individual achievement was
positively related (Nagengast and Marsh, 2012). Affirmative action-induced overmatching
among students leads to lower academic outcomes for those students who benefitted from it
(Sander, 2004).
66
3.3.4 STEM Attrition and Institutional Selectivity
Student and post-secondary institutional characteristics have been associated with STEM
degree completion. Academic achievement in high school has been linked to persistence in
STEM during college. Students with higher high school GPAs and SAT scores have lower rates of
attrition and higher achievement (Grandy, 1998; Bonous-Hammarth, 2000; Benbow et al, 2010).
The academic rigor of high school math and science courses and performance in those courses
also predict success in STEM majors (Ellington, 2006; Anderson and Kim, 2006; Tyson et al, 2007).
More selective post-secondary institutions have been found to be more efficient at
producing STEM graduates (Eagen, 2009; Chen, 2013). Yet non-Asian minority STEM students at
these institutions do not have the same rates of success (Elliot et al, 1996; Chang et al, 2008;
Chang et al, 2010). The same is true for women of color (Espinosa, 2011). The different levels of
institutional selectivity have their own comparative advantage where more selective institutions
are better at graduating more academically prepared STEM majors and less selective institutions
are more efficient at graduating less academically prepared students (Arcidiacono, 2013). In
contrast to non-Historically Black College and Universities (HBCUs), as selectivity increases
among HBCUs the persistence rate of non-Asian minorities in science increase (Chang et al, 2008).
In addition, the organizational climate around STEM education has been linked to student
outcomes. High Faculty-student interactions, inclusion in STEM peer groups, the size of the
graduate school population and participation in formal research or co-op programs and student
learning communities have been found to have serve as a positive influence on STEM persistence
(Leslie et al; 1998; Springer et al, 1999; Gloria et al, 2005; Hernandez, 2005; Cole and Espinoza,
2008; Hurtado et al, 2009; Ost, 2010; Griffith, 2010; Light and Micari; 2013). Grades in first
67
college STEM courses and their relative standing to non-STEM course grades have been positively
linked to STEM degree attainment (Crist et al, 2009; Rask, 2010; Ost, 2010). In addition, STEM
courses in which students are more actively engaged in learning and/or delivery of content are
associated with higher rates of retention for STEM majors (Watkins and Mazur, 2013; Freeman
et al, 2014).
Three fourths of STEM majors make their decision to pursue STEM majors in high school
and the college choice facing them is not easy (Harris Interactive, 2011). As discussed above, the
factors associated with successful completion of a STEM degree are many and slightly less than
half of STEM majors do not earn their bachelor’s degree within six years (Chen, 2013). Even if
students, parents, and counselors are aware of the various institutional attributes which may
exert positive or negative influences on desired academic outcomes they may not be able to
assess the status of each factor or a bundle of factors for specific institutions in their college
choice sets. In this regard, a rule of thumb may prove useful to help overcome the “dark side of
choice,” (Scott-Clayton, 2012). In my third essay, I propose to test the rule of thumb that, if in
doubt, “more selective is better.”
3.4 Data The Education Longitudinal Study (ELS) of 2002 tracks a representative sample of high
school sophomores in 2002 through high school and into college and/or the labor market. The
sample was drawn from 752 schools out of a sampling frame of approximately 1,200 public and
private schools from a total population of about 27,000 schools with 10th grades. From each
school, a sample of 26 to 30 sophomores was randomly selected. The slightly more than 16,000
68
students participated in the study represented more than three million sophomores in 2002
(Ingels et al, 2014).
The baseline survey and its first follow-up contained information on the high schools they
attended, their teachers, peers, and families. The first two follow-ups were conducted at two-
year intervals in 2004 and 2006. The former captured most students at the end of their high
school careers. In addition to high school outcomes, it gathered information on long-term career
plans and the college choice process. The latter follow-up encapsulated their first years in post-
secondary education or the workforce. The third wave conducted in 2012 concentrated on
gathering information on post-secondary and labor market outcomes (Ingels et al, 2014).
3.4.1 Defining Interest in STEM Interest in ELS of 2002/06/12
Student interest in STEM is defined by student responses captured in variable F2B15. The
second follow-up to the Education Longitudinal Study of 2002 asks post-secondary attendees
which field of study they were most likely to pursue upon entering (F2B15)9. Specifically,
students are asked “When you began at [name of institution], what field of study did you think
you would most likely pursue?”. Response choices are categorized into 14 broad fields. For this
essay, interest in STEM is identified when respondents opt for either of three categories:
engineering or engineering technologies, computer or information sciences, and natural sciences
or mathematics.
9
69
3.4.2 Defining a Highly Selective Post-Secondary Institution in ELS of 2002/06/12
The variable F2PS1SLC captures the highest selectivity category of the first post-secondary
institution for each student based on 2005 Carnegie classifications from IPEDS (US Department
of Education, 2008). The variable contains six valid values: highly selective (F2PS1SLC=1),
moderately selective (F2PS1SLC=2), 4-year inclusive (F2PS1SLC=3), 4-year not classified
(F2PS1SLC=4), 2-year not classified (F2PS1SLC=5), and less than 2-years (F2PS1SLC=6). Roughly,
the highly selective category identifies institutions whose student bodies have average SAT/ACT
scores in the highest quintile of the distribution while the moderately selective category houses
colleges and universities in the adjacent quintile of the distribution. The rest of the institutions
in the remaining categories either do not report average ACT/SAT scores or do not require them
for admissions (US Department of Education, 2008).
3.4.3 Defining a STEM Degree in ELS of 2002/06/12
The variable F3TZBCH1CP2 captures the first known bachelor’s degree for each student.
Students were identified as having earned a STEM degree under the most narrowly defined
definition of STEM if they had one of the following values: 10=Communication Technology and
Support, 11=Computer/Information Science Support, 14=Engineering, 15=Engineering
technologies/technicians, 27=Mathematics and Statistics, 40=Physical Sciences and 41=Science
technologies/technicians. To identify that degree was earned from the first-attended post-
secondary institution, the F3TZBCH1CP2 variable was coupled with the variable F3PS1RETAIN
which summarizes the status relative to first-attended postsecondary institution at the third
70
follow-up. If they had obtained a STEM degree under the above-criterion, only those with values
of 1 or 2 (1=Earned a credential from first post-secondary institution attended; still attending
post-secondary institution as of the third follow up, 2=Earned a credential from first post-
secondary institution attended; no longer attending post-secondary institution as of the third
follow up, 3= Did not earn a credential from first post-secondary institution attended; still
attending first post-secondary institution as of the third follow up; 4= Did not earn a credential
from first post-secondary institution attended; not attending first post-secondary institution as
of the third follow up; did attend another post-secondary institution; 5= Did not earn a credential
from first post-secondary institution attended; not attending first post-secondary institution as
of the third follow up; did not attend another post-secondary institution) were identified as
having received them from their first post-secondary institution.
3.5 STEM Students Interest in STEM careers among high school students fluctuates during their high school
experience. In the sample, approximately 13 percent of the 55 percent of tenth graders who
responded to the question in the baseline survey regarding occupational expectations at age 30
reported plans of having a STEM career.10 By the first follow-up two years later, the 10 percent
10 The Education Longitudinal Study of 2002 probes high school sophomores about their long-term career plans (US Department of Education, 2004). Question 64 of the baseline survey asks, “Write in the name of the job or occupation that you expect or plan to have at age 30.” Below the blank line designated for the answer, two other options are found: 1. I don’t plan to work when I am 30 and 2. I don’t know. The verbatim responses provided by student may be interpreted in different ways. A response of “engineer” may mean a “train conductor” or a “chemist.” In all such cases, I assumed that the responses to indicate a STEM career. In the cases in which multiple career aspirations are expressed by a single student such student is considered to be a STEM career aspirant as long as one of them is STEM. The same question was posed to students in the first follow-up in 2004.
71
of the 68 percent of students who replied to the same question reported STEM career
expectations.11
As Table 15 points out about 17 percent of the students in the sample entered post-
secondary institutions with the idea of becoming a STEM major. In contrast to their non-STEM
peers, these students opted to attend a) highly selective institutions at higher rates and b) two-
year colleges at lower rates.
Table 1512. First post-secondary institution (PSI) by STEM interest as freshmen
Selectivity of First PSI Attended
Freshmen: STEM Freshmen: non-STEM
n % n %
Highly - 4 year 96,820 30% 322,830 17%
Moderately - 4 year 94,020 29% 498,810 27%
Inclusive - 4 year 24,730 8% 131,100 7%
Selectivity not classified - 4 year 18,680 6% 90,680 5%
Selectivity not classified - 2 year 81,190 25% 744,870 40%
Selectivity not classified - less than 2 years 6,110 2% 66,010 4%
Total 321,550 100% 1,854,300 100%
Table 16 suggests that, in comparison to their peers at moderately selective institutions,
students interested in STEM as freshmen at highly selective institutions earned a degree at higher
rates by the third follow-up survey in 2012. They also transferred to other institutions or
dropped out at lower rates. Overall, this is suggestive of the potential advantage of highly
selective institutions over moderately selective institutions in the production of STEM degrees.
11 The pattern of missing data in STEM career aspirations does not allow for inclusion of STEM career aspirations in the matching/outcome models since they would severely restrict the number of observations. 12 See Appendix L for un-weighted counts.
72
Namely, students at highly selective institutions may be more likely to obtain the degree to which
they initially aspired because they are less likely to drop out or transfer out. The differences in
graduation and retention rates between the two types of institutions, however, may be the
results of systematic differences in academic readiness and motivation between the students
who attended the institutions.
Table 1613. Earning of credentials by first post-secondary institutions selectivity among STEM freshmen
Enrollment Status of STEM Freshmen at First PSI Attended at Third Follow-up
Selectivity
Highly Moderately
n % n %
Earned a credential from PSI; still attending PSI as of 2012 3,010 3% 2,500 3%
Earned a credential from PSI; no longer attending PS1 as of 2012 58,920 64% 34,210 41%
No cred from PSI; still attending PSI as of 2012 560 1% 3,380 4%
No cred from PSI; no longer attending PSI; did attend another PS institution 28,240 30% 36,990 44%
No cred from PSI; no longer attending PSI; did not attend another PS institution 1,960 2% 6,380 8%
Total 92,691 100% 83,461 100%
3.6 Methodology After conducting logistic regression, propensity score matching will also be used. The
complex survey design of ELS of 2002/12 will be accommodated in the propensity score matching
approach (Zanutto, Lu, & Hornik, 2005). Sampling weights will be applied in the matching and
outcome models (Zanutto, 2006). The matching model will be a binary logistic regression instead
of a multinomial logistic regression for two reasons. First, moderately selective institutions are,
13 See Appendix M for un-weighted counts.
73
theoretically, the closest substitute for highly selective institutions. Second, institutions “below”
the moderately selective line are less likely to require SAT/ACT scores for admissions, thus,
reducing the possibility of adequate matching. Thus, the analysis will compare STEM degree
attainment between comparable students attending highly selective institutions and students
attending moderately selective institutions. The binary matching model will take the following
general form:
(1) Yi= β0 + βdΣXd + βaΣXa+ βcΣXc + βoΣXo + ei
-where Y is attendance at a highly selective institution (1=highly selective institution
0=moderately selective institution), Xd is a vector of student demographic variables, Xa is a vector
of academic performance and exposure measures, Xc is a vector of college choice behaviors and
outcomes, Xo is a vector of contextual variables-and ei is an error term with the usual properties,
and the betas capture the cumulative association of each vector on the outcome. The
demographics variables include gender, race with four values -Asian, Multi-racial, Under-
represented Minorities (URM) and White – and income with three values: low income (low
[below $35,000], middle [$35,001 to $75,000], and high income [above $75,001]. The academic
readiness variables are Math and Verbal SAT scores and high school GPA. The exposure variables
capture the highest level of science and math courses completed in high school. The seven
categories of exposure for each subject are found in Appendix N. The college choice behavior
and outcome variables are total number of applications and number of admissions to a) highly
selective and b) moderately selective institutions. The contextual variables include school
control (public, Catholic and private), urban status (suburban, urban, and rural) and region
(Northeast, Midwest, South, and West).
74
Meeting the conditions of common support, the outcomes model takes the form:
(2) Yi= β0 + β1T + βrΣXr + ei
- where Y is the attainment of a STEM degree for student i at the first post-secondary institution
attended (where 1=STEM degree 0=else), T identifies attendance at a highly selective post-
secondary institution, β1 captures the average treatment on the treated (ATT) of T on the
outcome, Xr is vector of all the variables included in the matching model and βr captures its
cumulative impact of any residual differences between “treatment” and “control” groups on the
outcome in the matched sample (DuGoff, Schuler, & Stuart, 2012). The Mantel and Haenzel (MH)
test will be used to assess the sensitivity of treatment estimates (Becker and Caliendo, 2007;
Caliendo and Kopeinig, 2008).
The number of total applications and number of admissions to highly selective institutions
and the number of admissions to moderately selective institutions play an important role in the
identification strategy. Post-secondary institutions make their admissions decision on a variety
of factors. Some of those factors – like academic achievement and readiness - may be observed
by researchers but others like motivation, maturity, and cultural fit may only be observed by
admissions personnel through admissions essay, interviews, and campus visits (Dale and Krueger,
1998). Thus, admissions decisions signal/confer valuable information about applicants not
directly observed by the researchers. They act as proxies for non-observables. Failure to include
them would lead to biased estimates. By definition, highly selective institutions have higher
admission thresholds for academics and, and most likely also for, non-academic factors than
moderately selective institutions. If non-academic factors or their proxies are not included in the
75
models and are positively correlated with academic factors then the estimates on the impact of
attending a highly selective institution on STEM degree attainment would be upwardly biased
since it would be confounded with academic factors despite matching on academic factors.
Formally following Dale and Krueger (1998), for every student i applying to school j the
admissions committees grant admission or do not grant admission according the following rule:
(3) Zij= γ1X1i + γ2X2i + eij >=Tj then admit student i , else do not admit.
- where Z is the quality of the applicant, X1 are student traits observable to the researcher, X2
are student attributes not known to the researcher, their accompanying gammas are the relative
weights given by the admission committees to each type of characteristic, ei represents the
unique perspectives of personnel making admissions decisions at college j (and orthogonal to
STEM degree attainment) and T is the minimum threshold at which students gain admissions. If
X2 is not included in (1) as Xc then β1 in (2) will be upwardly biased if X2 is positively correlated
with X1.
3.7 Results The results section is structured as follows. It first identifies differences in academic
readiness/exposure measures and admission outcomes between the two types of attendees and
then presents the results of logistic regression analysis first for those interested in STEM (all
students and female students) and then for those who are not interested in STEM (all students
and female students). For the sake of brevity, the comparisons in the body of the paper are
restricted to SAT math scores, percentage of students exposed to the highest levels in the math
and science pipelines, and the number of acceptances to highly selective institutions.
76
Comparison of all the variables in the matching/outcome models can be found in the appendices.
Next, reduction of average differences in the afore-mentioned variables between students
attending the two types of institutions as result of propensity score matching is quantified and is
followed by unadjusted and covariate adjusted PSM “treatment” estimates for the those
interested in STEM (all students and females) and for those not interested in STEM (all students
and females). Finally, the stability of unadjusted PSM estimates are tested.
The analyses will be restricted to those students who were admitted to at least one highly
selective institution. The control groups of students who opted to attend moderately selective
institutions will be composed of students who had the chance to attend at least one highly
selective institution. In this way, the control and treatment groups will be comparable, at least,
in applying and getting admissions to at least one highly selective institution.
3.7.1 Logistic Regression Analysis
STEM covers students interested in physical and biological sciences, mathematics,
information and computer sciences, and engineering. Approximately 322,000 students reported
interest in such fields at the beginning of their collegiate careers. The differences in academic
readiness and exposure measures between the approximately 190,000 students attending highly
selective and moderately selectively institutions, in some cases, are considerable. Students who
opted to attend highly selective institutions score 50 points higher on the math section of the
SAT, have two percentage point advantage in exposure to the highest level in the math course
pipeline and an eight percentage point advantage in the science pipeline and have been accepted
by highly selective institutions by .6 more of an admission. See Appendix O for a more detailed
view of differences.
77
Logistic regression analysis suggests a positive association of attending a highly selective
institution with STEM degree attainment among those interested in STEM. The odds ratio is for
the highly selective status is 1.51. The corresponding marginal effect estimate suggest that for
every 100 students who attend highly selective institutions and every 100 students who attend
moderately selective institution highly selective institutions will graduate 22 more students with
STEM degrees. See Appendix P for a more detailed view of regression results. Marginal effects
for females interested in STEM were estimated at 30 percentage points. 14
Among students not interested in STEM fields upon entering post- secondary education,
attending a highly selective institution decreases the probability of graduating with a STEM
degree by three percentage points (marginal effect). Attending a highly selective institution
decreases the probability of obtaining a
STEM degree by one percentage point among females not interested in STEM fields (marginal
effect).
3.7.2 Matching Analysis
The matching models yield more comparable “treatment” and “control” groups.
Differences in SAT math scores are reduced by 56 percent, highest exposure in the science course
pipeline and number of acceptances to highly selective institutions are both reduced by 87
percent but differences in the highest exposure to math courses are increased by almost fourfold,
from two percent to almost eight percent. See Appendix Q. Most differences remain statistically
significant. About forty percent of the 12,276 students in the treatment group were not
14 Based on a linear probability model.
78
matched. The unmatched units were all at high end of the propensity score distribution. See
Appendix R.
The unadjusted PSM results suggest positive average treatment on the treated (ATT)
effects for attending a highly selective institution. As Table 17 points out, on average, the
probability of graduating with a STEM degree is 16 percentage points higher if one attends a
highly selective institution. Manzel-Haenszel bound results suggest that the unadjusted PSM
estimates are stable. This is a lower estimate than the marginal effect of the logistic regression
analysis. Despite the differences in the magnitude of the estimate both coincide on the positive
effect of attending a highly selective institution. Ideally, when there is less than perfect parity
between the synthetic control and treatment groups as in the case at hand, more precise
estimates can be calculated by running the matched sample through the matching regression
model. Covariate adjusted PSM estimates suggest a positive association of 30 percentage points
between attending a highly selective institution and STEM degree attainment. PSM estimates
for female students could not converge on an answer.
Table 17. Pre and Post Matching Differences in STEM Degree Attainment Variable Sample Treated Controls Difference S.E. t-stat
STEM Degree Unmatched 0.52 0.42 0.10 0.01 15.03
STEM Degree Matched 0.58 0.42 0.16 0.01 20.71
Finally, it should be noted that highly selective institutional status has the opposite effect
when the analytical sample is reduced to those who did not have STEM interest upon arriving at
their post-secondary institutions. Highly selective institutions are not as effective at generating
and maintaining STEM interest among those initially not interested in STEM. Unadjusted PSM
79
results suggest that students not interested in STEM at moderately selective institutions earn
STEM degrees at a rate of six percent while their counterparts at highly selective institutions earn
degrees at half the rate. The covariate adjusted PSM results put the difference between slightly
lower at one and half percent. PSM results suggest that females who attend moderately selective
institutions do not have an advantage in obtaining STEM degrees. The covariate adjusted
estimate suggests a one percentage point advantage. Although the possible positive association
between moderately selective attendance and STEM degree attainment is relatively small the
sheer number of students not interested in STEM at those institutions make it a non-negligible
source of STEM graduates.
3.7.3 Summary of Results
Table 18 summarizes results from logistic regression (Reg), propensity score matching
(PSM) and covariate adjusted PSM (CA PSM). Among all students initially interested upon
entering college, attending a highly selective institution is associated with a higher probability of
attaining a STEM degree from the first attended institution. Attending a moderately selective
college, on the other hand, is associated with higher probabilities of STEM degree attainment
among students not initially interested in STEM. This advantage may not extend to female
students or is considerably less for them.
Table 18. Summary of Results
Group
Initially Interested in STEM Initially Not Interested in STEM
Reg PSM CA PSM Reg PSM CA PSM
All Students 0.22 0.16 0.30 -0.02 -0.03 -0.02
Female Students 0.3015 - - 0.00 0.00 -0.01
15 Linear regression
80
3.8 Conclusion The rule-of-thumb that “more selective is better” overall holds true for students
interested in STEM. This suggest that parents and counselors should encourage students to
attend highly selective institutions when students have the chance of doing so. Scholarships
directed at students interested in STEM should also take this into account and provide incentives
for students to opt for more selective institutions. On the other hand, highly selective
institutions are not as effective at generating and maintaining interest in STEM among those
initially not interested. Parties involved in the college choice process of students who are
undecided about their major at the end of high school who believe these students could
potentially succeed in STEM should encourage them to attend moderately selective institutions.
81
Works Cited/Consulted Acemoglu, D. (2012). What Does Human Capital Do? A Review of Goldin and Katz's The Race between
Education and Technology (No. w17820). National Bureau of Economic Research.
Adamson, C., & Britt, R. (2009). Repeat testing with the HESI Exit Exam-sixth validity study. Computers
Informatics Nursing, 27(6), 393-397.
Aghion, P. A., Howitt, P. A., & Peñalosa, C. G. (1998). Endogenous growth theory. MIT press.
Altonji, J. G., Elder, T. E., & Taber, C. R. (2005). An evaluation of instrumental variable strategies for
estimating the effects of catholic schooling. Journal of Human Resources, 40(4), 791-821.
Ansalone, G. (2010). Tracking: Educational Differentiation or Defective Strategy.Educational Research
Quarterly, 34(2), 3-17.
Arathuzik, D., & Aber, C. (1998). Factors associated with national council licensure examination-
registered nurse success. Journal of Professional Nursing, 14(2), 119-126.
Austin, P. C., Grootendorst, P., & Anderson, G. M. (2007). A comparison of the ability of different
propensity score models to balance measured variables between treated and untreated subjects: a
Monte Carlo study. Statistics in medicine, 26(4), 734-753.
Autor, David & Dorn, David (2009). The Growth of Low Skill Service Jobs and the Polarization of the U.S.
Labor Market, NBER Working Papers 15150, National Bureau of Economic Research, Inc.
Ballantine, J.H., & Hammack, F.M. (2009). The sociology of education: A systematic analysis. Upper
Saddle River, NJ: Pearson.
Becker, S. O., & Caliendo, M. (2007). Mhbounds-sensitivity analysis for average treatment effects (No.
2542). IZA Discussion Papers.
Beeson, S. A., & Kissling, G. (2001). Predicting success for baccalaureate graduates on the NCLEX-
RN. Journal of Professional Nursing, 17(3), 121-127.
Benbow, C. P., & Minor, L. L. (1986). Mathematically talented males and females and achievement in the
high school sciences. American Educational Research Journal, 23(3), 425-436.
Betts, J. R., & Shkolnik, J. L. (2000). The effects of ability grouping on student achievement and resource
allocation in secondary schools. Economics of Education Review, 19(1), 1-15.
Bifulco, R. (2002). Addressing Self-Selection Bias in Quasi-Experimental Evaluations of Whole-School
Reform A Comparison of Methods. Evaluation Review, 26(5), 545-572.
Bondmass, M. D., Moonie, S., & Kowalski, S. (2008). Comparing NET and ERI Standardized Exam Scores
between Baccalaureate Graduates Who Pass or Fail the NCLEX-RN©. International Journal of Nursing
Education Scholarship,5(1), 1-15.
Bosch, P. C., Doshier, S. A., & Gess-Newsome, J. (2012). Bilingual nurse education program: Applicant
characteristics that predict success. Nursing education perspectives, 33(2), 90-95.
82
Bound, J., Lovenheim, M., & Turner, S. (2009). Why have college completion rates declined? An analysis
of changing student preparation and collegiate resources (No. w15566). National Bureau of Economic
Research.
Bowen, W. G., Chingos, M. M., & McPherson, M. S. (2009). Crossing the finish line: Completing college at
America's public universities. Princeton University Press.
Bray, M., & Thomas, R. M. (1995). Levels of comparison in educational studies: Different insights from
different literatures and the value of multilevel analyses.Harvard Educational Review, 65(3), 472-491.
Briscoe, V. J., & Anema, M. G. (1999). The relationship of academic variables as predictors of success on
the National Council Licensure Examination for Registered Nurses (NCLEX-RN) in a selected associate
degree program. ABNF journal, 10(4), 80.
Burdett, K. (2013). How students choose a college: Understanding the role of internet based resources in
the college choice process. Unpublished dissertation.
Bureau of Labor Statistics (2008). Occupational Employment Series Survey, (May 2004 to May 2008).
Washington, D.C.
Business Higher Education Forum (2010). Increasing the Number of STEM Graduates: Insights from the
U.S. STEM Education & Modeling Project.
Cabrera, A. F., & La Nasa, S. M. (2000). Understanding the College-Choice Process. New Directions for
Institutional Research, 2000(107), 5-22.
Cabrera, A. F., & La Nasa, S. M. (2001). On the path to college: Three critical tasks facing America's
disadvantaged. Research in Higher Education, 42(2), 119-149.
Caliendo, M., & Kopeinig, S. (2008). Some practical guidance for the implementation of propensity score
matching. Journal of economic surveys, 22(1), 31-72.
Calonico, S., Cattaneo, M. D., & Titiunik, R. (2014). Robust Nonparametric Confidence Intervals for
Regression-Discontinuity Designs. Econometrica, 82(6), 2295-2326.
Calvó-Armengol, A., Patacchini, E., & Zenou, Y. (2009). Peer effects and social networks in education. The
Review of Economic Studies, 76(4), 1239-1267.
Card, D., & Payne, A. A. (2002). School finance reform, the distribution of school spending, and the
distribution of student test scores. Journal of Public Economics, 83(1), 49-82.
Carrell, S., Sacerdote, B., & West, J. (2009). Beware of economists bearing reduced forms? an
experiment in how not to improve student outcomes.
Carrell, Scott, Page, Marianne, and West, James (2010). Sex and Science: How Professor Gender
Perpetuates The Gender Gap: The Quarterly Journal of Economics: 1101-1144.
Catsambis, Sophia (1994). The Path to Math: Gender and Racial-Ethnic Differences in Mathematics
Participation from Middle School to High School. Sociology of Education, Vol. 67 (3), 199-215.
83
Cattaneo, M. D., Frandsen, B. R., & Titiunik, R (2015) . Randomization Inference in the Regression
Discontinuity Design: An Application to Party Advantages in the US Senate.
Cattaneo, M. D., Frandsen, B. R., & Titiunik, R. Randomization Inference in the Regression Discontinuity
Design: An Application to Party Advantages in the US Senate.
Cattaneo, M. D., Jansson, M., & Ma, X. (2015). Simple Local Regression Distribution Estimators with an
Application to Manipulation Testing. working paper, University of Michigan.
Cavanagh, S. E., & Fomby, P. (2012). Family Instability, School Context, and the Academic Careers of
Adolescents. Sociology of Education, 85(1), 81-97.
CCNE (2013). Standards for Accreditation of Baccalaureate and Graduate Nursing Programs.
Chen, X. and Weko, T (2009). Students Who Study Science, Technology, Engineering, And Mathematics
(STEM) In Postsecondary Education. Stats in Brief (NCES 2009-161). National Center for Education
Statistics, U.S. Department of Education.
CNEA (2016). Accreditation Standards for Nursing Education Programs.
Coble and Allen (2005). Keeping America Competitive. Denver, CO, ECS.
Comparative Education, Vol. 41 (3), 329-350.
Crosnoe, Robert. 2009. “Low-Income Students and the Socioeconomic Composition of Public High
Schools.” American Sociological Review, Vol. 74: 709-730
Dale, S. B., & Krueger, A. B. (1999). Estimating the payoff to attending a more selective college: An
application of selection on observables and unobservables (No. w7322). National bureau of economic
research.
Dale, S., & Krueger, A. B. (2011). Estimating the return to college selectivity over the career using
administrative earnings data (No. w17159). National Bureau of Economic Research.
Daley, L. K., Kirkpatrick, B. L., Frazier, S. K., Chung, M. L., & Moser, D. K. (2003). Predictors of NCLEX-RN
success in a baccalaureate nursing program as a foundation for remediation. Journal of Nursing
Education, 42(9), 390-398.
Dawes, P. L., & Brown, J. (2002). Determinants of awareness, consideration, and choice set size in
university choice. Journal of Marketing for Higher Education, 12(1), 49-75.
DuGoff, E. H., Schuler, M., & Stuart, E. A. (2014). Generalizing observational study results: Applying
propensity score methods to complex surveys. Health services research, 49(1), 284-303.
Dynarski, Susan M. and Judith E. Scott-Clayton. 2006. The Cost of Complexity in Federal Student Aid:
Lessons from Optimal Tax Theory and Behavioral Economics. Cambridge, MA: National Bureau of
Economic Research.
Eder, D. (1981). Ability grouping as a self-fulfilling prophecy: A micro-analysis of teacher-student
interaction. Sociology of Education, 151-162.
84
Enders, C. K., & Tofighi, D. (2007). Centering predictor variables in cross-sectional multilevel models: a
new look at an old issue. Psychological methods,12(2), 121.
Engberg, Mark and Wolniak, Gregory C., "College Student Pathways to the STEM Disciplines"
(2013). School of Education: Faculty Publications & Other Works.Paper 10.
Flanagin, A. J., & Metzger, M. J. (2000). Perceptions of Internet information credibility. Journalism &
Mass Communication Quarterly, 77(3), 515-540.
Flanagin, A. J., & Metzger, M. J. (2007). The role of site features, user attributes, and information
verification behaviors on the perceived credibility of web-based information. New Media & Society, 9(2),
319-342.
Fleming, J., and Morning, C. (1995). Correlates of the SAT in Minority Engineering Students: An
Exploratory Study. Journal of Higher Education, Vol. 69(1), 89-108.
Fortier, M. E. (2012, July). Predictors of Success on the NCLEX-RN Among Transfer BSN Students.
In Sigma Theta Tau International's 23rd International Nursing Research Congress.
Frith, K. H., Sewell, J. P., & Clark, D. J. (2005). Best practices in NCLEX-RN readiness preparation for
baccalaureate student success. Computers Informatics Nursing, 23(6), 322-329.
Gardiner, J. C., Luo, Z., & Roman, L. A. (2009). Fixed effects, random effects and GEE: What are the
differences?. Statistics in medicine, 28(2), 221-239.
Gelman, A., & Imbens, G. (2014). Why high-order polynomials should not be used in regression
discontinuity designs (No. w20405). National Bureau of Economic Research.
Gelman, A., & Zelizer, A. (2015). Evidence on the deleterious impact of sustained use of polynomial
regression on causal inference. Research & Politics, 2(1).
Gemici, A., & Wiswall, M. (2014). Evolution of gender differences in post-secondary human capital
investments: College Majors. International Economic Review, 55(1), 23-56.
Giddens, J. and Groecker (2005). The relationship of critical thinking to performance on the NCLEX-
RN®. Journal of Nursing Education, 44(2), 85.
Grandy, Jerilee (1998). Persistence in Science of High-Ability Minority Students: Results of a Longitudinal
Study. The Journal of Higher Education, Vol. 69 (6), 589-620.
Grodsky, Eric and Melanie T. Jones (2004). Real and Imagined Barriers to College Entry: Perceptions of
Cost. Social Science Research 36:745–766.
Hanushek, E. A., Kain, J. F., & Rivkin, S. G. (2004). Why public schools lose teachers. Journal of human
resources, 39(2), 326-354.
Harris Interactive (2011). STEM Perceptions: Student & Parent Study Parents and Students Weigh in on
How to Inspire the Next Generation of Doctors, Scientists, Software Developers and Engineers.
Commissioned by Microsoft Corporation Corp. Seattle, WA.
85
Harwell, E. (2012). An Analysis of Parent Occupation and Student Choice in STEM Major. Project STEP-
UP. University of Illinois at Urbana-Champaign.
Hearn, J. C. (1984). The relative roles of academic, ascribed, and socioeconomic characteristics in college
destinations. Sociology of Education, 22-30.
Heller, D. E. (1997). Student price response in higher education: An update to Leslie and
Brinkman. Journal of Higher Education, 624-659.
Heller, D.E. (2006). Early Commitment of Financial Aid Eligibility. American Behavioral Scientist 49(12):
1719–1738.
Heroff, K. (2009). Guidelines for a progression and remediation policy using standardized tests to
prepare associate degree nursing students for the NCLEX-RN at a rural community college. Teaching and
Learning in Nursing, 4(3), 79-86..
Hoekstra, M. (2009). The Effect of Attending the Flagship State University on Earnings: A Discontinuity-
Based Approach. The Review of Economics and Statistics, 91(4), 717-724.
Hofmann, D. A., & Gavin, M. B. (1998). Centering decisions in hierarchical linear models: Implications for
research in organizations. Journal of Management, 24(5), 623-641.
Hong, G., & Raudenbush, S. W. (2005). Effects of kindergarten retention policy on children’s cognitive
growth in reading and mathematics. Educational Evaluation and Policy Analysis, 27(3), 205-224.
Hossler, D., & Palmer, M. (2008). Why understand research on college choice. National Association for
College Admission Counseling (NACAC). Fundamentals of college admission counseling: A textbook for
graduate students and practicing counselors, 42-53.
Hossler, D., Braxton, J., & Coopersmith, G. (1989). Understanding student college choice. Higher
education: Handbook of theory and research, 5, 231-288.
Hossler, D., Schmit, J., & Vesper, N. (1999). Going to college: How social, economic, and educational
factors influence the decisions students make. JHU Press.
Hoxby, C. M., & Avery, C. (2012). The missing" one-offs": The hidden supply of high-achieving, low
income students (No. w18586). National Bureau of Economic Research.
Hoxby, C., & Turner, S. (2013). Expanding college opportunities for high-achieving, low income
students. Stanford Institute for Economic Policy Research Discussion Paper, (12-014).
Hubbard, A. E., Ahern, J., Fleischer, N. L., Van der Laan, M., Lippman, S. A., Jewell, N., ... & Satariano, W.
A. (2010). To GEE or not to GEE: comparing population average and mixed models for estimating the
associations between neighborhood risk factors and health. Epidemiology, 21(4), 467-474.
Hughes, J. N., Chen, Q., Thoemmes, F., & Kwok, O. M. (2010). An investigation of the relationship
between retention in first grade and performance on high stakes tests in third grade. Educational
evaluation and policy analysis,32(2), 166-182.
Ingels, S.J., Pratt, D.J, Alexander, C.P., Jewell, D.M., Lauff, E. Mattox, T.L., and Wilson, D. (2014).
Education Longitudinal Study of 2002 Third Follow-up Data File Documentation (NCES 2014-364).
86
National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education.
Washington, DC. Retrieved [date] from http://nces.ed.gov/pubsearch.
Jean Giddens PhD, R. N. (2005). The relationship of critical thinking to performance on the NCLEX-
RN®. Journal of Nursing Education, 44(2), 85.
Karpiak, C. P., Buchanan, J. P., Hosey, M., & Smith, A. (2007). University students from single-sex and
coeducational schools: Differences in majors and attitudes at a Catholic university. Psychology of
Women Quarterly, Vol. 31: 282–289
Kelcey, B. M. (2009). Improving and assessing propensity score based causal inferences in
Koonce, D. , Anderson, D., Zhou, J. , Hening, D., & Conley, V. (2011). What is STEM? Paper presented to
the American Society for Engineering Education’s 2011 Annual Conference & Exposition.
Kurlaender, M., & Grodsky, E. (2013). Mismatch and the paternalistic justification for selective college
admissions. Sociology of Education, 0038040713500772.
Langdon, D. McKittrick, G., Beede, D., Khan, B., & Doms, M. (July, 2011). STEM: Good jobs now and for
the future. Washington D.C.: U.S. Department of Commerce, Economics and Statistics Administration.
Retrieved from www.esa.doc.gov/sites/default/files/reports/.../stemfinalyjuly14_1.pdf
Langford, R. and Young, A (2013). Predicting NCLEX-RN success with the HESI Exit Exam: Eighth validity
study. Journal of Professional Nursing. 29: S5–S9.
Lankford, H., Loeb, S., & Wyckoff, J. (2002). Teacher sorting and the plight of urban schools: A
descriptive analysis. Educational Evaluation and Policy Analysis, 24(1), 37-62.
Lau, S., & Roeser, R. W. (2002). Cognitive abilities and motivational processes in high school students'
situational engagement and achievement in science.Educational Assessment, 8(2), 139-162.
Lauder, H., & Hughes, D. (1990). Social inequalities and differences in school outcomes. New Zealand
Journal of Educational Studies, 25(1), 37-60.
Lee, A. (2013). Determining the Effects of Pre-College STEM Contexts on STEM Major Choices in 4-year
Post-Secondary Institutions Using Multilevel Structural Equation Modeling. Journal of Pre-College
Engineering Education,3(2).
Lee, James (2002). More Than Ability: Gender and Personal Relationships Influence Science and
Technology Involvement. Sociology of Education, Vol. 75 (4), 349-373.
Lee, V. E., & Ready, D. D. (2009). US high school curriculum: Three phases of contemporary research and
reform. The Future of Children, 19(1), 135-156.
Lent, R. W., & Brown, S. D. (2006). On conceptualizing and assessing social cognitive constructs in career
research: A measurement guide. Journal of Career Assessment, 14(1), 12-35.
Leslie, L. Larry, McClure, Gregory T., and Oaxaca, Ronald L. (1998). Women and Minorities in Science and
Engineering: A Life Sequence Analysis. The Journal of Higher Education, Vol. 69 (3), 239-276
87
Levin, H. M. (1985). Solving the shortage of mathematics and science teachers.Educational Evaluation
and Policy Analysis, 7(4), 371-382.
Lewis, C. C. (2005). Predictive accuracy of the HESI Exit Exam on NCLEX-RN pass rates and effects of
progression policies on nursing student Exit Exam scores (Doctoral dissertation).
Lim, N., Haddad, A., Butler, D. M., & Giglio, K. (2013). First Steps Toward Improving DoD STEM
Workforce Diversity. RAND Corporations
Lüdtke, O., Marsh, H. W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2008). The
multilevel latent covariate model: a new, more reliable approach to group-level effects in contextual
studies. Psychological Methods,13(3), 203.
Lyons, E. M. (2008). Examining the effects of problem-based learning and NCLEX-RN scores on the
critical thinking skills of associate degree nursing students in a southeastern community
college. International Journal of Nursing Education Scholarship, 5(1), 1-17.
Ma, X., & Klinger, D. A. (2000). Hierarchical linear modelling of student and school effects on academic
achievement. Canadian Journal of Education, Vol. 25: 41-55.
Ma, X., Ma, L., & Bradley, K. D. (2008). Using multilevel modeling to investigate school effects. In A. A.
O’Connell & D. B. McCoach (Eds.), Multilevel modeling of educational data (pp. 59-110). Charlotte, NC:
Information Age.
Manski, C. F. (1993). Identification of endogenous social effects: The reflection problem. The review of
economic studies, 60(3), 531-542.
Marcus, S. M., & Gibbons, R. D. (2012). Caution should be used in applying propensity scores estimated
in a full cohort to adjust for confounding in subgroup analyses: commentary on “Applying propensity
scores estimated in a full cohort to adjust for confounding in subgroup analyses”.
Pharmacoepidemiology and drug safety, 21(7), 710-712.
Marsh, H. W., Lüdtke, O., Robitzsch, A., Trautwein, U., Asparouhov, T., Muthén, B., & Nagengast, B.
(2009). Doubly-latent models of school contextual effects: Integrating multilevel and structural equation
approaches to control measurement and sampling error. Multivariate Behavioral Research, 44(6), 764-
802.
Marsh, J. L., Hutton, J. L., & Binks, K. (2002). Removal of radiation dose response effects: an example of
over-matching. BMJ: British Medical Journal, 325(7359), 327.
Martinez, Miguel (2014). Assessment Report: Center for Teaching Advancement and Assessment
Research.
Martinez, Miguel (2016). Climate Survey Report.
Mattern, K. D., Patterson, B. F., Shaw, E. J., Kobrin, J. L., & Barbuti, S. M. (2008). Differential Validity and
Prediction of the SAT®. Research Report No. 2008-4. College Board
McGann, E., & Thompson, J. M. (2008). Factors related to academic success in at-risk senior nursing
students. International Journal of Nursing Education Scholarship, 5(1), 1-15.
88
Michigan.
Miller, L. C., & Mittleman, J. (2012). High Schools That Work and college preparedness: Measuring the
model's impact on mathematics and science pipeline progression. Economics of Education Review.
31(6), 1116–1135.
Moogan, Y. J., & Baron, S. (2003). An analysis of student characteristics within the student decision
making process. Journal of Further and Higher Education,27(3), 271-287.
Morgan, S. L., Gelbgiser, D., & Weeden, K. A. (2013). Feeding the Pipeline: Gender, Occupational Plans,
and College Major Selection∗. Social science research.
Morris, T., & Hancock, D. (2008). Program exit examinations in nursing education: Using a value added
assessment as a measure of the impact of a new curriculum. Educational Research Quarterly, 32(2), 19.
Morton, A. M. (2006). Improving NCLEX scores with structured learning assistance. Nurse
Educator, 31(4), 163-165.
Muller, C. Farkas, G., and Riegle-Crumb, C. (2006). The role of gender and friendship in course-taking.
Sociology of Education, Vol. 79(3), 208-228.
multilevel and nonlinear settings. Unpublished doctoral dissertation. University of
Nagengast, B., & Marsh, H.W. (2012). Big fish in little ponds aspire more: Mediation and cross-cultural
generalizability of school-average ability effects on self-concept and career aspirations in
science. Journal of Educational Psychology, 104, 1033-1053.
Nibert, A. and Young, A. (2001). A third study on predicting NCLEX success with the HESI Exit Exam.
Computers in Nursing. 19: 172–178.
Nibert, A. T., & Young, A. (2006). A Third Study on Predicting NCLEX Success With the HESI Exit
Exam. Computers Informatics Nursing, 24, 21S-27S.
Nibert, A., Young, A., and Adamson, C (2002). Predicting NCLEX success with the HESI Exit Exam: Fourth
annual validity study. Computers, Informatics, Nursing. 20: 261–267.
Norfleet James, A., & Richards, H. C. (2003). Escaping stereotypes: Educational attitudes of male alumni
of single-sex and coed schools.Psychology of Men & Masculinity, 4(2), 136.
Norton, C. K., Relf, M. V., Cox, C. W., Farley, J., Lachat, M., Tucker, M., & Murray, J. (2006). Ensuring
NCLEX-RN success for first-time test-takers. Journal of Professional Nursing, 22(5), 322-326.
Parker, C.E. (2010). Project RISE Pilot Study: Methods in Longitudinal Studies of Youth in Informal STEM
Education. Presented at the American Educational Research Association Annual Conference, Denver,
Colorado, May 2010.
Parrone, J., Sredl, D., Miller, M., Phillips, M., & Donaubauer, C. (2008). An evidence-based
teaching/learning strategy for foreign nurses involving the Health Education Systems Incorporated
examination as a predictor for National Council Licensure Examination for Registered Nurses
success. Teaching and Learning in Nursing, 3(1), 35-40.
89
Pennington, T. D., & Spurlock Jr, D. (2010). A systematic review of the effectiveness of remediation
interventions to improve NCLEX-RN pass rates. Journal of Nursing Education, 49(9), 485.
Rabe-Hesketh, S., & Skrondal, A. (2006). Multilevel modelling of complex survey data. Journal of the
Royal Statistical Society: Series A (Statistics in Society), 169(4), 805-827.
Raudenbush, S. W. (2002). Hierarchical linear models: Applications and data analysis methods (Vol. 1).
Sage.
Raudenbush, S. W., & Willms, J. (1995). The estimation of school effects.Journal of educational and
behavioral statistics, 20(4), 307-335.
Robertson, K. F., Smeets, S., Lubinski, D., & Benbow, C. P. (2010). Beyond the Threshold Hypothesis Even
Among the Gifted and Top Math/Science Graduate Students, Cognitive Abilities, Vocational Interests,
and Lifestyle Preferences Matter for Career Choice, Performance, and Persistence. Current Directions in
Psychological Science, 19(6), 346-351.
Romer, P. M. (1990). Endogenous technological change. Journal of Political Economy, S71-S102.
Rosenbaum, J. E. (1980). Track misperceptions and frustrated college plans: An analysis of the effects of
tracks and track perceptions in the National Longitudinal Survey. Sociology of Education, 74-88.
Rothwell, J. (2013). The Hidden STEM Economy: The Surprising Diversity of Jobs Requiring Science,
Technology, Engineering, and Math Knowledge.
Rothwell, Jonathan and Ruiz, Neil (2012). H-1B Visas and the STEM Shortage. Brookings Institute.
Retrieved from http://www.brookings.edu/research/papers/2013/05/10-h1b-visas-stem-rothwell-ruiz
Ruggles, S., J. T. Alexander, K. Genadek, R. Goeken, M. B. Schroeder, and M. Sobek. (2010). 2010
American Community Survey data. Edited by University of Minnesota, Integrated Public Use Microdata
Series, Minneapolis, MN.
Rule, R. , Has, and Nugent (2004). The use of discriminant function analysis to predict student success on
the NCLEX-RN. Journal of Nursing Education, 43(10), 440.
Russell, M. L., & Atwater, M. M. (2005). Traveling the road to success: A discourse on persistence
throughout the science pipeline with African American students at a predominantly white
institution. Journal of Research in Science Teaching, 42(6), 691-715.
Salzman, H., Kuehn, D., & Lowell, B. L. (2013). Guestworkers in the High-Skill US Labor Market. Economic
Policy Institute Briefing Paper, 359.
Sander, R. H. (2004). A systemic analysis of affirmative action in American law schools. Stanford Law
Review, 367-483.
Sanders, Mark (2009). STEM, STEM Education, STEMmania. The Technology Teacher,
December/January: 20-26.
Santelices, M. V., & Wilson, M. (2010). Unfair treatment? The case of Freedle, the SAT, and the
standardization approach to differential item functioning. Harvard Educational Review, 80(1), 106-134.
90
Santiago, L. A. (2013). Forecasting nursing student success and failure on the NCLEX-RN using predictor
tests (Doctoral dissertation).
Sayles, S., Shelton, D., & Powell, H. (2003). Predictors of success in nursing education. ABNF
Journal, 14(6), 116.
Schreiner, B. Barton, L., Willson, P., & Langford, R (2014). Standardized Predictive Testing: Practices,
Policies and Outcomes. Administrative Issues Journal 4(2), 68-76.
Schroeder, J. (2013). Improving NCLEX-RN pass rates by implementing a testing policy. Journal of
Professional Nursing, 29(2), S43-S47.
Sifford, S., & McDANIEL, D. M. (2007). Results of a remediation program for students at risk for failure
on the NCLEX exam. Nursing Education Perspectives, 28(1), 34-36.
Smith, J., Pender, M., & Howell, J. (2013). The full extent of student-college academic
undermatch. Economics of Education Review, 32, 247-261.
Southworth, S. and Mickelson, R. (2007). The Interactive Effects of Race, Gender, and School
Composition on College Track Placement. Social Forces, Vol. 86(2), 497-523.
Spurlock Jr, D. R. (2008). A study of the usefulness of the HESI exit exam in predicting NCLEX-RN
failure. Journal of Nursing Education, 47(4), 157.
Stuenkel, D. L. (2006). At-risk students: do theory, grades, standardized examinations, promote
success?. Nurse Educator, 31(5), 207-212.
Sutherland, J. A., Hamilton, M. J., & Goodman, N. (2007). Affirming At-Risk Minorities for Success
(ARMS): Retention, graduation, and success on the NCLEX-RN®. Journal of Nursing Education, 46(8), 347-
353.
Tai, R. H., Liu, C., Maltese, A, and Fan, X. (2006). CAREER CHOICE: Enhanced: Planning Early for Careers in
Science. Life sci, 1, 1443-1444.
Taylor, H., Loftin, C., & Reyes, H. (2014). First-Time NCLEX-RN Pass Rate: Measure of Program Quality or
Something Else?. Journal of Nursing Education,53(6).
Thompson, Jennifer (2003). The Effect of Single-Sex Secondary Schooling on Women’s Choice of College
Major. Sociological Perspectives, Volume 46 (2), 257–278.
Tipton, P., Pulliam, M., Beckworth, C., Illich, P., Griffin, R., & Tibbitt, A. (2008). Predictors of associate
degree nursing students’ success students. Southern Online Journal of Nursing Research, 8(1), 8.
Toma, J. D., & Cross, M. E. (1998). Intercollegiate athletics and student college choice: Exploring the
impact of championship seasons on undergraduate applications. Research in Higher Education, 39(6),
633-661.
Tracey, T. J., Robbins, S. B., & Hofsess, C. D. (2005). Stability and change in interests: A longitudinal study
of adolescents from grades 8 through 12. Journal of Vocational Behavior, 66(1), 1-25.
91
Uyehara, J., Magnussen, L., Itano, J., & Zhang, S. (2007, January). Facilitating Program and NCLEX-RN
Success in a Generic BSN Program. In Nursing Forum (Vol. 42, No. 1, pp. 31-38).
Van Langen, Annemarie and Dekkers, Hetty, (2005). Cross-National Differences in Participating in
Tertiary Science, Technology, Engineering and Mathematics Education.
Wang, X. (2013). Why Students Choose STEM Majors Motivation, High School Learning, and
Postsecondary Context of Support. American Educational Research Journal.
Ware, N. C. & Lee, V. E. (1988). Sex differences in choice of college science majors. American Educational
Research Journal, Vol. 25: 593-614.
White House Office of Science and Technology Policy (2013). Preparing a 21st Century Workforce
Science, Technology, Engineering, and Math
Wilkinson, Ian, Judy M. Parr, Irene Y.Y. Fung, John A.C. Hattie, Michael A.R. Townsend (2002).
Discussion: modeling and maximizing peer effects in school. International Journal of Educational
Research, Vol. 37(5), 521-535.
Wiswall, M., & Zafar, B. (2015). Determinants of college major choice: Identification using an
information experiment. The Review of Economic Studies, 82(2), 791-824.
Wiswall, M., & Zafar, B. (2015). How do college students respond to public information about
earnings?. Journal of Human Capital, 9(2), 117-169.
Wray, K., Whitehead, T., Setter, R., & Treas, L. (2006). Use of NCLEX preparation strategies in a hospital
orientation program for graduate nurses. Nursing Administration Quarterly, 30(2), 162-177.
Young, A., & Willson, P. (2012). Predicting NCLEX-RN success: The seventh validity study HESI exit
exam. Computers Informatics Nursing, 30(1), 55-60.
Yu, Bing (2012). Variable Selection and Adjustment in Relation to Propensity Scores and Prognostic
Scores: From Single Level to Multi-level Data. Unpublished doctoral dissertation. University of Toronto.
Zanutto, E. L. (2006). A comparison of propensity score and linear regression analysis of complex survey
data. Journal of Data Science, 4(1), 67-91.
Zanutto, E., Lu, B., & Hornik, R. (2005). Using propensity score subclassification for multiple treatment
doses to evaluate a national antidrug media campaign. Journal of Educational and Behavioral Statistics,
30(1), 59-73.
Zinth, Kyle (2006). Recent State STEM Initiatives. Denver: Education Commission of the States.
Zweighaft, E. L. (2013). Impact of HESI specialty exams: The ninth HESI Exit Exam validity study. Journal
of Professional Nursing, 29(2), S10-S16.
92
Appendices Appendix A
Years 2009-2015
NCLEX Odds Ratio SE z P>|z|
Lower Limit
Upper Limit
Demographics
EOF Status 1.15 0.58 0.28 0.78 0.43 3.07
American Indian 1.00 (omitted)
Asian 1.13 0.64 0.21 0.83 0.37 3.46
Hispanic 0.48 0.29 -1.21 0.23 0.15 1.57
International 1.00 (omitted)
Multiracial 0.60 0.48 -0.64 0.53 0.13 2.87
Native 1.67 1.60 0.53 0.60 0.25 10.97
Other 1.00 (omitted)
Unknown 0.40 0.43 -0.86 0.39 0.05 3.28
White 1.08 0.64 0.13 0.90 0.34 3.44
Male 1.60 0.80 0.94 0.35 0.60 4.25
Newark Campus 0.54 0.24 -1.40 0.16 0.23 1.28
Blackwood Campus 1.00 (omitted)
Academic Readiness
SAT Verbal Score 1.00 0.00 0.85 0.39 1.00 1.01
SAT Math Score 1.00 0.00 -0.02 0.98 0.99 1.01
Academic Performance
HC Delivery 0.59 0.19 -1.60 0.11 0.31 1.13
Pathophysiology 1.40 0.39 1.21 0.23 0.81 2.40
Health Assessment 1.30 0.47 0.73 0.47 0.64 2.64
Foundations I 3.21† 1.24 3.04 0.00 1.51 6.83
Childbearing Family 1.54 0.63 1.05 0.30 0.69 3.45
Health and Illness ICA 3.03† 1.27 2.63 0.01 1.33 6.91
Health and Illness AOA I 2.50 1.11 2.06 0.04 1.05 5.95
Foundations II 0.64 0.27 -1.05 0.30 0.28 1.47
Pharmacotherapeutics 1.10 0.33 0.31 0.76 0.61 1.97
Psych Mental Health 2.55 0.95 2.51 0.01 1.23 5.30
Health and Illness AOA II 1.15 0.54 0.29 0.77 0.45 2.90
Year Fixed Effects
Year 2010 0.15 0.11 -2.56 0.01 0.04 0.64
Year 2011 0.30 0.26 -1.41 0.16 0.06 1.61
Year 2012 0.22 0.20 -1.70 0.09 0.04 1.25
93
Year 2013 0.51 0.50 -0.69 0.49 0.08 3.46
Year 2014 0.13 0.11 -2.43 0.02 0.03 0.68
Year 2015 1.00 (omitted)
Intercept 0.00 0.00 -4.82 0.00 0.00 0.00
Years 2009-2012
NCLEX Odds Ratio SE z P>|z|
Lower Limit
Upper Limit
Demographics
EOF Status 1.30 1.05 0.32 0.75 0.27 6.31
American Indian 1.00 (omitted)
Asian 2.31 1.88 1.03 0.30 0.47 11.36
Hispanic 1.95 1.69 0.77 0.44 0.36 10.62
International 1.00 (omitted)
Multiracial 2.43 3.40 0.64 0.53 0.16 37.66
Native 3.98 5.98 0.92 0.36 0.21 75.77
Other 1.00 (omitted)
Unknown 1.00 (omitted)
White 1.62 1.27 0.61 0.54 0.35 7.53
Male 1.52 1.24 0.51 0.61 0.31 7.56
Newark Campus 0.69 0.42 -0.60 0.55 0.21 2.30
Blackwood Campus 1.00 (omitted)
Academic Readiness
SAT Verbal Score 1.01 0.00 1.87 0.06 1.00 1.02
SAT Math Score 1.00 0.00 -0.39 0.70 0.99 1.01
Academic Performance
HC Delivery 0.48 0.23 -1.51 0.13 0.18 1.25
Pathophysiology 1.29 0.55 0.59 0.56 0.56 2.98
Health Assessment 0.95 0.59 -0.08 0.94 0.28 3.20
Foundations I 4.36† 2.41 2.66 0.01 1.47 12.90
Childbearing Family 2.46 1.58 1.40 0.16 0.70 8.69
Health and Illness ICA 5.02† 3.20 2.53 0.01 1.44 17.51
Health and Illness AOA I 2.79 1.79 1.60 0.11 0.79 9.81
Foundations II 0.45 0.30 -1.19 0.24 0.12 1.68
Pharmacotherapeutics 1.03 0.46 0.07 0.95 0.43 2.46
Psych Mental Health 1.45 0.85 0.63 0.53 0.46 4.60
Health and Illness AOA II 0.89 0.64 -0.16 0.87 0.22 3.60
Year Fixed Effects
Year 2010 0.47 0.48 -0.73 0.46 0.06 3.51
Year 2011 0.76 0.74 -0.28 0.78 0.11 5.15
Year 2012 0.49 0.45 -0.78 0.44 0.08 3.00
94
Intercept 0.00 0.00 -3.77 0.00 0.00 0.00
Years 2010-2013
NCLEX Odds Ratio SE z P>|z|
Lower Limit
Upper Limit
Demographics
EOF Status 1.30 1.05 0.32 0.75 0.27 6.31
American Indian 1.00 (omitted)
Asian 2.31 1.88 1.03 0.30 0.47 11.36
Hispanic 1.95 1.69 0.77 0.44 0.36 10.62
International 1.00 (omitted)
Multiracial 2.43 3.40 0.64 0.53 0.16 37.66
Native 3.98 5.98 0.92 0.36 0.21 75.77
Other 1.00 (omitted)
Unknown 1.00 (omitted)
White 1.62 1.27 0.61 0.54 0.35 7.53
Male 1.52 1.24 0.51 0.61 0.31 7.56
Newark Campus 0.69 0.42 -0.60 0.55 0.21 2.30
Blackwood Campus 1.00 (omitted)
Academic Readiness
SAT Verbal Score 1.01 0.00 1.87 0.06 1.00 1.02
SAT Math Score 1.00 0.00 -0.39 0.70 0.99 1.01
Academic Performance
HC Delivery 0.48 0.23 -1.51 0.13 0.18 1.25
Pathophysiology 1.29 0.55 0.59 0.56 0.56 2.98
Health Assessment 0.95 0.59 -0.08 0.94 0.28 3.20
Foundations I† 4.36 2.41 2.66 0.01 1.47 12.90
Childbearing Family 2.46 1.58 1.40 0.16 0.70 8.69
Health and Illness ICA† 5.02 3.20 2.53 0.01 1.44 17.51
Health and Illness AOA I 2.79 1.79 1.60 0.11 0.79 9.81
Foundations II 0.45 0.30 -1.19 0.24 0.12 1.68
Pharmacotherapeutics 1.03 0.46 0.07 0.95 0.43 2.46
Psych Mental Health 1.45 0.85 0.63 0.53 0.46 4.60
Health and Illness AOA II 0.89 0.64 -0.16 0.87 0.22 3.60
Year Fixed Effects
Year 2010 0.47 0.48 -0.73 0.46 0.06 3.51
Year 2011 0.76 0.74 -0.28 0.78 0.11 5.15
95
Year 2012 0.49 0.45 -0.78 0.44 0.08 3.00
Intercept 0.00 0.00 -3.77 0.00 0.00 0.00
Years 2011-2014
NCLEX Odds Ratio SE z P>|z|
Lower Limit
Upper Limit
Demographics
EOF Status 1.03 0.73 0.04 0.97 0.25 4.16
American Indian 1.00 (omitted)
Asian 1.56 1.15 0.60 0.55 0.37 6.61
Hispanic 2.40 2.11 0.99 0.32 0.42 13.51
International 1.00 (omitted)
Multiracial 1.66 1.69 0.50 0.62 0.23 12.21
Native 2.99 4.17 0.78 0.43 0.19 46.07
Other 1.00 (omitted)
Unknown 0.81 0.94 -0.18 0.86 0.08 7.93
White 0.98 0.74 -0.02 0.98 0.23 4.29
Male 0.66 0.46 -0.59 0.55 0.17 2.62
Blackwood Campus 1.00 (omitted)
Newark Campus† 0.27 0.16 -2.18 0.03 0.08 0.87
Academic Readiness
SAT Verbal Score 1.00 0.00 1.14 0.25 1.00 1.01
SAT Math Score 1.00 0.00 -0.64 0.52 0.99 1.01
Academic Performance
HC Delivery 0.83 0.37 -0.43 0.67 0.35 1.97
Pathophysiology 1.29 0.47 0.71 0.48 0.63 2.64
Health Assessment 2.04 1.03 1.42 0.16 0.76 5.50
Foundations I† 3.24 1.62 2.35 0.02 1.22 8.64
Childbearing Family 1.07 0.60 0.12 0.91 0.35 3.23
Health and Illness ICA† 4.00 2.33 2.38 0.02 1.28 12.52
Health and Illness AOA I 3.06 1.88 1.82 0.07 0.92 10.23
Foundations II 0.46 0.28 -1.29 0.20 0.14 1.49
Pharmacotherapeutics 1.45 0.56 0.95 0.34 0.67 3.10
Psych Mental Health† 4.12 2.12 2.76 0.01 1.51 11.28
Health and Illness AOA II 0.32 0.23 -1.58 0.12 0.08 1.32
Year Fixed Effects
Year 2011 1.40 1.22 0.38 0.70 0.25 7.75
Year 2012 1.44 1.27 0.41 0.68 0.26 8.09
Year 2013 3.88 3.02 1.74 0.08 0.84 17.83
Intercept 0.00 0.00 -3.70 0.00 0.00 0.00
96
Years 2012-2015
NCLEX Odds Ratio SE z P>|z|
Lower Limit
Upper Limit
Demographics
EOF Status 2.55 1.72 1.38 0.17 0.67 9.60
American Indian 1.00 (omitted)
Asian 1.62 1.21 0.65 0.52 0.38 7.00
Hispanic 0.63 0.50 -0.58 0.56 0.13 3.04
International 1.00 (omitted)
Multiracial 0.46 0.48 -0.75 0.45 0.06 3.50
Native 4.23 5.30 1.15 0.25 0.36 49.24
Other 1.00 (omitted)
Unknown 0.30 0.37 -0.96 0.34 0.03 3.45
White 0.90 0.69 -0.14 0.89 0.20 4.08
Male 0.99 0.65 -0.02 0.98 0.27 3.61
Newark Campus 0.31 0.18 -2.04 0.04 0.10 0.96
Blackwood 1.00 (omitted)
Academic Readiness
SAT Verbal Score 1.00 0.00 0.47 0.64 1.00 1.01
SAT Math Score 1.00 0.00 -0.14 0.89 0.99 1.01
Academic Performance
HC Delivery 0.80 0.36 -0.50 0.62 0.34 1.91
Pathophysiology 1.01 0.37 0.04 0.97 0.50 2.06
Health Assessment 1.71 0.80 1.14 0.25 0.68 4.30
Foundations I† 4.94 2.61 3.02 0.00 1.75 13.91
Childbearing Family 1.35 0.79 0.51 0.61 0.43 4.25
Health and Illness ICA 2.45 1.30 1.69 0.09 0.87 6.92
Health and Illness AOA I 2.65 1.55 1.67 0.10 0.84 8.33
Foundations II 0.42 0.24 -1.49 0.14 0.13 1.32
Pharmacotherapeutics 1.24 0.50 0.53 0.60 0.56 2.71
Psych Mental Health† 4.71 2.41 3.02 0.00 1.72 12.86
Health and Illness AOA II 0.80 0.53 -0.34 0.73 0.22 2.90
Year Fixed Effects
Year 2012 0.53 1.13 -0.30 0.77 0.01 33.81
Year 2013 1.00 1.99 0.00 1.00 0.02 50.03
Year 2014 0.38 0.75 -0.49 0.62 0.01 18.62
97
Intercept 0.00 0.00 -3.21 0.00 0.00 0.01
Appendix B
15 Point Bandwidth
7.5 Point Bandwidth
0
.00
2.0
04
.00
6
-1000 -500 0 500-1000 -500 0 500
Remediaton Not Required Remediation Required
Den
sity
Centered HESI Exit Exam Score
98
Appendix C
McCrary Test Results by Bandwidth.
Bandwidth for Wald Test Statistic Coef. SE z P>z LB UB
0 0.10 0.12 0.84 0.40 -0.13 0.33
25 0.00 (omitted)
30 0.00 (omitted)
35 0.00 (omitted)
40 0.00 (omitted)
45 0.00 (omitted)
50 0.00 (omitted)
55 0.00 (omitted)
60 0.10 0.12 0.84 0.40 -0.13 0.33
65 0.00 (omitted)
70 0.10 0.12 0.84 0.40 -0.13 0.33
75 0.10 0.12 0.84 0.40 -0.13 0.33
80 0.10 0.12 0.84 0.40 -0.13 0.33
85 0.10 0.12 0.84 0.40 -0.13 0.33
90 0.10 0.12 0.84 0.40 -0.13 0.33
95 0.10 0.12 0.84 0.40 -0.13 0.33
0
.00
2.0
04
.00
6.0
08
-1000 -500 0 500-1000 -500 0 500
Remediation Not Required Remediation RequiredD
en
sity
Centered HESI Exit Exam Score
99
100 0.00 (omitted)
105 0.10 0.12 0.84 0.40 -0.13 0.33
110 0.00 (omitted)
115 0.27 0.22 1.20 0.23 -0.17 0.71
120 0.27 0.22 1.21 0.23 -0.17 0.71
125 0.27 0.22 1.22 0.22 -0.17 0.71
130 0.27 0.22 1.22 0.22 -0.17 0.71
135 0.27 0.22 1.23 0.22 -0.16 0.71
140 0.27 0.22 1.23 0.22 -0.16 0.71
145 0.28 0.22 1.23 0.22 -0.16 0.71
150 0.28 0.22 1.23 0.22 -0.16 0.71
155 0.28 0.22 1.24 0.22 -0.16 0.71
160 0.28 0.22 1.24 0.22 -0.16 0.71
165 0.28 0.22 1.24 0.22 -0.16 0.72
170 0.27 0.22 1.24 0.22 -0.16 0.71
175 0.22 0.21 1.05 0.29 -0.19 0.63
180 0.19 0.20 0.92 0.36 -0.21 0.59
185 0.16 0.20 0.82 0.42 -0.23 0.56
190 0.15 0.20 0.74 0.46 -0.24 0.53
195 0.13 0.20 0.68 0.50 -0.25 0.52
200 0.12 0.19 0.63 0.53 -0.26 0.50
205 0.11 0.19 0.58 0.56 -0.27 0.49
210 0.11 0.19 0.55 0.59 -0.27 0.48
215 0.10 0.19 0.51 0.61 -0.28 0.48
220 0.09 0.19 0.49 0.63 -0.28 0.47
225 0.09 0.19 0.46 0.65 -0.29 0.46
230 0.08 0.18 0.46 0.64 -0.27 0.43
235 0.08 0.17 0.47 0.64 -0.25 0.40
240 0.07 0.16 0.47 0.64 -0.24 0.39
245 0.07 0.15 0.47 0.64 -0.23 0.37
250 0.07 0.15 0.46 0.64 -0.22 0.36
Appendix D
The CJM heaping test under triangular, uniform, and Epinochev kernel distributional
assumptions do not yield statistical evidence of manipulation of the running variable at the 95%
confidence level. The Table below summarizes the parameters associated with the CJM heaping
test under the triangular distribution.
100
Results of CJM density test
Cutoff c = 0.000 Left of c Right of c
Number of obs 3,404 2,105
Effective Number of obs 1,551 840
Order local polynomial 2 2
Order BC 3 3
Bandwidth 63 70
T Statistic -1.66
Appendix E
The table below summarizes the continuity of the demographics, performance in nursing
courses, and academic readiness measures for the 18 percent of the sample that has those data
available. Seventeen out of the eighteen variables do not yield any evidence of violating the
continuity assumption. The first order polynomial specification, however, picks up a negative
statistically significant (s) discontinuity for Psych/Mental Health course at the cutoff of 850. This
holds true even when the cut-off is moved to 825 (negative discontinuity) and 800 (positive
discontinuity). The discontinuities present at 825 and 800 suggest that the discontinuity at 850
may be a statistical artifact. None of the other 17 variables pick up significance when the cut-off
is artificially moved to 800 and to 825. In addition, second and third order polynomial
specifications detect only non-statistically significant (ns) discontinuities for Psych/Mental Health
at 850. Thus, the finding for the negative discontinuity for Psych/Mental Health is relatively
unstable. This leads to the conclusion that there is no discontinuity in pre-treatment measures
101
and extrapolating this finding from the subsample to the full sample of students in institutions
with required remediation. Nevertheless, it should be noted that if the finding were to be valid
then the discontinuity would downwardly bias the effect of remediation.
Continuity of Demographics and Academic Performance and Readiness at Cutoff
Order of Polynomials
1 2 3
Demographics
EOF Status ns ns ns
Minority ns ns ns
Male ns ns ns
Main Campus ns ns ns
Academic Readiness
SAT Verbal Score ns ns ns
SAT Math Score ns ns ns
Academic Performance
HC Delivery ns ns ns
Pathophysiology ns ns ns
Health Assessment ns ns ns
Foundations I ns ns ns
Childbearing Family ns ns ns
Health and Illness ICA ns ns ns
Health and Illness AOA I ns ns ns
Foundations II ns ns ns
Pharmacotherapeutics ns ns ns
Psych Mental Health s ns ns
Health and Illness AOA II ns ns ns
102
Appendix F
As the table below points out, second order polynomial bias-corrected estimates are not
sensitive to increases/decreases of bandwidth of 10 and 20 percent.
Falsification Exercise for Table 7 Statistically Significant Results
Polynomial Order Baseline 2nd: b=137 & h=74
Difference from Baseline -0.2 -0.1 0.1 0.2
Bandwidth (h) 60 67 81 89
Bandwidth (b) 110 123 150 164
Methodology β β β β
Conventional (b) 0.02 0.03 0.03 0.03
Bias-corrected (h) (0.22) †† (0.25) †† (0.29) †† (0.25) ††
Robust (h) (0.22) (0.25) (0.29) †† (0.25) ††
†statistically significant at 95% level ††statistically significant at 99% level
Statistically insignificant results are not sensitive to changes in bandwidth. The
falsification exercise was structured as follows. For each order polynomial n for bandwidth
selector type i on Table 4, the difference between the maximum and the minimum values is
103
calculated and then divided by four to get x. Three bandwidths for the falsification/sensitivity
exercises for each order polynomial n for bandwidth selector type i were calculated as:
Minimum value + x Minimum value + 2x Minimum value + 3x Thus, for each order polynomial, treatment effects are estimated at three additional bandwidths
within the values of the largest and smallest bandwidths produced by the various bandwidth
selectors.
Falsification Exercise for All Table 4 Results
Polynomial Order 1st 2nd 3rd
Bandwidth (b) 102 129 156 171 268 365 178 265 352
Bandwidth (h) 105 131 157 128 240 322 167 258 339
Methodology β β β β β β β Β β
Conventional (b) 0.02 0.02 0.02 0.03 0.02 0.02 0.02 0.03 0.03
Bias-corrected (h) 0.02 0.03 0.03 0.02 0.03 0.03 0.01 0.02 0.03
Robust (h) 0.02 0.03 0.03 0.02 0.03 0.03 0.01 0.02 0.03
†statistically significant at 95% level ††statistically significant at 99% level
All the estimates are statistically insignificant.
104
Appendix G
The observations of the sample that have demographics, SAT scores, and performance in
nursing courses suggest that covariate balance may exist within 10 points of the cutoff. Table 5
summarizes tests of proportions bandwidths of ten points and lower. At all bandwidths, the
first-time pass rates for students subject to remediation are lower than those who met or
surpassed the cutoff. None of the differences in pass rates are, however, statistically significant.
Overall, the results based on zero order polynomials do not suggest either a negative nor positive
impact.
Table 5. Results of Test of Proportions by Bandwidth
Treatment Control Bandwidth Pass Rate n Pass Rate n Difference
10 89.4% 104 93.1% 250 -3.7%
9 88.7% 97 92.2% 226 -3.6%
8 88.5% 87 93.0% 206 -4.5%
7 88.3% 77 92.9% 186 -4.6%
6 88.6% 70 93.2% 169 -4.7%
105
5 87.9% 58 92.6% 148 -4.6%
4 89.4% 47 92.1% 121 -2.7%
3 87.9% 33 90.6% 101 -2.7%
2 92.0% 25 93.5% 85 -1.5%
1 83.3% 12 93.3% 62 -10.0%
†statistically significant at 95% level ††statistically significant at 99% level
Localized Randomized Experiment Estimates: First, Second, and Third Order Polynomials
Results of Table 5 are sensitive to functional form and distributional assumptions. Table
6 summarizes the statistical significance of treatment estimates derived using bandwidths of 10
points or less. All estimates using bandwidths under each kernel function - uniform (Uni),
triangular(Tri), and Epanechnikov (Epa) – are statistically insignificant (ns). The only statistically
significant (s) result is positive is at the four-point bandwidth but it is sensitive to functional
assumptions.
Table 6. Results of Treatment Effect Estimates by Polynomial Order/Functional Assumption
1st Order Polynomial 2nd Order Polynomial 3rd Order Polynomial
Bandwidth Uni Tri Epa Uni Tri Epa Uni Tri Epa
10 ns ns ns ns ns ns ns ns ns
9 ns ns ns ns ns ns ns ns ns
8 ns ns ns ns ns ns ns ns ns
7 ns ns ns ns ns ns ns ns ns
6 ns ns ns ns ns ns ns ns ns
5 ns ns ns ns ns ns ns ns ns
4 ns ns ns ns ns ns s ns ns
3 ns ns ns ns ns ns ns ns ns
2 ns ns ns ns ns ns ns ns ns
1 -16 - - - - - - - -
16 - indicates that estimates could not be calculated
106
Appendix H.
Variable Odds Ratio Std. Err. z P>z UB LB
HESI Score 1.01 0.00 16.05 0.00 1.01 1.01
Treatment 1.14 0.18 0.82 0.41 0.83 1.56
Required Remediaton Policy 2.28 1.86 1.01 0.31 0.46 11.31
Appendix I
Variables No Remediation Remediation Difference
107
Student Characteristics
Non-White 66% 62% 5%
White 34% 38% -5%
Male 13% 9% 3%
Main Campus 51% 47% 4%
EOF 6% 10% -4%
Pre-admission Academics
Composite SAT 1117 1152 -37
SAT Verbal 543 564 -22
SAT Math 574 588 -15
Observations 98 93
% of treatment/control 23% 33%
**statistically significant at 95% level *** statistically significant at 99% level
Appendix J
Area of Common Support for All Observations with Scores Lower than 850
108
Variable Treated Control %bias T-Statistic p>t
HESI Score 742.13 737.65 5.7 1.35 0.178
BSN School 51% 51% 0 0 1
PN School 10% 11% -0.5 -0.13 0.895
P Score 0.59 0.59 0 0.01 0.993
Appendix K
Area of Common Support for All Observations with Scores Lower than 600
.4 .5 .6 .7 .8Propensity Score
Untreated Treated: On support
Treated: Off support
109
Variable Treated Control %bias T-Statistic p>t
HESI_score 553.67 557.91 -7.9 -0.73 0.469
BSN School 82% 81% 2.3 0.19 0.852
P Score 0.78 0.79 -1.5 -0.13 0.898
.4 .5 .6 .7 .8 .9Propensity Score
Untreated Treated: On support
Treated: Off support
110
Appendix L
Selectivity of First PSI Attended Freshmen: STEM
Freshmen: non-STEM
n % n %
Highly - 4 year 1,799 20% 569 36%
Moderately - 4 year 2,418 27% 449 28%
Inclusive - 4 year 610 7% 113 7%
Selectivity not classified - 4 year 445 5% 82 5%
Selectivity not classified - 2 year 3,318 37% 364 23%
Selectivity not classified - less than 2 years 288 3% 25 2%
Total 8,878 100% 1,602 100%
Appendix M
Selectivity
111
Enrollment Status of STEM Freshmen at First PSI Attended at Third Follow-up
Highly Moderately
n % n %
Earned a credential from PSI; still attending PSI as of 2012
15 3% 11 3%
Earned a credential from PSI; no longer attending PS1 as of 2012
336 64% 181 45%
No cred from PSI; still attending PSI as of 2012
4 1% 14 4%
No cred from PSI; no longer attending PSI; did attend another PS institution
161 30% 170 43%
No cred from PSI; no longer attending PSI; did not attend another PS institution
12 2% 24 6%
Total 528 100% 400 100%
Appendix N
Math Course Pipeline
112
No Math
Non-Academic
Low Academic
Middle Academic I
Middle Academic II
Advanced Academic I
Advanced Academic II/Pre-Calc
Advanced Academic III / Calculus
Science Course Pipeline
No Science
Primary Physical Science
Secondary Physical Science and Basic Bio
General Biology
Chemistry 1 or Physics 1
Chemistry 1 and Physics 1
Chemistry 2 or Physics 2
Chemistry and Physics and Level 7
Appendix O. Pre-matching Differences between Treatment and Control
Variables Moderately
Selective Highly
Selective Difference
113
SAT Math 580.06 629.70 -49.633***
SAT Verbal 554.42 589.73 -35.317***
Math Pipeline 2 0.00 0.00 -0.003***
Math Pipeline 3 0.00 0.00 0.00
Math Pipeline 4 0.01 0.01 0.002**
Math Pipeline 5 0.08 0.03 0.052***
Math Pipeline 6 0.10 0.10 -0.005**
Math Pipeline 7 0.19 0.24 -0.043***
Math Pipeline 8 0.61 0.63 -0.018***
Science Pipeline 2 0.01 0.00 0.008***
Science Pipeline 3 0.00 0.01 -0.006***
Science Pipeline 4 0.06 0.02 0.042***
Science Pipeline 5 0.16 0.13 0.030***
Science Pipeline 6 0.24 0.26 -0.019***
Science Pipeline 7 0.15 0.13 0.024***
Science Pipeline 8 0.37 0.45 -0.079***
Black 0.02 0.04 -0.019***
URM 0.13 0.06 0.071***
White 0.75 0.83 -0.077***
Female 0.28 0.30 -0.024***
Middle Income 0.52 0.37 0.151***
High Income 0.31 0.52 -0.205***
Moderately Selective Admissions 1.53 1.36 0.175***
High Selective Admissions 1.58 2.17 -0.597***
Total Applications 4.16 3.93 0.232***
Catholic 0.07 0.10 -0.026***
Private 0.04 0.09 -0.045***
Suburban 0.47 0.47 -0.01
Rural 0.27 0.18 0.089***
Observations
14,145
69,577
* Significant at 90% Confidence
** Significant at 95% Confidence
*** Significant at 99% Confidence
Appendix P: Unmatched Sample Logistic Regression Results
Variables Odds Ratio
Std. Error
z P>z Lower Bound
Upper Bound
114
Highly Selective 2.50 0.11 20.66 0.00 2.29 2.73
SAT Math 1.00 0.00 12.21 0.00 1.00 1.01
SAT Verbal 1.00 0.00 -5.83 0.00 1.00 1.00
Math Pipeline 2 1.00 (omitted) Math Pipeline 3 1.00 (omitted) Math Pipeline 4 1.00 (omitted) Math Pipeline 5 1.00 (omitted) Math Pipeline 6 0.87 0.08 -1.54 0.12 0.73 1.04
Math Pipeline 7 0.50 0.03 -13.37 0.00 0.45 0.55
Math Pipeline 8 1.00 (omitted) Science Pipeline 2 1.00 (omitted) Science Pipeline 3 1.00 (omitted) Science Pipeline 4 594.15 103.36 36.72 0.00 422.50 835.55
Science Pipeline 5 7.15 0.53 26.30 0.00 6.18 8.28
Science Pipeline 6 0.94 0.04 -1.40 0.16 0.85 1.03
Science Pipeline 7 2.67 0.18 14.81 0.00 2.35 3.04
Science Pipeline 8 1.00 (omitted) Black 2.92 0.35 8.96 0.00 2.31 3.69
URM 12.48 1.21 26.00 0.00 10.31 15.09
White 16.76 1.44 32.70 0.00 14.15 19.84
Female 0.04 0.00 -53.84 0.00 0.04 0.05
Middle Income 0.49 0.03 -13.00 0.00 0.44 0.54
High Income 0.16 0.01 -30.11 0.00 0.14 0.18
Moderately Selective Admissions
1.19 0.04 4.95 0.00 1.11 1.27
High Selective Admissions 1.77 0.05 19.40 0.00 1.67 1.87
Total Applications 0.81 0.02 -10.22 0.00 0.78 0.84
Catholic 0.27 0.02 -22.21 0.00 0.24 0.30
Private 0.09 0.01 -29.90 0.00 0.08 0.10
Suburban 0.80 0.04 -4.91 0.00 0.73 0.87
Rural 0.22 0.02 -20.14 0.00 0.19 0.25
Intercept 0.04 0.01 -15.28 0.00 0.03 0.06
Observations 20,166 Pseudo R^2 0.4848
115
Appendix Q. Post-matching Differences between Treatment and Control
Variables Highly
Selective Moderately
Selective Difference
SAT Math 579.63 599.40 -19.77
SAT Verbal 555.76 554.72 1.04
Math Pipeline 2 0% 0% 0%
Math Pipeline 3 0% 0% 0%
Math Pipeline 4 0% 0% 0%
Math Pipeline 5 12% 2% 11%
Math Pipeline 6 0% 0% 0%
Math Pipeline 7 23% 26% -3%
Math Pipeline 8 65% 73% -7%
Science Pipeline 2 0% 0% 0%
Science Pipeline 3 0% 0% 0%
Science Pipeline 4 7% 1% 6%
Science Pipeline 5 7% 7% -1%
Science Pipeline 6 34% 37% -4%
Science Pipeline 7 5% 11% -6%
Science Pipeline 8 48% 44% 4%
Black 5% 4% 0%
URM 15% 11% 4%
White 74% 82% -8%
Female 22% 22% 0%
Middle Income 42% 43% 0%
High Income 37% 40% -3% Moderately Selective Admissions
159% 140% 19%
High Selective Admissions
160% 147% 13%
Total Applications 457% 381% 76%
Catholic 11% 13% -2%
Private 4% 3% 0%
Suburban 41% 60% -19%
Rural 17% 13% 4%
Observations 8,447 8,447
116
Appendix R. Area of Common Support
0 .2 .4 .6 .8 1Propensity Score
Untreated Treated: On support
Treated: Off support