unc teacher quality research: 2013 teacher preparation ... · unc teacher quality research: 2013...

UNC Teacher Quality Research:2013 Teacher Preparation

Program Effectiveness Report- MAY 2013 -

Gary T. Henry, Vanderbilt University

Kristina M. Patterson, UNC–Chapel Hill

Shanyce L. Campbell, UNC–Chapel Hill

Pan Yi, UNC–Chapel Hill

UNC Teacher Quality Research:

2013 Teacher Preparation Program Effectiveness Report

May 2013

Gary T. Henry, Vanderbilt University

Kristina M. Patterson, UNC–Chapel Hill

Shanyce L. Campbell, UNC–Chapel Hill

Pan Yi, UNC–Chapel Hill

Contents

Acknowledgements ………………………….………………………… i

Introduction ………………………………………………….………. 1

Data and Methods……..…….………………………………….…….. 3

Results ……………………………………………………………... 8

Conclusion ………………………………………………………….. 17

Calculating Days Equivalency ……………………………………….….. 18

Tables:

Table 1: Standard Model Control Variables ……………………………….. 4

Table 2: Institution Abbreviations ……………………………………….. 5

Table 3: UNC Institutional Counts ………………………………….…… 7

Table 4: Summary of Results ………………………………….….….… 15

Table 5: Teacher Counts by UNC System School, Level and Subject …………… 16

Table 6: Key for the Interpretation of Coefficients (Days Equivalency) ……..…… 17

Figures:

Figure 1: UNC Programs vs. All Other Sources of Teachers: Elementary School ……. 12

Figure 2: UNC Programs vs. All Other Sources of Teachers: Middle School ……..… 13

Figure 3: UNC Programs vs. All Other Sources of Teachers: High School …….…… 14

i

Acknowledgements

We wish to recognize Alisa Chapman and Keith Brown with the University of North

Carolina General Administration for their vital contributions in providing data and working as

partners throughout the research and communication processes.

We also wish to thank the deans and department heads from the colleges, schools and

departments of education at the 15 UNC institutions engaged in teacher education for their

valuable input during the development of the models and discussions of the findings. We

gratefully acknowledge the many contributions made by our current and former researchers and

fellows at the Education Policy Initiative at Carolina (EPIC), including Kevin C. Bastian, C.

Kevin Fortner, David C. Kershaw, Jade V. M. Jenkins, Charles L. Thompson, and Rebecca A.

Zulli. Finally, we wish to acknowledge the editing and formatting work done by Elizabeth

D’Amico, who is responsible for the overall look and polish of the report. All authors accept

responsibility for any remaining errors in the report.

Department of Public Policy Education Policy Initiative at Carolina

University of North Carolina at Chapel Hill

140 Friday Center Drive, Chapel Hill, NC 27599-9000

919-962-0668 publicpolicy.unc.edu

http://www.publicpolicy.unc.edu/

Introduction

This report, produced in collaboration with the UNC General Administration, presents the third1

set of results assessing the effectiveness of undergraduate teacher preparation programs at North

Carolina public universities. UNC established this research agenda in 2009 in order to quantify

the effects of UNC traditional (undergraduate) preparation programs on student achievement in

North Carolina and to provide results to assist UNC Schools of Education in the evaluation and

improvement of their programs. The purpose of this analysis is to accurately estimate the

average test score gains for public school students taught by recent graduates of each of UNC’s

fifteen undergraduate teacher preparation programs, net of other variables that affect student

performance. Thus, this report applies the value-added models used in prior studies, with

extensive student, classroom, and school covariates, in order to isolate the effects of traditional

UNC teacher preparation programs on student achievement in the public schools of North

Carolina.

Generating quantitative estimates of program effectiveness allows the UNC institutions to see

where their program graduates perform well, provides a starting point for inquiry about program

characteristics that affect student achievement scores, and identifies programs that need

improvement. The information in this report covers programmatic effects across the entire range

of grade levels and subjects tested by public schools in North Carolina. Estimates of program

effectiveness vary widely across these subject and grade level combinations, and therefore, there

is no single measure of overall effectiveness for each program.

Two types of models were employed for this analysis: 1) models comparing the graduates of

each of the 15 public undergraduate preparation programs to all other teachers in the state, and 2)

individual campus models comparing each undergraduate program to 12 other categories of

teacher preparation, including alternative entry and teachers prepared out-of-state. These two

approaches serve different purposes. The first allows comparisons between each of the

undergraduate teacher preparation programs and a common reference group representing the

average of all other types of teacher preparation. Essentially, this first analysis shows how

effective each of the undergraduate programs is by comparing it to the teaching corps that North

Carolina would have if the traditional UNC programs did not exist. This analysis is the focus of

this report. The second set of models provides a more detailed comparison between

undergraduate prepared teachers from an individual UNC institution and teachers prepared

through the other major means by which teachers are prepared to teach in North Carolina. This

set of value-added models specifies each UNC institution as the reference group and makes

direct comparisons between the reference institution and 12 other categories of teacher

preparation, including teachers from other in-state public institutions, in-state private institutions,

1 See Henry, G.T., Thompson, C.L., Fortner, C.K., Zulli, R.A., and Kershaw, D.C (2010). The Impact of Teacher

Preparation on Student Learning in North Carolina Public Schools. Chapel Hill, NC: The Carolina Institute for

Public Policy. Available online at:

http://publicpolicy.unc.edu/research/Teacher_Prep_Program_Impact_Final_Report_nc.pdf and

Henry, G.T., Thompson, C.L., Bastian, K.C., Fortner, C.K., Kershaw, D.C., Marcus, J.V., and Zulli R.A. (2011)

UNC Teacher Preparation Program Effectiveness Report. Chapel Hill, NC: The Carolina Institute for Public Policy.

Available online at: http://publicpolicy.unc.edu/research/TeacherPrepEffectRpt_Final.pdf

http://publicpolicy.unc.edu/research/Teacher_Prep_Program_Impact_Final_Report_nc.pdf

http://publicpolicy.unc.edu/research/TeacherPrepEffectRpt_Final.pdf

2

out-of-state universities, Teach For America, and other alternative entry teachers. The 12 other

categories of teachers are defined in the 2011 UNC Teacher Preparation Program Effectiveness

Technical Report.2 We provide results from this second set of models to the deans of each

institution in a separate institution-specific report.

These estimates of the effects of traditional UNC teacher preparation programs reflect both the

knowledge and skills that teachers develop while they are prospective teachers at a particular

university and the capacity or learned ability that the prospective teachers bring with them into a

particular preparation program. While it may be useful to separate these effects to determine

which programs add more value during the preparation process, the education that a teacher

provides to a student in North Carolina public schools is a product of both preparation and

academic ability. Therefore, these estimates are intended to capture the total effect of teacher

preparation and selection in a single estimate of value-added model outcomes.

The effectiveness of teacher preparation programs should be judged, at least in part, on the extent

to which teachers prepared by these programs are able to produce gains in students’ knowledge

and skills. To assess these gains we use the standards-based state assessments as the measures of

student learning. In this report, we estimate the effects of teachers on student performance on the

available End-of-Grade (EOG) and End-of-Course (EOC) tests administered by the North

Carolina Department of Public Instruction (NC DPI). These test scores have strengths and

weaknesses as a method for estimating the effectiveness of teachers. The student assessment

program in North Carolina serves as a model for other states and the state’s test scores are widely

used in rigorous, high quality research and evaluation studies. North Carolina’s state tests are

aligned with the NC Standard Course of Study within each grade and/or subject, and tests in

reading and mathematics in grades 3 to 8 are vertically scaled to allow for comparisons across

years and grades. Student EOG and EOC test scores serve as the basis for the ABCs

accountability program in North Carolina as well as for calculating the value-added by teachers

and schools to student test score growth which will be used in the state’s new system for

evaluating teachers and principals. Because EOG and EOC tests are closely aligned with the

intended curriculum, the test score gains produced by program graduates are appropriate for

estimating the effectiveness of teachers who graduate from UNC teacher preparation programs.

The weaknesses of using student test score outcomes to measure teaching effectiveness include

the limited subjects and grades in which testing is conducted—students below grade three are not

tested; students in 3rd

through 8th

grade are tested only in reading and mathematics, annually, and

in science in only the 5th

and 8th

grade; and the number of EOC tests in high school has decreased

in recent years. This prevents any estimates of teacher effectiveness based on test scores in other

subject or grade level combinations and means that program effectiveness estimates are based on

a subset of program graduates and not on all teachers prepared by the preparation program (see

Table 3). Other important outcomes, such as graduation, attitudes toward school and learning, or

capacity to function as a citizen within a democracy, are not captured by these standardized tests.

The tests measure the extent to which individual teachers succeed in teaching their students the

state-adopted curriculum in specific grades and subjects. So while limited, they do provide

useful outcome-based information for assessing the effectiveness of teacher preparation

programs. 2 Henry, G.T., Thompson, C.L., Fortner, C.K., Bastian, K.B. and., Marcus, J.V. (2011) Technical Report: UNC

Teacher Preparation Program Effectiveness Report. Available online at:

http://publicpolicy.unc.edu/research/ProgramEffect_TechRpt_July2011.pdf

http://publicpolicy.unc.edu/research/ProgramEffect_TechRpt_July2011.pdf

3

This report provides an overview of the data and methodology of the current analysis and the

changes between this evaluation of teaching program effectiveness and the prior report.3

Subsequent sections include the results of the analysis—comparing each undergraduate program

to the aggregate of all other sources of teacher preparation—and a conclusion section. We

provide separate campus reports (not included here) to each institution with results from the

individual campus models.

Data and Methods

This 2013 edition of the UNC Teacher Preparation Program Effectiveness Report continues to

utilize a value-added modeling approach that compares the average student learning gains for

students whose teachers have different types of preparation prior to entering the classroom in

North Carolina public schools. These value-added models are estimated within a multi-level

model specification with extensive controls at the student, classroom, and school level. Limited

controls for teachers (years of experience and out-of-field teaching) are included in the models in

order to generate comparable estimates of the total effect of teacher preparation programs on

student achievement. For example, although we expect teachers with higher Praxis scores and

National Board Certification to generate greater learning gains for their students, on average, we

exclude these variables from the analysis since teachers from certain institutions may be more

likely to have some of these characteristics, and including them would downwardly bias program

effect estimates.

Table 1 includes a complete list of control variables included at each of the three levels in the

multi-level model. The definitions for three of the variables may not be obvious: structural

mobility refers to students who changed schools due to the grade configuration of a school (i.e.

6th grade students in a 6 - 8 middle school); between-year mobility refers to students who either

attended or completed assessments at a different school in the prior academic year; and within-

year mobility refers to students who were enrolled in the school where they took their tests for

substantially less than the full school year (more than two weeks less than a full year).

3 Henry, G.T., Thompson, C.L., Bastian, K.C., Fortner, C.K., Kershaw, D.C., Marcus, J.V., and Zulli R.A. (2011)

UNC Teacher Preparation Program Effectiveness Report. Chapel Hill, NC: The Carolina Institute for Public Policy.

Available online at: http://publicpolicy.unc.edu/research/TeacherPrepEffectRpt_Final.pdf

http://publicpolicy.unc.edu/research/TeacherPrepEffectRpt_Final.pdf

4

Table 1: Standard Model Control Variables

Student Classroom & Teacher School

1. Prior test scores (reading

& math)

2. Classmates prior test

scores (peer effects)

3. Days absent

4. Structural mobility

5. Between-year mobility

6. Within-year mobility

7. Gender

8. Race/ethnicity

9. Poverty

10. Gifted

11. Disabled

12. Currently limited English

proficient

13. Previously limited

English proficient

14. Overage for grade (held

back or retained at least

once)

15. Underage for grade

(promoted two grades)

16. Years of experience

17. Teaching out-of-field

18. Number of students

19. Advanced curriculum

20. Remedial curriculum

21. Dispersion of prior

achievement within

classroom

22. School size (ADM)

23. School size squared

24. Suspension rate

25. Violent acts rate

26. Total per pupil expenditures

27. District teacher supplements

28. Racial/ethnic composition

29. Concentration of poverty

Table 2 (below) lists the 15 campuses included in the program comparison results contained in

this report and the abbreviations for each campus used throughout the report. Separate reports

present the results of each of these campuses compared to 12 other categories of teacher

preparation. To identify graduates of traditional UNC preparation programs for this analysis, we

rely on data from the UNC General Administration, which includes information on the

undergraduate major(s) and campus enrollment for each individual who attended a North

Carolina four-year public institution. Individuals are classified as program graduates if they

graduated from a UNC institution with an education major or graduated with another major and

simultaneously received a teaching license. These individuals are considered traditionally

prepared teachers in this report regardless of any additional preparation (such as a Master’s

degree from another institution) obtained between graduating from the traditional teacher

education program and starting work as a classroom teacher. This definition allows the UNC

teacher preparation program comparisons to include all of the traditional undergraduate prepared

teachers that graduated from each institution, but creates slight differences between the teacher

categories specified in this report and the Education Policy Initiative at Carolina’s prior

publication on portal effectiveness.4

4 Portal Report: Teacher Preparation and Student Test Scores in North Carolina. (2010). Chapel Hill, NC: The

Education Policy Initiative at Carolina (EPIC) formally known as the Carolina Institute for Public Policy. Available

online at:

http://publicpolicy.unc.edu/research/Teacher_Portals_Teacher_Preparation_and_Student_Test_Scores_in_North_Ca

rolina_2.pdf

http://publicpolicy.unc.edu/research/Teacher_Portals_Teacher_Preparation_and_Student_Test_Scores_in_North_Carolina_2.pdf

http://publicpolicy.unc.edu/research/Teacher_Portals_Teacher_Preparation_and_Student_Test_Scores_in_North_Carolina_2.pdf

5

Table 2: UNC Institution Abbreviations

Teacher Preparation Program Abbreviation

Appalachian State University ASU

East Carolina University ECU

Elizabeth City State University ECSU

Fayetteville State University FSU

North Carolina Agricultural & Technical State University NCA&T5

North Carolina Central University NCCU

North Carolina State University NCSU

University of North Carolina – Asheville UNCA

University of North Carolina – Chapel Hill UNCCH

University of North Carolina – Charlotte UNCC

University of North Carolina – Greensboro UNCG

University of North Carolina – Pembroke UNCP

University of North Carolina – Wilmington UNCW

Western Carolina University WCU

Winston-Salem State University WSSU

The data used for this analysis includes student, classroom, and school characteristics from

public schools in North Carolina from the 2006-07 through the 2010-11 school years. This five-

year span updates the data used (2005-06 through 2009-10) in our previous analysis. In addition,

we restrict the models to teachers with less than five years of teaching experience. This allows us

to assess the effectiveness of relatively recent graduates from each program. The cutoff of less

than five years of experience balances the need for a sample of program graduates that is large

enough to calculate effects and recognizes that the effects of a teacher’s training program will

diminish over time as a teacher learns from classroom experience, principal and peer feedback,

and other professional development.

This report includes results from 11 distinct models for student test score outcomes:

elementary school mathematics, reading (grades 4 and 5), and science (grade 5 only); middle

school mathematics, reading (grades 6-8), science (grade 8 only), and algebra I; and high school

(grades 9 – 12) algebra I & II, English I, science (biology and physical science), and social

studies (US history and civics & economics). The previous report did not include analysis of

elementary school science. High school models are slightly different from the prior analysis due

to the elimination of a number of EOC tests (North Carolina discontinued the geometry,

chemistry, and physics tests during our study period). Finally, we eliminated the high school

overall model due to the changes in tests previously included in that analysis.

We present results in a graphical format that resembles a thermometer and displays institutions

or teacher preparation categories vertically, corresponding to their estimates of effectiveness.

These graphs provide an accessible visual presentation of the data, but introduce the need for

some explanatory notes. First, only comparisons with categories that contain at least ten teachers

5 Abbreviated as NCAT in Figures 1, 2, and 3

6

from a given teacher preparation program are reported. This count is based on unique teachers

across years and may include observations of student outcomes over numerous years that are

linked to a given teacher. Second, results which are statistically significant at the 0.05 level are

marked with asterisks and bolded to indicate they are significantly different from the reference

category, teachers from all sources other than undergraduate preparation programs at UNC

institutions. We should also note that these estimates are reported as a percentage of a standard

deviation unit, meaning for example, that a value of 25 indicates that the students taught by a

teacher from a specific program are expected to generate test score outcomes 25 percent of a

standard deviation higher than similar students in similar schools and classrooms taught by

teachers not prepared by undergraduate preparation programs at UNC institutions.

Table 3 presents the total number of teachers from each institution who were teaching in North

Carolina public schools during the five year study period, the number of teachers with less than 5

years of experience teaching in NC public schools over the time period, and the number of

teachers with less than 5 years of experience who were teaching in tested grades and subjects,

and therefore, could be included in the models. In cases of missing or unmatched data, teachers

are excluded from models. (See Table 5 for exact counts of teachers included in each model).

7

Table 3: UNC Institutional Counts

Teacher

Preparation

Program

Total Teachers

from

2006-07 to 2010-111

Total Teachers with Less than

5 Years’ Experience

from

2006-07 to 2010-112

Total Teachers in Tested Grades

and Subjects with Less than 5

Years’ Experience from

2006-07 to 2010-113

ASU 6850 3074 1530

ECU 6537 2828 1263

ECSU 716 238 101

FSU 1253 555 282

NCA&T 698 282 128

NCCU 862 356 167

NCSU 1852 1030 614

UNCA 381 211 135

UNCCH 1924 629 281

UNCC 3342 1783 940

UNCG 3956 2019 839

UNCP 1784 681 316

UNCW 3204 1630 808

WCU 2425 1053 514

WSSU 614 197 77 1Represents the total number of unique, traditionally prepared teachers paid as certified instructors in North

Carolina public schools at least one time between 2006-07 and 2010-11. Counts in this column are for all teachers,

regardless of experience or subject/grade taught. 2 Represents the total number of unique, traditionally prepared teachers with less than five years of experience paid

as certified instructors in North Carolina public schools at least one time between 2006-07 and 2010-11. Counts in

this column are for all teachers with less than five years of experience, regardless of subject/grade taught. 3Represents the total number of unique, traditionally prepared teachers with less than five years of experience paid

as certified instructors in North Carolina public schools at least one time between 2006-07 and 2010-11 who taught

in tested subjects and grade levels during that time period.

8

Results

Graduates of traditional teacher preparation programs at each UNC institution were compared to

all other sources of teachers on 11 different standardized tests (EOG or EOC), with three

outcomes in elementary school, four in middle school, and four in high school. Results are not

reported in cases where a program had fewer than ten graduates teaching the grade and subject

associated with the EOG or EOC test. For elementary and middle school mathematics and

reading we report results in terms of equivalent days of instruction gained (or lost) by

comparable students whose teacher graduated from a particular program compared to all other

sources of teachers. We cannot report days equivalency in elementary school science, middle

school science, middle school algebra I, nor any high school EOC tests because the tests are not

vertically equated to a prior test score in the same subject.

The Effects of UNC Teacher Preparation Programs on Student Achievement

Appalachian State University (ASU): On average, graduates of Appalachian State

University’s undergraduate teacher preparation program were more effective than all other

sources of teachers in one comparison – elementary school mathematics. It may be useful to

think of this impact in terms of days of learning. A student with a teacher prepared at ASU could

gain an average of 6 days of learning per year more than a student instructed by a teacher in the

reference category.6 ASU graduates performed similarly to all other sources of teachers in the

remaining ten comparisons – elementary school reading, elementary school science, middle

school mathematics, middle school reading, middle school science, middle school algebra I, high

school algebra I and algebra II, high school science, high school English I, and high school social

studies.

East Carolina University (ECU): On average, graduates of East Carolina University’s

undergraduate teacher preparation program were more effective than all other sources of teachers

in three comparisons – elementary school reading, middle school algebra I, and high school

social studies. In elementary school reading, a student with a teacher prepared at ECU could

gain an average of 5 ½ days of learning per year more than a student instructed by a teacher in

the reference category. ECU graduates performed similarly to all other sources of teachers in the

remaining eight comparisons – elementary school mathematics, elementary school science,

middle school mathematics, middle school reading, middle school science, high school algebra I

& II, high school English I, and high school science.

Elizabeth City State University (ECSU): On average, graduates of Elizabeth City State

University’s undergraduate teacher preparation program performed similarly to all other sources

of teachers in five comparisons – elementary school mathematics, elementary school reading,

elementary school science, middle school mathematics, and middle school reading. There were

insufficient (<10) ECSU graduates teaching middle school science, middle school algebra I, high

school algebra I and algebra II, high school science, high school English I, and high school social

studies to report results from these comparisons.

6 See Appendix for instructions on calculating days equivalency.

9

Fayetteville State University (FSU): On average, graduates of Fayetteville State


sources of teachers in one comparison – high school algebra I & II. FSU graduates performed

similarly to all other sources of teachers in seven comparisons – elementary school mathematics,

elementary school reading, elementary school science, middle school mathematics, middle

school reading, high school English I, and high school social studies. There were insufficient

FSU graduates teaching middle school science, middle school algebra I, and high school science

to report results from these comparisons.

North Carolina A&T State University (NCA&T): On average, graduates of North

Carolina A&T State University’s undergraduate teacher preparation program performed

similarly to all other sources of teachers in five comparisons – elementary school mathematics,

elementary school reading, elementary school science, middle school mathematics, and high

school algebra I & II. On average, NCA&T graduates were less effective than all other sources

of teachers in one comparison – middle school reading. There were insufficient NCA&T

graduates teaching middle school science, middle school algebra I, high school English I, high

school science, and high school social studies to report results from these comparisons.

North Carolina Central University (NCCU): On average, graduates of North Carolina

Central University’s undergraduate teacher preparation program were more effective than

teachers from all other sources in one comparison – elementary school science. NCCU

graduates performed similarly to all other sources of teachers in four comparisons – elementary

school mathematics, elementary school reading, middle school mathematics, and middle school

reading. There were insufficient NCCU graduates teaching middle school science, middle

school algebra I, high school science, and high school social studies to report results from these

comparisons.

North Carolina State University (NCSU): On average, graduates of North Carolina

State University’s undergraduate teacher preparation program were more effective than all other

sources of teachers in one comparison – high school social studies. NCSU graduates performed

similarly to all other sources of teachers in the remaining ten comparisons – elementary school

mathematics, elementary school reading, elementary school science, middle school mathematics,

middle school reading, middle school science, middle school algebra I, high school algebra I and

algebra II, high school English I, and high school science.

University of North Carolina – Asheville (UNCA): On average, graduates of

University of North Carolina at Asheville’s undergraduate teacher preparation program

performed similarly to all other sources of teachers in seven comparisons – elementary school


middle school reading, high school English I, and high school science. On average, UNCA

graduates were less effective than all other sources of teachers in one comparison – high school

algebra I & II. There were insufficient UNCA graduates teaching middle school science, middle

school algebra I, and high school social studies to report results from these comparisons.

University of North Carolina – Chapel Hill (UNCCH): On average, graduates of

University of North Carolina at Chapel Hill’s undergraduate teacher preparation program were

more effective than all other sources of teachers in one comparison – middle school

mathematics. In terms of days of learning, a student with a teacher prepared at UNCCH could

gain an average of 44 days of learning per year more than a student instructed by a teacher in the

10

reference category. UNCCH graduates performed similarly to all other sources of teachers in

five comparisons – elementary school mathematics, elementary school reading, elementary

school science, middle school reading, and middle school science. There were insufficient

UNCCH graduates teaching middle school algebra I, high school algebra I and algebra II, high

school science, high school English I, and high school social studies to report results from these

comparisons.

University of North Carolina – Charlotte (UNCC): On average, graduates of

University of North Carolina at Charlotte’s undergraduate teacher preparation program were

more effective than all other sources of teachers in two comparisons – high school science and

high school social studies. UNCC graduates performed similarly to all other sources of teachers

in eight comparisons – elementary school mathematics, elementary school reading, elementary

school science, middle school mathematics, middle school reading, middle school science, high

school algebra I & II, and high school English I. On average, UNCC traditionally prepared

teacher were less effective than all other sources of teachers in one comparison – middle school

algebra I.

University of North Carolina – Greensboro (UNCG): On average, graduates of

University of North Carolina at Greensboro’s undergraduate teacher preparation program were

more effective than all other sources of teachers in one comparison – elementary school

mathematics. A student with a teacher prepared at UNCG could gain an average of 7 ½ days of

learning per year more than a student instructed by a teacher in the reference category. UNCG

graduates performed similarly to all other sources of teachers in nine comparisons – elementary

school reading, elementary school science, middle school mathematics, middle school reading,

middle school algebra I, high school algebra I and algebra II, high school science, high school

English I, and high school social studies. There were insufficient UNCG graduates teaching

middle school science to report results from this comparison.

University of North Carolina – Pembroke (UNCP): On average, graduates of

University of North Carolina at Pembroke’s undergraduate teacher preparation program

performed similarly to all other sources of teachers in eight comparisons – elementary school


middle school reading, high school algebra I & II, high school science, and high school social

studies. There were insufficient UNCP graduates teaching middle school science, middle school

algebra I, and high school English I to report results from these comparisons.

University of North Carolina - Wilmington (UNCW): On average, graduates of

University of North Carolina Wilmington’s undergraduate teacher preparation program were

more effective than all other sources of teachers in two comparisons – elementary school science

and middle school mathematics. A middle school mathematics student with a teacher prepared

at UNCW could gain an average of 32 ½ days of learning per year as compared to a student

instructed by a teacher in the reference category. UNCW graduates performed similarly to all

other sources of teachers in the remaining nine comparisons – elementary school mathematics,

elementary school reading, middle school reading, middle school science, middle school algebra

I, high school algebra I and algebra II, high school science, high school English I, and high

school social studies.

11

Western Carolina University (WCU): On average, graduates of Western Carolina


sources of teachers in one comparison – high school English I. WCU graduates performed

similarly to all other sources of teachers in the remaining ten comparisons – elementary school


middle school reading, middle school science, middle school algebra I, high school algebra I &

II, high school science, and high school social studies.

Winston-Salem State University (WSSU): On average, graduates of Winston-Salem

State University’s undergraduate teacher preparation program performed similarly to all other

sources of teachers in five comparisons – elementary school mathematics, elementary school

reading, elementary school science, middle school mathematics, and middle school reading.

There were insufficient WSSU graduates teaching middle school science, middle school algebra

I, high school algebra I and algebra II, high school science, high school English I, and high

school social studies.

The figures below present the results of models that compare the average effectiveness of

teachers from each of UNC’s fifteen traditional undergraduate teacher preparation programs to

the average effectiveness of teachers from all other sources of preparation. Where estimates are

statistically different from all other sources of teachers (significant at the 0.05 level), institution

names appear in bold text, are noted with a *, and have points designated with diamonds.

Institutions with insufficient data to report results (fewer than ten teachers in the model) are

excluded from graphs. The scaling of graphs may differ to maximize the readability of each

graph. The results are summarized in Table 4.

12

Figure 1: UNC Traditional Preparation Programs vs. All Other Sources of Teachers – Elementary School

13

Figure 2: UNC Traditional Preparation Programs vs. All Other Sources of Teachers – Middle School

14

Figure 3: UNC Traditional Preparation Programs vs. All Other Sources of Teachers – High School

15

Table 4: Summary of Results

The results summarized in Table 4 (above) are based on the number of statistically significant

positive or negative comparisons for each institution across the 11 models. Table 5 (below)

displays the number of teachers, by institution, included in each model. Higher numbers of

teachers in a particular group decrease the standard errors and make statistically significant

findings more likely. Schools with small sample sizes are unlikely to show statistically

significant results, even when the estimated effect is large in absolute value terms.

Institutions with

Two or More

Positive Results

Outperforming the

Reference Group Comparisons

Underperforming the


ECU ES Reading; MS Algebra I;

HS Social Studies ----

UNCC HS Science; HS Social Studies MS Algebra I

UNCW ES Science; MS Mathematics ----

Institutions with

One Positive Result

Outperforming the


Underperforming the


ASU ES Mathematics ----

FSU HS Algebra I & II ----

NCCU ES Science ----

NCSU HS Social Studies ----

UNCCH MS Mathematics ----

UNCG ES Mathematics ----

WCU HS English I ----

Institutions with

No Statistically Significant

Results

Outperforming the


Underperforming the


ECSU ---- ----

UNCP ---- ----

WSSU ---- ----

Institutions with

One Negative Result and

No Positive Results

Outperforming the Reference

Group Comparisons

Underperforming the


NCA&T ---- MS Reading

UNCA ---- HS Algebra I & II

16

Table 5: Teacher Counts by UNC System School, Level and Subject

Teacher

Preparation

Program

ES

Math

ES

Read

ES

Science

MS

Math

MS

Read

MS

Science

MS

Algebra

I

HS

Algebra

I & II

HS

English

HS

Science

HS

Social

Studies

ASU 666 666 183 177 180 30 33 116 112 35 147

ECU 620 617 192 139 140 30 23 81 70 40 79

ECSU 59 59 20 16 10 2 1 1 5 2 3

FSU 102 102 32 43 48 5 4 39 18 4 18

NCA&T 58 58 18 17 11 2 6 16 8 3 3

NCCU 88 88 27 23 17 2 8 12 11 3 8

NCSU 31 31 13 86 122 32 34 146 58 43 65

UNCA 46 46 15 10 17 2 4 14 19 13 7

UNCCH 164 164 43 40 41 12 9 8 5 8 1

UNCC 497 497 138 113 91 28 21 60 34 12 50

UNCG 406 405 114 77 105 8 13 39 69 14 42

UNCP 143 143 42 24 34 7 1 29 9 13 25

UNCW 416 416 113 96 81 18 21 59 58 12 29

WCU 215 215 61 68 70 10 11 38 30 18 37

WSSU 39 39 10 10 12 1 1 3 1 0 3 *Highlighted cells have fewer than ten teachers and therefore do not have any results reported.

17

Results in elementary and middle school mathematics and reading models may be interpreted in

terms of the equivalent days of instruction gained (or lost) by comparable students whose teacher

graduated from a particular program compared to all other sources of teachers. Table 6 contains

values for interpretation of effectiveness estimates (coefficients) depending on the model under

consideration. For example, comparable students in similar classrooms and schools are expected

to score as if they had attended 14 and one-third extra days of school when they are taught by a

teacher whose effectiveness estimate (coefficient) is five percent of a standard deviation higher

than the reference group in elementary school mathematics. These estimates vary based on the

subject and grade level, and the formula for calculating values based on different results is found

in the appendix of this report.

Table 6: Key for the Interpretation of Coefficients (Days Equivalency)

Result Values

(Coefficient/Effectiveness) ES Math

ES

Reading MS Math

MS

Reading

15.00 43.13 days 49.16 days 97.71 days 79.35 days




Note: These result values show days equivalency in relation to the reference group; negative result values

have negative days equivalency results. See the appendix section for directions on calculating the days

equivalency.

Conclusion

Of the fifteen traditional teacher education programs at UNC institutions, most are performing

about as well, or better than, all other sources of teachers in terms of their estimated impact on

student EOG and EOC achievement tests. Only two of the campuses had results with more

negative model outcomes than positive, ten campuses demonstrated more positive model

outcomes than negative results, and three campuses had no statistically significant results.

Overall, on average, UNC traditionally prepared teachers are likely to outperform teachers from

all other sources combined, and some programs perform substantially better than others in

specific subjects or grade levels. We present the estimates comparing each campus to each of the

twelve “portals” or categories of teacher preparation in separate campus-specific reports.

18

Appendix: Calculating Days Equivalency

Table A.1. Necessary Information for Days Equivalency Calculations

End of Grade Test Standard Deviation Average Yearly Gains

Elementary School Mathematics 9.373 5.867

Elementary School Reading 9.614 5.280

Middle School Mathematics 9.174 2.535

Middle School Reading 8.931 3.039

Days Equivalency Equation:

(((Result value/100) x Standard Deviation)/ (Avg. Yearly Gain))) x 180

Example for Elementary School Mathematics:

Step One

Result value from institutional comparison graph = 2.10

Standard Deviation (9.373) and Average Yearly Gains (5.867) from the table above

Step Two

Insert the result value into the days equivalency equation

(((2.10/100) x 9.373)/(5.867))) x 180 = 6.04 days of learning

Days Equivalency Note:

Days equivalency values can be calculated for elementary and middle grades mathematics and

reading tests because the tests are interval scaled and students have prior test scores for the

subject. We cannot calculate days equivalency in elementary grades science, middle grades

science and algebra, and in all high school tests (EOC) because these assessments are not

vertically equated to a prior test score in the same subject.

publicpolicy.unc.edu

unc teacher quality research: 2013 teacher preparation ... · unc teacher quality research: 2013...

Documents