+

Questioning What We Know: Evaluating Predictability of Candidate Data

Megan Parker Peters, PhDDeborah Boyd, EdDJonathan Harrison, MBA

Lipscomb University

+Project Goals

nContinuous Improvement: Goal for all of our candidates

nDetermine how state measures educator effectiveness

nDetermine what factors predict success on measures of educator effectiveness

nProgram reflections and data-based changes

+Continuous Improvement

nAll of us attempt to improve our programs from year to year.

nUnit of measure for program success?

nOur completers!n How are they impacting student lives? Making positive

changes?

+Past Findings

n It’s difficult to identify an effective teacher (Strong et al., 2011)

n Grit & perseverance linked to effectiveness (Robertson-Kraft & Duckworth, 2014)

n Higher standardized test scores (e.g., ACT scores; Clotfelter, Ladd, & Vigdor, 2007; Ferguson & Ladd, 1996)

n Higher cognitive ability are more likely to transfer to higher performing schools or leave the classroom entirely (Guarino, Santibañez, Daley, & Brewer, 2004; Lankford et al., 2002)

n Non-cognitive personality factors moderate link to effectiveness (Rockoff et al., 2008)

+Measuring Educator Effectiveness

n Tennessee: TVAAS (Tennessee Value Added Assessment System)

n “Measures the impact schools and teachers have on their academic progress” (tn.gov)

n TVAAS measures growth rather than student proficiency, allowing students of all ability levels the opportunity to show strong scores on teachers’ assessments.

n We receive a yearly report of TVAAS scores for our completers.

+How is TVAAS Calculated?

nDeveloped to measure a teacher’s impact on student learning.

nTVAAS estimates student academic growth based on previous student achievement scores on standardized assessments. n Value-added scores count for 35 percent of the final

evaluation score. For teachers who do not receive an individual growth score (teachers in “non-tested grades and subjects”), the school’s value-added score counts as 25 percent of the overall evaluation score.

+How is TVAAS Calculated?

nTeachers’ TVAAS scores are determined by comparing students’ predicted growth to their actual growth over the course of the school year.

nGrowth = current achievement/current results compared to all prior achievement/prior results, with achievement being measured by a quality assessment such as the TCAP.

+TVAAS Examples

+TVAAS Data: What does it mean?

n TVAAS reports effectiveness scores with a range of 1 to 5

n What makes the difference in a 1 and a 5?

n Are there differences in these students that predict TVAAS scores? Are there program factors, candidate performance indicators, or rubric indicators that predict TVAAS scores?

n Knowing these correlations could help us better prepare our candidates to be effective educators.

+TVAAS: All Subject Areas

+Completer TVAAS Effectiveness

Moments

N 178 Sum Weights 178

Mean 3.51123596 Sum Observations 625

Std Deviation 1.7921321 Variance 3.21173745

Skewness -0.5307667 Kurtosis -1.5825141

Uncorrected SS 2763 Corrected SS 568.477528

Coeff Variation 51.0399221 Std Error Mean 0.13432598

+Graduating GPA: TVAAS Score

3

3.25

3.5

3.75

4

1 2 3 4 5

+TVAAS: GPA

+ Praxis II: PLT & CIA

+Type of Student: TFA

Pearson Correlation Coefficients, N = 178 Prob > |r| under H0: Rho=0

GrUgrad Effectiv TFA

GrUgrad

GrUgrad

1.00000

-0.10918

0.1469

-0.52327

<.0001

Effectiv

Effectiv

-0.10918

0.1469

1.00000

0.24051

0.0012

TFA

TFA

-0.52327

<.0001

0.24051

0.0012

1.00000

+TFA: TVAAS Effectiveness

+Graduate GPA: TVAAS Score

Graduate GPA: TVAAS Effectiveness Score

Dependent Variable: Effectiv Effectiv

Source DF Sum of Squares Mean Square F Value Pr > F

Model 48 195.3061639 4.0688784 1.41 0.0676

Error 129 373.1713642 2.8928013

Corrected Total 177 568.4775281

R-Square Coeff Var Root MSE Effectiv Mean

0.343560 48.43946 1.700824 3.511236

Source DF Anova SS Mean Square F Value Pr > F

GRGPA 48 195.3061639 4.0688784 1.41 0.0676

+Entrance Tests

Simple Statistics

Variable N Mean Std Dev Sum Minimum Maximum Label

tvaas 119 3.12605 1.51016 372.00000 1.00000 5.00000 tvaas

Combined 119 3.71042 0.32812 441.54000 2.16000 4.00000 Combined

GRECOMP 20 691.10000 446.90172 13822 291.00000 1370 GRECOMP

ACT 45 23.44444 6.03985 1055 0 32.00000 ACT

SAT 21 1055 115.69871 22150 860.00000 1320 SAT

+Entrance Tests

Pearson Correlation Coefficients Prob > |r| under H0: Rho=0

Number of Observations

tvaas Combined GRECOMP ACT SAT

tvaas

tvaas

1.00000

119

-0.02730

0.7682

119

0.01034

0.9655

20

-0.04332

0.7775

45

-0.35866

0.1104

21

Combined

Combined

-0.02730

0.7682

119

1.00000

119

0.33085

0.1542

20

0.18852

0.2149

45

0.65990

0.0011

21

GRECOMP

GRECOMP

0.01034

0.9655

20

0.33085

0.1542

20

1.00000

20

1.00000

.

2

1.00000

.

2

ACT

ACT

-0.04332

0.7775

45

0.18852

0.2149

45

1.00000

.

2

1.00000

45

-0.21080

0.3590

21

SAT

SAT

-0.35866

0.1104

21

0.65990

0.0011

21

1.00000

.

2

-0.21080

0.3590

21

1.00000

21

+Candidate Dispositions: TVAAS

Pearson Correlation Coefficients, N = 17 Prob > |r| under H0: Rho=0

TVAAS CommunProgress Reflective ConfEnthus PD

TVAAS

TVAAS

1.00000

0.49755

0.0421

0.40671

0.1052

0.48606

0.0479

0.41026

0.1019

CommunProgress

CommunProgress

0.49755

0.0421

1.00000

0.67763

0.0028

0.57103

0.0167

0.21660

0.4037

Reflective

Reflective

0.40671

0.1052

0.67763

0.0028

1.00000

0.85718

<.0001

0.42165

0.0918

ConfEnthus

ConfEnthus

0.48606

0.0479

0.57103

0.0167

0.85718

<.0001

1.00000

0.46622

0.0592

PD

PD

0.41026

0.1019

0.21660

0.4037

0.42165

0.0918

0.46622

0.0592

1.00000

+Dispositions

n J. Communicates student progress to students, their parents (when directed by the mentor teacher to do so), and appropriate others. (1.b)

n K. Reflects on teaching practice by evaluating continually the effects of instruction. (1.c & 1.i)

n C. Projects confidence, enthusiasm and initiative (2.d. & 2.e)

n G. Engages in professional development (2.b., 2.g)

+Limitations

nA large percentage of graduates are not included in the TVAAS report.

nTeachers’ TVAAS scores may be linked to a particular institution because they graduated from that institution, but it may not be where they received their training to be a teacher.

nTotal N (especially for Dispositions data)

+Future Directions

n Importance of application documents: ACT, SAT, GRE?

n Importance of in-major tests (Praxis)?

n Importance of GPA?

n Consideration of measurement: TVAAS

n Consider changes in focus; 2 Disposition areas moderately correlated with effectivenessn Apply greater focus in developing and targeting these areas?