Similar English LearnerSimilar English Learner Students, Different Results Students, Different Results

Why Do Some Schools Do Why Do Some Schools Do Better?Better?

EdSourceEdSourcewww.edsource.orgwww.edsource.org

Trish Williams, Kenji Hakuta, Ed Haertel, Mike Trish Williams, Kenji Hakuta, Ed Haertel, Mike Kirst, Jesse Levin, Robert Linquanti, Noli Brazil, Kirst, Jesse Levin, Robert Linquanti, Noli Brazil,

Mary Perry, Isabel OregonMary Perry, Isabel Oregon

Castañeda v. PickardCastañeda v. Pickard

Sound theory

ImplementationResults

examine

evaluate

reform

revise

Castañeda v. PickardCastañeda v. Pickard

Sound theory

ImplementationResults

examine

evaluate

reform

revise

Reading Comprehension

0 1 2 3 4 5 6GRADE

3

4

5

6

7

8

9

10

11

12A

ge E

quiv

alen

t Sco

re

English-OnlyBilingual

Program

0 1 2 3 4 5 63

6

9

12

Opportunity CostsOpportunity Costs

AIR / WestEd StudyAIR / WestEd Study

Staff capacity to address EL Staff capacity to address EL needs;needs;

Schoolwide focus;Schoolwide focus; Shared priorities and Shared priorities and

expectations; andexpectations; and Systematic assessment and data-Systematic assessment and data-

driven decision-making.driven decision-making.

The Research QuestionThe Research Question

Why do some schools serving similarly Why do some schools serving similarly challenged student populations vary challenged student populations vary in their performance on the API by in their performance on the API by

as many as 250 points? as many as 250 points?

Extended analysis in 2006-07:Extended analysis in 2006-07: Why do some schools serving similarly Why do some schools serving similarly

challenged EL student populations challenged EL student populations vary in their performance on the EL-vary in their performance on the EL-

API by as many as 250 points?API by as many as 250 points?

2005 English Learner Base 2005 English Learner Base Academic Performance Index (API) Academic Performance Index (API)

vs. School Characteristics Index vs. School Characteristics Index (SCI)(SCI)

300

400

500

600

700

800

900

1000

120 130 140 150 160 170 180 190 200 210

School Characterstics Index (SCI)

20

05

EL

Ba

se

Ac

ad

em

ic P

erf

orm

an

ce

In

de

x (

AP

I)

25th-35th Percentile

Number of Schools with EL API: 485

EL API Scores Max: 811 Median: 663 Min: 508

Schools with most challenges Schools with fewest challenges

Survey DevelopmentSurvey Development Review of effective schools literature Review of effective schools literature

updated for current CA standards-based updated for current CA standards-based policy environmentpolicy environment

Focused on concrete, actionable items, Focused on concrete, actionable items, neutrally phrasedneutrally phrased

Initial surveys reviewed by academics, Initial surveys reviewed by academics, state policymakers, K-12 educatorsstate policymakers, K-12 educators

Piloted study in 11 elementary schools Piloted study in 11 elementary schools

Size of SSDR Sample is Size of SSDR Sample is ImpressiveImpressive

257 elementary schools from 145 districts257 elementary schools from 145 districts 80% of K-5 teachers at each school80% of K-5 teachers at each school 5,500 total completed teacher surveys5,500 total completed teacher surveys All principals from each schoolAll principals from each school About 50% of the schools from the 25-35 About 50% of the schools from the 25-35

bandband Teacher survey was 31 pages with 63 multi-Teacher survey was 31 pages with 63 multi-

part questions, totaling 370 items eachpart questions, totaling 370 items each The principal survey had 442 itemsThe principal survey had 442 items

Geography of SSDR Sample Geography of SSDR Sample SchoolsSchools

AreaArea # of Districts # of Districts # of Schools # of Schools

Northern CaliforniaNorthern California 4747 7474

Central ValleyCentral Valley 3434 5353

Southern CaliforniaSouthern California 6464 129129

SSDR Sample Student SSDR Sample Student DemographicsDemographics

Average Values for:Average Values for:

SampleSampleSchoolsSchools

All CA All CA Elem. Elem. SchoolsSchools

English LearnersEnglish Learners 42%42% 32%32%

Low IncomeLow Income 78%78% 57%57%

African AmericanAfrican American 8%8% 8%8%

AsianAsian 6%6% 8%8%

HispanicHispanic 66%66% 50%50%

WhiteWhite 15%15% 29%29%

Restricting the Sample to Look at EL:

Restricting the Sample to Look at EL:

Survey Research Survey Research DomainsDomains

1.1. Prioritizing student achievementPrioritizing student achievement

2.2. Implementing a coherent, standards-based Implementing a coherent, standards-based instructional programinstructional program

3.3. Using assessment data to improve student Using assessment data to improve student achievement and instructionachievement and instruction

4.4. Ensuring availability of instructional Ensuring availability of instructional resourcesresources

5.5. Enforcing high expectations for student Enforcing high expectations for student behaviorbehavior

6.6. Encouraging teacher collaboration and Encouraging teacher collaboration and providing professional developmentproviding professional development

7.7. Involving and supporting parentsInvolving and supporting parents

FindingsFindings

Which schoolwide and district Which schoolwide and district practices differentiated schools with practices differentiated schools with

higher EL-APIs from schools with higher EL-APIs from schools with lower EL-APIs, serving similar EL lower EL-APIs, serving similar EL

student populations?student populations?

Findings: Use of Findings: Use of Assessment DataAssessment Data

Teachers report receiving CST data in Teachers report receiving CST data in variety of formatsvariety of formats

Principals report directly using data from Principals report directly using data from CST and other tests in many ways related to CST and other tests in many ways related to student learning and instructional student learning and instructional improvementimprovement

Principals report that district evaluates them Principals report that district evaluates them upon and provides support for site level upon and provides support for site level planning related to improving student planning related to improving student achievementachievement

Findings: Ensuring Findings: Ensuring Availability of Instructional Availability of Instructional

ResourcesResources Teacher competence is strong:Teacher competence is strong:

teachers report 5 years experience and teachers report 5 years experience and a standard credentiala standard credential

principal reports staff have strong skills principal reports staff have strong skills and positive attitudesand positive attitudes

Both report school has current texts Both report school has current texts for every student, adequate for every student, adequate materials, maintained facilitymaterials, maintained facility

Findings: Curriculum and Findings: Curriculum and Instructional Program is Instructional Program is

CoherentCoherent Based on CA academic content standards, Based on CA academic content standards,

especially math and ELAespecially math and ELA

Consistent within each grade and aligned Consistent within each grade and aligned from one grade to the nextfrom one grade to the next

District provides the grade by grade District provides the grade by grade curriculum, principal provides the curriculum, principal provides the oversight, and teachers make it happenoversight, and teachers make it happen

Findings: Prioritizing Findings: Prioritizing Student AchievementStudent Achievement

School vision = student learning School vision = student learning outcomesoutcomes

Principal makes expectations clearPrincipal makes expectations clear Teachers embrace responsibilityTeachers embrace responsibility Plans for improvement are well definedPlans for improvement are well defined API/AYP goals are a priorityAPI/AYP goals are a priority Measurable goals for exceeding the Measurable goals for exceeding the

API subgroup growth targetsAPI subgroup growth targets

Effect SizeEffect Size

Domain (#)Domain (#) Predicted Predicted variation in variation in

EL-APIEL-API

Predicted Predicted variation in variation in schoolwide schoolwide

API*API*

Using assessment data (3)Using assessment data (3) 20.420.4 16.716.7

Availability of resources (5)Availability of resources (5) 17.917.9 16.916.9

Coherent standards-based Coherent standards-based instruction (1)instruction (1)

17.317.3 17.617.6

Prioritizing student Prioritizing student achievement (8)achievement (8)

14.714.7 16.316.3

High expectations for High expectations for student behavior (6)student behavior (6)

10.310.3 12.312.3

Involving and supporting Involving and supporting parents (2)parents (2)

10.110.1 9.99.9

Teacher collaboration (4)Teacher collaboration (4) 9.49.4 11.011.0

School Sample (N)School Sample (N) 237 (high EL)237 (high EL) 257 (original 257 (original sample)sample)

Predicted variation in API scores (based on standard deviations) for each domain in

order of importance.

*Based on findings published in Similar Students, Different Results, 2005

Observation: Vertical Observation: Vertical Alignment Alignment

District to ClassroomDistrict to Classroom Strong district leadership, accountability, Strong district leadership, accountability,

and support appear to influence and support appear to influence schoolwide and EL APIschoolwide and EL API

Principal leadership is being redefined to Principal leadership is being redefined to focus on effective management of the focus on effective management of the school improvement processschool improvement process

Teachers get clear signals and support, Teachers get clear signals and support, work together to implement curriculum, work together to implement curriculum, and take responsibility for student and take responsibility for student achievementachievement

Analysis of EL-Analysis of EL-Specific Survey Specific Survey

QuestionsQuestions

FindingsFindings

Higher school EL–API was correlated with Higher school EL–API was correlated with principals who affirmed that in the last principals who affirmed that in the last four years, their school has implemented four years, their school has implemented a new program for EL students.a new program for EL students.

Also more positively correlated with Also more positively correlated with higher EL-API was response by a school’s higher EL-API was response by a school’s teachers that explicit English Language teachers that explicit English Language Development instruction was delivered to Development instruction was delivered to the teacher’s EL students through a pull the teacher’s EL students through a pull out program (e.g. resource teacher). out program (e.g. resource teacher).

FindingsFindings The response by the teachers at a school that The response by the teachers at a school that

their EL students were taught mathematics their EL students were taught mathematics using ESL or immersion techniques (SDAIE) using ESL or immersion techniques (SDAIE) was also more positively correlated with a was also more positively correlated with a higher schoolwide weighted mean scale score higher schoolwide weighted mean scale score for math on the CST.for math on the CST.

In addition, as was also found in our original In addition, as was also found in our original analysis, schools with higher EL–API more analysis, schools with higher EL–API more often had teachers who agreed strongly that often had teachers who agreed strongly that their district addresses the instructional needs their district addresses the instructional needs of English Learner students at their school.of English Learner students at their school.

Positive with AMAOPositive with AMAO

• • School principal responses School principal responses indicating that a new EL program indicating that a new EL program had been implemented at the had been implemented at the school within the past four years school within the past four years was positively correlated with was positively correlated with AMAO.AMAO.

Negative with AMAONegative with AMAO

Negatively and significantly correlated Negatively and significantly correlated with AMAO 1 and 2 were strong school-with AMAO 1 and 2 were strong school-level teacher responses that explicit ELD level teacher responses that explicit ELD instruction was delivered to EL students instruction was delivered to EL students by the classroom teacher herself, or by by the classroom teacher herself, or by ELD level through teacher teaming.ELD level through teacher teaming.

Also negatively correlated were teacher Also negatively correlated were teacher reports that they receive CELDT reports that they receive CELDT assessment data and that, besides CELDT, assessment data and that, besides CELDT, teachers use no other type of ELD teachers use no other type of ELD assessment for their EL students.assessment for their EL students.

An An InterpretationInterpretationWe interpret these results to complement We interpret these results to complement

those discussed elsewhere in this report. those discussed elsewhere in this report. Namely, having in place an explicit program Namely, having in place an explicit program

of instruction for ELs is fundamentally of instruction for ELs is fundamentally important to their English language important to their English language

development needs, and enhancing EL development needs, and enhancing EL students’ access to and comprehension of students’ access to and comprehension of core academic subjects such as math may core academic subjects such as math may

also yield benefits to their academic also yield benefits to their academic language development in English.language development in English.

Some SurprisesSome Surprises We were surprised that the We were surprised that the

presence in a school of more presence in a school of more teachers with CLAD/BLCAD teachers with CLAD/BLCAD certification did not have a certification did not have a relationship, in our analysis, to relationship, in our analysis, to schools’ EL–API scores.schools’ EL–API scores.

Some SurprisesSome Surprises Another surprise was that the Another surprise was that the

number of minutes teachers number of minutes teachers reported devoting to explicit reported devoting to explicit ELD instruction at the school ELD instruction at the school did not correlate with EL–API.did not correlate with EL–API.

An An InterpretationInterpretation

Quality trumps Quality trumps quantity.quantity.

Bigger Picture: Castañeda Bigger Picture: Castañeda v. Pickardv. Pickard

Sound theory

ImplementationResults

examine

evaluate

reform

revise

The Daunting The Daunting ChallengeChallenge

The rich get richer.The rich get richer.

THANK YOUTHANK YOU

For more information:For more information:

hakuta@stanford.eduhakuta@stanford.eduwww.edsource.org

EdSourceEdSource520 San Antonio Road, Ste. 200520 San Antonio Road, Ste. 200Mountain View, CA 94040-1217Mountain View, CA 94040-1217

(650) 917-9481 (650) 917-9481