evaluating the nyc core knowledge early literacy pilot: year 3 report
DESCRIPTION
Evaluating the NYC Core Knowledge Early Literacy Pilot: Year 3 Report. Research and Policy Support Group February 2012. 1. Executive Summary – Quantitative Analysis. - PowerPoint PPT PresentationTRANSCRIPT
11
Evaluating the NYC Core KnowledgeEarly Literacy Pilot: Year 3 Report
Research and Policy Support GroupFebruary 2012
2
Executive Summary – Quantitative AnalysisCore Knowledge Reading (CKR) students had significantly greater gains in
Year 3 than comparison school students on nearly all measures. Overall Achievement
Spring scores for 2nd grade CKR students were greater than that of comparison students on all tests.*
High vs. Low Performers The CKR intervention had an impact for all students, regardless of their incoming fall
reading scores, but the effects were strongest for students with lower incoming scores.
CKR Program Exposure Students new to the CKR program this year (who also had lower incoming fall scores)
displayed greater increases on the WJ Brief Reading than those who had been in the program longer. CKR students who had been in the program for multiple years displayed the highest spring scores.
Post- Pilot CKR Participation CKR Kindergarten and 1st grade classes implementing the CKR program this year
scored significantly higher than comparison schools in the spring on TerraNova Reading.
*All differences were statistically significant except TerraNova Reading
3
Surveys The teacher and administrator surveys indicate high levels of satisfaction with the
CKR Program and a preference for the program over other 2nd grade reading curricula.
Most teachers found that students were very engaged with the content of the curriculum and that it was successful at sparking enthusiasm for reading.
Site Visits Fidelity to the program was quite high. The main difficulty noted was in allotting an
uninterrupted 60 minutes for program implementation.
Most of the observed classrooms fully utilized the materials provided and students were largely attentive during lessons, participated in discussions, and were curious about the content of the materials.
In surveys and interviews, teachers noted difficulty with differentiation of instruction using the provided materials for students with special learning needs, such as English Language Learners and Students with Disabilities.
Executive Summary – Surveys & Site Visits
4
Methodology: A multi-method, longitudinal research designPROGRAM YEAR 3: 2ND GRADE
Assessments of 2nd Grade Students (at 10 CKR schools & 10 comparison schools)*
• Pre- and post-test of literacy skills • Tests of science and social studies skills at the end of the year• Additional test of content-specific literacy skills (the Core Knowledge Assessment for
CKR students only) at the end of the year
Teacher and Administrator Surveys (at 10 CKR schools):• Assesses satisfaction with and impact of CKR
Case studies (at 4 CKR schools):• Classroom observations, administrator & teacher interviews
Hypothesis: Students taught with the Core Knowledge Reading (CKR) Program will gain reading competencies and content knowledge (science and social studies skills) at a faster rate than their peers.
Focus of the Evaluation
• In Year 1 and 2, Reading First schools were used as a subgroup in the evaluation. However, these schools were no longer Reading First in Year 3. ** Kindergartners and 1st graders in CKR pilot schools no longer received full CKR program supports and two schools did not test additional students.
ADDITIONAL GRADE LEVELSAssessments of Kindergarten and 1st Grade Students** (at 9 CKR schools & 9 comparison schools)
• Tests of literacy, science and social studies skills at end of the third year
12.6
70.1
82.0
39.8
6.89.5
78.2
88.0
39.1
5.7
ELL Free/ Reduced Lunch
Black/ Hispanic Level 3/4 Students ELA
School Size (in Thousands)
Demographics: While the comparison schools for the evaluation were initially picked for their similarities to the pilot schools, analyses control for demographics to account for any demographic differences that have emerged over the past two years.
5
CKR Students (N = 700)
Comparison Students
Analyses in all slides exclude students with disabilities. Data from 08-09 was used to select comparison schools (data presented here are from the 2010-11 school year). * Difference between CKR and Control groups statistically significant. ** ELA percents based on total number of students who took the exams in the schools in this evaluation.
Percent of 2nd Grade Students
(N = 357)*
* N = the number of students for whom both fall and spring data were available for any WJ test.
A random sample of half of the students in the comparison schools were selected for testing. All students in CKR schools were tested.
*** *
Evaluation of
Achievement Gains
6
7
Quantitative Results Overview:CKR students displayed higher scores and gains on nearly all tests.
Woodcock-Johnson III TerraNova
Test W-J Brief Reading
W-J Word Attack
(Decoding)
W-J Spelling of Sounds
(Written Spelling)
TerraNova Reading
(Oral Reading Comprehension,
Vocabulary, Basic Reading,
Decoding)
TerraNova Social Studies
TerraNova Science
W-J Letter Word
Identification (Basic Reading
Skills)
W-J Passage Comprehension
(Oral Reading Comprehension)
Spring Scores at
CKR Schools
Compared to Comparison
Schools
CKR
Significantly Greater
Spring Scores
CKR Significantly
Greater Spring Scores
CKR
Significantly Greater
Spring Scores
CKR
Significantly Greater Spring
Scores
XNo Significant
Difference
(But CK students had
higher scores)
CKR
Significantly Greater
Spring Scores
CKR
Significantly Greater
Spring Scores
Change in Fall to Spring
Scores at CKR
Schools Compared toComparison
Schools
*CKR
Significantly Greater Fall to
Spring Change
CKR Greater
Significantly Fall to Spring Change
CKR Greater Significantly Fall to Spring
Change
CKR Greater Significantly Fall to Spring
Change
Not Administered in
Fall
Not Administered
in Fall
Not Administered
in Fall
Note: All analyses control for student demographic characteristics.
*Significant only at one-tailed level. Two-tailed p=.067
Quantitative Results Overview: CKR students’ literacy gains were more than double the gains of students at demographically similar comparison schools.
8
Average Fall-Spring Gainin Scale Score Points
Woodcock-Johnson (Brief Reading Test)CKR Students
Displayed ASignificantly Greater
Change FromFall to Spring
Than Comparison School Students
(N = 683) (N = 352)
Fall Spring
97.4
100.7
101.9
104.5104.4
105.2
1 Year/2nd Grade Only (N=86)2 Years/1st and 2nd Grade (N=162)3 Years/All Grades (N=435)
9
Impact of time in program: CKR had the largest impact among students who were new to the program (who had lower starting scores)However, the highest ultimate (spring) scores were found among those who had been in the program the longest.
Average Fall and Spring Scale Score PointsBy Years in CKR Program in Year 3 Woodcock-Johnson (Brief Reading)
+ 3.2 pts
+0.8 pts
+2.6 pts
10
Impact of time in program: The longer second grade students had been in the CKR program, the higher their TerraNova Reading scores were.
Average Spring Scale Score PointsBy Years in CKR ProgramTerraNova (Reading Test)
3 Years/All Grades (N=447) 2 Years/1st and 2nd Grade (N=169)
1 Year/2nd Grade Only (N=129)
540.0
550.0
560.0
570.0
580.0
590.0
600.0
610.0
594
582577
50th percentileNationwide (606)
Scores are not adjusted for demographics.
10th percentileNationwide (553)
11
Effects of CKR Following Participation in the Pilot: Kindergarten students in schools implementing CKR showed higher TerraNova Reading spring scores than students in comparison schools in both the year they participated in the pilot and the current year.
Spring ScoresTerraNova Reading Test
Scores are not adjusted for demographics.
CKR Kindergartners in 2008-2009Kindergarteners in Year 1
CKR Kindergartners in 2010-2011Kindergarteners in Year 3
CKR Schools (n=693)
Comparison Schools (n=307)
510.2
501.9
CKR Schools (n=547)
Comparison Schools (n=215)
512.2
501.7
12Scores are not adjusted for demographics.
CKR Schools (n=698)
Comparison Schools (n=391)
560.6
555.0
CKR Schools (n=600)
Comparison Schools (n=223)
557.6
541.0
Spring ScoresTerraNova Reading Test
CKR First Graders in 2009-2010First Graders in Year 2
CKR First Graders in 2010-2011First Graders in Year 3
Effects of CKR Following Participation in the Pilot: First grade students in schools implementing CKR showed higher TerraNova Reading spring scores than students in comparison schools in both the year they participated in the pilot and the current year.
13
Administrator and Teacher Surveys
13
1414
Administrators reported satisfaction with the CKR program.
No
7Yes
2
Will your 2nd grade classrooms be using the CK
Reading Curriculum next year?
(n = 9)
Would you recommend the CK Reading
curriculum to other school administrators
you know?
(n = 9)
1No
Not Sure
7Yes
Administrators’ overall
satisfaction with CK Reading
(n = 9) *
The writing connection is missing. We don't feel there is enough aligned to the new CCSS and although the children are able to get the Phonics, and Phonemic Awareness down; the comprehension and independent thinking skills are lacking…
VerySatisfied
SomewhatSatisfied
Neutral
We feel that it is not preparing them for the state standards for reading and Writing.
Each school is unique. I have a significant amount of ELL’s and SWD's other schools may not. As I have said before, children gain in Phonics and Phonemic Awareness. They need more basic vocabulary development and comprehension skills and making a writing connection.
1
* Administrators had five options for answering this question: Very Satisfied; Somewhat Satisfied; Neutral; Somewhat Dissatisfied; Very Dissatisfied .
33.3%38.5%
38.9%38.5%
1515
Teachers’ overall satisfaction with CK
Reading(n = 18)a
Percent of Respondents
77.0%72.2%
Much BetterSomewhat Better
Teachers’ overall opinion of CK Reading compared with other
reading programs(n =13)b
Very Satisfied
Somewhat Satisfied
Other Answers: Question a Neutral=2 (11.1%); Somewhat dissatisfied=2 (11.1%); Very Dissatisfied=1 (5.6%); Question b About the Same=0; Somewhat Worse=2 (15.4%); Much Worse=1 (7.7%). In Year 2, Overall satisfaction was 82.7% very/somewhat satisfied, while overall opinion of CK relative to other programs was 50% much/somewhat better
Teachers’ Views:“The students enjoyed learning new concepts
which sparked their reading and the need to know. Although their skills were limited they were always ready for a new concept or a new presentation of skills. Because this was a new program for me as a teacher it sparked my interests as well.”
“Over more than twenty years of teaching I have experienced a number of reading programs. Phonics, spelling, vocabulary and writing and comprehension skills were integrated throughout and the sequence made sense. Enrichment suggestions and remediation suggestions were presented throughout the materials.”
Most teachers were satisfied with CKR and reported that CKR was better than other reading programs.
Note: Calculations were based only on responses that selected an answer other than N/A, which resulted in a lower overall N count.
62.5% 50.0%50.0%
25.0%
25.0% 25.0%
87.5%
75.0% 75.0%
1616
Goals of lessons are clear
(n = 16)
Neutral: N=2 (12.5%)Disagree: N=1 (6.3%)Strong y Disagree: N=2 (12.5%)
I have enough time to complete
daily lessons(n = 16)
Neutral: N=1 (6.3%)Disagree: N=0 (0.0%)Strong y Disagree: N=2 (12.5%)
Students find activities engaging
(n = 16)
Neutral: N=2 (12.5%)Disagree: N=1 (6.3%)Strongly Disagree: N=2 (12.5%)
Somewhat Agree
Strongly Agree
Goals of lessons are
clear(n = 16)
Neutral: N=1 (6.3%)Disagree: N=0 (0.0%)Strongly Disagree: N=1 (6.3%)
I have enough time to complete
daily lessons(n = 16)
Neutral: N=1 (6.3%)Disagree: N=2 (12.5%)Strong y Disagree: N=1 (6.3%)
Students find activities engaging
(n = 16)
Neutral: N=1 (6.3%)Disagree: N=2 (12.5%)Strongly Disagree: N=1 (6.3%)
Skills Strand Listening and Learning Strand
Percent of Respondents
Teachers indicated that both CKR strands had clear goals, were engaging, and that they had sufficient time to complete the lessons. In Year 2, teachers had rated the Skills Strands higher on these measures.*
* In Year 2, those that rated Strongly Agree or Somewhat Agree: Goals of lessons are clear =96.6%;Students find activities engaging=78.6%; I have enough time to complete daily lessons= 72.4%. Note: Calculations were based only on responses that selected an answer other than N/A, which resulted in a lower overall N count.
68.8%50.0%
62.5%
12.5%
18.8%6.3%
81.3%
68.8% 68.8%
23.1%30.8% 30.8%
45.5%
25.0%
61.5%38.5% 38.5% 9.1%
25.0%
84.6%
69.2% 69.2%
54.5%50.0%
Somewhat Better Much Better
About the Same: N=1 (7.7%)Somewhat Worse: N=0Much Worse: N=1 (7.7%)
About the Same: N=1 (7.7%)Somewhat Worse: N=1 (7.7%)Much Worse: N=2 (15.4%)
About the Same: N=1 (7.7%)Somewhat Worse: N=2 (15.4%)Much Worse: N=1 (7.7%)
About the Same: N=2 (18.2%)Somewhat Worse: N=2 (18.2%)Much Worse: N=1 (9.1%)
About the Same: N=3 (25%)Somewhat Worse: N=2 (16.7%)Much Worse: N=1 (8.3%)
Compared to other programs they taught, most teachers found CKR much or somewhat better at teaching content/background knowledge.
Teachers’ Views:“I like the content reading, the
students really enjoyed learning about different areas and different people. I always got the question-what are we going to learn next?
17Note: Calculations were based only on responses that selected an answer other than N/A, which resulted in a lower overall N count.
Teachers believed the CKR program was better than other programs they have previously used at accommodating different learning needs and teaching vocabulary.
Teaching vocabulary(N=13)
Accommodations for special education
students(N=9)
Accommodations for English language
learners(N=11)
Teaching writing skills(N=13)
23.1%
55.6% 54.5%
23.1%
53.8%
11.1% 9.1%
15.4%
76.9%
66.7% 63.6%
38.5%
Somewhat Better Much BetterTeachers’ Views:“The writing component was weak and
did not really teach students how to write a multi paragraph writing. Most of your writing pieces were quick and easy. They didn't really show the writing process or teach students how to write extended responses.”
18
About the Same: N=0Somewhat Worse: N=2 (15.4%)Much Worse: N=1 (7.7%)
About the Same: N=0Somewhat Worse: N=2 (22.2%)Much Worse: N=1 (11.1%)
About the Same: N=2 (18.2%)Somewhat Worse: N=1 (9.1%)Much Worse: N=1 (9.1%)
About the Same: N=3 (23.1%)Somewhat Worse: N=2 (15.4%)Much Worse: N=3 (23.1%)
Note: Calculations were based only on responses that selected an answer other than N/A, which resulted in a lower overall N count.
19
Site Visits and
Interviews
19
20
Measuring fidelity to the CKR curriculum
• Examining implementation fidelity allows us to better determine:1) whether achievement gains can be attributed to the CKR program2) which components of the CKR program teachers are struggling to
implement and those they are implementing successfully
• We took several different approaches to measuring fidelity in the CKR Pilot schools:
Site visits with classroom observations• Site visits were conducted at 4 randomly selected Pilot schools• 11 classrooms were observed
• Interviews with administrators and teachers• Interviews were conducted with 16 teachers and 3 administrators
• Additional teacher and administrator survey questions
20
Examples from Checklist
• Check to see that CKR visuals are posted and that CKR materials are present and being used effectively
• Verify that the daily schedule allows for both the Skills and the Listening and Learning Strands to last a full 60 minutes each
• Whether or not students are familiar with language introduced in the domain
• Teacher engaging all students in exercises, practice, class discussions, and games
• Whether or not students are attentive to the teacher during instruction
• Whether or not students are eager to participate in the daily activities of the program
• Whether or not students are generally on task and complete assignments in a timely fashion
• Evidence of differentiated instruction and assessment techniques
21
During the site visits, we used a classroom observation protocol developed from the Core Knowledge Reading Pilot Observation Form
These components were taken from the Core Knowledge Reading Pilot
Observation Form
These components were added as additional measures of student engagement
21
22
10
11
9 8
1
1 2 3
10
Students are attentive to
their teacher during
instruction
Students respond
appropriately to their teacher’s
directions, questions, and assignments
Teachers have Skills Strand
materials posted and used Word Walls effectively
Small group and/or
individual practice time
Full 60 minutes is allotted for
both the Listening and Learning and
Skills Strands*
Number of Classrooms (N=11)
Observed During Visit
Not Observed During Visit
Nearly all classrooms demonstrated high fidelity to the CKR program. In particular, we observed high levels of student engagement in lessons.
* 60 minutes for the Strands typically not allotted uninterrupted.
23
Overall, teachers and administrators reported high fidelity to the program. One area where they reported struggling to implement the program was in finding time for daily small group instruction.
Fidelity Survey Questions Spring 2011 Teacher Responses
Please indicate your overall level of satisfaction with the CK Reading Curriculum so far. 72.2% Very Satisfied or Somewhat Satisfied
Typically, I find I have enough time to complete the daily Skills Strand Lesson 81.3% Strongly or Somewhat Agreed
Typically, I find I have enough time to complete the daily Listening and Learning lesson 75% Strongly or Somewhat Agreed
Core Knowledge Reading engages students and sparks enthusiasm for reading 69.2% Strongly or Somewhat Agreed
The Skills Strand materials are developmentally appropriate 75% Strongly or Somewhat Agreed
The Listening and Learning materials are developmentally appropriate 62.5% Strongly or Somewhat Agreed
The CKR curriculum accommodates different learning needs 54.5% Strongly or Somewhat Agreed
24
• Comparison of 3rd grade 2012 NY State ELA test scores among:• Students who participated in the CKR Pilot as second graders in the
2010-2011 vs. students who were assessed at comparison schools in that year.
• Students who had previously participated in the CKR Pilot as second graders (one year in the pilot) versus those who had previously participated in the CKR Pilot for multiple years.
• Exploration of associations between 3rd grade scores on the 2012 NYS ELA test and previous performance on tests of reading comprehension (WJIII, TerraNova) while participating in the pilot as second graders.
NYCDOE/RPSG’s Year 4 EvaluationTrack student achievement through 3rd grade on the NYS English
Language Arts (ELA) test
Next Steps: Year 4