1  

Literacy Design Collaborative i3 Validation Grant Proposal: Implementing College and Career Readiness Standards

A. SIGNIFICANCE (20 POINTS)

1. National Need and Priority (Absolute Priority 2, Competitive Priorities 2 and 3)

Overview: The Literacy Design Collaborative organization (LDC.org) seeks i3 Validation

funding that would meet:

• Absolute Priority 2: implementing internationally benchmarked, college- and career-ready elementary and secondary academic standards

• Competitive Priority 2: projects that enable the broad adoption of effective practices. This competitive preference priority rewards applicants that will implement systematic methods for identifying and supporting the expansion of these practices

• Competitive Priority 3: entities that have not previously received an i3 grant to apply

LDC meets these priorities with a set of resources and guided online tools that lead teachers

through a learning experience that enables educators to “operationalize” the instructional shifts

of the Common Core while creating Common Core-aligned curriculum. This curriculum design,

implementation, and reflection process is both systematic and capable of rapid expansion and

impact on American education. However, different from simply a curriculum solution, LDC’s

instructional process re-unifies typically separate CCSS and teacher effectiveness PD work

streams. Indeed, years of research confirm LDC’s positive impact on teacher and student skill

and engagement. By granting an i3 validation grant, the U.S. Department of Education would

support the codification, broad adoption, and expansion of these effective practices, improving

the outcomes for high needs and, indeed, all students.

Why LDC? National Need: Literacy Rates and CCSS Implementation. Overwhelming data

demonstrates that American students lag in essential literacy achievement. For example,

U.S. performance in reading and literacy on the 2012 international PISA exams reflects:

• Only 8 percent of U.S. 15-year-old students scored at proficiency level 5 or above, the top performing range of students.

  2  

• Seventeen percent of 15-year-olds in the U.S. performed below level 2, an OECD baseline of proficiency, lower than 33 education systems (out of 65 participating nations).

The Common Core State Standards (CCSS) offer a unique opportunity for addressing our

national literacy challenge. But standards alone are not sufficient. Research suggests that merely

raising standards will not have a significant impact on student performance. As professor

Richard Elmore articulated a decade ago:

There are only three ways to improve student learning at scale: You can raise the level of the content that students are taught [CCSS Standards]. You can increase the skill and knowledge that teachers bring to the teaching of that content. And you can increase the level of students’ active learning of the content. That’s it.

* * *

If you change any single element of the instructional core, you have to change the other two. If you raise the level of content without changing the level of knowledge and skill that teachers bring to the content, you get what we see with considerable frequency in American classrooms: low-level teaching of high-level content.1

This theory has played out in the reality of CCSS implementation. CCSS did raise standards.2

However, too frequently, the other two strands of Elmore’s construct have not been adequately

addressed – specifically, the need to support teachers and address student engagement. In short,

the problem has been in CCSS’s implementation.

In many places – such as New York state – the perceived top-down and incoherent

implementation process, as well as concerns about the overemphasis on testing, have created a

highly politicized environment around the CCSS.3 While there is a fraction of people on both

sides of the political aisle that rejects the CCSS outright, there are many administrators and

teachers (as well as parents and students) who support the Common Core as standards but need

                                                                                                               1 E.A. City, R.F. Elmore, S.E. Fiarman, L. Title (2009). Instructional Rounds in Education: A Network Approach to Improving Teaching and Learning. Cambridge, MA: Harvard University Press. 2 “What about content and performance standards? Standards only operate by influencing the level of the content that’s actually being taught; their effect in actual classrooms depends on whether there are materials that reflect the standards, whether teachers know how to teach what the materials and standards require, and whether students find the work that they are being asked to do worthwhile and engaging.” Ibid. 3 See, e.g., Simon, S. “New York fails Common Core tests,” Politico. August 7, 2013. www.politico.com

  3  

to be better engaged and provided opportunities and supports to both learn and share their

expertise. Many are asking for alternatives to the curricula that have been released so far –

curricula that are often perceived as prescriptive and not tailored to their students’ specific needs

and, therefore, alienating to many teachers; curricula that do little to engage student or teacher

thinking; or curricula that have little direct research on its effectiveness or impact on student

learning.4 Teachers and administrators are also asking for time and vastly increased support to

understand, reflect on, and improve how they use the CCSS and aligned curricula they select.

LDC Response to the Challenge

The Literacy Design Collaborative (LDC) addresses all three components of Elmore’s

Instructional Core – high standards, teacher efficacy, and student engagement. In large part this

derives from its genesis as a practice-based, educator-led approach to implementing college- and

career-readiness standards that holds at its core student engagement and growth as foundational.

LDC is not a publisher’s curriculum; it is not a “teacher-proof” black box. Instead, LDC works

because the tools and technical assistance engage teachers in a reflective and collaborative

process; we grow the expertise of teachers in understanding and using the CCSS in their

classrooms.5 In effect, LDC reconnects the planning and delivery of Common Core classroom

instruction to teacher learning and efficacy.

To do so, LDC builds practitioner expertise in the design and implementation of rigorous,

CCSS-aligned: (1) “writing in response to reading” assignments authentic to their subject area

(“teaching tasks”); (2) instructional plans that engage students in developing the skills necessary

to complete the teaching task over a two- to four-week period (an “LDC module”); (3) scoring of

student work through rubrics, scoring process, and nationally calibrated benchmarked examples

                                                                                                               4 http://www.npr.org/blogs/ed/2014/06/03/318228023/the-common-core-curriculum-void 5 Levin, S. and Poglinco. S. “Scale-Up and Sustainability Study of the LDC and MDC Initiatives” Philadelphia: Research for Action. September 2013.

  4  

by Measured Progress; and (4) a calibrated jurying process and protocol created by professor

Ray Pecheone and his team at the Stanford Center for Assessment, Learning and Equity

(SCALE) to ensure quality teaching task and module development. LDC’s focus on building

teacher capacity and engaging students derives from its genesis as a practice-based, educator-led

approach to implementing college- and career-readiness standards. With extensive investment by

the Bill and Melinda Gates Foundation (BMGF) – including significant third-party research

shaping its evolution over the past four years – LDC has created the tools and technical services

that build teacher capacity: a tested CCSS-aligned literacy framework; online interactive tools,

exemplars, and collaborative work spaces (called LDC CoreTools); technical assistance; and on-

site and online community opportunities for feedback and collaboration.

The Validation and Expansion of LDC

With the i3 grant, LDC will further refine and codify our tools and resources to support

teacher professional learning of CCSS through the LDC framework, as well as build out the

system to ensure interactive online tools fully: (1) assist teachers and school systems in designing

literacy-rich courses with LDC modules at the foundation and LDC mini-tasks used between

modules to continue to diagnose or build specific literacy skills as needed, and (2) assist

educators in using LDC modules to align literacy teaching and learning K-12 (vertically) and

across courses (horizontally).6 Through a  host  of  LDC  tools  and  resources  that  support  the  

teacher’s  learning  progression,  LDC  ensures  the  adoption  of  effective  literacy  standards,  

including  the  further  validation,  expansion,  and  scale  of:  

• LDC library of nationally juried CCSS assignments and instructional mini-tasks. Like

students, teachers profit by knowing what proficient and exemplary look like. LDC CoreTools

                                                                                                               6 The year-long planning can assume many typical forms: scope and sequence, curriculum maps, UbD unit plans, etc. For a discussion of LDC’s compatibility with typical curriculum planning systems like UbD, see McTighe, J. “LDC and UbD- Complementary Frameworks,” January 2013. Link.

  5  

includes a library of nearly 60 and growing (classroom teacher-created) exemplary modules

across grades and subject areas that have been juried and CCSS-approved by SCALE at Stanford

University.7 Likewise, the LDC “Mini-[literacy] Task Library” includes over 100 teacher-created

mini-tasks tested and refined through teacher use and vetted by literacy expert David Pearson’s

UC-Berkeley team. The LDC tools allow teachers to adopt and adapt both the modules and the

mini-tasks to their classrooms. LDC continues to grow these libraries, adding approximately 20

exemplary teacher-authored modules each quarter through a SCALE-led national jurying

competition, 40+ “Good-to-Go” modules each quarter that can be used in classrooms

immediately,8 and 3-5 teacher-authored mini-literacy tasks per week.

• LDC training and coaching processes and tools for teacher learning, collaboration,

reflection, and improvement of literacy practices. LDC supports teachers in CCSS

implementation through both dynamic online resources and LDC-partner, in-person job-

embedded training. The LDC CoreTools’ online experience provides two sets of tools and

resources to support teacher learning: (1) Online professional learning resources for individual

teachers, including a basic CCSS curriculum design experience with rudimentary prompts for

teacher reflection and links to helpful resources such as the LDC Guidebook, videos, module

exemplars, workstations, webinars, peer collaboration space, and opportunities for feedback.

This is the first step in providing individual teachers a beginning experience with LDC as a tool

for CCSS implementation and a connection to a community of likeminded educators. (2) A set of

coaching products for LDC professional development (PD) partners and district/school leads to

                                                                                                               7 Stanford Center for Assessment, Learning and Equity (SCALE) is a key advisor to one of the two CCSS assessment consortia, Smarter Balance Assessment Consortium. 8 A parallel “Good-to-Go” Module Library (against the SCALE CCSS-alignment and rigor rubric) will be launched in July 2014 with another nearly 100 modules that, while not exemplary, are strong, aligned to Common Core, and capable of teacher adoption, modification, and immediate use in the classroom.

  6  

use when providing technical assistance to teachers.9 By the end of the i3 grant, LDC seeks to

improve upon these tools, moving them from providing a fairly static experience to a dynamic,

interactive experience for users by: (1) validating and codifying a set of professional

development standards, processes, and tools from the early community efforts products to ensure

quality LDC technical assistance is scalable; (2) expanding the current online resources by

including technical assistance tools for course design and vertical/horizontal alignment and

mapping; (3) continuing to build out LDC community-sourced “best in class” static coaching

resources that can be embedded in more online dynamic teacher learning experiences; and (4)

transforming much of the static LDC coaching support into relevant and accessible asynchronous

opportunities to support teacher LDC viral adoption nationwide.10 This PD support includes

leadership orientation and induction for school administrators, school teacher-leaders, and central

office staff to create the necessary school contexts that support LDC implementation and, as

importantly, support a gradual release model in which district capacity builds through teacher

leaders, teacher teams, and individual teachers to drive future instructional sustainability.

• National jurying system. SCALE’s LDC module scoring rubric, jurying system, and

training protocol create three exceptional opportunities for teacher learning: (1) summative,

quarterly national jurying for teachers to have their modules reviewed for feedback and

potentially validated as exemplary to share by SCALE-trained experts; (2) regular in-school or

PD-setting formative feedback by coaches or peers; or (3) teachers can also use the tools for self-

assessment. If funded, the i3 grant would further enable LDC to validate, codify, and create more

                                                                                                               9 The national LDC organization does not provide direct PD. Instead, PD services are provided by LDC-certified teacher training organizations, many of which have been part of the BMGF-funded pilot work for years including the New Teacher Center, the Southern Regional Education Board, National Writing Project, Battelle, AFT, and several others. As new PD organizations seek to enter the LDC training environment, the national organization has an accreditation process for ensuring high-quality, Common Core-aligned PD supports. 10 For example, LDC is working on a “gamified” version of SCALE’s “Looking at Teacher Work” protocol to train teachers, coaches, and administrators to assess whether their curricula is rigorously Common Core aligned.

  7  

dynamic online SCALE-jurying resources for teachers to use and learn from each other,

producing a model of blended summative and formative professional learning around calibrating

a Looking at Teacher Work (LATW) process.

• Nationally tested LDC student work rubric and scoring processes. LDC also includes a

calibrated mechanism, created by Measured Progress through Gates Foundation funding, to

assess whether student writing (and inferentially reading) met CCSS. Currently, the student work

rubric and scoring process provides anchor papers, calibrated expert scores, and a student work

scoring protocol and process for calibrating student work against Common Core reading and

writing expectations. By the end of the grant, LDC will build out the benchmarked student work

examples for each grade level and subject area, offering to the education field both a collection

and a process for calibrating scoring of student work across classrooms, schools, districts, and

states. (Existing student work resources here.)

• Third-party evaluation. LDC practitioners also benefit from ongoing deep research funded

the past four years by the Gates Foundation. This practical research has and will continue to feed

new and ongoing iterative LDC implementations. To date, studies on LDC by Research for

Action have confirmed the significant improvement of teacher skill through their work with

LDC; identified the LDC key conditions for a successful LDC implementation;11 provided case

studies mapping out effective implementations across the country;12 articulated mechanisms for

effective jurying of LDC modules;13 identified the need to ensure sufficient scaffolds are in place

                                                                                                               11 Duffy, M. and Park, E. “Brief One: LDC and MDC Theory of Action and the Landscape of Implementation” Philadelphia: Research for Action. September 2012. Link. 12 “Enacting Common Core Instruction: How Intermediate Unit 13 Leveraged its Position as an Educational Service Agency to Implement and Scale the LDC Initiative” Philadelphia: Research for Action. December 2013. Link; “Enacting Common Core Instruction: Strong Central Management and Strategic Delegation of Responsibility Drove Implementation of LDC in Florida's Hillsborough County Public Schools” Philadelphia: Research for Action. May 2013. Link; “Enacting Common Core Instruction: How School District Leadership Drove Implementation of LDC and MDC in Kenton County, KY” Philadelphia: Research for Action. December 2012. Link. 13 Chung Wei, R. “Literacy Design Collaborative - Module Jurying: Innovating for High Quality Design” California: Stanford Center for Assessment, Learning and Equality. May 2014. Link.

  8  

for low literacy and special education and ELL students;14 and confirmed that teachers’ CCSS

skills accelerate if they continue to engage in LDC work after the first year.15 Additionally,

CRESST (National Center for Research on Evaluation, Standards & Student Testing at UCLA)

has begun to confirm LDC’s impact on improving students literacy skills and content knowledge

in key subject areas. CRESST will expand their studies to capture more detailed student and

teacher skill improvement outcomes across the course of the i3 implementation (Evaluation fully

described below in Section 3.)

2. Estimated Impact to Meet Unmet Demand and Scale.

Overview: LDC.org is uniquely situated to meet the demand and scale for implementing proven

tools and practices that engage teachers, administrators, and students in implementing the

CCSS. Evaluations of the LDC Framework and tools over the past four years show that teachers

overwhelmingly embrace the LDC module tools with evidence of their improved classroom

practice and improved student literacy skills.16 Moreover, the efficacy and impact of the LDC

Framework and module tools remains high even while the Collaborative grows to include more

teachers in more areas. Because of LDC’s emphasis on teacher and student engagement, there is

significant demand for LDC tools and services from states, districts, and individual teachers.17

The demand for LDC supports from our i3 partner sites – New York City and Los Angeles

Unified – is particularly important as they are the two largest school districts in the country,

                                                                                                               14 Levin, S. and Poglinco. S. “Scale-Up and Sustainability Study of the LDC and MDC Initiatives” Philadelphia: Research for Action. September 2013: p. 21.; Herman, J. and Epstein, F. “Supporting Middle School Content Teachers Transition to the Common Core: The Implementation and Effects of LDC” California: CRESST. 2014. 15 Levin and Poglinco. “Scale-Up and Sustainability Study of the LDC and MDC Initiatives,” p. 25 16 Levin and Poglinco. “Scale-Up and Sustainability Study of the LDC and MDC Initiatives.” 17 For example, two of the four states where LDC is a statewide Common Core approach – Kentucky and Colorado – are currently engaged in piloting the horizontal and vertical “wall-to-wall” LDC implementation across year-long courses. (Clark, A. and Marion, S. “Using a Theory of Action to Frame the Common Assignment Study Research Plan,” Common Assignment Study. Presentation at the Annual Meeting of the American Educational Research Association, April 7, 2014)

  9  

with significant and longstanding student achievement challenges for high needs students.18 In

both cities, CCSS has engendered significant backlash due in large part to the systems’ (both

states and districts) failure to adequately support teachers.19 The demand for CCSS-aligned

curricula that engages – rather than alienates – teachers is well documented.20 With i3 funding,

LDC would be able to successfully meet this demand and reach the proposed level of scale for a

validation grant thanks to LDC’s collaborative design (outlined in Program Design Section 2)

that will build the internal capacity of our partnering sites through a “gradual release”

technical assistance model. By demonstrating success in these high-profile settings, and

codifying that success into LDC tools, resources, and practices, LDC will then be able to

disseminate this effective work across the country through our partnering networks in 40-plus

states. It is important to note that currently LDC tools and resources are available (OER) to

teachers and systems at no cost, allowing them to be used openly and widely. Because the

experiences of the two i3 districts are not uncommon across the country, the LDC i3 Initiative

will serve as a model and impetus for similar work in other districts, with much lower cost

structures as a result of the tool set developed through this grant.

LDC Context and History: Growth of LDC and Evidence from Evaluations. Anticipating

the release of the CCSS, the Gates Foundation in 2009 convened a design team of literacy and

curriculum experts to develop a strategy that positioned literacy as the foundation of core subject

instruction, as the CCSS demands, rather than the too-frequent approach of haphazardly

assigning reading and writing to students as a supplement to teaching content. Based upon both

                                                                                                               18 New York’s 2012 percentage of free and reduced lunch citywide was 67%. It’s percentage of Black and Hispanic students citywide was 68%. http://schools.nyc.gov/Accountability/data/default.htm. LAUSD has similar high percentages of FRL (66.2%) link and Black and Hispanic populations (82%) link ensuring the LDC i3 implementation will have a significant impact on high needs students.  19 See, for example: Jones, B. “Los Angeles Unified's teacher's union gives Superintendent John Deasy failing marks”, UTLA. July 10, 2013. www.utla.net; and Simon, S. “New York fails Common Core tests,” Politico. August 7, 2013. www.politico.com 20 Weingarten in Ujifusa A. “State Chiefs Spar with AFT and NEA Presidents Over Common Core,” Edweek. March 18, 2014. www.edweek.org

  10  

research and the “wisdom of practice” from teachers in the field, the LDC Design Team

articulated a template framework that incorporated the CCSS and that could be applied to ELA,

social studies, and science courses. LDC piloted this framework in 2010–2011with six school

districts, a teacher network, and a network of schools. After incorporating teacher feedback from

the pilot, LDC expanded rapidly, including multi-year statewide adoptions in Kentucky,

Colorado, Louisiana, and Georgia in 2012–2013. Throughout LDC’s development, “teachers co-

created materials, tested tools in their classrooms, and offered real-time feedback to designers

about what worked and what didn’t.” (Phillips & Wong, 2012) The national LDC organization

began its work in late 2012 to build on the impact and momentum of the past few years while

continuing to grow and scale LDC.

From LDC’s inception over four years ago, research institutions have been on the ground

collecting data from teachers and schools that are using LDC tools. This data informs the design

of LDC tools and provides knowledge about what conditions at school, district, and state levels

are necessary for supporting effective implementation. Multiple research evaluations have found

that the LDC Framework and module tools have an overwhelmingly positive impact on teachers’

literacy instruction and students’ literacy skills. Even as the Collaborative has expanded and the

number of teachers involved in the research studies has grown, the evaluations on the

effectiveness of the LDC Framework and tools remain consistently strong.

According to Research for Action’s multiple qualitative studies and a quasi-experimental

study by CRESST, the LDC framework and tools show evidence of improving teacher literacy

instruction and student achievement.21 In their most recent evaluation in 2012–2013 involving

                                                                                                               21 Levin, S. and Poglinco. S. “Scale-Up and Sustainability Study of the LDC and MDC Initiatives” Philadelphia: Research for Action. September 2013: p. 21.; Herman, J. and Epstein, F. “Supporting Middle School Content Teachers Transition to the Common Core: The Implementation and Effects of LDC” California: CRESST. 2014.

  11  

1,801 teachers, 374 principals, and 257 district administrators, Research for Action found that22:

• Well over 90 percent of teachers report that LDC tools promote literacy instruction in science, social studies, or secondary classrooms.

• More than 80 percent of teachers report that students improved their understanding of content during their most recent LDC module.

• More than 80 percent of teachers report that LDC raised their expectations for the level of student work.

• Almost 80 percent of teachers reported use of tools resulted in higher-quality student writing.

• Almost 80 percent of teachers reported tools were effective in making instruction more engaging to students.

• More than 80 percent of teachers reported LDC tools are effective in encouraging the use of formative assessment to identify students’ strengths/weaknesses.

• 70% of ELL teachers agreed that LDC was an effective tool to differentiate instruction for their high-need students.

• 60% of Special Education teachers agreed that LDC was an effective tool to differentiate instruction for their high-need students.  

Moreover, equally high percentages of principals and district administrators reported that the

tools were positively affecting teaching and student engagement, demonstrating that critical

dimension of Elmore’s three strands.23

CRESST’s recent studies on LDC’s implementation and impact on teaching and learning

provide further insight into its effectiveness in improving classroom practice and student

achievement in various grades, subjects, and districts. The CRESST studies continue to verify

the ability of the LDC Framework and tools to foster literacy instruction and learning aligned

with CCSS – well situating LDC for further validation and scale. CRESST’s studies employ

specially crafted measures of LDC implementation and impact and feature a quasi-experimental

design (QED) to examine LDC impact on student learning. Focusing on the relationship between

                                                                                                               22 Ibid. 23 Ibid.

  12  

student achievement and specific instructional practices that create “opportunities to learn,”24 the

CRESST research asks four key questions:

• What is the effect of LDC on student learning? • How do teachers implement LDC? • How does the fidelity of LDC implementation affect student learning? • What conditions and context influence LDC learning?

CRESST conducted two studies: one examined the implementation and impact of LDC in

eighth-grade social studies and science classrooms in Kentucky and Pennsylvania and the other

(still in progress) looks at district-wide implementation of LDC in sixth-grade advanced reading

classes in Hillsborough County, FL. The studies collected and analyzed multiple measures of

both teacher implementation of LDC and student learning: teacher logs, surveys, classroom

artifacts, state assessments, and CRESST-designed Integrated Learning Assessments (ILA)

created to measure literacy skills against the CCSS and understanding of subject area content.

Overall, CRESST determined that the “Results [from LDC] are promising,” even

remarkable given that most teachers had only tried LDC for one year.25 CRESST’s findings on

LDC implementation, moreover, mirror much of the findings from Research for Action’s

evaluations. For Kentucky and Pennsylvania, “LDC teachers reported in logs and surveys that

they addressed a wide variety of reading and writing skills and used a variety of formative

assessment strategies in their instruction. Attitudes about the usefulness and effectiveness of the

intervention [LDC] were also very positive.”26 According to CRESST, across both reading and

writing, teachers in all three sites report engaging in frequent formative assessment, involving

multiple strategies for monitoring student learning to respond to student misunderstandings as

                                                                                                               24 Bryk, A. (2010). Organizing schools for improvement. Phi Delta Kappan, 91(7), 23-34; Rowan, B., & Correnti, R. (2009). Studying reading instruction with teacher logs: Lessons from the study of instructional improvement. Educational Researcher, 28(2), pp. 120-131; Winters, L., & Herman, J. (2011). Turn-around toolkit. Thousand Oaks, CA: Corwin Press. 25 Herman and Epstein, “Supporting Middle School Content Teachers Transition to the Common Core: The Implementation and Effects of LDC”; Levin and Poglinco. “Scale-Up and Sustainability Study of the LDC and MDC Initiatives,” p. 25 26 Ibid.

  13  

they occur. Further, based on CRESST’s survey findings, teachers in all three sites also reported

that they found LDC a helpful and effective tool in meeting a variety of goals, including

implementing the CCSS, using formative assessment, incorporating literacy into content

classrooms, and increasing the rigor of their writing assignments. CRESST noted a need for

more consistent high quality of teacher-designed modules, particularly as LDC scales – one of

the key needs we are addressing with the i3 Initiative through coaching, jurying/feedback, and

teacher collaboration, as well as the continued expansion of our exemplary module library.

Second, CRESST determined that LDC had a statistically significant impact on student

achievement. Through a quasi-experimental design in Kentucky involving 2,200 students taught

by 37 teachers implementing LDC and between 12,000 and 19,000 control students depending

on the outcome measure, CRESST compared the performance of LDC students on state

assessments to the performance of non-LDC students. Their analysis drew comparison students

from schools and districts throughout the state, controlling for student demographics, student

prior performance, teacher prior effectiveness, and school prior effectiveness. CRESST found

that the LDC treatment had a statistically significant positive effect on the reading achievement

of all students participating in the study, with particular positive impact on students who have

free/reduced lunch status and students who showed evidence of prior achievement.27

Additionally, CRESST found that the LDC treatment also had a statistically significant positive

impact on the learning of social studies content for the same sub-groups – students who receive

free or reduced lunch and students who showed prior achievement. CRESST’s conclusions and

recommendations included: “Positive QED [quasi-experimental design] results in reading are

encouraging for an intervention still in its early stages. The positive interaction effect with

                                                                                                               27 Again, this occurred notwithstanding that 30 of the 37 teachers studied had only taught one module in the fall and one in the spring of the school year.

  14  

free/reduced price lunch status is also encouraging, suggesting that the introduction of literacy

instruction into content area classrooms through the LDC intervention may particularly benefit

students from low socio-economic backgrounds.”28

Growing LDC to Support Demand and Scale.

Finally, LDC has strong capacity to support the demand and scale-up of LDC. As noted

above, LDC partners with multiple organizations across the country to provide on-site LDC

technical assistance to districts and schools. Additionally, LDC’s new website with tools and

collaboration space allows LDC to directly support teachers at scale – whether they are involved

in on-site district or state LDC work or are interested in LDC as individual teachers.29

In terms of our i3 sites, LDC’s work with the New York and Los Angeles school districts

would seek to reach 20,000 students a year for the five years of the grant in each district or

200,000 students overall and approximately 6,000 teachers in science, social studies, and ELA.

LDC CoreTools has the instructional planning and instructional infrastructure to support 6,000

new teachers and LDC’s PD partners will support the provision of regular and ongoing

professional learning.30 Equally important, the further codification and validation of the LDC

tools through our work with New York and LA will allow all current LDC partners to further

their implementation of LDC. For example, Kentucky (a national leader in CCSS) has a

statewide LDC Coordinator and she, along with the Kentucky Fund for Education and the

Pritchard Committee (statewide education intermediary) all participate in LDC’s national partner

                                                                                                               28 Herman and Epstein, “Supporting Middle School Content Teachers Transition to the Common Core: The Implementation and Effects of LDC,” 6. 29 Since the launch of LDC CoreTools this past January, nearly 10,000 teachers have already signed up and begun to use these edtech tools, a remarkable number. LDC CoreTools creates the first real ability for LDC to identify and track the instructional behaviors of its teacher-users. These are just the new online users; the Gates Foundation did not track teachers trained during the past four years and thus reliable estimates suggest that total LDC teacher-users virally and in adoption states and districts numbers closer to 40,000-50,000. 30 The likely lead PD partner in LA, Reach Associates, provided statewide trainings in Colorado, Kentucky, and Louisiana. The likely lead in NYC, The New Teacher Center, has a national if not international footprint. Content experts such as American Museum of Natural History (Sci), and Stanford History Education Group likewise have national capacity.

  15  

convenings and regular updates to secure resources for broad dissemination. Similar

relationships exist with the states of CO, GA, and LA to enable dissemination of codified tools

and new resources – meaning that the i3 work is likely to have the attendant affect of reaching

hundreds of thousands if not millions of needy students.31 In addition, because of LDC’s OER

structure, tools and resources developed through this grant will be freely and publically available

during and after the grant period, allowing for wide access and adoption by millions of U.S.

teachers.

3. Feasibility of National Expansion.

Based on past data and evaluations of LDC tools, we are confident that the evaluation of the

LDC tools and services related to the i3 Initiative will yield positive outcomes for teachers and

students. There are a number of features of LDC that make national expansion highly likely:

• LDC is structured but flexible. LDC is a framework that holds certain design elements

constant, such as hardwiring the CCSS into teaching tasks and requiring teachers to explicitly

identify the reading/writing skills students will develop (see below for LDC Framework

components). Within the structured framework, LDC is flexible in that it seeks to engage

teachers in the design of CCSS-aligned assignments and instructional plans. Using the

framework, teachers choose and incorporate state/local standards, particular content, and

instructional strategies that best meet the needs of their own students. The flexibility of the LDC

Framework ensures that teachers everywhere – whether in rural Pennsylvania, urban Denver,

already high-performing districts like Hillsborough County, FL, or sites with ongoing reform like

                                                                                                               31 A more complete roster of LDC partner organizations includes: American Federation of Teachers, ASCD, Aspen, Battelle STEM, Center for Inspired Teaching, Center for Teaching Quality, CO Education Initiative/CO DOE, ConnectEd, Digital Youth Network (Chicago), Educate Texas, Florida Academic Literacy Network (NLP), Georgia DOE, IU13 - Lancaster Lebanon, Kentucky DOE, Louisiana DOE, MetaMetrics, Measured Progress, New Tech Network, New Teacher Center, New Visions for Public Schools, Paideia, National Writing Project, SCALE, Reach Associates, Southeastern Regional Education Board, UC – Berkeley.

  16  

Baltimore – are able to successfully use LDC. At the same time, the common framework and

LDC language allows teachers to share and learn from each other’s work, – breaking down

classroom, school, district, and state barriers. 32

• LDC CoreTools are portable and constantly being improved upon. LDC interactive

tools are all online, accessible, and free. The LDC website is designed so that educators involved

in LDC have access to the tools and supports for implementing the tools such as guides,

examples, “how-to” videos, and other resources – from anywhere and at any time. Teachers

across the country, moreover, add each month to the content libraries (exemplary and good-to-go

modules, mini-literacy tasks, PD, and help resources).

• LDC is a community of support. LDC is a growing literacy community designed to

share practices, provide collegial feedback, and build expertise among members. Thanks to the

CCSS in setting common expectations, the LDC structured framework, and the new LDC

website, the LDC community has national reach. In addition to working online to co-design

modules, provide feedback on each other’s modules, and share instructional strategies, LDC

members meet at shared convenings such as the annual LDC conference hosted by SREB and the

national partners convening hosted by LDC. All of these opportunities lend themselves to a

successful national scale-up.

• LDC is a network of partners who communicate and support LDC implementation.

The Literacy Design Collaborative was developed as a collaborative so that many organizations

and educators across the country would engage as partners and “owners” of LDC. As the

Collaborative grew over four years, the Bill & Melinda Gates Foundation established the LDC

                                                                                                               32 For example, the teachers in two counties in CO and KY acknowledged that they were able to collaborate on authoring and implementing common units with common assessments over the past year because of the commonality of Common Core Standards and the “common chassis” of LDC. (Clark, A. and Marion, S. “Using a Theory of Action to Frame the Common Assignment Study Research Plan,” Common Assignment Study. Presentation At the Annual Meeting of the American Educational Research Association, April 7, 2014)

  17  

organization to facilitate and coordinate the LDC partners and ensure ongoing improvements and

quality as LDC scales across the country. The LDC partners form a unique and valuable

infrastructure to support scale-up nationally as the work is continued and validated.

B. QUALITY OF THE PROJECT DESIGN (25 POINTS)

1. Goals, Plan, and Risk Mitigation

Overview: The i3 Initiative’s goals and project plan seek to improve: (1) students’ literacy skills;

(2) the ability of teachers to design assignments and instructional plans; (3) the effectiveness of

teachers to implement CCSS-aligned instruction;(4) the ability of schools/districts in

implementing the CCSS as a system; and (5) our own ability to codify and scale LDC tools and

services to meet national demand. To meet these goals, LDC.org will work with two large

districts—New York and Los Angeles—to implement LDC as a system across 20 schools each

year for five years, expanding LDC’s reach by approximately 200,000 more students and 6,000

teachers by grant’s end. Through this collaboration, we will further evaluate, validate, and

codify current and new LDC tools and services, which, in turn, will be disseminated to a national

community of practice (40-plus states) to support national scale.

LDC i3 Initiative Purpose and Goals. The purpose of the LDC i3 Initiative is to increase

teacher skills and student skills as we validate, and scale the LDC system as a strategy for

improving K-12 literacy teaching and learning aligned with the CCSS. LDC will (1) partner with

the New York and LA school districts to implement and test LDC as a coherent system to

implement the CCSS, targeting 200,000 students and 6,000 teachers over the course of the five-

year grant33; (2) be evaluated by CRESST through a quasi-experimental design study to further

                                                                                                               33 The exact number of schools will be determined during school recruitment. Estimates of teachers and students were based on a target average secondary school size of 1,000 students with 30 ELA/Science/SS teachers. If that scenario holds, LDC would be

  18  

assess efficacy and impact of tools and services; (3) refine and codify current and new services,

processes, resources, and tools and implementation models to prepare them for broad distribution

and use; and (4) plan with LDC partners to use codified tools and processes to support national

scale.

Seeking to impact students, teachers, school systems, and our own organization, LDC has set

the following measurable goals and targets for the i3 Initiative:

Goals Measurable Objectives Evidence to Meet Objectives

1. Improve students’ literacy skills

All students (including SPED, ELL, FRL) in LDC classrooms will outperform matched-pair non-LDC classroom students statistically significantly after two years of the LDC treatment

- State test results are statistically significantly higher in the LDC treatment schools than the matched-pair schools after the end of Year 4.34 - Potential to add Intermediate Objectives such as growth on student mastery of literacy skills in LDC modules (currently under study SCALE and Center for Assessment) - Functionality to collect formative assessment data from LDC modules completed Year 1 and collected in succeeding years

2. Improve the ability of teachers to design and/or deliver CCSS-aligned LDC curricula

LDC-trained teachers will demonstrate the capacity to design and/or deliver CCSS-aligned curricula through the LDC Framework

- Number of LDC modules created by teachers that are determined by SCALE-vetted process to be Exemplary and/or Good-to-Go increases 20 percent each year starting in Year 2 (Year 1 baseline) - Number of teachers calibrated to SCALE jurying criteria against master scores increases 20 percent each year starting in Year 2 (Year 1 baseline) - Percentage of teachers who confirm that they are better able to implement CCSS through the LDC Framework increases 10 percent each year

                                                                                                                                                                                                                                                                                                                                                                     rolled out to 20 schools each year for five years or 100 schools in each district. LAUSD has confirmed that rough approximation though they may wish to explore a small set of elementary feeder schools as part of a middle school-focused pilot. NYC has indicated their schools may be of a smaller average size. 34 CRESST tentatively has concluded that only the first and second cohorts of teachers will have sufficient data to justify a statistical analysis that begins at the conclusion of Year 4.

  19  

- Number of teachers who download, clone, modify, or adjust instruction manifesting teacher reflection increases 50 percent each year starting in Year 2 (Year 1 baseline) - Number of district- and school-based staff who use LDC CoreTools to create, modify, or provide jurying or other feedback on LDC curricula increases 50 percent each year starting in Year 2 (Year 1 baseline)

3. Improve the ability of schools/districts to implement CCSS through LDC as a system

District builds capacity over time through administrators, teacher leaders, and teachers to capably deliver CCSS-aligned instruction (adult gradual release model)

- Percentage of administrators, teacher leaders, and teachers reporting the ability of teachers to more effectively deliver CCSS-aligned curricula increases 10 percent each year starting in Year 2 - Numbers of district- and school-based staff trained to provide deep CCSS PD: Shift in ratio in PD delivery from 80 percent LDC and 20 percent district to 80 percent district, 20 percent LDC by end of Year 4 - 10 percent annual increase in number of certified LDC coaches starting in Year 2 (Year 1 baseline)

4. Improve LDC.org’s ability to scale LDC tools and services

Number of LDC.org and CoreTools.LDC.org users increases every year during each of the three years of the grant

50 percent annual increase in number of LDC CoreTools users nationwide accessing LDC resources generated during the LDC i3 Initiative following Year 2 of the grant

LDC Plan: A Closer Look at the LDC Theory of Action, Framework, Tools, and Resources.

LDC was founded on the theory that if teachers are provided with the necessary literacy tools

that align instruction with standards and foster teacher learning—and teachers and students are

provided supportive conditions—teachers’ knowledge about literacy instruction will increase and

their beliefs about student learning will change. This will result in stronger teacher literacy

  20  

practices and increased student literacy skill in the short term and increased student college- and

career-readiness in the long term. RFA visualized LDC’s Theory of Change as:

Duffy, M. and Park, E. “Brief One: LDC and MDC Theory of Action and the Landscape of Implementation,” p. 3

The LDC Framework captures the key areas that our suite of tools and services support

related to the day-to-day district, school, teacher, and student experience:

3 | B r i e f O n e : T h e o r y o f A c t i o n

Figure 1. Theory of Action

  21  

• LDC Teaching Tasks (or student performance assignments): LDC tools guide

teachers to use their own content (subject area content, reading texts, etc.) to create, adopt, or

adapt their own CCSS-aligned reading/writing teaching tasks to address identified student need.

• LDC Modules: A host of LDC tools support teachers in creating, adopting, or adapting a

quality, CCSS-aligned instructional plan in which teachers engage students in reading, writing,

speaking, and listening mini-tasks and explicit instruction to guide them to completing the

assignment over two to four weeks. LDC online support includes embedded teacher guidance on

how to design the two- to four-week instructional ladder, including articulating the explicit

literacy teaching necessary to support students’ mastery of the skill and sub-skills central to

completion of the culminating module teaching task.

Courses'and''Ver,cal/Horizontal'Alignment'

Modules'

Teaching''Task'

• ""Courses"anchored"by"rich"module"teaching"tasks"

• """"Standard,"skills,"and"module"assignments""aligned"across"grades"and"courses"

• Student"Work"Calibra9on"cross:district"

• ""Task"• ""Skills/"Standards"""• ""Mini:Tasks"• ""Instruc9on"• ""Results"–"Student"Work"• ""Module"and"Mini:task"libraries""

• ""Fill:in:the:Prompt"• ""Standards"Alignment"• ""Rubric"

'Suppor,ng'Condi,ons'Leadership"Support"Web:based"supports"

Jurying"to"ensure"quality"Time"for"teacher"planning"and"collabora9on"

  22  

• LDC Courses and Vertical/Horizontal Alignment: A beta LDC tool (in testing) to

guide teachers and administrators in designing literacy-rich courses and K-12 alignment.

Currently a structured, manual exercise in Excel, the new tool guides teams of teachers through a

curriculum mapping process that identifies key (and spiraled) Common Core focus standards

(horizontal and vertical), disciplinary core conceptual content, related texts, student performance

products that demonstrate student mastery, an attendant student performance rubric

(SCALE/Measured Progress-designed), and an LDC teaching task that ably weaves them all

together.

• LDC Professional Learning: LDC resources from key partners support coaches,

trainers, principals, and other leaders in creating a supportive context for LDC implementation as

a system. Though mostly static resources to date, LDC has embedded many such resources in the

guided LDC CoreTools experience and continues to create both blended and asynchronous

versions of these teacher support tools. An i3 grant would radically advance such work.

Together these components are at the center of an intensive teacher-supported LDC i3

implementation: training teachers and administrators how to implement the LDC Framework,

utilizing LDC online tools and resources to build their capacity to sustain the work over time.

• Implementation Strategies to Meet the LDC i3 Initiative Goals: Building upon past

success in engaging teachers, districts, and states in CCSS-aligned LDC implementation, the

following provides an overview of the LDC program design strategies to meet the LDC i3

Initiative goals and objectives, including an overview of implementation plans (see Management

Section for additional detail on rollout activities/timeline/staffing/budget): (1) Developing

Administrator Capacity for Creating the Conditions for LDC Success; (2) Developing Teacher

Capacity for Implementing LDC; (3) Developing Teacher Leaders: Instructional Capacity to

  23  

Sustain and Expand LDC Within the District; (4) LDC i3 National Dissemination and Scaling of

Codified/Validated Strategies and Resources.

• Developing Administrator Capacity for Creating the Conditions for LDC Success:

Research for Action’s studies found that for LDC work to be successful, district and school

administrators needed to create the right conditions that impact classroom teaching and the

success of students: (1) Engage principals, school leaders, and district leaders to ensure LDC

alignment with other instructional priorities; (2) Consistently message the importance of LDC as

an essential CCSS implementation; (3) Build school-based leadership’s understanding that LDC

module planning and instruction are deep changes in the classroom to inform their classroom

observation practices; and (4) ensure teachers have designated time for professional learning and

collaboration to make the CCSS shifts embedded in LDC.35

LDC will put these four work strands into action in partnership with agreed-upon PD partners

starting in the first few weeks by creating a comprehensive, detailed, strategic rollout plan with

district staff, representatives of the union, school-based administration, and classroom teachers

that is district-contextual. The strategic rollout plan will be based on where the district is in

CCSS implementation, teacher needs, and agreed-upon roles of central office and principals. The

first activity with all school leaders involved in Year 1 will likely be an introduction to LDC as

an intensive instructional strategy and leaders’ role in supporting LDC (for example, finding

professional learning time, communicating LDC, what to look for in the classroom, etc.). This

initial meeting will be followed by monthly check-ins and modification, as necessary, to the

rollout plan to address new learning, challenges, or other feedback to maximize LDC’s impact

and improve teacher skills and student outcomes. Leaders will also be expected to become

                                                                                                               35 Levin, S., Duffy, M. and Dever, K. “Brief Four: Conditions for Scale and Sustainability,” Philadelphia: Research for Action. September 2012. Link.

  24  

familiar with other core LDC activities (attending certain PD sessions with teachers) as they

emerge in implementation such as the crosswalk of LDC to teacher effectiveness (for example,

Danielson, student skill growth), SCALE’s jurying system to assess curriculum, the critical need

for teachers to come together in calibrated sessions to look at student work (for example,

Measured Progress/SCALE LASW), etc.

Evaluation and Codification: LDC will work in tandem with CRESST to validate and codify

through online portable artifacts a series of these practices and products on LDC leadership

development for dissemination through national partners and our online portal.

• Developing Teacher Capacity for Implementing LDC in Their Classrooms: In the first

year, LDC—through its PD partners—trains cohorts of teachers (potentially 600 if average

participating school is 30 content teachers per 1,000 students), typically through two guided

cycles of module development, feedback, implementation, scoring of student work, and

refinement of their module. Through our guided iterative process of designing LDC assignments

and modules, teachers engage with and develop an understanding of implementing CCSS.

Professional development activities and concepts led by the LDC PD partner36 typically include:

• Introducing teacher teams to the LDC Framework’s grounding in the CCSS instructional

shifts

• Introducing teachers to mini-tasks as a way of understanding LDC and CCSS focus on

student performance tasks, moving to broader picture of LDC assignments and modules

• Engaging teachers in a first cycle of module development through creating new module

(or adopting/adapting existing district or exemplary modules) with feedback

                                                                                                               36 As noted above, LAUSD has indicated its interest in using Reach Associates, a longstanding PD provider who supported several statewide and many other LDC rollouts. NYC has indicated its interest in working with the New Teacher Center. Both are LDC national partners and qualified LDC PD providers

  25  

• Engaging teachers in teaching the LDC module and then guiding them in reviewing,

scoring, and calibrating the student work generated, reflecting on their instruction, and,

after making meaning from the student data on implications for how they instructed their

students, modifying the modules

This collaborative scoring process with validated CCSS-aligned rubric and benchmarked

samples guides teachers in understanding the level of written work that meets CCSS demands.

Teachers' engagement in collegial and experienced feedback on their LDC modules, as trainers

and other experts use the LDC jurying module and process to provide feedback on the teacher-

designed work, ensures CCSS alignment, rigor, and student engagement. This combination of

teacher design, looking at student work, and feedback creates an intensive learning system for

teachers to develop their knowledge and expertise of the CCSS. Monthly virtual “office hours”

staffed by the PD providers provide an opportunity for additional “real-time” coaching for

teachers engaged in module implementation and reflection.37

Evaluation and Codification: CRESST and LDC will examine the first-year implementation

experience throughout the year and identify factors to refine second-year implementation and

develop additional products to support teacher implementation of LDC in their classrooms.

Year 2 introduces a second cohort to the first cohort’s experience.

• Developing Teacher Leaders: Instructional Leadership to Sustain and Expand LDC as a

Broad Instructional Strategy. One of the essential deliverables of each LDC implementation

around the country is the building of leadership and teacher capacity to deliver CCSS-aligned

instruction post-grant. LDC and its PD partners must leave a district with an increasing and

                                                                                                               37 Several online coaching platforms have been used in different LDC implementations across the country, including Teaching Channel’s Deeper Learning Labs pilot, Center for Teaching Quality’s Collaboratory, and Reach Associates R-Group Space. Several districts have also simply used Google Hangouts or GoToMeetings. LDC will weigh the technology solution that makes the most sense in consultation with each district and the PD partner.

  26  

ongoing ability to drive constant improvements in instruction through the continuous

improvement lens embodied in all of LDC’s tools and strategies. One part of this strategy is to

develop teacher leaders who, over time, are able to lead the ongoing work without external LDC

PD partner coaching. Teachers learn best from other teachers – this has certainly been LDC’s

experience. Thus, after the first year, strong teachers in every school in the three content areas –

and preferably a teacher focused on students' special needs – come together for an LDC Summer

Institute (est. 80 teachers across 20 schools) in which they deepen their knowledge and

experience in implementing literacy-rich LDC modules and begin the process of examining year-

long curriculum to unpack the opportunities for vertical and horizontal curriculum mapping and

alignment.

This process of thinking through how CCSS unfolds across the entire next year (Year 2) only

begins at the summer institute. LDC curriculum design experts support this preliminary and year-

long ongoing work to create coherent horizontal and eventually vertical alignment of curricula in

schools utilizing LDC’s beta planning tools and system. In Year 2, teams of teachers continue to

identify and ensure specific “focus” standards are addressed at key points in the year and in key

subjects (horizontal alignment), spiraled to ensure certain essential CCSS literacy skills and sub-

skill development are assessed multiple times, and review standard coverage vertically to ensure

skills build progressively as the Common Core contemplates. The summer following Year 2 and

into Year 3 continue this deep instructional thinking/creation, enabling teachers to refine and

build dynamic but structured curricula – not workbooks or textbooks – with which individual

schools and teachers can modify, adapt, and/or adopt to support their students' literacy and

content development. The resultant curricular products reflect teacher choice points but leave

residual flexibility for teachers to instruct as dictated by the needs of their classrooms.

  27  

During the Year 1 summer institute, a subset of the first cohort teachers (approximately 20;

one per school) are identified as strong potential leaders to receive a partially reduced schedule

supported by the i3 grant to engage in peer LDC/CCSS support during the school year. (The

same happens for subsequent cohorts as schools get added each year to engage in Summer

Institutes.) These school-based teacher leaders conduct peer led, in-school CCSS/LDC PD

supports for their colleagues throughout Year 2, critical for building district/school capacity to

sustain CCSS implementation post-grant and also essential for teacher buy-in and adoption of the

Common Core Standards.

• Disseminating Validated/Codified Strategies for Scale: This i3 application is certainly

about supporting high-needs students in both NYCDOE and LAUSD to improve their chances

for post-secondary success. However, these sites are perhaps even more valuable as classroom

labs to validate and codify a set of research-based strategies, processes, and artifacts that have

supported LDC implementations around the country for the past four years, leading to improved

teacher skill and student outcomes. Through a rigorous CRESST evaluation and LDC

documentation process, the result of this i3 implementation is a host of validated/codified LDC

resources and CCSS-aligned strategies that will be disseminated through LDC’s vibrant national

partner network and growing teacher viral community network. LDC knows from its new LDC

CoreTools online identity verification functionality that teachers in 40-plus states are trying

LDC. We know from our most recent LDC Partners convening in Chicago (June 2–3, 2014) that

LDC has over 40 PD, channel, and district/state partners eager to take the best that is occurring

around the country to support their own efforts to build teacher capacity to improve outcomes for

students. CRESST’s rigorously planned research underpinning this i3 (with annual reports

reflecting ongoing learning) would radically accelerate the potential for adoption of research-

  28  

based effective Common Core implementation practices. LDC’s national organization would

deploy all of our social media and other marketing and communication capacity to leverage the

powerful work from this well-studied, carefully planned set of implementations in NYC and

LAUSD.

Mitigating Risks: RFA identified three essential conditions (or, reciprocally, risks) to

successful implementation: (1) leadership support; (2) alignment to existing district, state, school

initiatives; and (3) multiple opportunities for teacher professional learning.38 The LDC i3

Initiative is designed to strategically address and mitigate these potential risks:

• Leadership: Each district has agreed to name full-time leads and central lead coordinating

teams to co-design and co-facilitate each of the LDC strands of work, ensuring district and

school ownership of the LDC rollout and implementation.

• As noted above, LDC will use the i3 Initiative to finish documenting and codifying a draft

leadership professional learning strand that better involves principals and district leaders

in: (1) basic understanding, prioritizing, and messaging of LDC, so that teachers continually

hear/see/experience that their LDC work aligns with district and school priorities and

expectations; and (2) on a deeper level, creating the day-to-day conditions, through resource

and scheduling optimization, that commits elusive teacher time for collaboration, planning,

scoring student work, and reflection during the school day.

• Multiple Teacher Learning Opportunities: The district commitment to allocate school-day

time to this work serves as a key signal from school and district leadership that LDC is

important and that they are committed to their teachers’ efforts to expand teachers’ ability to

implement CCSS and deepen their classroom skill expertise. This includes supporting teacher

                                                                                                               38 Levin, S., Duffy, M. and Dever, K. “Brief Four: Conditions for Scale and Sustainability,” Philadelphia: Research for Action. September 2012. Link.

  29  

leaders, coaches, or other designated leads in facilitating teacher collaboration, planning,

sharing, and reflecting on designing and implementing LDC modules and scoring student

work. Part of this work includes looking at ways to leverage the LDC.org workspace and

collaboration capabilities of LDC CoreTools. By heavily supporting teacher leaders in this

facilitation role, LDC ensures that the newly found precious time is well used and that

teachers see value in this work.

• Instructional Systems Alignment: The agreement to take on the LDC i3 Initiative was

expressly agreed to by Superintendent Deasy and Chancellor Farina. The Chief Academic

Officers of both districts participated in discussions with LDC on implementation and signed

off on the outline of our joint efforts. Both districts have publicly voiced interest in aligning

their instructional and teacher effectiveness systems to support teachers. Both have

acknowledged inadequate supports in the past for the teacher growth necessary to implement

Common Core while expressing frustration at the quality and static nature of publishers’

putative Common Core-aligned content. LDC provides a strong, research-based mechanism

for honoring teacher professionalism while providing additional instructional (teacher-

created) supports.

• LDC CoreTools: LDC CoreTools is an explicit set of “continuing-to-grow” tools, processes,

and resources designed to address all three risks noted above. Specifically, LDC CoreTools

has embedded help resources; ability for real-time collaboration (coach, admin, peer); and

access to multiple resource libraries of high-quality, SCALE-denominated, CCSS-aligned

curricula, all of which constitute multiple learning opportunities in the guided and prompted

curriculum design process. New functionality coming online in the coming months and

supported by this i3 grant includes the ability to automate the teacher-student, student-teacher

  30  

assignment process (a real challenge for a ninth-grade history teacher with 140 students),

scoring of student work, real-time feedback on student writing, rolled-up data analytics, and a

resultant teacher portfolio of effectiveness. All of the existing and hoped-for planned

technology would empower teachers to take leadership in their own professional growth and

learning as a practitioner supporting students in reaching college and career learning

standards (CCSS).

2. Grant Funds to Address Barriers for Reaching Scale

Overview: LDC.org was established by the Bill and Melinda Gates Foundation to bring the

tested LDC curriculum tools, resources, and learning process for teachers to scale and to

design, test, and nationally scale new tools and solutions related to the LDC Framework. The

Foundation formally and legally spun off its $20 million-plus grant investment in creating and

research-validating effective Common Core implementation resources that have “a record of

significantly improving student achievement, attainment, or retention through its record of work

with an LEA or schools.”39 LDC.org has begun this work by solidifying existing implementations

and converting some paper and pencil work to online work. However, the potential for LDC to

scale from thousands to millions of teachers rests in a broader set of tools and resources

currently out of reach of LDC.org and the LDC national community of practice.

The Gates Foundation transferred its “research-validated record of success” to the LDC

organization to continue the mission-focused effort to validate, codify, and scale the proven LDC

strategy and framework for implementing CCSS. During the past 1½ years, the organization has

made significant progress in creating a context that will accelerate scaling:

• Continued evaluation and improvement of LDC’s original tools and services, iterating

based on teacher feedback (e.g., release of Template 2.0, Rules of the Road 2.0, etc.);                                                                                                                39 The Foundation’s assignment letter is attached in Appendix J.

  31  

• Created an online platform that is user friendly and offers interactive tools for teachers to

create, adopt, and adapt assignments and modules with students while creating a

mechanism for LDC to identify and track teacher choices to inform further online

investments;

• Started the process of engaging LDC PD partners and Learning Forward in establishing

evidence-based standards for effective LDC PD implementation;

• Completed the SCALE work to create a quality control curriculum content mechanism

that supports a national profile through quarterly jurying events that identify exemplary

modules and mini-tasks to build a portable and usable library for teachers as well as

formative jurying practices for schools, districts, and partners to use to foster teacher

instructional insight and improved teacher instructional design and implementation skills;

and

• Created an organizational structure that will support scale-up, including positions

dedicated to communications/marketing and partnership facilitation.

As noted above in the Significance of Project section (and incorporated by reference), the

LDC Framework itself is particularly well-suited to scale by virtue of its being “structured yet

flexible,” capable of supporting teachers K-12, in high-performing or “challenged” districts;

increasingly portable and modular while constantly being improved upon based on teachers’

“wisdom of practice”; and connected to a national teacher community of impassioned,

supportive educators, with over 40 high-capacity and nationally respected teacher support

organizations.

With i3 funding, LDC.org will overcome potential barriers, propelling LDC to scale:

Barriers i3-Funded Solutions

  32  

Consistent professional development offered by numerous partners

i3 funding will enable LDC to finish codifying professional development standards in partnership with Learning Forward to include key practical “look-fors” based on objective evidence (for example, number of exemplary modules produced) that at the same time respects partners’ diverse PD toolsets to ensure consistent professional learning at scale, including collecting strong samples from partners to share with each other.

Incomplete tool sets and codified practices/processes for teachers, leaders, and partners to support LDC as broad instructional strategy

LDC.org will be able to support the testing, validation, and codification of current and new LDC tools to have a complete series of tools and processes related to all components of the LDC Framework and system (more below).

Nominal module and mini-task libraries that offer a variety of exemplars for teachers in all grades and subjects that are constantly being added to with vetted resources from teachers across the country

LDC i3 Initiative will provide the necessary funding to support extensive feedback and jurying of teacher-designed modules and mini-tasks in the LDC i3 Initiative to contribute to the growing LDC libraries.

Incomplete validation of LDC’s efficacy and impact on teachers and students as a full Instructional System

i3 funding will support the CRESST quasi-experimental evaluation including assessing efficacy and impact of improving teacher and student skills.

Current and Proposed Tools and Services to be Validated and Codified.

LDC resources and LDC CoreTools for literacy assignments (teaching tasks), modules, and

courses, as well as tools for creating a supportive context for teachers and students, will be

further codified and validated by this i3 funding. The following chart provides an overview of

products already developed and offered on the LDC.org website (LDC CoreTools section) that

will be brought to scale and extensions on such products and services to be tested, codified, and

prepared for scale through our work with districts in this i3 Initiative and across the country.  

LDC Framework LDC CoreTools: Products for further codification Teaching Tasks (Assignments)

• Collections of assignment teaching tasks organized around CCSS writing areas (argumentation, informational, and narrative) and skills for teachers to design assignments that could be used as “cold prompt” diagnostics of student CCSS abilities, starting points for teacher or teams of teachers (schools) to create their own student-centered curriculum

• Interactive tools, guides, and videos for teachers using templates to create

  33  

rigorous, student-engaging teaching tasks Mini Literacy Tasks

• Collections of mini literacy tasks, organized by CCSS reading, writing, speaking, and listening skills, for teachers to use within modules, units of study, and courses. Collections populate grade level and discipline.

• Interactive tools, guides, and videos to support teachers in understanding how to find, use, and create their own mini literacy tasks specific to the assignments they are building to ensure they are targeted and precise in design (grade/discipline).

Modules • Interactive module tool with collaboration features to create, adopt, or adapt existing modules

• Expanded Exemplary and Good-to-Go module libraries for module adaptation/ adoption

• My Curriculum Library that enables teachers to customize and maintain their own library of usable modules and mini-tasks

• Rubric, scoring process, and expanded benchmarked student work samples (LASW)

• Integration of the LDC Jurying tool, scoring guide, and process for providing feedback on modules into LDC CoreTools and school PD processes to identify exemplary and good-to-go modules and protocols that support teacher collaboration around Looking at Teacher Work (LATW).

Courses and Vertical/Horizontal Alignment

Test and move beta design for courses and K-12 CCSS and curriculum alignment into LDC CoreTools release including: • Course designer tool • Curriculum mapping tool that allows teachers to build courses out of

sequences of modules, mini-tasks, common assignments, common assessments, etc.

• Tagging capacity to indicate time of year, sequences of CCSS, and skills as part of designing courses

Conditions That Support LDC Implementations

• Codified online training and coaching tools (for example, finalized Learning Forward/LDC professional development standards, additional videos, webinars, revision of current tools, and expansion of PD resources to include support in the effort to create vertical/horizontal alignment

• Tools and resources for principals and school administrators to create conditions that support LDC teaching and learning of CCSS, with a particular focus on supporting teacher collaboration, sharing, and reflection during shared time during the day, including ways to use and build out the LDC CoreTools collaboration space

Help Resources • Expanded, embedded online help content to guide teachers and coaches through a professional development experience. Specific and targeted video support that demonstrates the uses and implementation of modules and mini-tasks in classrooms, a series of decision-making articulation that enables teachers to identify their ability to integrate CCSS into their instructional design and helps to bridge the connection between instructional design and implementation in the classroom.

  34  

• Expanded online help content for leaders that builds their capacity to create the conditions necessary for deep LDC work, and their capacity to support CCSS work in sustainable and ongoing ways

C. QUALITY OF MANAGEMENT PLAN

Overview: LDC has the internal expertise, organizational infrastructure, and critical

partnerships to successfully lead the LDC i3 Initiative to ensure it meets the overarching goals of

supporting high-need students in meeting internationally benchmarked college/career standards.

1. Key Responsibilities, Objectives, Metrics on Progress with Annual Performance

Targets and Timeline with Milestones.  LDC identified in the prior Project Design section

goals, metrics, and activities to meet those goals, all relevant to this section. As previously noted,

the LDC i3 Initiative quite simply seeks to have a measurable impact in increasing teacher skill

to implement CCSS and student success in meeting those higher expectations. Also, as noted

above in Project Design, LDC will be managing four basic work streams through its partners and

the districts’ staff:

(1) Leadership strategic planning and orientation/support for LDC implementation throughout the five-year grant; (2) Schools and their teachers cycling through LDC PD in person and virtually across a three-year development arc and five cohorts of teachers, (3) Building Teacher Leader (as well as admin and other classroom teacher) capacity as part of the broader strategy to expand district/school capacity to sustain the CCSS work post-grant, and (4) LDC’s national dissemination efforts: taking the validated and codified rich CCSS resources, practices, and content to funnel them through LDC’s partners and national community of practice.

In general, Year 1 of the grant is used to introduce administrators and teachers to the LDC

framework and to collect baseline data; progress against that benchmark data becomes salient to

  35  

feed back into program design and implementation beginning at least before the Year 2

school/teacher cohort commences. Annual and Initiative progress goals and metrics include:

1. Improved students’ literacy skills   • Initiative goal: CRESST-validated, statistically significant increase in student state test scores

• Intermediate: Student growth measure (under study) • Intermediate: Student formative data on literacy skill

mastery 2. Improved ability of teachers to design and/or deliver CCSS-aligned LDC curricula  

• 20 percent annual increase in number of quality modules created/implemented

• 20 percent annual increase in number of teacher leaders calibrated to SCALE curriculum jurying ability

• 10 percent annual increase in teacher CCSS efficacy • 50 percent annual increase in reflective LDC tool use

3. Improved ability of schools/districts to implement CCSS through LDC as a system  

• 10 percent annual increase in teacher efficacy reported by admin staff

• Shift from external PD provider to internal district staff including teacher leaders (80/20 to 20/80)

4. Improved LDC ability to scale LDC tools and services  

• 50 percent annual increase nationwide in LDC CoreTools use of i3 generated resources

 

These goals and metrics will be achieved during the course of the five-year grant with the

first cohort schools, teachers, and students finishing the phased sequence first cycle of training

by the end of Year 3 with succeeding teacher cohorts following.

Timeline with Milestones

  36  

Years Fall (Sept-Dec) Winter (Jan-Mar) Spring (April-

June) Summer

Year 1: Intro to LDC Processes and Leadership Planning

• District, union, and other stakeholder engagement

• Needs assessment • Individualized

planning of technical assistance based on needs assessment, including phase-in plan based on district priorities

• Beginning engagement of teacher teams

LDC Cycle 1: Module development and feedback

Cycle 1: • Module

implementation • Scoring student

work

Cycle 1: Revision and submission for consideration as national exemplar Cycle 2: Module development

• Master schedule audit

• Scenario planning Development of master schedule in select schools that supports teacher planning and student reading/writing over time

Continued support in logistics of new master schedules in select schools

  37  

Year 2: Begin District and School Capacity to Lead LDC and Scheduling

LDC Cycle 2: Module implementation and scoring student work Revision and submission for consideration of national exemplar Beginning planning around Common Core courses and vertical/horizontal alignment

Planning for Common Core courses and vertical/horizontal alignment Teachers continue to implement past or new LDC modules with feedback. Student scoring from module implementation. LDC technical providers begin to transition module training to district/school experts (co-training model)

Planning for Common Core courses and vertical/horizontal alignment, including new module development and identification of modules from LDC library Teachers implement past or new LDC modules with feedback. Student scoring from module implementation. Continued transition to district/school leaders for module development and student scoring.

Planning for Common Core courses and vertical/horizontal alignment, including new module development and identification of modules from LDC library Student scoring from module implementation Revision and submission for consideration as national exemplar District leads train new round of teachers in LDC modules with support/feedback from LDC teams

Technical guidance for logistics related to the implementation of new master schedules Work with district/school leadership to build capacity for master schedule audit

Begin to build school/leadership capacity in scenario planning with current schools for revision of schedules and/or new schools for new master schedules Coordinate scenario planning around new courses and horizontal/vertical alignment

Work with school/district leadership to design master schedules developed in coordination with LDC-based courses and vertical/horizontal development

Continued support in logistics of new master schedules in select schools Support for district/school leadership in transition of resource allocation leadership

  38  

requirements

Year 3: Transition to District and School Leadership

LDC

Coaching to district and school leadership on module implementation, student scoring, and national submissions

Coaching for district/school leads for teacher module trainings for new teams, if any

Review and refinement to Common Core courses and vertical/horizontal alignment

Coaching to district and school leadership on module implementation, student scoring, and national submissions

Coaching for district/school leads for teacher module trainings for new teams, if any

Review and refinement to Common Core courses and vertical/horizontal alignment Support for district leaders on module implementation, student scoring and national submission

Coaching for district/school leads for teacher module trainings for new teams, if any

Support for district and school leaders as they lead logistics of new resource allocations/scheduling

Support for district and school leaders as they lead scenario planning—either refinements of past schools or new scheduling for schools new to the work

Support for district and school leaders as they lead master schedule development

Phase-out to district and school leadership

 In Years 4 and 5, succeeding teacher and school cohorts complete the Year 2/3 targets noted

above. These PD cycles—and LDC’s efforts to nationally disseminate the work as it emerges

and is codified—will be supported by district, LDC, and external partner consultants:

Learning Strands Staffing/ Resources

Identification

Central Office Guiding Committee: Stakeholder

Strategic Planning to Align Existing Instructional Initiatives

LDC Project Lead, Assistant Project Lead, CAO,

PD Partner LAUSD: Reach Associates

  39  

Orientation and Introduction to LDC: District Leadership, Union, School-based Leadership, Classroom Teachers

NYC: New Teacher Center Resource Optimization Review and Master Schedule Review

LDC Project Lead, Assistant Project Lead

Technical Consultant

Districts choice including potentially: Timewise, Targeted Leadership

Engagement and Understanding of LDC Implementation and Connection to Teacher Effectiveness

LDC Project Lead, Assistant Project Lead, CAO

PD Partner LAUSD: Reach Associates NYC: New Teacher Center

School Leadership Resource Optimization - Teacher Collaboration Time Review

LDC Assistant Project Lead Technical Consultant

District Choice: Timewise, Targeted Leadership

Teachers Engagement and Understanding of LDC Implementation and Connection to Teacher Effectiveness

LDC Project Lead, Assistant Project Lead, CAO

PD Partner LAUSD: Reach Associates NYC: New Teacher Center (Districts current tentative choices)

Ensuring High-Quality Disciplinary Content in LDC Modules

LDC Assistant Project Lead, CAO Content Experts SS: Stanford History Education

Group Science: American Museum of Natural History ELA: Author Eleanor Dougherty, NWP

LDC National Dissemination of i3 Codified Products

LDC Project Lead, CAO, Communications Director, COP Manager

 In Year 2, Cohort 2 teachers and school leadership would experience a similar training track

modified based on learning from the first year of implementation. In addition, a subset of

teachers at each school, identified as strong teachers, would come together in a summer institute,

(four from each school or 80 in total; ELA, Science, SS, Sped or ELL). These teachers would

engage in a weeklong intensive institute intended to support their ability to return to their schools

  40  

to help deepen the work as teacher leaders. The weeklong sessions would include LDC work,

both explicit and implicit (for example, leading others in adult learning, productive teacher teams

structures, protocols, etc.). These Cohort 1 teachers would also start the work of examining

CCSS standards vertically and horizontally as they connect to curriculum. This beginning

curriculum mapping and scope and sequence work would continue throughout the year as

teachers came together to build out yearlong course artifacts that may be adapted to different

school settings, have instruction tailored to high- and special-need children, have different texts,

etc., but generally include common assignments (for example, LDC modules) to anchor units. By

year's end, the goal would be a strong step toward yearlong scope and sequences by

grade/content populated with lesson and unit plans as appropriate for teachers in many more

schools to adapt/adopt. Year 3 continues to build out this deep instructional planning while

ensuring high-needs/special-needs students are supported by the CCSS curricular work.

Succeeding cohorts would follow this three-year sequence.

A description of many of the roles referenced above includes:

Roles Responsibilities Project Director Leads the partnership among the leaders of the NYC DOE and LAUSD

and the LDC organization. Responsible for overseeing professional services and tool development, ensuring the LDC i3 Initiative milestones are met, and allocating grant funding accordingly.

Software Developers Lead the development, refinement, and codification of current and new LDC CoreTools in collaboration with project director, LDC coaches, and representatives from each district.

LDC Trainers Lead the professional development and coaching of the NYC DOE and LAUSD. Trainers will lead the PD Annual Trainings and LDC Summer Institutes, working with teachers and transferring knowledge and expertise to Teacher Leaders.

Teacher Leaders Attend intensive LDC training sessions to gain expertise and serve as local coaches and experts for teachers in each school. Of 80 Teacher Leaders in each district, 20 teachers will devote 25 percent of their schedules to supporting teachers and LDC work.

Chief of Instruction The CID will use her deep knowledge of instructional design and teacher

  41  

and Design practice to help design and facilitate Annual Trainings, Summer Institutes, and Virtual Coaching. The CID will also oversee the creation and dissemination of all PD artifacts and teacher resources generated in the course of the grant.

Assistant Project Lead

Capture teacher processes, feedback, and best practices to support software developers with content to codify practices and support tool development. Additionally, facilitate the national jurying system to provide feedback to teachers, identify potential jurors, and expand the module exemplar and mini-task libraries.

Professional Development Partners

Reach Associates New Teacher Center American Museum of National History Stanford History Education Group

Evaluator The UCLA National Center for Research on Evaluation, Standards and Student Testing will design and implement the evaluation plan for the proposed intervention.

Financial Managers Leads the financial management and reporting for i3 funding, if granted. The LDC staff listed above (the “project team”) will evaluate the success or challenges of the

project and use that feedback to make improvements to the project, particularly through the data

LDC routinely collects from teachers using the LDC CoreTools. LDC analyzes trends and

statistics for internal purposes to measure the effectiveness of and to improve the supports

provided to teachers. Teachers and districts have access to data to improve teaching and

ultimately to help students meet Common Core expectations. Teacher profile information

supports demographic and other analysis. Profile information tracked includes grade level,

teacher’s teaching discipline, school, and LDC Professional Development Partner.

LDC also collects activity information from teachers as they plan, instruct, and reflect

upon their teaching practice. Rich information streams provide insights into teachers’

collaborative practices; instructional choices such as standards selected, template choices, and

student skills targeted; reflective improvements post instruction; and many other online

actions.40 This objective, formative evidence of what real teachers do will constantly be cycled

                                                                                                               40 Early in the 2014–2015 school year, LDC will expand data capture capabilities to include feedback provided to students and rubric-based scores. Teachers will choose which student products will be scored and can score both draft and final work

  42  

back into the LDC project team’s continuous improvement process for program modification and

improvement as the LDC i3 grant unfolds.

The project team will expand the project to the national or regional level by the end of the

grant by funneling all valid and codified resources, tools, artifacts, research findings, etc., into

LDC’s well-developed national community of practice. As noted above, over 40 partners in over

40 states support LDC as part of their teacher support work. The Communications Director

utilizes social media, partner pages, national and regional meetings, white papers, and other

mechanisms to communicate throughout the grant to our partners engaged in LDC

implementation around the country.

2. Multi-Year Financial and Operating Model

The LDC i3 Initiative’s financial and operational models are fully aligned to ensure the

implementation, support, expansion, and sustainability of LDC work in NYCDOE and LAUSD.

Financial resources directly map to each of the four primary LDC i3 work strands:

Work Strand

District Funding Supports

LDC Funding Supports

i3 Funding Supports Partner Resources

Leadership Support

• FTE District Coordinators to support all work

• LDC Project Lead (ProjD) (.5 FTE), CID (.15 FTE)

• Funding for PD partner work with Leadership including technical resource optimiza-tion audit

Teacher Training

• School Schedule Review ensures that PD activities are planned for existing teacher PD days and common planning time.

• Summer Institute

• Proj Lead, Asst Proj Lead, and CID

• Technology Supports and extensions

• PD Partners • Content Partners

                                                                                                                                                                                                                                                                                                                                                                     products. This student data will be directly connected to instruction, forming rich evidence of teacher practice and enabling tight feedback loops to teachers to help improve their practice.  

  43  

funding for 80 teachers annually

Teacher Leader Capacity Building

• Funding to reduce teaching schedule to release select teacher leaders to provide peer to peer PD support (20 teachers annually)

• Proj Lead, Asst Proj Lead, and CAO

• Technology Supports and extensions (e.g. online scope and sequence tool)

• PD Partners • Content Partners

National Dissemination

• LDC CoreTools, LDC Jurying

• Asst Proj Lead: codification of valid resources, practices, content

• Marketing and Social Media sharing

• Technology codification and extensions

• $1.55m budget for CRESST (Independent Evaluator)

The financial model also maps to the operational model in that it reflects an expectation that as

district, school, and teachers build their capacity to understand and implement LDC, that external

contractor coaching reduces and the district takes on that work to create sustainability post grant.

Cost estimates wherever possible were based on existing fees and salaries.41

3. Experience of Project Director

Chad Vignola, the LDC i3 Initiative’s project director, brings extensive knowledge and

expertise in leading and managing large-scale operations and bringing projects to scale (see

resume attached). After earning a B.S. from Wharton at UPenn and a J.D. from UPenn, and after

a successful legal career in local and federal roles, Chad served as the General Counsel and

Executive Director of the New York City Department of Education and a member of the

                                                                                                               41  For example, all except one LDC salary is based on current salaries, district salaries were based on instructional leaders on commensurate lines currently, CRESST’s evaluation costs are based on current billing rates, the PD partner costs were based on current billing rates, the technology estimates were based on current billing rates and estimated work hours against scope, etc.  

  44  

Chancellor’s leadership team from 1999–2004. Supporting the Department’s system-wide

strategic planning and implementation of all major policy initiatives, Chad was directly

responsible for a $24 million budget and for all legal affairs (55 staff), labor relations (25 staff),

Auditor General (45 staff and $2 million Deloitte contract), Special Investigations (17 staff), and

Office of Equal Opportunity (7 staff). During this time, Chad completed Ed Leadership

coursework toward an Ed.D. from New York University. Shortly thereafter, Chad became the

Vice President for the Princeton Review, where he managed the nationwide implementation of

$35 million in K-12 services (formative assessment platform, PD, AIS). He later became the

Vice President at New Visions for Public Schools, where he led a staff of nearly 40 experienced

educators in overseeing 75 NYC public schools, providing leadership development, college

readiness services, professional development, SIG implementations, and other school

improvement. During the last two years of Chad’s tenure, he implemented both a new,

transformational teacher effectiveness strategy (in partnership with The New Teacher Project)

and CCSS, including a 14-school LDC pilot. Over 80 percent of the school’s under Chad's aegis

had the most significant historical improvement in student outcomes for high-needs children

during his tenure.42

Now, as the LDC founder and CEO, Chad led the development of a five-year strategic

business planning process to expand LDC services and tools to scale LDC nationally, securing

an initial $13 million Gates Foundation multi-year commitment. As the CEO, Chad leads a core

team of 12 full-time staff and technology consultants and coordinates over 40 national partners

who are involved in LDC, providing technical assistance, supporting tool development, and

providing a host of resources to enable national scaling. LDC’s i3 plan clearly would fully

                                                                                                               42 Most of the 75 schools were Bronx and central Brooklyn high schools, with generally 100% minority populations and significantly high FRL (average 80 percent) and SPED/ELL populations (average 24 percent).

  45  

support the organization’s overarching strategic plan to nationally scale effective CCSS

resources, tools, and processes. With his extensive expertise in managing large organizations and

extensive partnerships, Chad is well situated to successfully lead and manage LDC’s i3 work to

ensure the tools and services continue to positively impact student learning as they begin to scale

to more schools, teachers, and students.

D. PROJECT EVALUATION: LDC’s i3 application will be evaluated by an independent third

party, UCLA’s Center for Research on Evaluation, Standards and Student Testing (CRESST).

CRESST brings to the effort strong capacity and history in rigorous qualitative and quantitative

methodologies and wide experience in evaluating and supporting the improvement of state,

district, and local programs. Dr. Joan Herman, Principal Investigator (PI), CRESST Co-Director

Emeritus, will lead the research and is a nationally known expert on educational assessment and

evaluation. Project Director Dr. Jia Wang has over a decade of experience in educational

evaluation specializing in statewide research design and methodology.

The proposed study will be a comprehensive mixed-method evaluation to understand the

impact of LDC on teacher effectiveness and student learning using a quasi-experimental

design.43 Aligned to the LDC implementation cycle, CRESST’s analyses in Years 1 and 2 will be

primarily descriptive, while the accumulated data in Years 3 and 4 will allow for more advanced

statistical analyses to compare the performance of LDC students on a range of indicators to that

of propensity matched comparison students.

During Year 1, the evaluators will provide an updated, comprehensive evaluation plan

within 100 days of the award, as required by the Department; work with LDC staff to develop,

                                                                                                               43 A copy of the complete 35-page CRESST evaluation plan is in the Appendix. This is excerpted from that plan.

  46  

refine, and augment study instruments; initiate and build relationships with participating districts

and schools; and survey school site coordinators and teachers on the readiness of the schools and

teachers, both logistically and knowledge-wise, for the full implementation of LDC in Year 2, in

addition to the challenges and obstacles they have or expect to encounter. Through teacher

survey, CRESST will also try to establish a baseline measure of teachers’ classroom practices via

survey questions and skills by asking for lesson plan examples on one of the few core

standards/topics that will be part of the LDC module development for later years.

The Year 1 report will focus on describing the program characteristics of the participating

schools and teachers, and the findings based on the survey data collected from the school site

coordinator and teachers. CRESST may also run some descriptive analyses on the evaluation

data from LDC on professional development, coaching, and other supports teachers received.

CRESST’s full-scale evaluation work (for example, site coordinator survey, teacher

survey/log, teacher-developed modules, CRESST-developed student assessment data, district-

wide student assessments) will be implemented in Years 2 and 3, during which CRESST will

collect and analyze all data sources to answer the evaluation questions proposed below. Both

qualitative and quantitative analyses will be conducted. Quantitative analyses will draw on

statistically sophisticated propensity matching and multi-level analysis techniques to compare the

performance of LDC students on a range of indicators to that of propensity matched comparison

students. The annual report for Year 2 will focus on presenting findings on all outcomes except

district-collected student outcome data as it typically is not available until September/October

after the spring testing. Reports, Years 3 and 4, will provide preliminary data analysis with the

final report attempting to answer all the proposed evaluation questions by analyzing the last

  47  

wave of student achievement data to triangulate results from all sources for the final report.

CRESST is confident about its approach because of its success in earlier LDC evaluation studies.  

Evaluation Questions

Evaluation questions, as noted above, focus on three areas: program characteristics and

implementation, contextual factors and implementation, and program impacts.

I. Program Characteristics and Implementation

a. Who are the participating teachers and schools? Are they representative of the teacher/school populations of the respective district on years of teaching, education level, etc.?

b. How well do teachers, school administrators, and district administrators work with LDC and its PD partners?

c. What professional development opportunities are offered and utilized by teachers at each school/district that build teacher capacity? Are teachers and schools satisfied with the professional development opportunities they received?

d. What and how do teachers implement the LDC tools in the classrooms? What is the

alignment between teacher practices and LDC practices?

e. How often do teachers access the LDC support system? Which are the most frequently and least-used features? Does this vary by a teacher’s prior experience, by subject area, etc.?

f. How do teachers feel about the LDC tools? Do teachers perceive them as useful and having a

positive influence on their learning and effectiveness? Do teachers perceive them as having a positive influence on their students’ learning?

g. To what extent are LDC’s core strategies being realized? What are primary tactics for

achieving them? What are apparent facilitators and inhibitors to implementation? What is needed to improve the implementation at the teacher, school, and district level?

h. Will teachers/schools continue their LDC-influenced practice after the LDC support ends? What contributes to the continuation and abandonment?

II. Contextual Factors and Implementation

a. What other educational reforms are being implemented in the participating schools and

districts? What are their influences on the LDC adoption in the schools and districts? Are schools able to align reform efforts?

b. What are the roles of school and district leadership in shaping the LDC implementation?

  48  

c. What do teachers and administrators perceive as the factors that contribute to successful

LDC implementation? What are challenges and obstacles to a successful LDC implementation?

III. Program Impacts

a. What is the impact of LDC on the academic performance of participating middle and/or

high school students as measured by the state/district assessments?

b. Do the academic impacts vary by student subgroup including race, ethnicity, socio-economic status, gender, and/or disability? Does LDC help close the achievement gap between student subgroups?

c. What is the impact of LDC on the teacher learning as measured by the quality of LDC

modules they produce, compared to the lesson plans they previously used? What is the impact of LDC on teacher learning as measured by teacher self-reported survey questions?

d. What is the relationship between the fidelity of LDC implementation with teacher and student learning, and the conditions and contexts under which the LDC tool use is most effective?

e. What are some of the reported changes in teacher practices after LDC intervention? f. Are participating schools and teachers planning to continue their LDC-influenced practice

after the LDC support ends? What contributes to their decision to continue or to abandon?

Evaluation Design

Guided by the primary evaluation goal to validate LDC impact on teacher and student

learning, CRESST’s design and sampling for treatment-control comparisons is as follows: The

treatment group also will be the target for implementation measures. The treatment group will

consist of at least 50 middle and/or high schools and their teachers and students in at least two

districts (NYC/LAUSD). CRESST will work with LDC and the participating districts to identify

a suitable comparison group. The comparison group would be teachers/schools implementing an

alternative CCSS implementation in science, social studies, and/or English Language Arts.

CRESST will seek to convince districts to randomly assign schools to treatment and

comparison groups to enable a true experimental design—either because there are more schools

  49  

willing to participate that can be accommodated by the i3 grant and/or districts are willing to

mandate involvement. If districts and schools are amenable, each district could have a stratified

random sampling plan to assign schools—and within them, teachers and students—to the two

conditions (LDC treatment/control).

However, CRESST’S experience suggests that they likely will need to use a quasi-

experimental design, with propensity score matching techniques used to compose a comparison

group that is matched on core school, teacher, and student characteristics. Propensity scoring is

used to make causal inferences about a treatment in the absence of random assignment. It is a

statistically-based weighting method used to minimize the effects of existing differences in

student background characteristics. First, propensity scores are computed from a large reservoir

of potential controls by applying a systematic weighting procedure. Then these scores are

estimated to account for potential differences in student background characteristics. In other

words, the propensity score is the conditional probability of being assigned to the treatment

condition given a set of observed covariates. It is commonly estimated using a logistic link

function or using ordinal logistic regression.44 Matching helps ensure the comparisons are among

students with similar prior demographic background and schooling experiences.

Participating Districts: Sample Size and Existing District Surveys

Consistent with the What Works Clearinghouse Standards, CRESST’s study aims to

conduct the evaluation using a large sample with at least 50 or more LDC teachers for each

cohort, each teacher having 25 or more students, and a corresponding group of comparison

teachers and students. (Likely sample pool much higher.)

                                                                                                               44 Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70 (1), 41-55.

  50  

CRESST’s study would have multiple cohorts of teachers and students with one cohort

starting in 2014–2015, the second cohort staring in 2015–2016, etc. Per the LDC implementation

plan, LDC would start the introduction to Cohort 1 LDC teachers/schools in 2014–2015, have

the schools do the vertical and horizontal alignment and teacher full implementation in 2015–

2016, and have teachers continue to conduct full implementation in 2016–2017 and succeeding

years while LDC’s PD partners transition out of schools. Cohort 2 LDC teachers and schools

would start in 2015–2016 and finish their second-year in 2016–2017, etc. Starting with Cohort 1

teachers/schools, CRESST would assess any growth that occurred in student and teacher learning

during Years 2 and 3 and so on.

The New York City Department of Education (NYCDOE) is the largest public school

system in the nation, serving about 1.1 million students in almost 1,800 schools in 2013–2014

(475 schools serving middle school grade students and 369 schools serving high school grade

students).45 The Los Angeles Unified School District (LAUSD) enrolls more than 640,000

students and has 76 middle schools and 68 high schools.46

CRESST Instruments and Data

For the proposed evaluation study, the CRESST team will create school site surveys,

update existing teacher implementation measures, and update LDC module quality rubrics.

School site implementation. CRESST will specially create a school site implementation

survey to be completed by school site coordinators. The survey will explore in depth how core

strategies (professional development and collaboration, support network, leadership support, and

vertical and horizontal alignment of CCSS standards) are being implemented and the nature and

                                                                                                               45 http://schools.nyc.gov/AboutUs/default.htm 46 http://search.lausd.k12.ca.us/cgi-bin/fccgi.exe

  51  

perceived effectiveness of specific strategies. Questions on program implementation challenges

and obstacles during the multi-year support period will be included in the survey also.

Teacher implementation. The existing “fidelity-of-the-LDC-implementation” survey

developed for CRESST’s current LDC evaluation studies funded by the Gates Foundation

provides tools for both research and for improving teacher classroom practice. For this research,

CRESST will collect teacher logs/survey data to capture variables such as time spent, use of

assessment, feedback provided to students, availability and use of implementation support,

drawing on protocols used in prior CRESST studies. 47 The derived contextual variables

(leadership support, ownership, learning community involvement, etc.) will be used to examine

relationships between implementation and impact on teacher and student learning. These

measures will be refined during the first year of the proposed work.

Teacher Module Quality Rubrics. CRESST’s existing rubrics were designed to examine

the rigor and content literacy present in the teacher module materials (for example, template task,

supplemental reading and writing materials, student work samples, and descriptions of the pacing

and goals of the modules). The nine dimensions examined were each scored on a scale of 1-5 and

covered the following themes: effective writing; alignment to literacy and content standards, text

alignment, appropriateness, and rigor; fidelity to LDC module instruction; quality instructional

strategies, coherence and clarity of module; and, overall impression. The modules were scored

by specially trained teachers in the relevant subject area—history-social studies, science, and/or

English Language Arts. Generalizability, factor analysis, and decision study methodology were

used to evaluate the measurement quality of the scores.48

                                                                                                               47 Herman, J. L., Rickles, J., Hansen, M., Thomas, L., Gualpa, A., & Wang, J. (2011). Evaluation of Green Dot’s Locke Transformation Project: Findings from Cohort 1 and 2 Students. Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST). 48 Reisman, A., Herman, J., Luskin, R., and Epstein, S. (2013). Summary report: Developing an assignment measure to assess quality of LDC modules. Los Angeles: National Center for Research on Evaluation, Standards, and Student Testing.

  52  

The existing rubrics will be refined during the first year of the proposed work, along with

the LDC staff and district staff. During the scoring session, we will resolve any coding questions

through consensus.49 Approximately 35 percent of materials will be double coded by multiple

members to provide a coder reliability check.

LDC Instruments and Data

CRESST may explore the data generated by teachers’ online user experience in LDC

CoreTools which may be inferential formative data about teacher instructional choices,

collaborative experiences, and what supports teacher-effective CCSS implementation. In

addition, LDC CoreTools captures teacher profile information to support demographic and other

analysis. Profile information tracked includes grade level, teacher’s teaching discipline, school,

and their LDC Professional Development Partner.

Available Data at the Participating Districts

School districts routinely collect background and school outcome data on their students,

and background and evaluation data on their teachers. Both LAUSD and NYC also conduct

annual surveys of their students, teachers, and parents. CRESST may seek to leverage the

existing survey processes to lessen teacher/student burdens by adding a small set of questions

about teachers' comfort in implementing CCSS.

Teacher Data. CRESST will analyze the available LDC and district data on teacher

demographics, experience, education background, retention, annual evaluation results, etc.

Student Data. Student background and demographic variables collected by the school

districts typically include race/ethnicity, gender, eligibility to receive free or reduced-fee lunch,

                                                                                                               49 Carlson, N. M., & McCaslin, M. (2003). Meta-inquiry: An approach to interview success. The Qualitative Report, 8(4): 549-569.

  53  

special education status, etc. Districts also track students in their school attendance, course-

taking and course-completion information, interim test scores when appropriate, and test scores

on the annual district/state tests. Starting 2015–2016, the district/state tests could be English

Language Arts and mathematics tests developed by the Smarter Balanced Assessment

Consortium or Partnership for Assessment of Readiness for College and Careers.

Analysis Strategies

Multiple analytic procedures, including both quantitative and qualitative analytic

methodologies, will be applied to the data to discern potential changes in student and teacher

learning and to answer the proposed evaluation questions. The following paragraphs describe the

specific strategy that will be used to answer our three groups of evaluation questions.

Evaluation Questions I and II (“Program Characteristics and Implementation” and

“Contextual Factors and Implementation”). Descriptive analyses will examine the distribution

of schools, teachers, and student characteristics in the LDC and comparison sites. The descriptive

analyses will document differences in teacher experience, student demographics, and prior

academic performance relative to students in similar comparison schools. Analyses will be

disaggregated for student subgroups. Existing district survey data will also be analyzed to

capture the school differences reported by students, teachers, and parents. Close-ended survey

items, as well as checklists and ratings, will be analyzed using descriptive statistics such as

means, frequencies, and/or percentages. Standard qualitative methodologies will be applied in

the analysis of the open-ended survey items.

Evaluation Questions III (“Program Impacts”). Although specific impact analysis

strategies will be influenced by the final research design and sampling frame decisions that will

  54  

be made upon the award, a general set of steps will be undertaken to address the evaluation

questions on student learning:

Baseline treatment and comparison group equivalence. If non-equivalence between

treatment and comparison group equivalence is found, CRESST will include the variables on

which statistically significant differences are found in their subsequent analyses to help adjust for

baseline differences. CRESST also will examine descriptive statistics before fitting the data to

hierarchical models (HMs) and will calculate bivariate correlation coefficients for outcomes,

implementation, and all the student characteristics. These preliminary results will provide useful

information for specifying HMs and may also suggest qualitative analyses.

Descriptive analyses to assess impact on teacher learning. Based on the teacher-

developed instructional module, CRESST will evaluate, generate, and analyze the teacher

module quality indicators to track teacher growth in designing high-quality module. CRESST

will also analyze treatment teachers’ national jurying submissions for quality.

Hierarchical modeling analyses to assess impact on student learning. Depending on

the final design and sample, CRESST’s plan will employ a three-level hierarchical model—

students nested within teacher, teachers nested within schools—to estimate program effect on the

outcomes of interest: student performance on state/district assessment results. The use of HLM

solves potential problems of misleadingly small standard errors for treatment effect estimates and

the failure to detect between-site heterogeneity in program effects.50

Estimate of the effect of fidelity of implementation. CRESST will examine the

reliability, dimensionality, and validity of implementation measures based on teacher

                                                                                                               50 For details see Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. 2nd edition. Newbury Park, CA: Sage; Seltzer, M. (2004). The use of hierarchical models in analyzing data from experiments and quasi-experiments conducted in field settings. In D. Kaplan (Ed.), The Handbook of Quantitative Methods for the Social Sciences (pp. 259-280). Thousand Oaks, CA: Sage Publications; and Snijders, Tom A.B., Bosker, & Roel, J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London etc.: Sage Publishers.

  55  

surveys/logs using factor analysis and decision study methodology. These analysis results will

then be used to create reliable measures of fidelity of implementation. Incorporating this

information, CRESST will use HM to estimate the effects of different levels of treatment

implementation (for example, how much of an increase in effectiveness can be expected given a

high level of implementation?). These analyses also will include other school and teacher

characteristics collected at the school level such as the percentage of free/reduced price lunch

eligible students, demographic composition, leadership support, ownership, quality of learning

community, etc., to examine their effects on implementation and impact.

Proposed Evaluation Timeline

The proposed evaluation covers four years and six months for a total of 54 months.

YEAR 1 YEAR 2 YEAR 3 YEAR 4 YEAR 5

Submit UCLA IRB application

Collect baseline data (2014–2015) in all Cohort 1 participating districts

Collect Year 2 student outcome data (2015–2016) in all Cohort 1 participating districts

Collect Year 3 student outcome data (2016–2017) in all Cohort 2 participating districts

Analyze all prior year student outcome data for all cohorts

Work with LDC to get school districts' approval to access existing district data on students and teachers

Work with LDC and school districts to administer the evaluation instruments

Collect baseline data (2015–2016) in all Cohort 2 participating districts

Collect Year 4 student outcome data (2016–2017) in all Cohort 1 participating districts

Prepare final report

Finalize design and sampling plan

Data collection, entry, coding, and scoring

Work with LDC and school districts to administer the evaluation instruments

Work with LDC and school districts to administer the evaluation instruments

Refine evaluation study plan and submit the revised plan to U.S. Department of Education

Data analysis Data collection, entry, coding, and scoring

Data collection, entry, coding, and scoring

Develop, adapt, and administer various instruments

Prepare annual report

Data analysis Data analysis

Work with LDC and school districts to administer the evaluation instruments

Prepare annual report Prepare annual report

Data collection, entry, coding, and scoring

  56  

Data analysis

Prepare annual report