Melda N. YildizUsing Computer Generated Data Analysis to Drive Classroom Instruction Dec 6, 2007

Please watch the letters from Teddy http://www.makeadifferencemovie.com/

Vocabulary average of a 14-year-old dropped from 25,000 words in 1950s to only 10,000 words in 1999.

“Numbers.” Time Magazine 155, no 6 (Feb 14, 2000); 25

Vocabulary Average for 14-Year-Old

25,000

10,000

0

5,000

10,000

15,000

20,000

25,000

30,000

Year

Nu

mb

er o

f Vo

cab

ula

ry

VocabularyRate

VocabularyRate

25,000 10,000

1950 1999

Statistics

In political Washington, Statistics are weapons of war. That’s why they get manipulated, massaged, and twisted until any connection to reality is strictly coincidental.

Peter Carlson

CNN.com posted misleading graph showing poll results on Schiavo case

http://mediamatters.org/items/200503220005

The Truth but not the Whole Truth

0102030405060708090

100

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

EastWestNorth

020406080

100

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

EastWestNorth

0102030405060708090

100

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

EastWestNorth

What is data-driven decision making? DDDM is the process of making choices

based on appropriate analysis of relevant information. School district decision makers are using technology and professional expertise to improve instruction and operations.

Why use data for decision making in K-12 education? Decisions in school districts have been made

according to tradition, instinct, and regulations. More access to better information enables educational professionals to test their assumptions, identify needs, and measure outcomes. Schools are using data-driven decision making to provide more individualized instruction to students, track professional development resources, identify successful instructional strategies, better allocate scarce resources, and communicate better with parents and the community.

How does a district decide what data to collect?

Most districts are data rich. They have too much information in too many places to effectively use it. They have information about student records, student assessment, transportation services, food service, human resources, library automation, student health, special education, and curriculum management, to name a few. The challenge is to integrate these disparate systems and make the information available in timely, easy-to-understand reports so that decision makers can affect student performance.

What is a data warehouse?

A data warehouse is a storage facility integrating sources of vital information about every student and staff member in the school system. Providing easy access to this data is a crucial element of a data warehousing solution. At the same time, much of the information is highly confidential. Finding the right balance between access and security, flexibility and control, is an ongoing challenge for K-12 IT departments.

Ways in which data reports can be structured Reports need to be timely, tied to objectives, and available

to people with the responsibility and ability to act on them. Data reports that show data in different ways such as

tables, charts, graphs, and trends enable more people to access and understand the information. Most districts create a standard set of reports based on the key questions and indicators identified in the planning process.

If possible reports should include longitudinal data so that teachers, principals, and administrators can compare results over time.

What common data report formats are most useful to teachers? Web-based systems enable teachers to log-in and view a

class or drill down to a student profile or flexible groupings of students. They can view assessment results tied to standards and assessment items.

Teachers filter by period, course or NCLB filters such as ethnicity, gender, or second language learners.

They have access to current and historical data as well as contact information for student, parents, and email links to other teachers. One district enables teachers to export contact information for mail merges.

What common data report formats are most useful to principals? Principals use data on attendance, enrollment,

student/teacher/parent satisfaction surveys, and test results to assess progress, allocate resources, and create school improvement plans.

They look for information that is organized numerically rather than alphabetically, includes objective descriptions of data, visual displays of information, and query tools.

Common data report formats are most useful to district personnel District personnel use data to report results to federal and state agencies,

most notably NCLB and state assessments. Data also helps district personnel determine the appropriate allocation of

district resources, plan professional development, analyze district level interventions to achieve desired results, create school improvement plans, and assess the overall progress toward strategic goals.

To use data more effectively in decision making, district personnel need access to data across information systems, for example: linking financial data with student assessments helps to refine resource allocations, connecting human resources with student assessment helps identify professional development needs, etc. The data must be available in both aggregate and disaggregate formats, allowing administrators to drill down by school, department, classroom, student demographics, etc.

Why is data-driven decision making so important to No Child Left Behind? With the right data at the right time to inform decisions about

resources, grouping, and instruction, schools are more likely to meet their Adequate Yearly Progress (AYP) requirements and comply with NCLB.

The first years of No Child Left Behind (NCLB) required school districts to collect more data, in more detail and disaggregate it to show the progress toward achieving state standards. If teachers and administrators are going to be able to keep students from falling behind, they need to know what’s working and what students are learning during the year.

What data is being collected by states and districts? The State Educational Technology Directors

Association (SETDA) created a set of data elements to help state education departments meet the data reporting requirements of NCLB and to generate comparative national data. The data elements are divided into sections based on NCLB requirements. Each section contains key questions, indicators of the answer, and data elements that can be collected to measure the results. (See the Data Collection Project at www.setda.org.)

Major barriers to effective use of data in decision making for school districts Lack of training and interoperability are the main barriers to more

effective data-driven decision making, according to a survey conducted by Grunwald & Associates on behalf of CoSN in 2004. Lack of training: 50% Interoperability—systems that are unable to share or exchange data: 42% Lack of understanding of what to do with the data: 39% Absence of clear prioriies on what data should be collected: 36% Failure to collect data in a uniform manner: 35% Outdated technology/legacy systems: 31% Low quality data – inaccurate or incomplete: 24% Timing of data collection: 24% User interface is too complicated to understand reports: 22%

Major misconceptions about effective use of data in decision making Build it and they will use it. It is not enough to make data available. The district

has to have a process in place for analyzing the information and getting it to the right decision maker at the right time with the power and resources to act on it.

Teachers need to know how to analyze data and query systems. Teachers want to teach not crunch numbers. Districts that have successfully

implemented DDDM in the classroom provide teachers with on-site support, timely reports, analytic tools, and planning teams.

Test scores determine the quality of a school and a child’s education. Many factors contribute to the success or failure of a student. Emphasis on test scores can give the community the wrong impression about a school. It is up to the superintendent and principals to frame the discussion so that parents and community members understand how well schools are doing and what they need to do to improve.

What is necessary for the systematic use of data for decision making? The district strategic planning process provides

the framework for data-driven decision making. The district data warehouse is designed to aggregate and disaggregate data needed for the planning process. An interface is developed to give different members of the educational organization access to the reports and data related to their work.

Data-driven decision making can be divided into three functional areas: collection, integration and dissemination of data;

analysis and reporting of data, and; process and procedures for acting on the data.

Once the system is in place, a process is developed for review, analysis, and planning. School teams meet with a cross-functional district team to define requirements for disaggregation and determine interim measures.

Experts from the district planning and evaluation division meet with the area superintendent and members of the school planning team to discuss the specific data and help school teams understand it within the context of their school. The district sets benchmarks to help area superintendents and principals set goals and meet expectations.

How long does it take for a district to establish a process? Districts using data-driven decision making estimate that they spend at least one

year planning the system and developing community support for it. Building the system and rolling it out will take at least two years. Veteran planners recommend aligning the process with a major initiative that the stakeholders have control over and responsibility for, such as recruitment, achievement, or enrollment. Some risk should be required in order to prove to skeptics that the process works.

Data-driven decision making is an on-going process rather than a one-time project. District staff members need to be open and honest about results and have the freedom and responsibility to test and try new strategies for improvement. Legitimate concerns about assessment tools, data, and curriculum should be acknowledged and addressed as the district refines the process. The result is a common understanding of what goes into the aggregate data and a process for helping each student meet the same standard for success.

Type of technology is needed to implement systemic data processes The hardware may include secure servers for storage and computing

devices for input and output, and a secure network to store and access data. As computing devices evolve and develop, more options with increased mobility, security, and lower cost will most likely be available.

To manage information about every student over time requires sophisticated data warehouse systems with integrated student information and assessment systems. At their most comprehensive, these integrated systems combine standards-based instructional resources with multiple assessment tools, data management and analysis systems, and professional development. Districts must also adopt methods for authenticating and validating data, safeguards and security to comply with privacy legislation and protect data, and business continuity plans in case of loss or system failure.

Types of skills are needed to implement systemic data processes Perhaps the most important part of data-driven

decision making is enabling decision makers to use it. Colorful reports and expensive assessment packages will have no effect unless they are combined with leadership and effective professional development.

The district needs both organizational and individual capacity for improvement. Administrators need training in continuous improvement

processes and the opportunity to share ideas with peers to learn how to ask the right questions.

Faculty and staff members need training to learn how to read data and apply it to their goals and objectives.

Instructors need training in different instructional strategies to apply when the data shows that traditional methods are not working.

Many districts have created staff positions within the district or at the school site to provide analysis and training. The hands-on support helps decision makers become more sophisticated in their use of data, and as analytic and instructional tools come online, they are ready to use them.

Key decision makers at the district level should be involved in the DDDS process? The superintendent sets priorities and leads the effort by setting measurable,

realistic goals for using data. The IT department is responsible for managing the technology infrastructure, coordinating system planning and development, and providing access tools. Many districts have established research and assessment divisions to oversee testing, reporting, and evaluation. This group provides analysis to help principals and teachers use the data and develop assessment tools that are aligned to standards. Curriculum and instruction administrators often provide the training necessary for teachers and principals to use data reports for intervention and planning.

Involving classroom teachers during the design and testing of systems builds support from the people who will use it to reach students. How much analysis teachers do themselves depends on the availability of tools, support, and training.

Who are the key decision makers at the school level who should be involved in the data-driven decision making process?

Principals are the change agents at the school site. Without their commitment, it will be difficult for data to become an integral part of instruction.

Principals model data use and encourage it by sharing the benefits and successes. They help teachers become data-driven decision makers by scheduling time for teams to meet, plan, train, and conduct evaluation.

Site-based specialists or support teams assist principals and teachers with data mining and analysis. They may have special expertise or training to query the data systems and produce reports needed to inform decisions.

Who are the key decision makers at the classroom level who should be involved in the data-driven decision making process? In addition to using data for determining

instruction, teachers can engage students in the decision making process by helping them view appropriate reports, set learning goals, and make decisions about how to meet their goals.

How are assessment data used for decision making?

Assessments used during the year may be formative assessments used for instructional interventions or benchmark assessments to determine progress against an external measure. It is essential for teachers, principals and others to know what kind of assessment they are using and the proper method of analysis based on the reliability and validity of the measure.

How can teachers use assessment data to inform instruction? Benchmark assessments tied to state and district standards

provide quick snapshots of where students are with regard to the progress they are expected to make. Interventions used by educators include reemphasizing skills, utilizing additional diagnostics to get at the root cause, changing instructional materials, and creating cohort groups within schools and classrooms of students who have a similar achievement gap or pattern to apply instructional strategies.

Although teachers have always used tests and quizzes to track student progress, these measures did not necessarily relate to standards or the assessment systems did not provide results in a timely manner.

How does a district keep momentum to change going? Planning for DDDM can become an all-consuming

process with perpetual refinement of processes and adjustment to the technology. At some point, district leaders need to choose an area for improvement and begin the process. Staff members will have different levels of experience and interest in changing their practice to incorporate data. Leaders need to choose specific areas and celebrate success to keep the momentum toward long-term change.

How can you use this data to guide instruction?

Student data on a specific item can be valuable to teachers. Inferences based on the data can be used to guide classroom instruction.

Teachers might want to explore the following questions: What core learning goal indicator is this item testing? Is this indicator included in the curriculum in my local school system? To what extent is this indicator being taught? To what extent have I assessed this indicator? How do the results of my classroom assessment correlate with the field

test? How familiar are the students with the rubric used to score performance (for

constructed responses items only)? What common errors do you see in the way students respond? What do the distractors tell you about instructional needs?

http://www.kuhrs.com/Files/Final%20FOS%20Brochure.pdf

"Data helps you make changes. And when you see data, it really puts [student achievement] right in your face." —Virginia Lawton, 6th-grade teacher in Wisconsin

http://www.3d2know.org

Data-Driven Instruction

3D2Know: Data-Driven Decision MakingCoSN launched the Data-driven Decision Making Initiative: Vision to Know and Do building upon its role in providing key K–12 school district managers with the knowledge and skills necessary for effective leadership.

http://www2.edc.org/asap/

Getting Started with Assessment

"Effective educators make effective decisions, decisions based on accurate information. If knowledge is power, then studying the current abilities, skills, attitudes, and learning styles of students empowers educators to adjust the curriculum to achieve whatever goals the school and district have chosen.

When educators study their schools and classes, they seek an answer to an ageless question: Is it good because we've been doing it for a long time, or is it good because we have tangible evidence of its worth? In many instances one must conclude the former because no evidence exists to support the latter." James Johnson

Assessment of learning vs. Assessment for learning Data can be used not only to evaluate and

track student performance but also to assess instructional effectiveness and various other factors that influence student learning.

Data Quality Campaign (DQC)A national, collaborative effort to encourage

and support state policymakers to improve the collection, availability and use of high-quality education data and implement state longitudinal data systems to improve student achievement.

http://www.dataqualitycampaign.org

Buried Treasure: Developing a Management Guide From Mountains of School DataThis report (in PDF format) provides a practical

discussion of what is required to develop a school district "management guide," along with an actual guide built on evidence-based indicators.

http://www.crpe.org/pubs/pdf/BuriedTreasure_celio.pdf

http://www.wcboe.org/teachers/cmolnar/data_driven_instruction.htm When I first started teaching, I would teach a

lesson and assess my students to see if they had learned the objectives of the lesson.  If they did, then I moved on.  If they did not then I had to re-teach.  That was the fledgling beginnings of data driven instruction.  Using the data to make decisions about what is best for our students is what we are expected to do with increasing precision.   

No longer can I look at my class as a whole and say, "Did they get it?"  I have to look at each individual student and say, "What are his/her areas of strength and weakness?"  "How can I improve those weaknesses?"  "How can I use those strengths to the child's advantage?"  I need to have data to support every lesson I teach.  If I am re-teaching something, I should have data that supports that decision.  When you stop to think about it, that is a daunting task. 

For those elementary school teachers who have 20 - 30 students and teach them all subjects, and they are expected to know each child's needs in each subject.  Or for the middle school or high school teacher who could possibly have 170 or more  students and needs to know all of their specific needs.  How to organize that information and what to do with it once it has been organized is overwhelming!  One of the problems that educators face in this task is that there are many software programs that assist in organizing such information. 

However, they, are not compatible, or do not all work together, which can create more problems than the program was intended to alleviate.  In addition once a system of data organization is in place, all stake holders do not have access to all information which can make the teacher's job frustrating.  I only teach one subject and this year alone I have already created many different spreadsheets using Excel.  Excel has been a lifesaver for me for organizing and analyzing data. 

NCREL: School Improvement Through Data-Driven Decision MakingDesigned to give educators—and others

involved in using data in a classroom, school, or district—a variety of places to find resources, tools, and action steps to foster school improvement.

http://www.ncrel.org/datause/ http://www.ncrel.org/datause/howto.php

It has allowed me to organize the data for each student and their test scores.  I have additional columns for the objectives that I will be teaching so that when I have that data, I can add it to the snapshot I am creating of each individual student.  I can then use that snapshot to assess students' needs.  I can create groups of students based upon their individual needs.  Hopefully with this data at my disposal, I can increase my students' learning.   

http://www.ncrel.org/

Statewide Longitudinal Data Systems Grant ProgramThis website acts as a resource for grantee and

non-grantee states regarding the grant program, and the development of longitudinal data systems in general.

http://165.224.221.98/Programs/SLDS/index.asp

Guide to Using Data in School Improvement EffortsA Compilation of Knowledge From Data

Retreats and Data Use at Learning Point Associates

December 2004 by Learning Point Associates

http://www.learningpt.org/pdfs/datause/guidebook.pdf

More…

http://www.success.co.il/is/dik.html

http://www.fcrr.org/science/pdf/kosanovich/jrf_leadership.pdf

Spreadsheet Project

Some sample projects that students can complete using spreadsheets:

Collect, analyze and graph lab science projects. Setting up word problems (formula) and verifying answers. Collect coin toss results for demonstrating the laws of probability. Tracking stock market. Analyzing election data from the electoral college. Predict prices given the Consumer Price Index. Prepare a Profit and Loss statement for a mythical company. Prepare a home budget. Perform a comparative shopping demonstration. Keep a check register. Perform compound interest calculations. Perform statistical calculations (correlations, standard deviations)

Infusing Spreadsheets Spreadsheets can be used in an amazing number

of ways, in virtually every subject area. This page and the next will provide you with some ideas.

http://www.internet4classrooms.com/on-line_excel.htm North Carolina Education Lesson Plans

http://www.dpi.state.nc.us/ A spreadsheet lesson plan for the appreciation for

children's literature.http://volweb.utk.edu/Schools/tdalton/lesson2.html

TeacherVision Lesson Plans

Weather Station (http://www.teachervision.com/lesson-plans/lesson-331.html)

M & M Math (http://www.teachervision.com/lesson-plans/lesson-1.html)

Star Magnitude Graphing (http://www.teachervision.com/lesson-plans/lesson-1530.html)

Country of Origin Comparison (http://www.teachervision.com/lesson-plans/lesson-696.html)

http://www.ncpublicschools.org/mentoring_novice_teachers/Module1/Math/MathHandouts.pdf

Suggested Titles: