Download - Data Informed Decision Making
1
Data Informed Decision Making
Dr. Robert Rodosky
Dr. Marco Muñoz
Jefferson County Public Schools (Kentucky)
Presented at the National Evaluation Institute
October 4-6, 2007
2
Presentation Overview
• Wallace Foundation• Leadership Interest Groups (LIG) • Data-Driven Decision Making • Georgia, Kentucky, Michigan, New
Mexico, and Ohio• In the fall of 2006, all principals in
Kentucky were surveyed regarding data access to support their roles as instructional leaders.
3
Legislative Pressures• Demands of No Child Left Behind (NCLB)
legislation—meeting AYPs.• Demands of KERA—meeting goals.• Assessment and accountability that
require knowledge and skills in measurement and data use for school improvement.
• Principals must use data in order to inform their decision-making as instructional leaders.
• Understandings of how data analysis can inform instruction and ultimately improve student achievement.
4
Data Informed Decision Making
Project Purposes
What are the data needed to support/make key decisions that leaders need to make to improve student achievement?
What are the best practices of how leaders need the data analyzed to make it useful/actionable?
Main FocusNon-negotiable decisions that can increase student achievement.
Objectives
1. Identify the sets of data questions asked by school, district, and state leaders.
2. To classify the data questions in ways that help us understand how to make data actionable.
3. To identify best practices in answer to these data questions.
4. To develop tools and training related to these data questions.
5
Survey Development: Helping Principals Make Data Actionable
Modules Examples of StatementsKinds of
Response Formats
Key Sources of Data
AYP math and reading results for the different groups
Importance
Availability
Steps in the Improvement
Process
Help my teachers analyze data to understand achievement gaps
Preparation
Frequency of use
Data-Informed Interventions
Instructional strategies when not achieving academic standards
Importance
Support
Constraints Principals
Face
The data I need are available in a timely fashion
Rate agreement
Level of Accountability
Literacy
I can communicate data effectively in different settings
Rate agreement
6
Data Collection System
State Format Participants
GA2 paper-based surveys: Part 1 and Part 2 given to 2 groups at 4 different times.
249 principals in 27 districts
KY 1 web-based survey 481 principals state wide
MIInterviews & 1 web-based survey
16 principals interviewed; 258 principals responded to the survey
NM 1 web-based survey 79 principals in principal support network in 28 districts
OH 3 web-based surveys 186 principals state wide
Total Participants 1285
7
Survey Domains• Introductory Section: Demographic• Section 1:
• Student demographic data • Student achievement data• Teacher and program data• Climate/perception data• Financial resource data
• Section 2: Principal preparation and use• Section 3: Importance and support • Section 4: Data access and quality
8
Important Data for Principals
State or District
Student Demographic
Data
Student Achievement
Data
Teacher & Program Data
Climate & Perception Data
Financial Resource Data
GeorgiaAbsentee rates; retention rates.
AYP results by subgroup.
Instructional Practice. Student Satisfaction.Level of funding For
my school.
KentuckyStudent needs or
disability information.
Individual proficiency results from state
assessments.
Instruction effectiveness as measured by student
growth scores.
School safety indicators.
Level of funding for my school;
Adequacy of funding for my school.
OhioWhat are the special
needs/disability profiles in my school?
What are the specific achievement gap
profiles for the different groups of students in
my school?
How does our instruction align with the
assessments used for accountability?; What
instructional practices/programs are
my teachers using?
How safe do my students feel in my
school?
How adequate are the technology, facilities, and instructional
materials for my school?
MichiganSpecial
needs/impairment profiles of students.
Student proficiency results on MEAP
broken down by State Standards.
Teachers' instructional effectiveness.
School culture.
Level of funding for my school;
Adequacy of funding for my school.
9
Difficult Questions that Principals Face after Data Analysis
Top 3 Killer Questions
Georgia Kentucky Ohio New Mexico
1
What instructional strategies do I need to implement when students are not
achieving academic standards?
What can I do to help my teachers
become more effective?
What instructional strategies do I need to implement when
my students are not achieving academic
standards?
What instructional strategies do I need to implement when
my students are not achieving academic
standards?
2What can I do to help my teachers become
more effective?
What instructional strategies do I need to implement when students are not
achieving academic standards?
What can I do when teachers are not
making reasonable progress?
What instructional strategies will work best with students
with special needs?
3How do I find more financial resources
for my school?
How do I increase the level of my staff's satisfaction with my
school?
What instructional strategies and/or
programs will work best with students
with special needs?
What can I do to help my teachers become
more effective?
10
Support for Data-Informed Decisions
Georgia (n=235)
Kentucky (n=360)
Ohio (n=28)New Mexico
(n=64)
Number 1 Killer Question
What instructional strategies do I need to implement when students are not
achieving academic standards?
What can I do to help my teachers
become more effective?
What instructional strategies do I need to implement when
my students are not achieving academic
standards?
What instructional strategies do I need to implement when
my students are not achieving academic
standards?
% who answered "Less support than I need or no support" in response to "How
Much Support Do You Get To Answer
This Question?"
22% 24% 36% 40%
11
District and State Supportwhen Using Data
86%
65%
41%
47%
62%60%
10%
0
0.2
0.4
0.6
0.8
1
Georgia (n=248) Kentucky (n=363) Ohio (n=40) New Mexico (n=63)Most principals receive adequate support from the school district in using dataMost principals receive adequate support from the state in understanding what to do with data
Note: Georgia did not include a question on district support. Kentucky asked “I receive adequate support from the district.” and “I receive adequate support from the state.”
12
I have the assistance I need to use data effectively
81%
66%
72%
28%
0%
20%
40%
60%
80%
100%
Georgia (n=249) Kentucky (n=361) Ohio (n=40) New Mexico (n=73)
Completely or Somewhat Agree
13
I would like more support in understanding what to do with teacher effectiveness and program data
78%
86%82%
0%
20%
40%
60%
80%
100%
Kentucky (n=352) Ohio (n=42) New Mexico (n=63)
Completely or Somewhat Agree
14
What are the steps in a continuous improvement process that Principals use most frequently?
Note: Number in parentheses are the percent of respondents who stated that they used this step very frequently or frequently during the school year.
3 Most Fequently Used Steps
Georgia Kentucky Ohio New Mexico
1
Prioritize which issues are most important to my
school (94%)
Help my staff make decisions that will improve student
achievement (98%)
Develop hypothesis about which issues
are most important to my school (86%)
Prioritize which issues are most important to my
school (94%)
2Gather data in an effective manner
(91%)
Effectively monitor teachers’
implementation of instructional plans
(96%)
Help my staff make decisions that will improve student
achievement (87%)
Help my teachers understand the goals
and standards for student achievement
(93%
3
Help my staff make decisions that will improve student
achievement (89%)
Gather data in an effective manner
(96%)
Help my teachers select interventions
for different subgroups of students (85%)
Gather data in an effective manner
(93%)
15
How prepared were first-year Principals to use these steps in
the Continuous Improvement Process?
Georgia Kentucky Ohio New Mexico
Most Frequently Used Step
Prioritize which issues are most important to my school (94%)
Help my staff make decisions that will improve
student achievement
(98%)
Develop hypothesis about which issues are most important to my school (86%)
Prioritize which issues are most important to my school (94%)
% who reported that they were not too prepared or
not at all prepared to use this step
41% 23% 9% 36%
16
What Are The Most Common Constraints That Principals Face In Using Data
Top 3 Constraints
Georgia Kentucky Ohio New Mexico
1I have the time
necessary to use data effectively (45%)
I have the time necessary to use
data effectively (58%)
I have the time necessary to use data
effectively (50%)
I have the time necessary to use data
effectively (76%)
2
I have the resouces to act on the needs
revealed by the data (28%)
I have the resouces to act on the needs revealed by the data
(45%)
I have the resouces to act on the needs
revealed by the data (44%)
The data I need are available in a timely
fashion (75%)
3
I have the tools and the training I need to use data effectively
(23%)
The data I need are available in a timely
fashion (30%)
I have the tools and the training I need to use data effectively
(22%)
I have the resouces to act on the needs
revealed by the data (67%)
Note: Number in parentheses are the percent of respondents who somewhat disagreed or disagreed with the statements.
17
The data I need are available in a timely fashion
90%
69%
92%
25%
0%
20%
40%
60%
80%
100%
Georgia (249) Kentucky (n=363) Ohio (n=40) New Mexico (n=63)
Completely or Somewhat Agree
18
My Preparation/Certification Program prepared me to deal effectively with data
43%
54%
38%
28%
0%
20%
40%
60%
80%
100%
Georgia (n=251) Kentucky (n=361) Ohio (n=40) New Mexico (n=62)
Completely or Somewhat Agree
19
I can communicate data effectively in Political, Professional, and Community Settings
89%92%
89%
74%
0%
20%
40%
60%
80%
100%
Georgia (n=251) Kentucky (n=360) Ohio (n=43) New Mexico (n=62)
Completely or Somewhat Agree
20
JCPS StudyElementary, Middle, and High School Principals
21
JCPS Survey Procedures
• Web-based survey
• Population: JCPS principals
(N = 133)
• Response rate: 55%
(N = 73)
• Respondents by school level:• 39 elementary schools• 19 middle schools• 15 high schools
22
Section 1: Demographic Data
• Importance vs. Availability of Data:
• Race (100% vs. 100%)
• Absentee rates (100% vs. 100%)
• ECE information (100% vs. 95.4%)
23
Section 1: Student Achievement Data
• Importance vs. Availability of Data:
• Individual proficiency results from state assessment (98.4% vs. 98.4%)
• AYP results by subgroup (96.8% vs. 95.0%)
• Student results on other district administered tests (93.7% vs. 93.3%)
24
Section 1: Teacher and Program Data
• Importance vs. Availability of Data:
• Instructional practices (100% vs. 91.9%)
• Instructional effectiveness as measured by student growth scores (100% vs. 83.9%)
• Professional development effectiveness (95.2% vs. 72.6%)
25
Section 1: School Climate Data
• Importance vs. Availability of Data:
• Teacher satisfaction with school (100% vs. 95.2%)
• Student satisfaction with school (100% vs. 96.8%)
• School safety indicators (100% vs. 96.8%)
26
Section 1: Financial Resource Data
• Importance vs. Availability of Data:
• Level of funding for school (100% vs. 91.9%)
• Adequacy of funding for school (100% vs. 77.4%)
• Availability of discretionary funds (100% vs. 75.4%)
27
Section 2: Preparation vs. Frequency of Use
• Principal Preparation:• Rate how prepared you were –as a first
year principal- for using each of these data steps to make decisions
• Frequency of Use:• Rate how frequently you use each of these
data steps in your current work as principal
28
Principal Preparation vs. Use:Strengths and Opportunities
• Help teachers analyze data to understand where our achievement gaps are (90% vs. 96%)
• Help teachers select interventions for different subgroups of students (72% vs. 96%)
• Guide teachers’ use of student achievement data to plan effective instruction (86% vs. 94%)
29
Section 3: Importance, District, and State Support
• Importance:• Rate how important these questions are to
you in your work as a principal
• (District and State) Support:• Rate how much support you get to answer
these questions which are revealed by data
30
Importance Compared with Availability of District and State Support
• Instructional strategies that need to be implemented when students are not achieving academic standards (100%, 82%, 58%)
• Helping teachers become more effective (100%, 78%, and 46%)
• Finding more financial resources for school (100%, 57%, and 27%)
31
Section 4: Data Access & QualityStrengths and Opportunities (%
Agreement)• Data needed are accurate and reliable
(87%)
• Data needed are available in a timely fashion (79%)
• I have the time necessary to use data effectively (49%)
32
Data Use – Strengths and Opportunities (Agreement %)
• I have the authority to act on the needs revealed by the data (79%)
• My preparation/certification program prepared me to deal effectively with data (68%)
• I have the resources to act on the needs revealed by the data (63%)
33
More Data Issues – Strengths and Opportunities (Agreement %)
• I receive adequate support from my school district in using data (91%)
• Most experienced principals are well prepared to use accountability data effectively (88%)
• Most first-year principals are well prepared to use accountability data effectively (54%)
34
EffectiveSchoolLeadershipUses Data About: Student Demographics Student AchievementTeachers & ProgramsClimate & PerceptionFinancial & Human Resources
To Answer The Difficult Questions
Improved Teaching
& Student
Achievement
Conclusion: Data Informed Decision Making in a Leadership System
Standards Training Conditions
Set clear expectations about what leaders should be able to do with data to improve instruction & learning
Collaborate with higher education to provide quality preparation in the use of data
Provide continuing PD & support to leaders in using a wide variety of data
Ensure that leaders have the time, authority, and resources to make use of data
Data
Data
Data
35
How is the DID LIG Impacting the work of the State and District?
State Standards Training Conditions
JCPS
JCPS Human Resources will require data-informed decision-making components in all principal preparation programs.
IDEAS, Principals for Tomorrow, and Internship Program will include accountability data tools and their effective use.
JCPS Principal candidates will use real CATS data reports in demonstrating mastery of data knowledge and skills.
KY
The Education Professional Standards Board will require data-informed decision-making components in all principal preparation programs.
Statewide Principal Academy piloted in four districts will include accountability data tools and their effective use as part of the program.
Principal candidates will use real data reports in demonstrating mastery of data and to meet standards for licensure.