Five Steps for Structuring Data-Informed Conversations and Action in Education

October 1, 2014Music City Center, Nashville, TN

Welcome and Overview

Lydotta Taylor, Ed.D.Research Alliance Lead, REL Appalachia

3

Agenda

• Quick overview of REL program and REL Appalachia• Local practitioner perspectives—Panel discussion • Overview of Five Steps for Structuring Data-Informed Conversations

and Action in Education• Step 1—Setting the Stage; Activity• Break• Step 2—Examining the Data; Activity• Lunch• Step 3—Understanding Findings; Activity• Sharing your district’s current data practices• Expert feedback; Considerations for action • Wrap-Up; Stakeholder Feedback Survey

4

Regional Educational Laboratory (REL) Program

• U.S. Department of Education, Institute of Education Sciences.• RELs provide regional support for:

– Applied research and evaluation.

– Technical support and information sharing to build capacity to use data for improved education outcomes.

5

REL Appalachia’s Mission

• Meet the applied research and technical support needs of Kentucky, Tennessee, Virginia, and West Virginia.

• Bring evidence-based information to policymakers and practitioners:– Provide support for a more evidence-reliant education system.

– Inform policy and practice for states, districts, schools, and other stakeholders.

– Focus on high-priority, discrete issues and build a body of knowledge over time.

6

How We Work: Research Alliances

• What is a research alliance?– A partnership between education stakeholders and REL Appalachia.

• What is the purpose of a research alliance?– As partners, REL Appalachia and alliance members develop and carry out a

research and analytic technical assistance agenda on priority topics.

• Who are the education stakeholders in an alliance?– May include representatives from one or more schools, divisions, state

education agencies, and other organizations (e.g., colleges and universities).

7

Goals of Today’s Bridge Event

• Understand the structure of an evidence-based data conversation.• Understand how to examine data and findings.• Understand a variety of practical strategies for having structured data

conversations.

Local Practitioner PerspectivesPanel discussion

Chris Causey, Ed.D.Supervisor of Accountability, Robertson County

Mary Laurens Seely Data Coach Coordinator, Metropolitan Nashville Public Schools

Vicky Smith, Ed.D. Regional Data Analyst, Mid-Cumberland CORE, TN Department of Education

Overview ofFive Steps for Structuring Data-Informed

Conversations and Action in Education

Shelby Maier, Ph.D.Systems Transformation Researcher, REL Pacific at McREL

About the Guide

10

11

About the Guide

• Initially created by REL Pacific as a technical assistance tool for research alliance members.

• Guidebook on how to bridge education data and practice. • Applications of the guide are field-wide and have the potential to

empower education stakeholders across the Pacific and beyond.

12

Background and Development of the Guide

• Developed by REL Pacific in 2012/13.• Influenced by:

– Using Data to Guide Action for School Improvement—guidebook released by the Nebraska Department of Education.

– McREL’s The Power of Data handbook.

• Connections to:– The NCES Forum Guide to Taking Action.

– The Pacific Comprehensive Center’s webinar series on data-driven decisionmaking.

Need for the Guide

• Educators increasingly have access to more and more data.– We all use data to inform our decisionmaking.

– Increasing availability due to No Child Left Behind, Race to the Top, other public policy—let’s leverage this power!

13

14

How the Guide Has Been Used

• Trainings and workshops throughout the Pacific Region and beyond:

– Data analysis trainings with the Palau Ministry of Education to understand how to examine questions of interest.

– The Palau 20th Education Convention; tailored for classroom teachers.

– College and career readiness trainings with the Commonwealth of the Northern Mariana Islands Public School System to understand, create, and examine college indicators.

Poll

How often do you currently use data to guide your conversations and decisions?

15

The Five Steps

Step 1Setting the

Stage

Step 2Examining the

Data

Step 3Understanding

Findings

Step 4Developing a Plan of Action

Step 5Future Actions

16

The Five Steps—What We Will Focus on Today

Step 1Setting the

Stage

Step 2Examining the

Data

Step 3Understanding

Findings

Step 4Developing a Plan of Action

Step 5Future Actions

17

Data Teams

• Data teams …– Are groups of individuals with a common goal to use data to drive action and

monitor performance.

– Have a topic of interest that they are interested in exploring.

– May include school administrators, teachers, analysts, parents, students, and other stakeholders from the community.

18

Data Teams and Today’s Workshop

• This workshop and the Five Steps guide capitalize on the collective input and wisdom of multiple perspectives.

• Today we will work in groups of 4-5 people that will serve as examples of “data teams.”

• Following this workshop, you may wish to convene a data team within your school system. – Teams can be formed by grade level, by subject, or around a topic of common

interest.

– Teams can be diverse. School-, district-, and state-level personnel. Policymakers, researchers, data managers. Family and community members.

– See what works for your school and for your topic.

19

Step 1. Setting the Stage

20

Step 1. Setting the Stage

• What is the question?• What information is needed to answer the question?• What information is available to answer the question?

21

22

Step 1. Setting the Stage

• What is the question?– Start by framing your question to answer “what” the issue is.

– Four types of questions:

1) Description. “What percentage of my students are at or above

proficiency in my quarterly math assessment?”2) Relationship.

“What is the relationship between students who participate in after-school tutoring and their math homework completion?”

3) Difference. “What is the difference between reading scores from

quarter 1 and quarter 3 on my quarterly reading assessment?”

4) Prediction. “Do ACT scores predict GPA in college?”

Step 1. Setting the Stage

• What is the question?– Start by framing your question to answer “what” the issue is.

– Questions may start off as broad, but should eventually be narrowed, using the four elements of a research question.

Group Measure

Benchmark or Evaluation

MethodTime

topic

23

Step 1. Setting the Stage

• What is the question?

“What do the data tell us about our

middle school students’ math

achievement level?”

“What percentage of middle school students are achieving at or above proficiency on standardized math achievement tests in December 2013?”

24

Step 1. Setting the Stage

• What information is needed to answer the question?– Using the question that your data team has just defined, brainstorm what data

are needed to answer it.

– Consider whether the question requires data to be separated out for special groups of interest.

Example:Standardized mathematics test scores for middle school students for December 2013

25

Step 1. Setting the Stage

26

Classroom/Program DataIncludes information related to schools’ efforts to promote high achievement.

Perception DataStudents’ attitudes, beliefs, and interests are important because people act in congruence with what they believe.

Performance DataHow students are doing at this point in time.Acquisition of knowledge and skills.

Demographic DataWhat students bring helps to understand students and their unique situations.

Step 1. Setting the Stage

Classroom/Program Data• Class size.• Availability of communication in

parents’ or patrons’ primary language.• Performance expectations for students.• Nature and frequency of classroom

assessment.• Time allotted to specific subjects.

Perception Data• School climate.• Parent satisfaction.• Students’ views about the level of

personal safety.• Level of support students feel they have

at school.• Newspaper editorials and letters.

Performance Data• Portfolios.• State assessments, standardized tests.• Dropout and retention rates.• Student achievement on standards.• Discipline referrals, suspensions, and

expulsions.• Graduation and promotion rates.

Demographic Data• Free/reduced price lunch status.• Student ethnicity.• Daily rate of attendance.• Student language proficiency.• Family size and composition.

27

Step 1. Setting the Stage

• What information is available to answer the question?– Previously analyzed data.

– Raw or individual data.

– Data collected at single points in time.

– Data that can be combined to show how the same student, or groups of students, is performing over a period of time (i.e., longitudinal data).

28

Step 1 ActivityRecording Questions of Interest

and Data Sources

Chris Causey, Ed.D. Supervisor of Accountability, Robertson County

Margie Johnson Business Intelligence Coordinator, Metropolitan Nashville Public Schools

Scenario

• Scenario: At the beginning of the year, Tennessee Comprehensive Assessment Program (TCAP) data were released. Last week, students took the Benchmark Assessment for monitoring student progress toward mastering Common Core State Standards.

30

Break

Step 2. Examining the Data

32

Step 2. Examining the Data

• Looking for patterns and making observations.• Exploring limitations of the data.• Identifying key strengths and/or challenges.

33

Step 2. Examining the Data

• Looking for patterns and making observations– Start with the “snapshot.”

– Make sure your observations relate back to your question.

– Observations can be made about strengths and challenges.

34

Step 2. Examining the Data

• Exploring limitations of the data.– Comparing different sources of data.

– Data that are not the same.

– Rigorous analysis to draw causal conclusions.

– Data that were not collected or stored in a way that allows for certain levels of analysis.

35

Step 2. Examining the Data

• Identifying key strengths and/or challenges.– Done through observations that are specific, factual, and related to your

question of interest.

Observation verification questions ExplanationIs it specific? Does the observation focus on only one item in the data?

Is it factual? Does the observation incorporate ONLY the data? This means the observation is NOT an assumption, inference, or bias.

Is it related? Does the observation begin to answer the question?

36

Step 2 ActivityObservation Tracking;

Strengths and Challenges

Shannon Black Director of Talent Management, Metropolitan Nashville Public Schools

Mary Laurens Seely Data Coach Coordinator, Metropolitan Nashville Public Schools

38

Examining the Data—Data Observations

• Remember, JUST THE FACTS! • Avoid the following phrases:

– Therefore…

– Because…

– It seems…

– However…

Examining the Data—Strengths and Challenges

• What strengths and challenges arise from the data?

39

Lunch

Step 3. Understanding Findings

41

Step 3. Understanding Findings

• Choose a key challenge.• Brainstorm possible factors.• Identify key strengths and/or challenges.

42

Step 3. Understanding Findings

• Choose a key challenge that is:– Actionable.

– High priority.

43

Step 3. Understanding Findings

• Brainstorm possible factors:– Consider possible causes of the challenge.

– Stick to available facts.

– Ask a series of “why” and “because” questions to get to possible causes of the challenge.

44

45

Step 3. Understanding Findings

46

Step 3. Understanding Findings

• Let’s look at some examples:

47

Step 3. Understanding Findings

Step 3 ActivityIdentifying the Driving Factors

Margie Johnson Business Intelligence Coordinator, Metropolitan Nashville Public Schools

Activity Facilitators

• Shannon Black, Director of Talent Management, MNPS• Chris Causey, Ed.D., Supervisor of Accountability, Robertson County• Mary Laurens Seely, Data Coach Coordinator, MNPS• Vicki Smith, Ed.D., Regional Data Analyst, Mid-Cumberland CORE, TN DOE

49

Understanding Findings—Identifying Driving Factors

• Use the assigned protocol to brainstorm driving factors or causes. (Why?)

• Prioritize and circle 1‒3 contributing factors/causes that seem most plausible.

Why?

Because…

50

Understanding Findings—Protocol Debrief

51

Sharing Your District’s Current Data Practices

Lydotta Taylor, Ed.D.Research Alliance Lead, REL Appalachia

Table Discussion on Your District’s Current Practices

• Discuss with those at your table the following:

– How do your district’s current data practices compare with those discussed today?

– What are the immediate next steps/actions you will need to take to begin what we’ve discussed today in your school or classroom ?

53

Expert Feedback,Considerations for Action

Shelby Maier, Ph.D.Systems Transformation Researcher, REL Pacific at McREL

Wrap-UpStakeholder Feedback Survey

Lydotta Taylor, Ed.D.Research Alliance Lead, REL Appalachia

Resources

Kekahio, W., & Baker, M. (2013). Five steps for structuring data-informed conversations and action in education (REL 2013‒001). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Pacific. Retrieved from http://ies.ed.gov/ncee/edlabs

Mid-continent Research for Education and Learning. (2010). The power of data. Denver, CO: Author. Retrieved from http://www.mcrel.org/products-and-services/services/service-listing/service-42

National School Reform Faculty. (2002). Data-driven dialogue. Retrieved from http://www.nsrfharmony.org/

Nebraska Department of Education. (2012). Using data to guide action for school improvement: Facilitator’s guide. Denver, CO: Mid-continent Research for Education and Learning. Retrieved from http://www.esu1.org/downloads/misc/Facilitator.pdf

56