data literacy: introduction to the data literacy handouts€¦ · data literacy: introduction to...
TRANSCRIPT
[email protected] Version 2
Data Literacy: Introduction to the Data Literacy Handouts
Why do we need Data Literacy?
The Center for Applied Research and Educational Improvement (CAREI) completed a statewide
needs assessment in 2015 with input from approximately 800 individuals and 13 professional
organizations in Minnesota. Results indicated that professional educators need help building
their capacity to use data to improve educational outcomes. The problem is not that school
districts do not have enough data, rather, they need to understand how to use data to
inform decision-making.
A common theme in the needs assessment was that schools are data rich but information poor.
For example, districts have state assessments, local assessments, and teacher-made
assessments. However, each of these assessments have a different purpose. Districts need to
understand how to use the various assessments in a correct manner. Needs assessment
participants identified the need to improve in three areas:
1. Data literacy: What are the purposes of assessment? What data will help answer
specific district questions?
2. Program evaluation: How to use data to answer the questions “Did what we do work?
How can we improve?”
3. Data management: What are efficient and effective data management resources such as
data dashboards and analytic tools? How can we access these resources?
These results were further confirmed with the release of an evaluation of standardized student
testing by the Office of the Legislative Auditor. One question asked in the audit was “How useful
are state test results to policy makers, school districts, schools, teachers, and students?”
Although 85 percent of principals and 77 percent of teachers report that the scores are at least
somewhat useful, over half reported that they were not confident in their ability to correctly
interpret the scores. The report authors suggest that more training is needed for school staff on
how to use test scores at the local level.
[email protected] Version 2
What are the Data Literacy Fact Sheets?
The data literacy fact sheets are organized around the purposes of assessment. The fact sheets
are easy to read (e.g., 2-3 pages) and include links to other resources. Each fact sheet is
organized in similar format: (1) What is it? (2) How can we use data? (3) What does it look like?
(4) Discussion questions. Currently, there are five fact sheets:
1. Data Literacy: Overview
2. Data Literacy: Screening
3. Data Literacy: Diagnostic
4. Data Literacy: Progress Monitoring
5. Data Literacy: Systems Outcomes
How do we use the Data Literacy Sheets?
These sheets were developed to use as supplements to district and building professional
learning activities. They were not developed to be “stand-alone” documents to provide in-depth
knowledge about each area. We envision that these sheets could be very useful for principals to
use with their building teams (e.g., grade level teams, PLC’s, problem solving teams, etc.) to
facilitate discussion, identify areas in need of further professional learning, and provide activities
for teams to engage in to promote higher-order thinking in each area.
Where are the Data Literacy Sheets Located?
The data literacy sheets may be found on the CAREI website in the members only portal.
Please contact CAREI ([email protected]) if you need instructions for accessing these forms.
[email protected] Version 2
Data Literacy: Overview
What is it? Just as literacy refers to “the ability to read for knowledge, write coherently and
think critically about printed material”, data-literacy is a term that refers to the ability to
consume for knowledge, produce coherently and think critically about data. Data literate
educators:
➔ know the different kinds of data that exist and which kind of data to use for various
decisions,
➔ evaluate the accuracy and sufficiency of each kind of data they will use,
➔ transform data from a variety of sources (classroom, school, district, state) into
actionable information to guide decisions, and
➔ hold themselves accountable for ethical generation, interpretation, and application of
assessment data.
What are the purposes of assessment? A first step toward becoming data literate is
understanding that there are four purposes of assessment.
Individual/Groups of Students System
Screening Identify students or groups that may need additional support.
Identify specific programs and practices that may need additional supports.
Progress Monitoring Monitor the current impact of instruction and interventions.
Monitor the implementation of a specific program or action plan activities.
Diagnostic Identify specific skill, curriculum, instructional, or environmental needs.
Use data to identify why a specific program or practice is not successful.
Systems Outcomes Identify the success of instruction and/or intervention.
Identify the results of a plan.
What does it look like?
Each type of data collection serves a different purpose. Utilizing the right tool is an important
part of valid and reliable data collection. Sometimes one tool can serve multiple purposes. For
example, Curriculum Based Measures (CBM) are screening and progress monitoring tools, and
these data may be used to evaluate system outcomes. Many assessment systems such as
AIMSweb, DIBELS, easyCBM, and FAST utilize CBMs as an assessment tool.
[email protected] Version 2
Examples Non-examples
Screening ● Curriculum-based measurement (CBM) ● Brief social-emotional survey tools ● Office discipline referrals (ODR)
● Student work samples ● Teacher developed measures ● Minnesota Comprehensive
Assessment (MCA)
Progress Monitoring
● Direct behavior ratings (DBR) ● Attendance data ● Daily behavior report cards
● Functional behavioral assessments (FBA)
● Strand scores from MCA
Diagnostic ● Developmental reading assessment (DRA)
● Individual reading inventories ● Functional behavioral assessment (FBA) ● Office discipline referrals (ODR) by
infraction and setting, time of day, teacher, etc.
● Strand scores from the MCA ● Direct Behavior Ratings (DBR)
Systems Outcomes
● MCA scores for all students and disaggregated by subgroups of students
● Screening data by classroom or grade level
● Office discipline referrals for all students and disaggregated by subgroups of students
● Individual student work ● Functional behavioral
assessments (FBA)
What else do we need to know?
When selecting assessment tools, the reliability and validity must be evaluated. Reliability is
the extent to which an assessment yields consistent scores. Reliability can refer to consistency
across time, raters, items, and alternate forms. Validity is the extent to which the use of an
assessment actually measures what it intends to measure. It refers to the appropriateness of
interpreting the test scores as a measure of a particular concept.
The Center on Response to Intervention provides ratings on academic screening tools at
http://www.rti4success.org/resources/tools-charts/screening-tools-chart.
The National Center for Intensive Intervention provides reviews of academic and
behavioral progress monitioring tools through the Tools Charts:
http://www.intensiveintervention.org/resources/tools-charts.
[email protected] Version 2
Data Literacy: Overview
Discussion Questions
1. With your team, discuss the assessments used at each grade level. Identify the
assessments used at each level and indicate the purpose of each assessment. An
MTSS Assessment Inventory organizer can guide the process. Additional tools for
conducting the inventory are available at http://www.achieve.org/assessmentinventory.
2. With your team, identify any gaps or redundancies that appeared in your inventory. Is
there a way to streamline your assessments?
[email protected] Version 2
Data Literacy: Screening
What is it? The screening process is designed to allow for the efficient and early identification
of needs. Screening involves administering brief, reliable and valid assessments (see Overview
definitions) to all students at multiple points per year. Many districts screen students three times
per year (e.g., fall, winter, and spring). Screening provides a quick way to identify which
students are expected to exceed, meet, or fall below grade level standards. Students who are
well-above or well-below grade level standards can be provided additional support and
differentiation. Without screening, all students who may need additional support are unlikely to
be identified.
How can we use screening data?
● Academic: Identify how students are performing in basic academic areas such as
reading, math, or writing. Identify students who are on-track for college and career
readiness.
● Social-emotional: Identify students with social-emotional needs that interfere with
learning and forming positive relationships due to internalizing or externalizing
behaviors.
● System level: Evaluate the impact of universal instruction. Use to evaluate programs
and instructional practices for both academic and social-emotional domains. Identify
grade levels and programs that need additional support.
What does it look like?
Examples Non-examples
Academic ● Curriculum based measurement (CBM; e.g. AIMSweb, DIBELS, FAST)
● Broad measures of achievement (FAST aReading, aMath, NWEA MAP)
● Developmental Reading Assessments ● Student work samples ● Teacher developed measures ● Minnesota Comprehensive
Assessment (MCA) ● Running Records
Social- Emotional
● Brief social-emotional survey tools (e.g. SAEBRS, SRSS, BASC-3 BESS)
● Office discipline referrals (ODR)
● Functional behavioral assessment (FBA)
[email protected] Version 2
Screening tools should be evidence-based, reliable, highly correlated with desired outcomes,
and accurate in predicting risk status. Therefore, student work samples or teacher developed
measures are not sufficient to use for screening. Screening tools should be brief and sensitive
to growth over time. Screening tools that take a long time to administer result in a loss of
instructional time for students. Time intensive assessments such as the MCA or an FBA do not
make a good screening tool. The Center on Response to Intervention provides ratings on
academic screening tools at http://www.rti4success.org/resources/tools-charts/screening-tools-
chart. When making screening decisions, multiple sources of data should be used.
What else do we need to know?
Screening data should not be used to make important decisions in isolation. Decisions
should be supported by at least two other data sources such as classroom observations, state
assessments, classroom work samples, or progress monitoring data.
Benchmarks: Screening data can be presented as benchmarks to determine which students are
above, at, or below an expected level of performance. Benchmark scores compare student
performance to an established criterion and are used to predict success on another measure
such as a statewide accountability test.
Local norms: Since screening measures are given to all students, screening data can also be
used for local comparisons. Local percentiles or local norms compare the scores between
students in the class, school, or district.
A screening score will not provide all of the information needed to identify
interventions, but it will help teachers determine which students need extra support
and differentiation in an efficient manner.
[email protected] Version 2
Data Literacy: Screening
Discussion Questions
1. Consider the following scenario: Your PLC team is meeting to look at fall screening data
for Reading. There are 100 students in your grade and you have resources to provide
individual intervention to 5 students and small group interventions to 15 students. Your
data coach passes out the results of the benchmark scores.
On Track 60 students
At Risk 25 students
High Risk 15 students
How does your team respond? How will you make decisions about which students will
receive small group and individual interventions?
2. Why is it important to screen all students? Why not only screen students who scored in
the at risk or at high risk range during the previous screening window? (hint: Why would
take a child to the doctor for well-child check-ups?)
3. What screening tools are currently being used in your building? To what extent are they
reliable, valid, simple and quick to administer, and inexpensive?
4. How could you use screening tools to evaluate universal instruction?
[email protected] Version 2
Data Literacy: Diagnostic
What is it? Diagnostic assessments are used to understand individual learning strengths and
needs. Diagnostic data are used to answer the question of why students are performing at their
current level and help inform the target for instruction and intervention. Because the diagnostic
process is more time intensive than screening, it is only used for those students who have
needs identified through the screening process. Diagnostic assessment sometimes includes
administering tests, but also includes observing instruction and students, reviewing materials
used, and interviewing the student and others. It may be helpful to use a framework for data
collection such as the RIOT/ICEL matrix to organize data from multiple sources (Review,
Interview, Observation, and Testing) on multiple domains of learning (Instruction, Curriculum,
Environment, and Learner).
How can we use diagnostic data?
● Individual level: Teachers can use diagnostic data to identify specific skill areas for
intervention or enrichment and identify materials or instructional strategies that may be
successful for a student.
● System level: Teams can collect diagnostic data to identify programs and practices are
not achieving desired outcomes.
What does it look like?
Diagnostic assessments should identify strengths and needs within specific sub-skills (e.g.
number sense, operations, measurement, or fractions, decimals, and percents). Diagnostic data
may also assist in identifying social, cognitive, or environmental factors that contribute to the
occurrence of a behavior.
Informal Assessments: Informal diagnostic assessments are used by teachers on a frequent
basis to identify if students have mastered the lesson, guide instruction, and assign grades.
These assessments may be created by a teaching team or provided as part of the curriculum.
An example of an informal diagnostic assessment would be Running Records or placement
tests that come with a published curriculum.
Formal Assessments: Formal diagnostic assessments provide data through a standardized
measure and have been tested for reliability and validity. These formal measures are usually
individually administered and are more time consuming than screening assessments.
Formal diagnostic assessments should only be administered to some students when
additional information is needed to inform intervention or identify why a student is not
making progress. Collecting diagnostic information on all students would be an inefficient
use of instructional time.
[email protected] Version 2
Examples of Formal Diagnostic Assessments
Non-examples
Academic ● Developmental Reading Assessment (DRA)
● Individual reading inventories
● Strand scores from the MCA
Social- Emotional
● Functional behavioral assessment (FBA)
● Office discipline referrals (ODR) by infraction and setting, time of day, teacher, etc.
● Attendance data ● Direct Behavior Ratings (DBR)
Discussion Questions
1. What diagnostic assessments does your team have available to identify learning
strengths and needs in reading? In math? For social-emotional skills and behaviors?
2. Consider the following scenario. When reviewing the fall screening data for
mathematics, your team identifies 7 students who had scores in the high risk range. You
wonder if you could provide supplemental services to these students in small groups of 3
or 4 students. How could you use formal diagnostic assessments to identify instructional
groupings and sub-skills for targeted intervention?
3. What are ways you use informal diagnostic assessment in your classroom? How does
this inform your teaching?
[email protected] Version 2
Data Literacy: Progress Monitoring
What is it? Progress monitoring is a measure of student progress toward an end of the year
outcome goal. Progress monitoring data can be used to determine if instruction and
interventions are successful for an individual student and to evaluate programs for groups of
students. Additionally, screening data can be used as a progress monitoring check on universal
instruction (Tier 1) three times per year.
How can we use progress monitoring data?
● Individual level: Teachers can use progress monitoring to track a student’s growth when
receiving instruction or intervention. The progress monitoring data can be used to
determine whether to maintain, intensify, or change instruction or intervention.
● Group level: Teachers can use progress monitoring data to determine if programs are
successful for a group of students receiving the same intervention.
● System level: Districts can track student progress over time to assess the effectiveness
of universal, supplemental, and intensive instruction. Progress monitoring data can also
be used to evaluate the implementation of a program or practice such as MTSS.
What does it look like?
A good progress monitoring tools should meet a number of criteria. Most importantly, progress
monitoring tools should be reliable and valid. In addition, they should be simple and quick to
administer, easy to understand, and should be able to be administered on a frequent basis (e.g.,
weekly or bi-weekly). The tools should be sensitive to changes in instruction so the data may be
used to change instruction when student progress is insufficient.
Examples Non-examples
Academic ● Curriculum based measurement (CBM) or general outcome measures (GOM)
● Grades ● Running Records ● Strand scores from MCA ● Teacher made assessments
Social- Emotional
● Direct behavior ratings (DBR) ● Attendance data ● Daily behavior report cards
● Functional behavioral assessments (FBA)
[email protected] Version 2
Since progress monitoring requires frequent assessment, progress monitoring tools should have
alternate forms of equal difficulty. If there are not a sufficient number of alternating forms, the
student could become familiar with the content resulting in a “practice effect.” Progress
monitoring tools should provide suggested weekly or monthly growth rates and benchmarks for
predicting acceptable end-of-year performance. A student’s rate of improvement (ROI) is
compared to these suggested growth rates and benchmarks to determine if progress is
satisfactory.
What else do we need to know?
The National Center for Intensive Intervention provides reviews of academic and
behavioral progress monitoring tools through the Tools Charts:
http://www.intensiveintervention.org/resources/tools-charts.
Progress monitoring data are formative if used to make changes to instruction or intervention
when student progress is unsatisfactory. Progress monitoring and the review of progress
monitoring data should be planned into instructional and team routines. Making sound decisions
based on progress monitoring data requires collecting a sufficient number of data points to both
(a) give the instruction and intervention time to work and (b) increase confidence that the
student’s performance is an accurate reflection of their skill. The sensitivity of the measures also
needs to be considered. For example, curriculum-based measures (CBM) in math are typically
not as sensitive as CBM for reading. The desire for additional data points should be balanced
with the effort to use instructional time effectively.
Decision making guidelines:
● Progress monitoring data should be collected at least monthly for students receiving
supplemental services and at least weekly for students receiving intensive services.
● The more concerned you are about a student, the more frequently you should monitor
progress.
● A sufficient number of data points should be collected before making decisions
regarding the effectiveness of instruction or intervention.
● When using progress monitoring data, give at least 8 weeks of instruction and use
multiple sources of information to evaluate intervention effectiveness.
Discussion Questions
1. What progress monitoring tools are currently being used in your building? To what
extent are they reliable, valid, simple and quick to administer, and inexpensive?
2. How do various teams in your building incorporate the review of progress monitoring
data? Are there times scheduled to review the data?
3. How does your team make decisions about whether interventions are successful? Does
your team use any decision-making rules?
[email protected] Version 2
Data Literacy: Systems Outcomes
What is it? Outcome assessments provide information regarding performance at the system
level (universal, supplemental, intensive instruction) for the evaluation of programs and
practices.
How can we use it?
● Student data: Districts may use student data to evaluate the effectiveness of universal,
supplemental, and intensive instruction or monitor student progress over time. Student
outcome data is often reviewed in aggregate and disaggregated by subgroups of
students.
● Program evaluation: Systems outcome assessments can also be used to evaluate the
success of a specific program or practice such as the implementation of a new school-
wide behavioral system.
What does it look like?
Systems outcome assessments may come from multiple sources including screening, progress
monitoring, and summative assessments such as the Minnesota Comprehensive Assessment
(MCA). For the successful and sustainable implementation of MTSS, it is encouraged for
districts to establish short- and long-term goals. System outcome data are used to track
progress toward these goals.
Examples Non-examples
Academic ● MCA scores for all students and disaggregated by subgroups of students
● Screening data by classroom, grade level, or building.
● Individual student work
Social- Emotional
● Office discipline referrals for all students and disaggregated by subgroups of students
● Functional behavioral assessments (FBA)
[email protected] Version 2
What else do we need to know?
Data-based decision making with systems outcomes data involves the allocation of resources
and support. Screening data can be used to identify trends or areas at high-risk for low
performance at the school, grade, or classroom level.
Discussion Questions
1. How do you currently evaluate system outcomes at the district, building, grade, and
class levels?
2. How can systems outcome data inform data-based decision making?
3. To what extent are you currently evaluating the effectiveness of instruction and
intervention in your classroom/school/district using systems outcome data?