Data Tools for Nancy Love’s Sessions
Research for Better TeachingActon, Massachusetts, USA
Near East South Asia Council of Overseas SchoolsBangkok, Thailand
5-6 April 2014
Research for Better Teaching, Inc. · One Acton Place, Acton, MA 01720 · +1-978-263-9449 · [email protected]
Copyright © 2014 by Research for Better Teaching, Inc.
The material in this handout from or adapted from Nancy Love, Katherine E. Stiles, Susan Mundry, & Kathryn DiRanna, The Data Coach’s Guide to Improving Learning for All Students: Unleashing the Power of Collaborative Inquiry (Thousand Oaks, CA: Corwin Press, 2008) is used with the permission of Corwin Press.
Research for Better Teaching, Inc.One Acton PlaceActon, MA 01720
President: Jon SaphierExecutive Director: Sandra SpoonerProgram Director: Nancy Love
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
i
Data Tools for Nancy Love’s Sessions
Table of Contents
After Analyzing Formative Assessment Data, Now What? ................................................................... 1
Building Blocks of Collaborative Inquiry Self-Assessment ................................................................... 2
Choice-Points for Effective Feedback ....................................................................................................... 4
Criteria Analysis Data Display Tool .......................................................................................................... 6
Data-Driven Dialogue ................................................................................................................................ 8
Engage in Data-Driven Dialogue with Item Data: Checklist ................................................................ 9
Engage in Task Deconstruction and Data-Driven Dialogue with Student Work: Checklist ........... 11
Error Analysis Protocol ............................................................................................................................ 13
Error Analysis Template ........................................................................................................................... 14
No-Because Sign ........................................................................................................................................ 15
Norms of Collaboration ............................................................................................................................ 16
Short-Cycle Action Plans for Grade-Level or Content Teams ............................................................ 18
Short-Cycle Action Plans for Grade-Level or Content Teams [Template] ........................................ 19
Stoplight Highlighting .............................................................................................................................. 20
Student Error Analysis .............................................................................................................................. 24
Verifying Causes ........................................................................................................................................ 26
Verifying Causes Graphic ......................................................................................................................... 27
Verifying Causes Template ....................................................................................................................... 28
Data Tools Organizer ................................................................................................................................ 29
Page
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
ii
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
1
After Analyzing Formative Assessment Data, Now What?
Findings from Formative Assessment
1. Based on these findings, what is our next step as a team? q Feedback: Will we give feedback to students? If so, how and to whom? q Investigation: Do we need to further investigate causes of students’ errors and misconceptions? If
so, how will we do so? q Reteaching/re-engaging/regrouping: Do we need to do one or all of these? If so, how will we do
so? How will we manage time and tasks to do so? q Moving on: Can we move on? Based on what criteria? q Extension: Are there students ready for an extended learning opportunity? How will we create it?
2. Which of the above actions would we most like to learn more about as a team?
3. Which is a priority for taking action?
4. What will we do at our next team meeting?
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
2
(cont. next page)
© 2012 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • 978-263-9449 • [email protected]
17
© Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • (978) 263-9449 • www.RBTeach.com
Building Blocks of Collaborative Inquiry Self-Assessment Assess the status of collaborative inquiry in either your school or your district, as applicable, with the following questions. 1. Leadership and Capacity How many teachers and administrators have the knowledge, skills, and dispositions to use data effectively, collaboratively, and continuously to improve teaching and learning? 1 2 3 4 Few Some Most All 2. Structured Collaboration a. How often do teachers engage in collaborative inquiry with their colleagues during the school day? 1 2 3 4 Once a year Quarterly Monthly Weekly b. To what extent is collaborative team time productive and focused, making use of processes, tools, and protocols? 1 2 3 4 Not productive Somewhat productive Productive Highly productive 3. Frequent and In-Depth Data Use How often are teachers using common formative assessments to analyze students’ strengths and immediate needs? 1 2 3 4 Annually 2-4 times a year Monthly Weekly 4. Instructional Improvement For how many teachers does collaborative inquiry have an immediate and direct impact on improvements in curriculum, instruction, and assessment practices? 1 2 3 4 None Few Some All
2.1 Collaborative Inquiry Self-Assessment
(cont. next page)
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
3© 2012 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • 978-263-9449 • [email protected]
18
© Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • (978) 263-9449 • www.RBTeach.com
5. School Culture a. How many teachers and administrators act in alignment with the following value: “We are collectively responsible for the learning and achievement of each and every student and adult in our system—no excuses”? 1 2 3 4 Few Some Most All b. To what extent is constructive dialogue about race, class, and culture a norm in your setting? 1 2 3 4 Not at all Somewhat Mostly All the time c. To what extent are relationships among staff characterized by trust, candor, openness, and collaboration? 1 2 3 4 Not at all Somewhat Mostly All the time
Questions for Reflection:
1. What are your areas of strength?
2. What areas are in need of improvement?
3. What are the most important next steps you can take to strengthen the bridge between data and results?
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
4
Cho
ice-
Poi
nts
for
Eff
ecti
ve F
eedb
ack
Que
stio
ns t
o C
onsi
der
Opt
ions
to
Con
side
rT
imin
gA
re th
e st
uden
ts s
till m
indf
ul o
f th
e le
arni
ng ta
rget
?Is
ther
e st
ill ti
me
for
them
to a
ct o
n yo
ur f
eedb
ack?
q
Imm
edia
te:
For
basi
c ri
ght/w
rong
que
stio
ns q
As
soon
as
feas
ible
: Fo
r m
ore
com
plex
pro
duct
s lik
e pa
pers
or
proj
ects
q
Cum
ulat
ive:
Aft
er o
bser
ving
a p
atte
rn o
f er
rors
or
mis
conc
eptio
ns
Qua
ntit
y/Q
ualit
yH
ow m
uch
feed
back
can
the
stud
ent
abso
rb a
nd a
ct u
pon
at o
nce?
If th
ere
are
mul
tiple
are
as n
eedi
ng
impr
ovem
ent i
n th
e w
ork
can
they
be
pri
oriti
zed
to m
ake
the
feed
back
m
anag
eabl
e fo
r th
e st
uden
t?Is
the
feed
back
spe
cific
eno
ugh
to
mak
e th
e st
uden
t aw
are
of h
ow h
is/
her
wor
k co
mpa
res
to th
e cr
iteri
a?
Will
the
feed
back
hel
p th
e st
uden
t to
kno
w w
hat t
o do
to im
prov
e th
e w
ork?
You
r de
cisi
ons
abou
t qua
ntity
and
qua
lity
mig
ht b
e di
ffer
ent f
or d
iffe
rent
stu
dent
s an
d di
ffer
ent a
ssig
nmen
ts:
Qua
ntit
y: (
see
“Sca
ffol
ded”
abo
ve)
q
Smal
ler
chun
ks q
All
crite
ria
Qua
lity:
(M
ake
sure
that
you
r fe
edba
ck m
eets
all
of th
e C
hara
cter
istic
s of
Eff
ectiv
e Fe
edba
ck)
q
Cle
ar q
Rel
evan
t q
Non
judg
men
tal
q
Supp
ortiv
e q
Tim
ely
q
Use
ful
q
Scaf
fold
edM
ode
Whi
ch m
ode
of f
eedb
ack
will
bes
t su
ppor
t the
stu
dent
in b
eing
abl
e to
un
ders
tand
and
act
on
it?W
ill th
e st
uden
t be
able
to r
ead
and
unde
rsta
nd w
ritte
n fe
edba
ck?
Will
stu
dent
s ne
ed to
ref
er b
ack
to th
e fe
edba
ck in
ord
er to
mak
e ch
ange
s?Is
ther
e a
need
or
a w
ay to
mod
el
wha
t the
fee
dbac
k is
foc
usin
g on
?
Whi
ch f
orm
of
feed
back
mig
ht b
est
mat
ch th
e st
uden
t’s le
arni
ng s
tyle
?
q
Wri
tten
: St
uden
ts c
an m
ore
easi
ly r
efer
bac
k to
it a
s th
ey w
ork
beca
use
it is
m
ore
perm
anen
t. C
an b
e w
ritte
n in
key
pla
ces
dire
ctly
on
stud
ent w
ork,
or
on a
ru
bric
or
assi
gnm
ent c
over
she
et.
q
Ora
l: W
orks
bes
t for
ver
y yo
ung
stud
ents
or
thos
e w
ho m
ay li
kely
not
rea
d w
hat i
s w
ritte
n. A
lso
good
whe
n th
e te
ache
r ha
s so
muc
h to
say
that
it m
ay b
e in
timid
atin
g if
wri
tten.
Thi
s is
a g
ood
chan
ce to
let t
he s
tude
nt d
ecid
e w
hich
fe
edba
ck h
e/sh
e w
ill a
ct o
n. E
asy
to g
ive
in s
mal
ler
dose
s as
a c
ompl
emen
t to
wri
tten
feed
back
, as
stud
ents
mak
e re
visi
ons.
q
Dem
onst
rati
on: A
nyth
ing
that
invo
lves
a p
hysi
cal s
kill
lend
s its
elf
wel
l to
dem
onst
ratio
n (e
.g.,
hold
ing
an in
stru
men
t or
usin
g a
tool
). I
t is
also
a g
ood
way
to “
show
” a
stud
ent h
ow to
use
hig
her
cogn
itive
ski
lls s
uch
as ju
stif
ying
an
answ
er in
mat
hem
atic
s. S
tude
nts
will
be
mor
e lik
ely
to c
ompa
re th
eir
own
wor
k to
wha
t you
mod
el if
you
r de
mon
stra
tion
is c
oupl
ed w
ith o
ral f
eedb
ack.
(cont. next page)
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
5
Aud
ienc
eW
hat d
o m
y st
uden
ts n
eed
feed
back
ab
out?
D
o I
see
a pa
ttern
of
sim
ilar
mis
take
s ac
ross
my
clas
s?A
re th
ere
smal
l gro
ups
of s
tude
nts
who
wou
ld b
enefi
t fro
m g
ettin
g th
e sa
me
feed
back
?
q
Indi
vidu
al: W
orks
bes
t whe
n st
uden
ts n
eed
spec
ific
feed
back
that
oth
ers
may
not
nee
d or
if y
ou k
now
that
a s
tude
nt m
ay b
e em
barr
asse
d by
it. I
f yo
ur
clas
sroo
m c
limat
e is
stil
l dev
elop
ing
in a
ccep
tanc
e of
cri
tique
, thi
s m
ay b
e th
e be
st p
lace
to s
tart
. Thi
s al
so h
as th
e ad
ded
bene
fit o
f le
tting
the
stud
ent k
now
th
at th
e te
ache
r ha
s re
view
ed a
nd th
ough
t car
eful
ly a
bout
his
/her
wor
k an
d va
lues
and
car
es a
bout
his
/her
pro
gres
s. q
Smal
l gro
up: W
hen
seve
ral s
tude
nts
can
bene
fit f
rom
hea
ring
the
sam
e fe
edba
ck, i
t can
be
good
tim
e fo
r pu
lling
them
toge
ther
in a
flex
ible
gro
up a
nd
prov
idin
g a
“min
i-le
sson
.” W
hen
deliv
ered
wel
l, sm
all-
grou
p fe
edba
ck c
an in
m
any
case
s le
ad to
stu
dent
s fe
elin
g le
ss a
lone
in th
eir
conf
usio
ns a
nd b
enefi
t fr
om le
arni
ng to
dis
cuss
them
with
thei
r pe
ers.
q
Who
le g
roup
: The
re a
re s
ome
occa
sion
s w
hen
the
who
le g
roup
nee
ds to
hea
r th
e sa
me
thin
g. T
his
is a
goo
d tim
e to
beg
in a
less
on w
ith f
eedb
ack
from
the
prev
ious
less
on (
or f
rom
exi
t tic
kets
!). B
e ve
ry c
aref
ul th
at it
is r
elev
ant t
o al
l; it
can
turn
off
stu
dent
s w
ho k
now
they
don
’t n
eed
it, c
onfu
se s
tude
nts
who
are
n’t
sure
if th
ey n
eed
it, a
nd o
ften
be
easi
ly ig
nore
d by
the
ones
you
are
inte
ndin
g to
re
ach.
Ada
pted
Mos
s an
d B
rook
hart
, 200
9, p
p. 4
8-50
.
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
6
Criteria Analysis Data Display ToolExperiment 4: Criteria Analysis
Data Display Tool
What product or performance are you using as a formative assessment? ___________________________ Criteria for success for this formative assessment (What evidence will the product or performance have as confirmation of student mastery of the objective?) 1
2
3
4
5
Student Name Criteria for Success 1 2 3 4 5
Objective for the lesson: By the end of the lesson students will be able to….
Data Display P= Proficient performance -- = Not yet proficient
(cont. next page)
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
7
Identify several possible root causes for the student learning problem you identified above: Criteria Not
Yet Met Possible Root Causes
(Why do you think the students did not meet standard on this particular criteria?) I wonder if they didn’t meet standard because…
I wonder if they didn’t meet standard because…
I wonder if they didn’t meet standard because…
How could you gather additional data to verify the hypothesized root cause(s). Criteria Not yet
Met Further Data Collection to Test Hypotheses
(How could you gather additional data to verify the accuracy of your hypothesized root cause[s])?
(same as above)
Based on the analysis of the data, identify your next instructional steps for those students who have met the criteria as well as those students who have not yet met the criteria. Consider: • Different presentation • Different materials • Providing exemplars and models
After performing your criteria analysis, identify a student learning problem that needs attention:
Next instructional steps for students who have met the criteria:
Next instructional steps for students who have not yet met the criteria:
-
-
Yet
?)
?
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
8
89
89
Dat
a-D
riven
Dia
logu
e
Ada
pted
from
B. W
ellm
an a
nd L
. Lip
ton,
Dat
a-D
riven
Dia
logu
e: A
Fac
ilita
tor’
s G
uide
to C
olla
bora
tive
Inqu
iry, S
herm
an, C
T:
Mira
Via
LLC
, 200
4. A
s fo
und
in N
. Lov
e, K
.E. S
tiles
, S. M
undr
y, a
nd K
. DiR
anna
, The
Dat
a C
oach’s
Gui
de to
Impr
ovin
g Le
arni
ng fo
r All
Stu
dent
s, T
hous
and
Oak
s, C
A: C
orw
in P
ress
, 200
8. A
ll rig
hts
rese
rved
.
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
9
©2012 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • Phone 978-263-9449 • www.RBTeach.com
Engage in Data-Driven Dialogue with Item Data: Data Coach Checklist
Preparation
¨ Collect item data for the content area, grade level, and time frame being analyzed from state or local assessments. You may also want to collect item data for a particular strand, standards within a strand, and/or multiple years of data to establish trends over time.
¨ Check data for accuracy. ¨ Prepare the data in table form as illustrated below:
¨ Prepare a predictions template (see sample in Institute Handouts). ¨ Provide a copy of the test blueprint or item map and relevant standards for each team member. ¨ Provide released items that correspond to item data being analyzed. ¨ Provide meeting agenda to team in advance. ¨ Prepare necessary materials (e.g., chart paper, pink, yellow, and green highlighters, markers, LCD
projector). Meeting Protocols
¨ Review purpose/agenda. ¨ Assign group roles (e.g., timekeeper, recorder, dialogue monitor, materials manager). ¨ Agree to norms on which the team will focus. ¨ Start and end on time. ¨ Review tools or protocols being used (e.g., Data-Driven Dialogue, Stoplight Highlighting). ¨ Review criteria for effective Data Team meetings (see last section below).
Item Data Analysis
¨ State questions that guide inquiry into item data: o What kinds of items are on the test? In what content strands? At what level of difficulty? o What knowledge, skills, and concepts are required for students to be successful with a
particular item? o What specific skills and understandings are our students’ strengths? Which pose
difficulties for them? o For which items are students frequently giving the same incorrect answers? o On what types of questions, such as short answer, extended response, or multiple-choice,
do our students perform well? Which pose difficulties? o Why are our students doing well or missing points on their open-response questions?
Engage in Data-Driven Dialogue with Item Data: Checklist
(cont. next page)
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
10
©2012 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • Phone 978-263-9449 • www.RBTeach.com
¨ Ask team to study released test items. ¨ Ask team to Predict (Phase 1) based on the following questions:
o What are our predictions about students’ performance on these items (standards)? o Which items (or standards) do we think they will do well on? Which will they have
difficulty with? o What trends will we see over time (using multi-year data)? o Based on what assumptions?
¨ Record predictions on chart paper or on the predictions template provided. (Note: predictions can be quantitative or descriptive.)
¨ Go Visual (Phase 2): o Provide team with a poster-size (paper) graph (for percentage correct and patterns over
time) and/or table (for percentage correct and distractor patterns) or electronic graph or table projected onto a screen or Smartboard.
o Ask team to check data for clarity and accuracy. o Ask team to determine criteria for Stoplight Highlighting (e.g., cut points to distinguish
urgent areas, team’s vision of an excellent school, or comparisons with state, district, or similar schools) for percentage correct and for distractor (incorrect responses) patterns
o Have team Stoplight Highlight their graph or table accordingly. ¨ Ask team to Observe (Phase 3) based on the questions below. Observations are best made without
looking at the released test items, just the table or chart. o What important points seem to pop out? o What is surprising or unexpected? o What are items of relative strength? Weakness? o What trends do we observe over time (if analyzing multiple years of data)?
¨ Record observations on chart paper. ¨ Ask team to refer back to released items that they are most interested in studying further and
Infer/Question (Phase 4) based on the following questions: o What would students need to know/do to be successful at this task? o Why might so many of our students have done well at a particular item? o What might students have been thinking to make the errors they did? o How can we find out which of our hypotheses is right? o What questions do we have? o What additional data might we need?
¨ Reflect on next steps and implications for actions. Reflect on the Criteria for Effective Data Team Meetings
¨ Did we follow protocols (e.g., Data-Driven Dialogue)? ¨ Did we observe our norm/s? ¨ Did we avoid blame and culturally blind or destructive behaviors? ¨ Did we “look for love in the all the right places,” that is, look for possible explanations and
actions in those areas that impact student learning: curriculum, instruction, assessment, equity practices, and critical supports?
¨ Did we determine clear next steps that will impact students and their learning? ¨ How can we improve our Data Team meetings in the future?
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
11
Engage in Task Deconstruction and Data-Driven Dialogue with Student Work: Checklist
(cont. next page)
(See also The Data Coach’s Guide to Improving Learning for All Students, pp. 218-226, and Unleashing the Power of Collaborative Inquiry: A Program for Data Coaches, Course Handouts, Task 10.)
Preparation q Ask team members to bring the task or item and the student work they will be analyzing (one set
for each team member) or collect and prepare student work yourself. q Bring copies of relevant standards and rubrics related to the task. q Provide meeting agenda to team in advance. q Prepare necessary materials (e.g., chart paper, markers, Post-its).
Meeting Protocols q Review purpose/agenda. q Assign group roles (e.g., timekeeper, recorder, dialogue monitor, materials manager). q Agree to norms on which the team will focus. q Start and end on time. q Review tools or protocols being used (e.g., Data-Driven Dialogue). q Review criteria for effective Data Team meetings (see last section below).
Task Deconstruction with Student Work Analysis q State questions that guide inquiry into student work:
o What evidence are we seeing of student mastery of the knowledge and skills required by the task?
o What errors are students making? o What knowledge and skills seem to be missing? o What additional insights into student thinking are we gaining?
q Deconstruct the task. Ask team to: o Do the task and share solutions or strategies. o Brainstorm, drawing on our own experience doing the task:
• What do students need to know and be able to do to be successful at this task? • Write each piece of knowledge and each skill on a large Post-it, one item per Post-it.
o Refine what we have generated based on:• Consulting relevant standards and rubrics.• Focusing on the three to six key concepts/skills in the content area being assessed.• Focusing on ideas and skills that would inform reteaching and extension.
q Ask team to Predict (Phase 1) based on the following questions:• What do students need to know and be able to do to be successful at this task?• How do we think our students performed?• What do we think they had trouble with?• What kinds of errors or misconceptions do we anticipate?• Based on what assumptions?
q Record predictions on chart paper. q Pass out samples of student work.
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
12
Go Visual (Phase 2): Recreate the table below on chart paper.
Student(list below)
Know/Can Do
Adapted from Research for Better Teaching, Sstudying Skillful Teaching Course Handouts, Acton, MA: Research for Better Teaching, 2011.
o Record students’ names in left-hand column. o Place Post-its with the knowledge and skills identified in task deconstruction in the top row,
over the words “Know/Can Do.” o Next to each student’s name, place a check in each column where there is evidence that the
student has demonstrated the requisite knowledge or skill identified. o Note student errors or misconceptions in the last column.
q Ask team to Observe (Phase 3) based on the following questions: o What patterns or trends do we observe across several pieces of work (examine the table by
columns)? o What patterns in errors and misconceptions are emerging (examine last column)? o What do we notice about individual students (examine the table by rows)?
q Record observations on chart paper. q Ask team to Infer (Phase 4) based on the following questions:
o What new insights have we gained about the student-learning problem? o What might be contributing to students’ lack of understanding or skill? What errors are we
noticing? What misconceptions are we seeing evidence of? o What additional questions are raised by the student work? o What additional data could be helpful? o If relevant, consider if examination of student work confirms or refutes the tentative conclusions
we drew from other data analysis o Reflect on next steps and implications for action
Reflect on the Criteria for Effective Data Team Meetings q Did we follow protocols (e.g., Data-Driven Dialogue)? q Did we observe our norm/s? q Did we avoid blame and culturally blind or destructive behaviors? q Did we “look for love in the all the right places,” that is, look for possible explanations and actions
in those areas that impact student learning: curriculum, instruction, assessment, equity practices, and critical supports?
q Did we determine clear next steps that will impact students and their learning? q How can we improve our Data Team meetings in the future?
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
13
Error Analysis Protocol
Advanced Preparation• To select an item for error analysis, identify a frequently missed item in a benchmark, matching pre-
post, or short readiness or diagnostic assessment, or in a quiz given after instruction. If the item is multiple-choice, bring a report that indicates that percentage of students who answered each of the distractors as well as the correct answer. You can use the Error Analysis Template as you step through the process.
• Alternatively, bring in student work from any of the other pre-assessment (or post-assessment) sources. Provide 6-10 samples (or more if the student work is short) that illustrate a range of student thinking and proficiency.
At the Meeting • Do the problem or item first individually or in pairs. Share solutions and strategies.• Engage in Data-Driven Dialogue, guided by the questions below. Note that your team might need to
gather more data (question 4) before moving to questions 5–7. Be sure to record team responses to the questions on newsprint or projected on a screen or white board.
1. What will students need to know and be able to do in order to be successful at this item? What kinds of errors or misconceptions do we anticipate students will make? (Predict)*
2. What errors are students making? (Observe)
3. What might they have been thinking to make these errors? (Infer)
4. How can we find out which hypothesis is true? (Investigate/Verify Causes)
5. What different teaching (reteaching) strategies could we use to help students understand their errors, unravel their confusion, and/or correct a misconception? (Generate Solutions)
6. How can we manage time, tasks, and student groups to assure that students receive the instruction they need? How can the team help? (Generate Solutions)
7. What individual and collective action do we commit to?
*Note: If you have already determined how students performed on the item through a previous item analysis, skip the prediction step.
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
14
Error Analysis TemplateNote: This template can be used for error analysis on multiple-choice assessment items. Note that before progressing to “Verified Hypotheses” on the template, teams might need to collect additional data, e.g., ask students to provide a written or verbal explanation of why they chose the answer they did and why they did not choose the others.
Assessment Date
Item # A B C DPercentage Responding to Each ChoiceIndividual Students Who Selected Each Response
Hypotheses About Student Thinking: How Can We find Out?
Verified Hypotheses
Strategies for Teaching, Reteaching, Grouping, and/ or Extending Learning
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
15
BECA
USE
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
16
�
A DAtA CoACh’s GuiDe to improvinG LeArninG for ALL stuDents CopyriGht © 2008 Corwin press
Handout H1.6Norms of Collaboration
Two sets of collaborative norms that have been useful to a variety of school teams are the Seven Norms of Collaboration (Garmston & Wellman, 1999) and the Four Agreements of Courageous Conversations (Singleton & Linton, 2006). Each is described briefly below. After discussing these sets of norms, the Data Team can decide to (1) adopt one, the other, or both, (2) adapt one or both, or (3) generate your own new set of norms for working together.
The Seven Norms of CollaborationPausing: Pausing slows down the “to and fro” of discussion. It provides for “wait time,” which has been shown to dramatically improve thinking. It signals to others that their ideas and comments are worth thinking about, dignifies their contributions, and implicitly encourages future participation. Pausing enhances discussion and greatly increases the quality of decision making.
Paraphrasing: To paraphrase is to recast into one’s own words, to summarize, or to provide an example of what has just been said. It helps members of a team hear and understand each other as they evaluate data and formulate decisions, and it helps to reduce group tension by communicating the attempt to understand. Signal your intention to paraphrase (“So, you’re suggesting…”), and choose a level for the paraphrase: (1) acknowledge and clarify; (2) summarize and organize; or (3) shift the focus to a higher or lower level.
Probing for specificity: Probing seeks to clarify something that is not yet fully understood. More information may be required or a term may need to be more fully defined. Clarifying questions can be either specific or open ended, depending upon the circumstances. Ask for clarification of vague nouns and pronouns (e.g., “they”), action words (e.g., “improve”), comparators (e.g., “best”), rules (e.g., “should”), and universal quantifiers (e.g., “everyone”).
Putting ideas on the table and pulling them off: Ideas are the heart of a meaningful discussion. Members need to feel safe to put their ideas on the table for discussion. To have an idea be received in the spirit in which you offer it, label your intentions: “This is one idea…” or “Here’s a thought….” The other half of this norm is equally important: knowing when an idea may be blocking dialogue or “derailing” the process and therefore should be taken off the table.
Paying attention to self and others: Collaborative work is facilitated when each team member is explicitly conscious of self and others—not only aware of what he or she is saying, but also how it is said and how others are responding to it. We need to be curious about other people’s impressions and understandings but not judgmental. As we come to understand someone else’s way of processing information, we are better able to communicate with them.
Presuming positive intentions: This is the assumption that other members of the team are acting from positive and constructive intentions, even if we disagree with their ideas. Presuming positive presuppositions is not a passive state; rather, it needs to become a regular part of one’s verbal responses. The assumption of positive intentions is an aspect of the concept of a “loyal opposition,” and it allows one member of a group to play “the devil’s advocate.” It builds trust, promotes healthy disagreement, and reduces the likelihood of misunderstanding and emotional conflict.
(cont. next page)
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
17
A D
ata
Coac
h’s
Gui
de to
Impr
ovin
g Le
arni
ng fo
r All
Stud
ents
�
Ch
apte
r 3, T
ask
�: H
ando
ut H
�.6
Pursuing a balance between advocacy and inquiry: Both advocacy and inquiry are necessary components of collaborative work. The intention of advocacy is to influence others’ thinking; the intention of inquiry is to understand their thinking. Highly effective teams consciously attempt to balance these two components. Inquiry provides for greater understanding. Advocacy leads to decision making. Maintaining a balance between advocating for a position and inquiring about the positions held by others helps create a genuine learning community.
Adapted from Robert J. Garmston and Bruce M. Wellman, The Adaptive School: A Sourcebook for Developing Collaborative Groups. 1999. pp. 37-
47. Norwood, MA: Christopher Gordon. Used with permission.
The Four Agreements of Courageous ConversationsStay engaged: Staying engaged means “remaining morally, emotionally, intellectually, and socially involved in the dialogue” (Singleton & Linton, 2006, p. 59).
Experience discomfort: This norm acknowledges that discomfort is inevitable, especially in dialogue about race, and that participants make a commitment to bring issues into the open. It is not talking about these issues that creates divisiveness. The divisiveness already exists in the society and in our schools. It is through dialogue, even when uncomfortable, that healing and change begin.
Speak your truth: This means being open about thoughts and feelings and not just saying what you think others want to hear.
Expect and accept nonclosure: This agreement asks participants to “hang out in uncertainty” and not rush to quick solutions, especially in relation to racial understanding, which requires ongoing dialogue.
Adapted from Glenn E. Singleton & Curtis Linton, Courageous Conversations about Race: A Field Guide for Achieving Equity in Schools. 2006. pp.
58-65. Thousand Oaks, CA: Corwin.
Source: Nancy Love, Katherine E. Stiles, Susan Mundry, and Kathryn DiRanna, The Data Coach’s Guide to Improving Learning for All Students: Unleashing the Power of Collaborative Inquiry, Thousand Oaks, CA: Corwin Press, 2008, CD-ROM, Handout H1:6.
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
18
Ann
ual S
MA
RT
Stud
ent-L
earn
ing
Goa
l: B
y Ju
ne, 2
014,
impr
ove
6th
grad
e st
uden
ts’ p
erfo
rman
ce in
com
preh
ensi
on o
f inf
orm
atio
nal t
ext a
s ev
iden
ced
by in
crea
sed
perc
enta
ge o
f stu
dent
s sc
orin
g pr
ofic
ient
from
50
% to
75%
ann
ual a
sses
smen
ts…
(lis
t act
ions
). In
terim
SM
AR
T St
uden
t Lea
rnin
g G
oal:
In s
econ
d qu
arte
r ben
chm
ark,
in
crea
se p
erce
ntag
e pr
ofic
ient
in c
ompr
ehen
sion
of i
nfor
mat
iona
l tex
t to
70%
. In
stru
ctio
nal o
r Ret
each
ing
Stra
tegy
We
Agr
ee O
n M
onito
ring
Impl
emen
tatio
n: H
ow W
ill
We
Kno
w W
e D
id It
?
Mon
itorin
g S
tude
nt L
earn
ing:
H
ow W
ill W
e K
now
It
Wor
ked?
• Tea
ch u
se o
f gra
phic
or
gani
zers
• P
rovi
de s
tude
nts
with
at
leas
t fou
r pra
ctic
e op
portu
nitie
s th
is m
onth
• Brin
g st
uden
t ex
ampl
es o
f gra
phic
or
gani
zers
to o
ur
next
mee
ting
• Ana
lyze
info
rmat
iona
l te
xt c
ompr
ehen
sion
on
open
-res
pons
e as
sess
men
t
Shor
t-Cyc
le A
ctio
n Pl
ans
for G
rade
-Lev
el
or C
onte
nt T
eam
s
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
19
Shor
t-Cyc
le A
ctio
n Pl
ans
for G
rade
-Lev
el o
r C
onte
nt T
eam
s
Ann
ual S
MA
RT
Stu
dent
-Lea
rnin
g G
oal:
Inte
rim S
MA
RT
Stu
dent
-Lea
rnin
g G
oal:
In
stru
ctio
nal o
r R
etea
chin
g S
trate
gy
We
Agr
ee O
n
Mon
itorin
g Im
plem
enta
tion:
How
Will
W
e K
now
We
Did
It?
Mon
itorin
g S
tude
nt
Lear
ning
: How
Will
We
Kno
w It
Wor
ked?
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
20
�
A DAtA CoACh’s GuiDe to improvinG LeArninG for ALL stuDents CopyriGht © 2008 Corwin press
Purpose To focus the Data Team on data that indicate a need for urgent attention.
Overview Stoplight Highlighting helps Data Teams analyze data they have represented in the Go Visual phase of Data-Driven Dialogue. Based on relevant criteria, Data Teams use highlighters to mark positive data as a “green light,” data that represent caution as a “yellow light,” and data that demand immediate attention as a “red light.”
Audience Data Team.
Use Primary Tasks: Tasks 6–9 and 11.
Advance Preparation1. In addition to the data noted above, gather information about the Data
Team’s, school’s, or district’s student-learning and achievement criteria for growth and improvement. Examples include your Data Team’s vision of a great school; any national, state, or local criteria already established for expected percentage of students reaching proficiency, of items correct, or of annual improvement.
Procedure 1. Direct the Data Team’s attention to the data on the chart they have
created for Phase 2: Go Visual. Depending on the task, this chart may focus on aggregated, disaggregated, strand, or item-level data.
2. Introduce Stoplight Highlighting as a process that enables the Data Team to highlight student-learning needs and successes. Explain the analogy of a stoplight: some of the data is “good to go” and is noted as green; some represents “caution” and is noted as yellow; some is in need of “immediate attention” and is noted in red. Stoplight Highlighting is a tool to guide and inform the team’s observations.
3. Share the school/district criteria for student learning and achievement growth and improvement. Facilitate a discussion about realistic criteria and bring the group to consensus about which criteria they will use. Write this information on a wall chart similar to Resource TR1 (Example of a Stoplight Highlighting Criteria Table). (Note that aggregated and disaggregated data are often reported as a percentage in each proficiency
29
Time �0-�5 minutes with each data set.
Materials Resources
TR�—Example of a Stoplight Highlighting Criteria Table
TR2—Stoplight Highlighting Vertical Plot Example
Data
Tables or graphs representing aggregated, disaggregated, strand, or item-level student-learning data
General
Chart paperHighlighters (green, yellow, red)Masking tape
Stoplight Highlighting
(cont. next page)
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
21
A D
ata
Coac
h’s
Gui
de to
Impr
ovin
g Le
arni
ng fo
r All
Stud
ents
2
To
olki
t: St
oplig
ht H
ighl
ight
ing
level, while strand or item-level data may be reported as a percentage correct; adjust your criteria accordingly.)
4. Ask team members to use the criteria to highlight their data. What is the range for “green,” for “yellow,” and for “red”?
5. Ask the team members to continue in their Data-Driven Dialogue, making observations and inferences about the highlighted data. In what areas can they celebrate student progress? What areas are in need of improvement?
6. If there are several “red” areas, have members discuss/determine a priority for addressing these areas. Do some naturally align themselves with others? If some were addressed, would others fall into place?
7. Use the “red” areas to target areas of focus for use in subsequent tasks and data sets. For example, if the Data Team is engaged with aggregated data and sixth-grade science is in the “red” zone, home in on disaggregated sixth-grade science in the next task.
Adapted from The ToolBelt: A Collection of Data-Driven Decision-Making Tools for Educators. Copyright ©
2004, Learning Point Associates. All rights reserved. Used with permission.
Facilitation Note
Stoplight Highlighting works best with line graphs for aggregated and disaggregated data. Bar graphs can be confusing because it will appear that some of the proficient students are in the red or yellow zone.
(cont. next page)
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
22
A D
ata Coach’s Guide to Im
proving Learning for All Students
� Toolkit: Stoplight H
ighlighting
TR1
Example of a Stoplight Highlighting Criteria Table: CRT Aggregated and Disaggregated Data
HigHligHT coloR Meaning PRoficiency level
green go! Meets expectations above 70%
yellow caution! Below expectations
Between 60% and 69%
RedUrgent!
in immediate need of improvement
Below 60%
Adapted from The ToolBelt: A Collection of Data-Driven Decision-Making Tools for Educators. Copyright © 2004, Learning Point Associates. All rights
reserved. Used with permission.
(cont. next page)
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
23
A D
ata
Coac
h’s
Gui
de to
Impr
ovin
g Le
arni
ng fo
r All
Stud
ents
�
To
olki
t: St
oplig
ht H
ighl
ight
ing
Stoplight Highlighting Vertical Plot Example: Percentage Proficient on Eighth-Grade Mathematics
Strand on State CRT, Year 3
PERCENT
MATHEMATICS STRANDS AND LEVELS OF UNDERSTANDING
Number of Students: 191
100
90
80 Geometry 78%
70
60 Data analysis/probability 61%sis/pMeasurement 58%
50Number sense 54%
40 Algebra 40%Computation 38%Knowledge and sk ills 35%
30 Conceptual understanding 30%Application/problem solving 27%
20
10
0
Go! (green)
Caution! (yellow)
Urgent! (red)
TR2
Source: Nancy Love, Katherine E. Stiles, Susan Mundry, and Kathryn DiRanna, The Data Coach’s Guide to Improving Learning for All Students: Unleashing the Power of Collaborative Inquiry, Thousand Oaks, CA: Corwin Press, 2008, CD-ROM, Toolkit.
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
24
Student Error Analysis
Name _________________
STUDENT ERROR ANALYSIS Assessment: ___________
Directions: Please review your assessment. For any questions you are correcting, please complete the following:
1. Write the number of the question you are correcting 2. Classify your error 3. Re-do the problem correctly 4. Explain how to do the corrected problem
A =
Arithmetic (You made a
calculation error - added, subtracted,
multiplied, or divided incorrectly)
C = Careless
(You made a silly mistake)
D = Directions (You didn’t follow the directions)
E= Explanation
(Your explanation was incomplete)
U = Understanding
(You did not understand how to do the problem)
Problem # _________ Error Classification: _______
Re-do the problem correctly
Explain the error you made
Note: This tool is to be used with students as a way for them to analyze and correct their own errors.
(cont. next page)
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
25
Problem # _________ Error Classification: _______
Re-do the problem correctly
Explain the error you made
Problem # _________ Error Classification: _______
Re-do the problem correctly
Explain the error you made
©2013 Michelle A. Savage This document is for individual use only. For any other purposes, please contact Michelle Savage
at [email protected] for permission.
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
26
Verifying Causes
Advance Preparation• Make a copy of the Verifying Causes Graphic to illustrate the process or show it on a slide.
• Make one copy of Verifying Causes Template for the team to fill out together.
At the Meeting(s)• Generate a hypothesis or possible cause for patterns of errors, misconceptions, or gaps in student
understanding that you observed in your data analysis (using any of the protocols provided).
• If the team determines that they do not have enough information from which to draw a sound conclusion about student thinking, use this Verify Causes protocol. Use the Verify Causes graphic provided to illustrate and explain the process. Note: It may be important to explain to the team why this extra step of verifying causes is important. If so, point out the danger of rushing to a premature conclusion without adequate data. This may lead to wasted time and effort attempting to solve the wrong problem.
• Using the Verifying Causing Template provided, insert the team’s hypothesis in the first section of the template.
• Determine what additional evidence is needed in order for the team to have confidence in their hypothesis and to generate appropriate solutions. For example, you might ask students to explain their answers verbally or in writing, observe students as they are working, or look to research on misconceptions.
• Insert what additional data will be collected and how in the next section of the template labeled “Additional Data/Research to Collect” and “How, By Whom, and When.”
• Collect evidence between meetings and come to the next meeting ready to analyze evidence, draw conclusions, and plan next steps.
• Use the “Findings (Observations)” section of the template to record observations of the data.
• Use the “Verified Hypothesis” section of the template to record the conclusion drawn from the data analysis. If the data confirm the team’s hypothesis, they can proceed to generating solutions. If the data refutes their hypothesis, the team revises their hypothesis, collects additional data if needed, and proceeds to generating solutions. (Note: Initial data collection may be sufficient to support a revised hypothesis.) And, if the data are inconclusive, they may decide to collect additional data.
• Record next steps in the final row of the template.
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
27
Verifying Causes Graphic
This is a graphic representation of the Verify Causes process.
• “Hypothesize Possible Cause” represents the team’s inference about what is causing the errors, misconceptions, or gaps in student thinking observed through data analysis.
• “Collect Additional Data” represents the various data sources the team might use to verify their hypothesis.
• Once they have collected the data or research, the team analyzes the data to determine if the data confirms or refutes their hypothesis or is inconclusive.
• The green arrow labeled “Data Confirms” illustrates that if the team has verified their hypothesis, they move on to generating solutions.
• The red arrow labeled “Data Refutes” shows that if the data do not support the hypothesis, the team revises their hypothesis.
• The yellow arrow labeled “Data Inconclusive” illustrates that if the data neither confirm nor refute their hypothesis, the team undertakes additional data collection.
Hypothesize Possible Cause
Collect Additional Data, e.g. • student interviews • observations • additional work • research
Generate Solutions
DATA CONFIRMS
DATA REFUTES
DATA IN
CO
NC
LU
SIV
E
Verify Causes – Short Cycle
© 2012 Research for Better Teaching, Inc. www.RBTeach.com
Verify Causes
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
28
Verifying Causes Template
Hypothesis to Test:
Additional Data/Research to Collect (e.g., student work, interviews, research on misconceptions):
How, By Whom, and When:
Findings (Observations):
Conclusion:____Hypothesis confirmed (go to template sections below)____Hypotheses refuted (write revised hypothesis below or repeat sections above as needed)____Hypothesis inconclusive (redo sections above as needed)Verified or Revised Hypothesis:
Next Steps:
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
29
Data Tools Organizer
Tool Where Can I Find It?
Notes
© 2014 Research for Better Teaching, Inc. • One Acton Place, Acton, MA 01720 • +1-978-263-9449 • [email protected]
30