the dangers of using data too much (pdf)
TRANSCRIPT
Presented by: Nichole Hertel
EDRS 8381 Research Methods in Education II
Relying on Data
TOO Much in
Your Research
Laughter is always the best medicine!
Define the term “the new stupid”
Answer the question – “What exactly
does it mean to use data or research to
inform decision-making policies?
How much data needs to be included in a
research paper or article?
What are the dangers of the data itself?
But in an age when information is literately
at our fingertips, don’t we have an obligation
to provide as much information as possible
on a specific topic?
Objective – To put together the puzzle found
in the envelopes
“Data-based decision making”
“Research-based practice”
They can stand-in for the careful cognitive thought
process necessary to improve the system
They can serve as dressed-up rationales for the same
ole fads, i.e. different words for the same idea
They can be used to justify incoherent proposals to
decision-making bodies
What exactly does it
mean to use data or
research to inform
decisions?
The Old Stupid = resistance to performance
measures
Today over-reliance on performance measures
has lead to a dependence on a few simple
metrics
Graduation rate
Expenditures
Reading and Math test scores of 3rd through
8th grade students
Element #1 - Using Data in Half-baked Ways
Element #2 – Translating Research Too
Simplistically
Element #3 – Giving Short Shrift to
Management Data
Future Superintendent - “Day one, we’re going to start
identifying those higher value-assessed teachers and moving
them to the schools that aren’t making AYP.”
Can we be confident that teacher who are effective in t heir
current classrooms would be equally effective elsewhere?
What effect would shifting teachers to different schools have on
the likelihood that teachers would remain in the district?
Are the measures in question good proxies for teacher quality?
What steps might either encourage teachers to accept
reassignment or improve recruiting for underserved schools?
Truly embrace the data by asking and
answering the hard questions
Consider organizational realities
Contemplating unintended consequences
Even policies or practices informed by rigorous
research can prove ineffective if the translation
is clumsy or ill considered
Most vexing problem in “research-based
practice” is the failure to recognize the limits of
what even rigorous scientific research can tell us
Decisions about governance, management and
compensation cannot be examined under
controlled conditions
Data-driven system is the result of leaders
embracing student achievement data instead
of paying attention to collecting/using more
relevant data to improve the performance of
schools and districts
Current achievement data are of limited use
for management purposes
Student achievement measures are largely
irrelevant to judging the performance of
many school district employees
Not simply indentify effective teachers or
struggling students, but should also help
render schools and school systems more
supportive of effective teaching and learning
Require that data assessments need to be
returned quickly, rather than years after it
was first administered
Key #1 – Educators should be wary of allowing data/research to substitute for good judgment
Key #2 – Schools must actively seek out the kind of data they need as well as the achievement data external stakeholders need
Key #3 – Educators should have an understanding of the limitations of research as well as its uses, especially when crafting policy
Key #4 – School systems should reward educational leaders and administrators for pursuing more efficient ways to deliver services rather than punish them
Research and data are powerful tools
In this era, educators stand to benefit enormously from advances in research and data systems
Be careful of catching “data enthusiasm” – you might miss the bigger picture
Hess, F. M. (2009). The new stupid. Educational
Leadership , 66 (4), pp. 12-17.