reporter's guide to research: how to read and write about studies

Reporter's Guide to Research: How to Read and Write About Studies

Holly YettickPostdoctoral Fellow, University of Colorado Denver School of

Public AffairsContributing Writer, Education Week

May 18, 2014

WHY COVER ED RESEARCH?

Because, for the most part, you don’tJan-June 2010: Percent of Articles that Mentioned Educational Research

or Experts

Daily Newspa-

pers

Online-Only Outlets

Education Week

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

Among Articles that Mention Research: Percent that Mention Research From Peer-Reviewed

Journals

Daily News-papers

Online-Only Outlets

Education Week

0%1%2%3%4%5%6%7%

Why Prioritize Peer Review?

It’s not perfect by any means but there is some quality control.

Editor decides if topic is important, appropriate for publication and , if so, sends to reviewers.

Reviewers (usually 3, sometimes 2) read, pick apart your article, recommend revisions and whether to accept as is or with minor modifications, revise and resubmit or reject. Editor makes final decision on basis of reviewer comments.

Reviewers don’t know who you are. You don’t know why they are. They make many, many suggestions.

Like your editor..but worse• More information (if there is any) should be included about

parents' ideas of high quality education for their children especially in minority groups.

• Is there any way to include the racial/SES demographics of the choice schools? For example, would a Black family select a slightly-lower performing school over a higher performing one if more Black students attended the lower-performing school? Is race/class important to families when choosing schools, especially when families are given access to the same school information?

Should reporters cover research? Why or why not?

• You’re missing good stories

• You’re missing a chance to add perspective, which can make it seem like news is an endless stream of unconnected events.

• You’re missing a chance to help people make decisions about schools on the basis of some of the better evidence out there

Ways to Cover Research

• Cover findings of a study or studies.

• Use findings to provide perspective about the topic you’re covering. (Truly, there is research about almost EVERYTHING you cover.)

WHY DO YOU THINK REPORTERS OFTEN AVOID COVERING OR MENTIONING EDUCATIONAL RESEARCH?

LOCALISMOBSTACLE 1

FOIA the research proposals submitted to your local school district(s) or state education departments

RFPs FROM DISTRICTS, STATES

Search for studies conducted by local researchers and/or in local schools

• Google scholar

• Search journals recommended in the handout.

• Search conference programs.

• Look at the web pages of local professors. What have they published lately? Find it.

WHAT ARE THE BIG ONGOING EDUCATION TOPICS IN YOUR AREA?

LIT REVIEWS!!!

Review of Educational Research (good source, top-rated journal)

• Consider buying this book:• Visible Learning: A synthesis of over 800 meta-

analyses relating to achievement, John A.C. Hattie, Routledge, 2009

Types of lit reviews

Meta-analysis: Combines effects of quantitative studies to come up with overall effectNarrative/qualitative: Summarizes without necessarily quantifying overall effects

• Sometimes you will also see a lit review of lit reviews or a meta-analysis of meta-analyses

WHY LIT REVIEWS?

• Any one study, no matter how good, can be wrong. Better to look at the accumulated knowledge.

• Lit reviews help you identify what we know and what we do not.

• If new studies emerge on your big topic, you will know how they fit in with past research.

I DON’T UNDERSTAND WHAT THEY’RE TALKING ABOUT!

OBSTACLE 2

Educate Yourself

• Take a masters-level or undergrad class on research methods or statistics.

• (BONUS…You’ll probably get free access to the university’s library, including subscription-only databases you can use from home.)

Educate yourself online

• Statisticshell.com

I like this book

Wanted: Personal Tutor• Someone who understands statistics

• Someone who understands education

• Someone who is good at breaking down complex topics in an understandable manner (i.e., you need a good teacher!)

• Examples: Psychometrician (studies testing), economist of education, professor of quantitative methodology/research methods in an education school

DISCLAIMER!

• I HAVE NO IDEA IF THESE PEOPLE ACTUALLY WANT TO BE YOUR PERSONAL TUTOR…

THEY’RE JUST EXAMPLES OF THE TYPE OF PERSON TO CONSIDER!

HOW TO SCAN AN ACADEMIC ARTICLE TO SEE IF IT’S INTERESTING: THE ABSTRACT, THE DISCUSSION & THE CHARTS

I DON’T UNDERSTAND WHAT THEY’RE TALKING ABOUT..CONTINUED..

ABSTRACT

• Single-paragraph summary of the article.

• If it’s any good, it’s all you need to determine if the article is what you’re looking for.

• Almost always free, even if the article is behind a paywall.

DISCUSSION

• The results section after the results section

• May be called different things in different fields (E.g. “Conclusion”)

• Repeats findings then puts them in context so you can tell what they mean

CHARTS

Quantitative study Often tell the story faster than the textCan be used to illustrate your story

Qualitative studyMay provide examples of types of responses received. Examples can be used as quotesMay Describe the sample. (E.g., how many people interviewed, how old were they, etc?)

AN ASIDE: A PLEA FOR COVERAGE OF QUALITATIVE RESEARCH

• Yes, it’s kind of like journalism…on steroids• It’s much more systematic. You must justify why

you decided to interview these people, emphasize these quotes etc., then justify it.

• It can help you answer the question of “why” and “how” something happens.

• EG: WHY does KIPP have high test scores?• HOW do middle class parents create educational

advantages for their children?

THE ARTICLE IS INTERESTING AND RELEVANT…WHAT DO I NEED TO KNOW TO WRITE A GOOD STORY ABOUT IT AND WHERE CAN I FIND IT?

I DON’T UNDERSTAND WHAT THEY’RE SAYING..CONTINUED..

WHO?

Q: How many people were in the study?Source: Methodology section

Q: Who were these people? Especially important: What characteristics of these people could effect how the study turned out?(E.g., SES-school type homework example)Source: Methodology section

WHO? (For studies of interventions… like a new reading program)

• Who are the subjects of the study being compared to? (Source: Methodology section)

• Some possible responses:– A group of people was randomized into a “treatment”

group that got the intervention and a “control” group that did not

– Statistical methods used to identify a group that is similar to those who got the treatment

– Themselves: A treatment is abruptly introduced and researchers compare results before and after the intervention

WHAT?

What was the study outcome?(Source: Results, Discussion sections)

How BIG was the outcome? How BIG was the outcome for certain groups (eg. English learners)?(Source: Results, Discussion)

SIZE MATTERS…DON’T BE AFRAID TO ASK FOR A TRANSLATION!

Would this be the equivalent of moving, for instance, from the 50th to the 60th percentile in reading?

How many days/years of learning did the students gain or lose as a result of the intervention?

WHY?

• Why does the researcher think she got the results she got? (Source: Discussion)

• Why did the researcher do the study in the first place? What hole in the body of knowledge was she trying to fill with her study? (Source:Literature review, conceptual framework)

• How do the results of this study compare with the results of past studies? (Source:Literature review, Discussion)

HOW DO I KNOW IF IT’S OBJECTIVE?

OBSTACLE 3

Good Signs

• Publication in a peer-reviewed journal or presentation at an academic conference. (Again, not perfect!)

• Authors describe the study’s limitations. (Every study has them.)

• Author is not presenting the study as the greatest thing since sliced bread.

You got a news release about the study

• Someone thinks it’s worth paying some $$$ to promote the study. This could be entirely innocent. But it’s worth asking why.

• The problem is often not the research, it’s the news release

• The person who promotes the study is often not the person who did the study. The two sometimes have different motives.

READ THE STUDY! NOT JUST THE NEWS RELEASE!

Too good to be true?

• Is the news release implying that someone has discovered a silver bullet or a root cause of a problem?

• Does that bullet or cause just happen to match the agenda of the organization producing the report?

NONE OF THESE ARE GUARANTEES OF SHODDY RESEARCH! THEY’RE JUST SIGNS YOU SHOULD ASK

QUESTIONS AND DIG DEEPER!

Other Questions

• If the report assigned letter grades (e.g. to districts, states):

Always ask: What is the basis for these grades?And, of course, put the answer in your story

Is the study suggesting that there is an association between two things that seem utterly unrelated to you? Is at least one of these things supported by the advocacy organization that published the report? Again, it’s not a guarantee of shoddy research. But..Do a little digging.

Objectivity: Suggestions

• Don’t assume that just because an organization has a reputation for being centrist , right or left-leaning in some fields of research (e.g. international affairs) that its education research is also centrist, right or left-leaning.

Funding

• Find out who funded the study and report it!

• But don’t automatically assume that the funding source biases the research. Sometimes it does. Sometimes it does not.

(Analogy: Is all of your coverage about a local college or school biased because they advertise in your publication?)

Is it Really Bias?

• Sometimes what looks like biased results is actually a difference in the types of questions askedExampleIs this intervention providing equal levels of opportunity to families of different income levels and racial and ethnic backgrounds?VERSUSIs this intervention providing families with opportunities to make their own decisions about how they’d like to educate their children?

When in Doubt, Time Out

First, ask someone knowledgeable about the topic to vet the study. If you find out it’s seriously flawed or biased, write about that. Or just ignore the study.

If you do decide to just put it out there and let readers judge for themselves, tell them that’s what you’re doing. Don’t assume they know the difference between the time and effort you devote to..say..your Twitter feed or your blog versus your big takeout.

View from the dark side: Researcher Pet Peeves

• Warring experts frame, especially when it involves personal attacks

• Quote from a researcher, who summarizes relevant studies, is juxtaposed with quote from a kindergartener who had an experience that does not align with the overall research findings. OR anecdotal lede does not reflect research findings reported in study, implying that a personal anecdote raise serious questions about hundreds of studies’ results.

Other Peeves: Researchers

• The Eduttante: That one person in your town who is quoted over and over again in the media on every education-related topic known to man even though he is not actually much of an expert on any of them.

• Describing something as an “experiment” when it is not, technically, an experiment.

Common Researcher Misconceptions of Our Work

• We select the topics and frames of our articles largely on the basis of influence from our funders (if non-profit) or corporate ownership (if for-profit).

• There is not a wall between editorial and news. Also, columns=articles.

• We mainly care about conflict and controversy• We are not very bright and probably can’t

understand their research. (Note: This is a minority view!)

Reporter Misconceptions of Researcher Work

• Researchers base conclusions based almost entirely on who funds them

• Research is really just another form of opinion• If the data analyzed is more than a few months

old, it’s out of date and not worth covering.• Qualitative work=journalism. So why cover it?• Research is impossible to understand without an

advanced level of knowledge of statistics.

Research Can Even be a Good Read..Give it a try!

• Here are just a few of my favorites• Annette Lareau, Unequal Childhoods• Elizabeth A. Armstrong & Laura T. Hamilton,

Paying for the Party: How College Maintains Inequality

• Jay MacLeod, Ain't No Makin' It: Aspirations and Attainment in a Low-Income Neighborhood