Crunching The Crunching The numbersnumbers

The secrets behind big data The secrets behind big data journalism projectsjournalism projects

Jennifer LaFleur, Center for Jennifer LaFleur, Center for Investigative ReportingInvestigative Reporting

Andy Lehren, The New York TimesAndy Lehren, The New York Times

July 2014July 2014

It’s all about the story

Great data stories are not just a bunch Great data stories are not just a bunch of numbersof numbers

Like any investigation, you are trying Like any investigation, you are trying to find the best storyto find the best story

If you have all the data, you have all If you have all the data, you have all the anecdotes to choose from. Find the anecdotes to choose from. Find the most compelling.the most compelling.

Find anecdotes that fit with Find anecdotes that fit with overarching themesoverarching themes

The New York Marathon

Combining dataCombining data

Put together more than one data set Put together more than one data set to get sparks to flyto get sparks to fly

Look for as many data sets as Look for as many data sets as possible that might tell the talepossible that might tell the tale

Helps with figuring out holes, Helps with figuring out holes, verifyingverifying

The Snowden FilesThe Snowden Files

Smartphones, leaky apps, locations

Stories do not stop with Stories do not stop with datadata

All the traditional tools applyAll the traditional tools apply Documents, interviewing, freedom of Documents, interviewing, freedom of

information requests information requests Storytelling and narratives: How you Storytelling and narratives: How you

tell the tale remains importanttell the tale remains important Data can allow you to tell those tales Data can allow you to tell those tales

with more authoritywith more authority

Amazing thingsAmazing things… can be amazing …… can be amazing …

…or wrong…or wrong Verify everything Verify everything Outliers – extremes – are often the best Outliers – extremes – are often the best

stories. They may also be wrongstories. They may also be wrong Bad data? Incomplete data? Changes in Bad data? Incomplete data? Changes in

the way data was recorded? the way data was recorded? The limits of data: It can tell you what’s The limits of data: It can tell you what’s

possible to report on. And the limits of possible to report on. And the limits of how far you can go. how far you can go.

Military SuicidesMilitary Suicides Worse than previously reported. Military Worse than previously reported. Military

playing with statistics, underplaying playing with statistics, underplaying severity.severity.

Those who kill themselves are Those who kill themselves are disproportionally those who disproportionally those who never served in never served in a war zone.a war zone.

Disproportionally young, white males.Disproportionally young, white males. Often with troubled backgrounds.Often with troubled backgrounds. Problems tracking soldiers harming efforts Problems tracking soldiers harming efforts

to prevent suicidesto prevent suicides Military loses track of reservistsMilitary loses track of reservists

Data trollingData trolling

Magic unicorn? Magic unicorn? There is no one button to hit and find There is no one button to hit and find

storiesstories This can be very frustrating. This can be very frustrating. Have an idea for a story before you Have an idea for a story before you

dive into datadive into data Always exceptions Always exceptions

Inspiration from Johan Inspiration from Johan MueleggMuelegg

?

Finding documentsFinding documents

USOC data of all US Olympic competitors for three USOC data of all US Olympic competitors for three decadesdecades

Database built from scraped athlete biographies Database built from scraped athlete biographies Congressional bills hot wiring citizenshipCongressional bills hot wiring citizenship Determined athletes whose Olympic dreams were Determined athletes whose Olympic dreams were

thwartedthwarted Country medal counts for each Olympics to show effectsCountry medal counts for each Olympics to show effects

Reporting The Story

World-class athletes compete at the Olympics not for their native

land

Documenting country hopping

Newsroom organizationNewsroom organization

Getting support for projectsGetting support for projects Teamwork: Collaborating for Teamwork: Collaborating for

efficiency, skillsefficiency, skills Convincing editors a project is Convincing editors a project is

worthwhileworthwhile A lot of us had to do work on our own A lot of us had to do work on our own

time to build our skillstime to build our skills