web science - how is it different?

Daniel

Web Science: How is it different?

Daniel TunkelangHead of Query Understanding

tl;dr:

The scientific method is alive and well.Big data has just changed the economics.

How have the web and big data changed science?

Let’s ask some of the experts.

“You have to kiss a lot of frogs to find one prince. So how can you find your prince

faster? By finding more frogs and kissing them faster and faster.”

Mike MoranDo It Wrong Quickly: How the Web Changes the Old Marketing

Rules, 2007

Cited by Kohavi in Online Controlled Experiments at Large Scale, 2013

Web Science = faster, cheaper experiments.

“The cost of experimentation is now the same or less than the cost of analysis. You can get more value…by doing a quick experiment than from doing a

sophisticated analysis.”

Michael SchrageValue-Creation, Experiments, and Why IT Does Matter, 2010

Web Science = more experiments, less analysis?

“with massive data, this approach to science — hypothesize, model, test — is becoming obsolete… Petabytes allow us to say: "Correlation is enough." We can

stop looking for models…analyze the data without hypotheses…throw the numbers into the biggest computing clusters the world…and let…algorithms find patterns

where science cannot.”

Chris AndersonThe End of Theory, 2008

RIP

ScientificMethod

1600 BCE –late 20th century

Killed by Big Data

?

No.

Let’s rewind.

What makes it science?

Hypothesis

Model

Test

The scientific method still works today.

What’s changed is the economics.

Scientific Method1747

Scientific MethodToday

It’s the economy, science.

YesterdayExperiments are

expensive,choose hypotheses

wisely.

TodayExperiments are cheap,do as many as you can!

What about Web Science?

A/B testing: everybody’s doing

it.

Google: 20k search

experiments per year

hypotheses

The Myth of Insight

Scientists gain insight

by staring at data.

Big data tools improve

data exploration.

In hypothesis generation,

quantity trumps quality.

Except when it doesn’t.

Easier to analyze data than research

humans.

But we pay the price.Example: search engine improvements in

batch evaluations don’t always predict real user benefits.

[Hersh et al, 2000] Do Batch and User Evaluations Give the Same Results?

[Turpin & Hersh, 2001] Why Batch and User Evaluations do not Give the Same Results

[Turpin, Scholer, 2006] User Performance versus Precision Measures for Simple

Search Tasks

But also see…

[Smucker & Jethani, 2010] Human Performance and Retrieval Precision Revisited

When local optimization is cheap, you neglect the rest.

To summarize: how is web science

different?•Online testing is cheaper and scalable.

•Data exploration tools make hypothesis generation cheaper and easier.

•But the experiments that are easy and cheap aren’t always the most valuable.

•Easy to forget our biases as scientists.

Take-Aways•The scientific method is alive and well.

Big data has just changes the economics.

•Cheaper hypothesis testing and generation has already been transformative. That’s why big data matters.

•But we neglect the human side of scientific experimentation at our peril.

Daniel Tunkelangdtunkelang@linkedin.comhttps://linkedin.com/in/

dtunkelang