statistical concepts every ppc specialist should know - hero conf london 2016

Statistical Concepts Every PPC Specialist Should Know

Frederik HyldigHead of PPC | s360 | @frederikhyldig

Who am I?Frederik Hyldig

Frederik Hyldig

Head of PPC at s360 - one of the leading search agencies in Denmark

Works with some of the largest advertisers in Denmark and Scandinavia

Writes for PPC Hero, Wordstream, Moz, SEMrush, 3Q Digital, and the Certified Knowledge blog

SEO & PPC inall the Nordic

countries

www.s360.dk

Some of the companies we work withFrederik Hyldig

In this presentation:- Statistics: a sexy skill- The Pareto principle- The law of large numbers- Relative and absolute numbers- Simpson’s paradox- How (not) to lie with statistics

Frederik Hyldig

A sexy skill

Frederik Hyldig

Frederik Hyldig

At least according to this guy...

Frederik Hyldig

I keep saying the sexy job in the next ten years will be statisticians. People think I’m joking, but who would’ve guessed that computer engineers would’ve been the sexy job of the 1990s? The ability to take data—to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it—that’s going to be a hugely important skill in the next decades [...]

- Hal Varian, Chief Economist at Google

The ability to take data- to be able to understand it- to process it- to extract value from it- to visualize it- to communicate it

Frederik Hyldig

Reflected in the quality of your work

Reflected in the relationships with clients (or bosses)

Frederik Hyldig

Data isn’t going anywhere. Our top skill category last year, statistical analysis and data mining, is still sitting comfortably at #2.

It is the only skill category that is consistently ranked in the top 4 across all of the countries we analyzed.

We still live in an increasingly data-driven world, and businesses are still aggressively hiring experts in data storage, retrieval and analysis.

-The 25 Skills That Can Get You Hired in 2016

https://blog.linkedin.com/2016/01/12/the-25-skills-that-can-get-you-hired-in-2016

Frederik Hyldig

Meaning businesses will be like...

Statistical Concepts Every PPC Specialist Should Know

Frederik Hyldig

The Pareto principle

Frederik Hyldig

Concept 1

The Pareto principle Frederik Hyldig

For many events, roughly 80% of the results come from 20% of the effort.

The Pareto principle Frederik Hyldig

Vilfredo ParetoItalian economist

Also known as the 80/20 rule

The Pareto principle Frederik Hyldig

stickfigureeconomics.blogspot.com/2012/06/understanding-pareto-principle-aka-8020.html

The Pareto principle Frederik Hyldig

stickfigureeconomics.blogspot.com/2012/06/understanding-pareto-principle-aka-8020.html

The Pareto principle Frederik Hyldig

How is this relevant in PPC?

Frederik Hyldig

The Pareto principle Frederik Hyldig

Focus on the few most important campaigns, ad groups, or keywords.

● Improving ads● Restructuring accounts● Negative keyword● Etc.

The Pareto principle Frederik Hyldig

Example:477 campaigns367 conversions

According to the Pareto principle:

293 conversions (80% of the 367 conversions)

should come from

95 campaigns (20% of 477 total campaigns)

Frederik Hyldig

92% of the conversions (338) came from 20% of the campaigns

The Pareto principle

The Pareto principleFrederik Hyldig

Concept:

Take away:A very large part of the results will often come from a small part of the activities.

Prioritize your work on the few keywords/campaigns with the largest impact.

For many events, roughly 80% of the results come from 20% of the effort.

The law of large numbers

Frederik Hyldig

Concept 2

The law of large numbers Frederik Hyldig

If you repeat a random experiment often enough, the average of the outcomes will converge towards the expected value.

The law of large numbers Frederik Hyldig

Coin tossing example: 10 tosses

The law of large numbers Frederik Hyldig

Coin tossing example: 1,000 tosses

The law of large numbers Frederik Hyldig

The law of large numbers Frederik Hyldig

>

Frederik Hyldig

Frederik Hyldig

If the statisticians who reported to the Gates Foundation had asked about the characteristics of the worst schools, they would have found that bad schools also tend to be smaller than average. The truth is that small schools are not better on average; they are simply more variable.

- Daniel Kahneman

How is this relevant in PPC?

Frederik Hyldig

The law of large numbers Frederik Hyldig

Sorted by highest conversion rate

These are all very small cities

The law of large numbers Frederik Hyldig

Sorted by lowest conversion rate

These are also small cities

The law of large numbers Frederik Hyldig

Seems like we have our most valuable hours at night!

Frederik Hyldig

Frederik Hyldig

The law of large numbers Frederik Hyldig

No.... It’s just more variable due to low volume

Frederik Hyldig

ROAS

#1 #2 #3 #4 #5 #6 #7 #8

Look at the return in country #1! Let’s allocate more of our spend to this account!

Countries

Frederik Hyldig

Ad spend

#1 #2 #3 #4 #5 #6 #7 #8

Not so fast! The volume is much lower than the other countries and therefore we should expect more variance.

Countries

The law of large numbers

Frederik Hyldig

Concept:

Take away:When randomness is involved (i.e: always) you should expect more

variance from low volume.

Put more trust in numbers when volume is higher.

If you repeat a random experiment often enough, the average of the outcomes will converge towards the expected value.

Relative and absolute numbers

Frederik Hyldig

Concept 3

Relative and absolute numbers

A new drug reduces the risk of getting a dangerous disease by 25%

Frederik Hyldig

Relative and absolute numbers Frederik Hyldig

No drug: 20 in 1,000 get disease (2.0% risk)

With drug: 15 in 1,000 get disease (1.5% risk)

25 percent fewer people get the disease

Actual risk of getting disease is only 0.5 percentage points lower

What you will read in the newspapers

What you really want to know when deciding

How is this relevant in PPC?

Frederik Hyldig

Relative and absolute numbers Frederik Hyldig

Relative and absolute numbers Frederik Hyldig

Relative and absolute numbers

Frederik Hyldig

Concept:

Take away:Percentage must not be confused with percentage points.

Percentages can be misleading when the absolute numbers are omitted.

Simpson's paradox

Frederik Hyldig

Concept 4

Simpson’s paradox Frederik Hyldig

A trend appears in different groups of data but disappears or reverses when these groups are combined.

Simpson's paradox Frederik Hyldig

1973 admission figures for the University of California, Berkeley

Simpson's paradox Frederik Hyldig

1973 admission figures for the University of California, Berkeley - by departments

How is this relevant in PPC?

Frederik Hyldig

Simpson’s paradox Frederik Hyldig

Variant 1: Low prices on XX Variant 2: Save 20% on XX

Ad group 1

Ad group 2

Ad group 3

Ad group 4

Simpson’s paradox Frederik Hyldig

Ad 2 is winning in every ad group

But when aggregating the numbers, ad 2 is actually worse!

Simpson’s paradox Frederik Hyldig

Ad 1 volume is concentrated in the two ad groups with generally higher CTRs.

Simpson’s paradox Frederik Hyldig

Simpson’s paradox Frederik Hyldig

Even though ads are shown evenly, a group with high volume can still cause Simpson’s paradox

Simpson’s paradox

Frederik Hyldig

Concept:

Take away:Beware when aggregating data.

Make sure to run properly randomized trials (rotating ads evenly).

How to lie with statistics

Frederik Hyldig

Concept 5

not

How (not) to lie with statistics Frederik Hyldig

Graph is from “How to Lie With Statistics” by Darrell Huff

Manipulating graphs

How (not) to lie with statistics Frederik Hyldig

Graph is from “How to Lie With Statistics” by Darrell Huff

Manipulating graphs

How (not) to lie with statistics Frederik Hyldig

Graph is from “How to Lie With Statistics” by Darrell Huff

Manipulating graphs

How (not) to lie with statistics Frederik Hyldig

Graph is from “How to Lie With Statistics” by Darrell Huff

Manipulating graphs

Same numbers. Different impressions.

How (not) to lie with statistics Frederik Hyldig

Manipulating graphs

Pay attention to the vertical axis in AdWords - especially when looking at shorter intervals.

How (not) to lie with statistics Frederik Hyldig

Manipulating graphs

A graph should start a zero.

How (not) to lie with statistics

An increase by 20% followed by a decrease by 20% does not bring you back to the starting value.

● 100 + 20% is 120

● 20% of 120 is 24

● 120 - 20% is 96

Frederik Hyldig

Shifting baseline

How (not) to lie with statistics Frederik Hyldig

Shifting baseline

How not to lie with statistics

Frederik Hyldig

Concept:

Take away:Pay attention to the vertical axis when looking at (or creating) graphs.

Be aware of shifting bases when increasing and decreasing.

Summary- The Pareto principle- The law of large numbers- Relative and absolute numbers- Simpson’s paradox- How (not) to lie with statistics

Frederik Hyldig

Thanks for listening!

Frederik HyldigHead of PPCE-mail: frederik@s360.dkTwitter: @frederikhyldig

s360SEO & PPC acrossall the nordic countriesWeb: www.s360.dk