measurement module 100704

47
Session 3: Auditing Agency Performance

Upload: adtech

Post on 06-May-2015

214 views

Category:

Business


1 download

TRANSCRIPT

Page 1: Measurement module 100704

Session 3: Auditing Agency Performance

Page 2: Measurement module 100704

Todays Agenda• Integrated Media Evaluation

– Offline Vs. Online – Changes to the Purchase Funnel – Best Practices– Case study: How to attribute credit for success?

• Interpreting Plans and Reports– Success factors– What to look for– Reporting typology

• Creative Testing – Testing Methodology’s – Best practices

Page 3: Measurement module 100704

Media Integration

Page 4: Measurement module 100704

Media Integration Background

• Offline media is a lean back medium, online is lean forward.

• TV is often a shared experience, digital is a personal experience.

• Many people are online while they consume other media.

• Research proves that there is a strong link with offline media driving online activity i.e. a TV ad triggers curiosity in a product, the consumer uses the internet (often a search engine) to find out more.

• China* TV content is often consumed online e.g. Toudu, Youku etc. Many under 30’s don’t consume analog media.

Page 5: Measurement module 100704

Traditional Versus Digital Media

Page 6: Measurement module 100704

Offline vs. Online Advertising Key Differences

Online

• Vast number of sites• Long tail• Global• Precise measurements• Measure quality

Offline

• Limited channels• typically few free to air stations• Local-centric• Estimated measurements• Measure R&F

Page 7: Measurement module 100704

Offline vs. Online Reach & Frequency

Reach - the number of different people who are exposed an advertising message at least once

Frequency - the number of times they are exposed to the message

Offline advertising

• Buy “Media Impressions” – based on overall reach• Calculate frequency by how many opportunities each person

will see the ad

Online advertising

• Buy “Ad Impressions” – actual unique impressions on the ad• Calculate frequency by tracking each repeat visits using

“cookies” technology

Page 8: Measurement module 100704

Online R & F Methodology

Measuring Online Share Of Voice (SOV)% of the publisher’s total inventory (based on impressions) in a given period of time.

•Measurement might not be consistent from site to site – is SOV relative to:

• Impressions?

• Unique visitors?

• Particular media targets?

• Total impression volume available on the site?

Issues:

•Disabled / deleted “Cookies” in browsers

•Shared computer terminals

•Publisher counting tactics and methodology differences

Page 9: Measurement module 100704

Social Media & the Purchase Funnel

• While the shape of today’s funnel can take many forms depending on the author, its clear that a feedback loop that is powered by consumer generated content is an ever present addition to this model.

• Word of Mouth has always been important in influencing consumer decisions, the difference is that digital media has amplified this voice and empowered consumers to listen at a higher volume.

• Instead of asking 2 neighbors which soap they use you can go online and find out what thousands of people think.

Page 10: Measurement module 100704

Digital Age Model is Non-Linear

• Today the impact of digital media has created multiple new tools for consumers to use to find out more information about potential purchase on their own terms.

• Blogs, search engines, BBS, video, comments/rankings, podcasts all offer opinions about brands that are separate to the message form the brand owners

Page 11: Measurement module 100704

The New McKinsey Model

• McKinsey rejects the linear approach in favor of a circular model of ongoing exposure leading to a trigger moment

• Consumers begin with a narrow consideration set due to ad clutter and being time poor. Then a trigger occurs when the consumer decides they want to buy something new

• Once they decide to buy the move into active evaluation. Here the brands considered increases (opp of linear model) and intense research is undertaken-checking out sites, paying more attention to certain kinds of relevant media they already consumer.

• Third stage is closure. This is where the consumer goes to purchase although many have not made a final decision on which item to buy.

Post Purchase experience dictates the degree of loyalty the consumer has:– Active loyalty: Prefers same brand and will recommend to others.– Passive loyalty: Is comfortable with repurchase but also open to other choices

Page 12: Measurement module 100704

McKinsey ‘Consumer Decision Journey’

Page 13: Measurement module 100704

Making 2+2=5 via Media Integration

• Effectiveness of offline media can be measured by immediate performance of online media.

• Both mediums should ALWAYS be synchronized (flighting, messaging, placement) to deliver optimal synergies.

• Online media can be used to increase ROI of offline media.

• The critical link in this approach is SEM- keyword bidding, copy & linking pages must be comprehensive to catch and redirect qualified traffic from offline.

• In this way online media supports and leverages the larger offline media spend to make it more effective

Page 14: Measurement module 100704

How the integrated road to purchase worksTV Ad: 8pm to 10pm June 4th

Internet Ad: 8pm to 10pm June 4th?

For the treasure hunt to succeed clues must be left along the way

Page 15: Measurement module 100704

Measuring Success• Media channels must be evaluated together to determine how they

effect one another. • Performance should be monitored before, during & after a

campaign to establish baseline performance and to enable benchmarking.

• If an ongoing PPC campaign usually yields 5,000 clicks per month (the baseline) but records 10,000 clicks during an offline media burst, then the campaign contribution is 5,000 clicks. Scrutiny should also be paid to changes in other digital media performance to assign credit appropriately.

• An ongoing challenge is to determine the impact of offline awareness building which drives subsequent online media response.

• If a TV campaign is driving a lot of brand searches on Baidu why do a PPC campaign when you can get organic clicks for free?

Page 16: Measurement module 100704

Does BMW need to do PPC?

?

Page 17: Measurement module 100704

InterpretingPlans and Reports

Page 18: Measurement module 100704

Critical Success Factors

Effectiveness and ROI depend on…– Creative strategy – Targeting– Ad type– Ad Size– Frequency– Website and landing pages

Optimizing these variables makes the difference between success and failure

Page 19: Measurement module 100704

What to look for in a media plan

• There must be a clear strategy that explains how the tactics will work together in harmony to achieve the objective.

• The online strategy should roll up into the overall integrated campaign strategy.

• Does the media approach support the creative strategy? Often they do not compliment one another.

• Is the plan a ‘cut and paste job’ from a previous campaign or a different client?

• Does this agency/team consistently recommend the same publishers? If so why and are there better alternatives?

Page 20: Measurement module 100704

What to look for in a creative strategy

• Is there a clear rationale justifying the creative concept?• Ask yourself, does this strategy make YOU think more

highly of this product/brand? What about someone that you know well that is part of the target audience profile?

• Does the online creative approach support and leverage the offline thinking so that it translates online?

• If larger sizes are underperforming smaller sizes its likely the concept or execution is sub-optimal.

• Multiple resizes are common in this market, each additional edition should take no more than 20-30 mins of billable hours.

• If the creative concept is a western import, once initial translation is done there should be no extra charges.

Page 21: Measurement module 100704

What to look for in tracking reports• GAP Analysis: Did the campaign begin and end on time and if it’s

a CPD campaign was each flighting verified?• Formatting: Manually check the formulas and that the fields all

tally. Look closely at ‘Gross’ and ‘Net’ media costs.• Value added: How much of the final delivery is bonus and what is

the nature of it- often its remnant inventory that’s dumped at the back end of a campaign to meet quotas.

• Ad serving: This is usually marked up by the agency. It should be no more than $0.07 CPM. If the agency is using allyes it may be free in return for media booking fees collected from media owners.

• Quality score: Media owners will usually pad a proposal with unsellable inventory to get to a desirable eCPM. Evaluate each line item for prominence, exclusivity and scale.

• Optimization: All publishers allow mid campaign changes to placements and creative. Ideally performance should always improve over time

Page 22: Measurement module 100704

Tips on interpreting reports

Daily (or weekly)

Fortnight

End Campaign

PurposeReport Type

Monitor delivery – assist in identifying delivery issues

Monitor performance – LUM analyze performance across a 2 weeks span on sites and placements

In depth analysis on entire campaign performance and take learnings into future campaigns

Page 23: Measurement module 100704

Type of Charts

Typical Charts

• By Banner Size• By Placement• By Day Part• By Unique Users• Post Click Tracking• Impressions vs. CTR• Interaction Rate vs. Impressions• Interaction Rate vs. CTR• CPC vs. CPM

Page 24: Measurement module 100704

Type of Charts

By Banner Size

0.29%

0.59%

0.25%

0.09%

0.88%

0.05%0.07%

42.09%

0.39%0.33% 0.08% 0.11%0.00%

0.10%

0.20%

0.30%

0.40%

0.50%

0.60%

0.70%

0.80%

0.90%

1.00%

CT

R

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

Inte

ract

ion

Rat

e

CTR 0.29% 0.59% 0.25% 0.07% 0.09% 0.88% 0.05%

Interaction Rate 0.33% 42.09% 0.39% 0.08% 0.11%

160x600 Polite Banner

300x250 Polite Banner

728x90 Polite Banner

145x110 Standard Banner

290x190 Standard Banner

463x130 Polite Banner

700x90 Polite Banner

Page 25: Measurement module 100704

25

Type of Charts

By Placement

Page 26: Measurement module 100704

26

Type of Charts

By Day Part

0500000

100000015000002000000250000030000003500000400000045000005000000

Impressions

Impressions

Page 27: Measurement module 100704

19,261,879

1,108,198

9,189,766

362,605

2,655,936 2,684,882

0.06% 0.15%

21.37%

71.62%

12.66%

0.06%

5.63% 5.17%2.64%

0

5,000,000

10,000,000

15,000,000

20,000,000

25,000,000

-5.00%

5.00%

15.00%

25.00%

35.00%

45.00%

55.00%

65.00%

75.00%

Servered Impressions 19,261,879 1,108,198 9,189,766

Unique Impressions 2,655,936 362,605 2,684,882

CTR % 0.34% 0.46% 0.30%

Unique Clicks % 0.06% 0.06% 0.15%

Interaction Rate 21.37% 71.62% 12.66%

Unique Interactions 5.63% 5.17% 2.64%

Hong Kong Singapore Taiwan

27

Type of Charts

By Unique Users

Page 28: Measurement module 100704

28

Type of Charts

Post Click Tracking

0.29% 0.31%

0.52%

0.17%

0.10%

0.07%0.14%

0.19%

0.46%

0.00%

0.10%

0.20%

0.30%

0.40%

0.50%

0.60%

Attack

Per

cen

tag

e

Conversion Rate 0.52% 0.29% 0.31%

Post Imp Conversion 0.07% 0.10% 0.17%

Post Click Conversion 0.46% 0.19% 0.14%

1 2 3

Page 29: Measurement module 100704

29

Type of Charts

Impressions vs. CTR

0.00%

0.10%

0.20%

0.30%

0.40%

0.50%

0.60%

0

50000

100000

150000

200000

250000

300000

350000

CT

R(%

)

Imp

ress

ion

s

Page 30: Measurement module 100704

30

Type of Charts

Interaction Rate vs. Impressions

0.00%

1.00%

2.00%

3.00%

4.00%

5.00%

6.00%

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

Lowyat Mainpage Lowyat Forum Hotmail msn messenger

Inte

ract

ion

Rat

e (%

)

Imp

ress

ion

s

Page 31: Measurement module 100704

0.19%

0.30%

0.08%

12.66%

2.49%

0.00%

0.05%

0.10%

0.15%

0.20%

0.25%

0.30%

0.35%

Attack

CT

R %

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

14.00%

Inte

ract

ion

Rat

e

CTR % 0.30% 0.19% 0.08%

Interaction Rate 12.66% 2.49%

Attack 1 Attack 2 Attack 3

31

Type of Charts

Interaction Rate vs. CTR

Page 32: Measurement module 100704

32

Type of Charts

CPC vs. CPM

Page 33: Measurement module 100704

33

Tips on interpreting reports

Snapshot in time Over time(1 month cumulative) (across 1 month)

Placement Analysis

- measures Impression vs. Click- assist in analyzing placements / creative

Run Charts

- measures Impression vs. CTR- across a set time period- assist in identifying trends or patterns

Page 34: Measurement module 100704

34

Tips on interpreting reports

Placement Analysis by Site

• Some sites / creative combination will be more effective in reach / impressions(e.g.: in the example below “Gamer ROS”– ad placements may appear on any pages of the target site)

• Some sites / creative combination will be more efficient in generating click-through(e.g.: below “Gamer Home Page”)

273 281300

16

774

169

51

845

47

0.59%

0.29%

0.08%

0.24%

0.05%

0.88%

0.05%

0.10%

0.01%

-

100

200

300

400

500

600

700

800

900

Imp

res

sio

ns

('0

00

0s

)

0.00%

0.10%

0.20%

0.30%

0.40%

0.50%

0.60%

0.70%

0.80%

0.90%

1.00%

CT

R %

Imp ('0000) 273 281 47 300 16 774 169 51 845

CTR 0.59% 0.29% 0.10% 0.08% 0.24% 0.05% 0.05% 0.88% 0.01%

MSN Home Page

MSN Game / Movie /

GameBase Home Page

GameBase Home Page

Yahoo Games Home

Wretch Online Photo

Wretch Forum

Gamer Home Page

Gamer ROS

Page 35: Measurement module 100704

35

Tips on interpreting reports

Placement Analysis by Banner Type

• Performance of banner types can be compared by Impressions vs. CTR

• CTR is typically reflective of the effectiveness of placement and/or creative

• CPC can be included in the analysis to measure placement efficiency

4,158,155

17,887,617

52,765,4420.18% 0.17%

0.15%

0

10,000,000

20,000,000

30,000,000

40,000,000

50,000,000

60,000,000

Imp

ress

ion

0.00%0.02%

0.04%0.06%

0.08%0.10%0.12%

0.14%0.16%

0.18%0.20%

CT

R (

%)

Impression 4,158,155 52,765,442 17,887,617

CTR 0.18% 0.15% 0.17%

Expandable Polite Standard bannerBanner types are grouped together for easier reading

Page 36: Measurement module 100704

36

Tips on interpreting reports

Run Charts – identify problems and trends

– Is there a significant trend or pattern that should be investigated?

– Are they just normal process variations?• Look at data for a long enough period of time (at least 2 weeks is

recommended for online), so that a “usual” range of variation is evident

• Is the data within the usual range of variation?• Is there a daily pattern? Weekly? Monthly?

Page 37: Measurement module 100704

37

Tips on interpreting reports

Identifying Trends

– Upward and downward (seesaw) movements are common – typically 6 or more consecutive jumps in the same direction (up or down)

indicates a trend– In display ads, a downward trend typically signifies “creative wear out”

Creative change

Page 38: Measurement module 100704

38

Tips on interpreting reports

Report date range

– Notice behaviour shifts – over weekends, holidays, etc.

In the example above, we can observe that business tech sites gain more impressions over weekends.

02000000400000060000008000000

10000000120000001400000016000000

11/2

4/20

07

11/2

5/20

07

11/2

6/20

07

11/2

7/20

07

11/2

8/20

07

11/2

9/20

07

11/3

0/20

07

12/1

/200

7

12/2

/200

7

12/3

/200

7

12/4

/200

7

12/5

/200

7

12/6

/200

7

12/7

/200

7

12/8

/200

7

12/9

/200

7

Cricketnext imps Mouthshut imps MSN media imps

NNE education imps Tech2 imps Yahoo imps

weekend

Page 39: Measurement module 100704

Tips on interpreting reports

Adopt a holistic view– Do not interpret data in isolation

In the example above, CTR was extremely high on Oct 24th.

On closer examination, we find that impression was very low for that day, therefore resulting in an anonymity in the reports.

As it turns out there were technical difficulties on that site in that particular day.

Page 40: Measurement module 100704

Example of Testing for Banner

• Effectiveness & Efficiency vary by site and format

High CTR not Equal High Action

Page 41: Measurement module 100704

Creative Testing

Page 42: Measurement module 100704

Background to Testing• Digital is a highly measureable medium, *everything can be

tracked and analyzed• The more tracking data you have the more learning’s you have,

however data overload can obscure this. • Rigorous testing and analysis is time consuming and is often seen

as only a ‘nice to have’ beyond basic metrics.• The key question is how much testing is optimal to provide the

most cost effective outcomes.• There are eye tracking studies, creative A/B and multivariate

testing, media trials and panel based surveys all to consider.• This aspect of digital marketing is the most important and

undervalued: Why Invest in a medium famous for its unique measurability when you don’t leverage its core advantage???

• Offline thinking (i.e focus group think) at the C-Level still precludes this elephant in the room from ever being properly acknowledged.

Page 43: Measurement module 100704

8 Step Closed Loop Creative Testing Approach

2. The Measurement StepMeasure the success of the banners.

3. The Selection and Speculation StepEvaluate each banner and try to assess why some worked and others didn't by size, idea, call-to-action, execution, etc. Why are the top/bottom performers winning/losing?

1. The Create StepCreate six different ideas, executed in four sizes each

4. Codify Learning’s and AssumptionsBuild on industry best practices to develop a set of client-specific & audience-specific best practices. Refine and record what is working

5. Create and Measure AgainCreate a second raft of banners, based on learning’s from previous Phase. Evaluate performance using same criteria

6. Select Top Performers AgainAnnotate learning’s and optimize

7. Leveraging the Library Rerun top performing banners from round 1 and 2. Run the tweaked or optimized mid-level banners from round 1 and 2. Mix in new banners .

8. Leveraging the Library Long Term Perform bi-weekly or monthly measurement reviews ,Ongoing measurement and analytics followed by optimization and codified learning combine to continue driving performance.

Page 44: Measurement module 100704

•Multivariate Testing

Main Types of Land Page Testing

•A/B Testing

Multivariate tests (MVT) allow you to test multiple elements of a single page at the same time. A/B tests as simplified versions of them.

Page 45: Measurement module 100704

6 Key Web Page Testing Mistakes1. Using the wrong type of test (A/B vs. multivariate test)

A/B tests are good for alternate designs of an entire page, whereas multivariate tests are helpful in determining the most successful elements at different locations on a page.

2. Testing too many elements

software solutions permit thousands or even millions of combinations on a single page. Testing fewer than 100 scenarios can boost conversion rates from 4% to 15% in a matter of a few months.

3. Running tests that take too long

Keep track of external factors and understand that they might have an impact on your results. Minimize the time it takes to run your experiments to four weeks at the most.

4. Failing to monitor your test as it takes place

Digital is not a set and forget medium. Closely monitor experiments to decide if you should eliminate some elements when there is enough evidence.

5. Failing to conduct follow-up experiments

Designing follow up experiments, and learning which elements worked and which elements did not, is at the heart of conversion optimization.

6. Thinking that testing is a silver bullet

Testing is a great tool to increase your conversion rates However, it is only one step in the process.

Bottom Line: Select the right type of test to conduct based on the number of visitors and other data collected from analytics. Limit the number of scenarios through a holistic approach to testing. Get beyond single-digit improvement by continuing to test. Approach any project from a number of different angles, because testing alone will not maximize your results.

Page 46: Measurement module 100704

Eye Tracking Testing Eye tracking works by reflecting invisible infrared light onto an eye, recording the reflection pattern with a sensor system, and then calculating the exact point of gaze using a geometrical model. Once the point of gaze is determined, it can be visualized and shown on a computer monitor. The point of gaze can also be used to control and interface with different machines. This technique is referred to as eye control.

Page 47: Measurement module 100704

Summary• Offline and online media are fundamentally different mediums

which are only truly effective when strategically synchronized.• Everything is measurable with digital, the key is to determine what

the most important metrics are to focus on and then apply as actionable learning's.

• The ‘devil is in the details’ with digital: strive to look beyond the obvious and question every single detail RUTHLESSLY to uncover the true value of a program.

• If a chart, statistic or concept does not make sense in less than 10 seconds to you its not meaningful.(KISS)

• ABT (Always Be Testing): digital thinking changes every 6 months and useable learning's must be deployed to make 2+2=5.

• Each of you together at R3 are Chinas most important thought leaders in making our marketing industry more effective, more respected and more important. Its up to you to make it happen!