from good to great emetrics_dc_keynote

From Good to GreatHow the testing program is evolving at Dell

Dell Online Analytics & Testing Team

Oct. 6th, 2010

2 Online Analytics, Testing & Targeting

Agenda

What’s the vision?(Ed Wu – Senior Manager, Global Testing)

What’ve learned?(Dave Nelson – Global Testing Manager)

Got process?(Carol Gohl – PMO)

Online Analytics, Testing & Targeting3

What’s the vision?


Dell Online Today

Online: Dell.com + Premier + Global Portal + B2B + Channel Portal + eSupport +

Community

An online orderis placed

every2 seconds

34 supported languages

Googled 1M times per day

58Kchannelpartnersonline

170M visitsto community properties

190KPremierPages

725K B2Btransactions per year

13Bpage views

in FY’09

1Bvisits per year to

Dell.com


Our journey started more than a decade ago…

Regional implementation

Limited expertise and talent bench

Regional KPI Standard KPI’s – global dashboard

Internal testing capability

Global implementation

Work closely with vendors & consultants

Full funnel testing

1:1 BT experience

Advanced segmentation

Early Adoption Standardizatio

nMaturity

VOC solution

1996-2007 2008-2009 2010-

Low

Global team across BU

Industry leading testing capability

Regional team Global team by BU

Grow in-house expertise

In-house clickstream tool


Great technology

Strong back-office support

Committed to testing

Good people

Our Strengths


“GOOD IS THE ENEMY OF GREAT”

Jim Collins


To be thetechnologye-commerce leaderby offering the mostrelevant experienceand solutions todrive high customerloyalty

Dell.comOnline Analytics,

Testing & Targeting

To become an innovation engine through insightful analytics & the best-in-class testing capability to enable constant and rapid changes


People

Double the size of the team

On-boarding process to provide support

and create connections

On-going training & development plan

If I were running a company today, I would have one priority above all others: to acquire as many of the best people as I could.

Jim Collins

Online Analytics, Testing & Targeting

Team

Global Analytics

Global Testing

eDell Programs

Reporting

PMO

10


Accountability

Because they lack accountability, they fail to achieve credibility, and they have no authenticity.

Jim Collins

2x Revenue upside Y/Y

400-600 tests globally

Monetization model based on test results

with statistical significance

Complete transparency & alignment with

finance & business

# of Tests Rev Upside

FY10

FY10

FY11

FY11


Got Process?


Holy Grail of Web Analytics


Entering Carol Gohl…

• 10 years experience in PMO’s– Information technology

deployments

– Sales force transformation

– Construction and infrastructure

– Retail sales

– Product launches

• But, knew nothing about web analytics!


Key Findings in the First 30 Days

• Entrepreneurial vs organizational strategy

• Same people ….different roles!

• Multiple “unique” processes!

• Great testing ideas…capability challenges

• Investment in Analytics tools – but not process and workflow tools

Look…

Listen…

Learn…


Planning cadencePipeline management

Standardized methodologies

PRIORITIZE the elephants and the cows….

“We’ve never done that”

“They” wont let you do that

“That’s too risky” “We can’t do that”


Process Transformation Strategy

PMO

Management

Personnel

Define

Align

Execute

Define

Align

Execute

Arch

itecte

d

Colla

bora

tive

Faster deploymentNear term benefit

Not readily adoptedHard to sustain

Slower deploymentLasting benefit

More readily adoptedBecomes organic


Big Picture First

Test Plan

Design

Code

QA

LaunchMonitor

Analyze

Brainstorm Recipes

Analytics Manager Lead

Test Manager Lead

Sign off required

Documentation required

Communicate

pipeline

roadmap

implement

Institutionalize learning


…and then get into Weeds

L1 Activity Activity What/deliverable Frequency When Who How/Format Doc/tool needed

Strategic PipelineGather internal developments/benchmarks

Global guidance Quarterly w eek 5-6 Analytics Lead

Internal reports and performance review (BoB tracker, Global Test Plan, Omniture reports, X-segment reporting)

Strategic PipelineGet COE input from COE leads on trends and opportunties

Partial presentations from COE leads (Benchmarking, CE etc)

Quarterly w eek 5-6Global Analytics Planning Manager (Joel) w ith input COE leads

Pow er point presentation(Partial) presentation on latest COE findings

Strategic PipelineLock testing needs from eDell w ith eDell manager (Glenn)

Planning document eDell Quarterly w eek 5-6Global Analytics Planning Manager (Joel)

Pow erpoint presentationE-dell planning presentation (Glenn)

Strategic Pipeline Analyze Global Trend reports

(Partial) presentation(s) on internal trends, industry trends, effectiveness/opportunities and BP on key online funnel elements (ODG, Consideration, Conversion, CE). Includes status on Analytics & Test plan.

Quarterly w eek 5-6Global Analytics Planning Manager (Joel)

Pow erpoint presentation

Ammo dashboard, Omniture Dashboard, X segment reports, Industry reports, Opportunity calculator

Strategic PipelineGenerate and ID Global Focus areas for analysis

Presentation on summarized key strategic areas including global long list of prioritized test ideas using/synthysizing COE findings, eDell input and global trends reports

Quarterly Week 5-6Global Analytics Planning Manager (Joel)

Pow erpoint presentation to Analytics managers

Input previous process steps

Strategic Pipeline BoB updateUpdate Bob tracker/w iki w ith outcome of global focus areas

QuarterlyMonthly testing/analytics calibration meeting

Global Analytics Planning Manager (Joel) and Global Testing Planning Manager (Dave)

PPT/Wiki · PPT/Wiki

Strategic PipelineQuarterly Planning Schedule Comms

Week 6 and forward until week 1 E-Dell lock – weekly meeting

QuarterlyRegional Analytics Manager to Regional Stakeholders

Weekly meeting discussion /PP

Quarterly planning schedule review; Weekly meeting minutes

Schedule, progress timeline etc

Strategic PipelineProvide global guidance on Global focus areas

Review meeting w ith Regional Analytical managers using Presentation on Global Focus areas

Quarterly Week 7Global Analytics Planning Manager (Joel)

Pow erpoint presentation to Analytics managers

Pow er point and conf call

Strategic PipelineUpdated global guidance based on results trends, needs of the business

Monthly email and update in staff meeting Monthly M1/M2/M3Global Analytics Planning Manager (Joel)

Email providing pow erpoint Pow erpoint and conf call

Strategic Pipeline Assess regional business priorities Notes of regional business priorities Monthly Week 8 Regional Analytics managerSolicit regional priorities from stakeholders in meetings

Template/Agenda

Strategic Pipeline Review Global Guidance no documentation Quarterly Week 8 Regional Analytics managerCompare idea pipeline to tops dow n and regional business guidance

-

Strategic Pipeline

Determine and prioritize key improvement areas (pages) based on global guidance and regional analysis

Partial presentation based on analysis MonthlyWeek 8 for next QM1, w eek 13 for next QM2, Week 4 for next QM3

Regional Analytics managerList of strategic areas to optimize (4-8 areas)


Strategic PipelineSubmit ideas for M1/2/3 for pipeline visibility

Goal, Test name, hypothesis, region, page MonthlyWeek 8 for next QM1, w eek 2 for next QM2, Week 6 for next QM5

Regional Analytics manager Sharepoint - new idea status ·

Strategic PipelineDraft pipeline of test ideas by key improvement area

Quarterly list of 20-30 test ideas to cover each improvement area and to be ready at M1 presentation. Including 5-10 top ideas for M1. At M2, M3 review the list is updated. Value H, M, L and $ and prioritized

QuarterlyWeek 8 for next QM1, w eek 2 for next QM2, Week 6 for next QM5

Regional Analytics managerPow erpoint format one slide per improvement area w ith test ideas for Q and for M1/2/3


Strategic PipelineBrainstorm regional pipeline for Q and M1

Lock f irst 2 test ideas for M1. QuarterlyWeek 8 for M1 then per iterative frequency

Regional Analytics manager Pow erpoint presentation Pow erpoint

Strategic Pipeline

Value ideas H,M,L w ith $ and prioritize w ithin key improvement area for any new ideas from above brainstorming

Revised valued pipeline for 13 w eeks (3 months) - Recommendation for how many days to run the test;

QuarterlyWeek 11 for next QM1, w eek 2 for next QM2, Week 6 for next QM6

Regional Analytics manager

Strategic PipelineLock strategic opportunities and test ideas for the Quarter and M1/2/3 in regional review session

Presentation to stakeholders of strategic areas including the regional pipeline for total Q (at least 1 test idea per w eek - idea description (one liner)). Ensure that M1 presentation contains Quarterly view . Then update for M2/3 review s. Include Q value estimation is included.

Quarterly Week 11 for next QM1 Regional Analytics manager

Collaborate w ith regions to generate prioritized quarterly and monthly pipeline in regional brainstorming session


Strategic PipelineSign off f irst tw o test plans for M1W1 and 2

Approved test plans Quarterly Week 11 for next QM2 Regional Analytics manager Present locked test plans Pow erpoint presentation

Regional Analytics Manager


Check Major Milestones

WK11 WK2 WK6 WK11WK13

Analy

tics

M

anager

Test

ing

Manager

• Strategic Areas (NQ)

• List of test ideas (NQ)

• Annualized upside (NQ)

M2 testing priorities– wireframes / feasibility


• Strategic Areas (NQ)

• List of test ideas (NQ)

• Annualized upside (NQ)

Lock M1 schedule; First test in M1 ready to launch

WK4 WK8



Test Execution

Idea Generation



Weekly Optimization Meeting

Analytics Manager LeadTesting Manager Lead

20


Establish a Feedback Mechanism

Overall Satisfaction

Improvement Areas


Process Change is Iterative…just like testing!

“Don't let the perfect be the enemy of the good”

Voltaire


Lesson Learned


Focusing on Strategic Areas

24

Key Pages

Cart and Checkout

Targeting


Understanding Customer Intent

25

Specific product offers for customers desiring an efficient shopping

experience

Improved site navigation for

customers want to take more time to find

the right product


Targeting Relevant ContentConsumer Customer

s

Small & Medium Business Customer


US Cons

US Cons

Checkout Testing - July

US SMB

UK Cons

US Cons

US Cons

DE Cons

JP Cons

AU Cons

CN Cons

BR SMB

CN SMB

JP SMB

AU SMB

US SMB

MX Cons

US Cons

MX SMB

BR Cons

Checkout Testing - August

US ConsCN

Cons

AU Cons

US Cons

US Cons

MX SMB

CA SMB

FR Cons

CA Cons

AU SMB

Checkout Testing - Sept

Testing Globally…an Iterative Process


Understand Why Something Works (or Doesn’t)

28

Simplified Cart Test – We changed too

much

vs.


Wrap it Up


Key Takeaways

• People first – invest more!

• Hold yourself accountable – be brave!

• Process is a necessity, not an option – start now!

• Scalability is key to success – leverage it!

• Failure is just as valuable as success – communicate it!


Embracing a New Culture:testing as a “religion”

Testing baked into dell.com projects from beginning

Tests are run rigorously with clearly defined

process

Results captured and shared widely, good

or bad


“It took Einstein ten years of groping through the fog to get the theory of special relativity, and he was a bright guy.”

Jim Collins


Thank You