JPK

Gro

up2015 Organizational Intelligence Forum

December 7-8 • San Francisco, CA

Keynote: Bridging the Gap betweenData Science & Market Intelligence

Best Practices in Collaboration Between Intelligence & InsightsProfessionals & Data Scientists: The Autodesk Experience

December 8, 9:45am

Adam Sugano serves as the Head of Predictive Modeling and Advanced Analytics atAutodesk. In this role, he leads a team of both internal and external data scientists chargedwith delivering innovative, actionable data driven solutions that help empower Autodesk’s

customer retention and engagement optimization efforts across the customer lifecycle.Prior to joining Autodesk, Adam was the Director of Analytics for Experian MarketingServices’ Cross-Channel Marketing organization where he leaned on his quantitative

marketing background to help clients understand the value of customer intelligence anddelivered tailored marketing analytics solutions..

View presentation online at:

https://jpkgroupsummits.com/attendee

Adam Sugano – Autodesk

© 2014 Autodesk

Best Practices in Collaboration between MI Professionals and

Data Scientists – The Autodesk Experience

Adam Sugano Head of Predictive Modeling and Advanced Analytics

Autodesk, Market Intelligence & Customer Analytics

© 2015 Autodesk

Introduction to Market Intelligence & Customer

Analytics

What is Data Science?

Case Study: Inception, Development and

Implementation of the Early Warning System (EWS)

and Account Opportunity Alerts (AOA)

Agenda overview

© 2015 Autodesk

Intro to MICA

© 2015 Autodesk

We strive to be a compass for Autodesk marketing.

In collaboration with Sales and Marketing, we understand, segment, and predict

customer behavior and needs; identify and prioritize business opportunities and

recommend go-to-market strategies for existing and new business models; and

support the implementation and optimization of customer engagement and

retention programs.

Our multidisciplinary team of diverse, innovative, and result-oriented market

analysts, data scientists, and marketing technologists crosses organizational

boundaries to deliver unbiased customer centric analytical insights.

MICA FY16 Charter

© 2015 Autodesk

MICA FY16 Functional Setup

• Segment Intelligence

• Sweet spot Intelligence

• Channel / Named account

Intelligence

• Market sizing

• Competitive Benchmarking

Market and Customer

Intelligence / Strategy

• Nurture Campaign

Intelligence &

Optimization

Cross Channel Customer

Engagement Analytics &

Optimization

• Retention modeling

• Nurture stream design & content recommendation

• Campaign targeting & prioritization

• Lead scoring

• Customer LTV, etc.

• Dataset exploration

Marketing Data Science and

Predictive Modeling

• Data quality and curation

for self-service access

• Reporting

• Analytics requirements

for marketing systems

and predictive model

score integration and

implementation

Analytics Solutions Product Management

⌂ ⚙ ☀

© 2015 Autodesk

What is Data

Science?

© 2015 Autodesk

Defining Data Science

Evolution from statistics

Computer science/machine learning

Increased computational power

Rapidly increasing volumes of data

© 2014 Autodesk

What does the Data Science process look like?

Research

&

Insights

Data Exploration &

Preparation Modeling

Actionable Insights /

Operational Product New

Discovery

• Define the question

• Understand problem,

scope

• Meetings, planning

• Analyze results from

model

• Define the ideal data set

• Determine what data is

accessible

• Obtain the data

• Clean the data

• Exploratory data

analysis

• Predictive,

unsupervised methods

• Debugging, exploring

alternatives

• Challenge results

• Interpret results

• Document results

• Create reproducible

code

• Deploy product or

system

• Use models to drive

business

• Reporting

© 2015 Autodesk

Case Study:

Early Warning

System

© 2015 Autodesk

EWS monitors customer activity, behavior and

usage to assess subscription retention risk and

identify possible causes and remedies

What is the Early Warning System?

© 2015 Autodesk

EWS & the Subscriber Lifecycle

365

RENEWAL

ADOPTION

90 Days In Day 1 90 Days Left

ONBOARDING

Signed In,

Downloaded,

Installed

Communities, Autodesk Knowledge

Network, Tailored Content, Support

Renewal

reminders,

Renewal

transactions

Last 90 Days

Retention Score

Low

Risk

High

Risk

First 90 Days

Onboarding Score

Low

Risk

High

Risk

Entire Term

Adoption Score

Low

Risk

High

Risk

© 2015 Autodesk

A High-Level Overview of the Process

INDUSTRY &

CUSTOMER

ANALYSIS

REQ

GATHERING

& DATA

DISCOVERY

1

2

PREDICTIVE

MODELING

3

TEST &

ITERATE

4

IMPLEMENT

5

In-depth research &

analysis to develop a

list of assumptions

that drive a

customer’s decision

to renew a

subscription contract.

Key indicators or data

points to measure

assumptions

developed.

Availability and

accessibility of data

explored

Statistical

modeling

Results from the

model shared back to

the industry analysts

for validation and

iteration. Pilots

launched.

Scores brought into

backend system.

Program

established to

formally roll-out

model company-

wide.

© 2015 Autodesk

Through research and analysis, eight main factors impacting a

customer’s decision to renew subscription were identified

Subscription Benefit Utilization

Product Utilization

Broken Relationship

Competitive Pressure

Company Performance

No Longer In Business

Assets Left from Migration Dif

fic

ult

y t

o a

dd

res

s

Licenses Transferred

What are the early

warning signs?

Can we measure the

signs?

Can we predict

renewal?

1

© 2015 Autodesk

Main reasons for dropping Subscription Subscription Adoption

Product Adoption

Broken relationship

Competitive pressure

Company

Performance

Business entity gone

Asset no longer in use

10%

15%

2%

20%

Breakdown of

seats D

ifficu

lty t

o a

dd

ress

10%

• Low Awareness of Benefits

• No perceived need for yearly upgrades

• Slow adoption of subscription benefits

• Low Product adoption at individual and department level

• Standalone seats left from migration to suite

• Questioned ROI,

• Wrong product. Poor partner follow up

• Poor deployment

• Not happy with Autodesk policies

• Pressure for Standardization

• Lack of interoperability with other solutions

• New hire prefers the competitive solution

• Downsizing/ over licensed

• Financial crunch/ tight budget

• Lack of projects

• Bankrupt, division dissolved

• Company acquired

• Restructuring

• Standalone seat left out from migration to suites

• Project based need

• Unsuccessful evaluation/ poor implementation 28%

5%

Others • Contact is not with company anymore

• Some licenses have been purchased locally

10%

© 2015 Autodesk

Data discovery was a collaborative process,

leveraging all MICA practice areas

Subscription Benefit Utilization

Product Utilization

Broken Relationship

Competitive Pressure

Company Performance

No Longer In Business

Assets Left from Migration

Licenses Transferred

What are we measuring? What are we

hypothesizing?

What metric could be used to test the hypothesis?

How important is this metric?

How accessible is the data?

2

Pre-EWS view of a subs renewal pipeline

$120,000

$20,00

0

$40,000

$60,000

$80,000

Co

ntr

act

Valu

e

$100,000

Subs Contract

Today, EWS is leveraged by sales & marketing to prioritize

resources and inform a customer-centric approach to

growing subscriber loyalty

Additionally, EWS can

support in sales

enablement: Insights on

the leading indicators

that contributed to the

accounts overall risk

score are provided.

5

© 2015 Autodesk

Score Drivers help inform a course of action

Product Utilization Product and service utilization and

adoption measures

Direct customer to training

materials/learning sessions, etc…

What it is… Possible Actions…

Subscription

Engagement

Historical subs buying and renewal

patterns, adoption of subs benefits

Ensure subs access and educate

customer on the benefits of the

subscription program

Customer Profile Customer characteristics, location,

industry, purchase channel, reseller

Share cases studies of similar profile

successful customers, make

connections w/ same at events, etc

Products & Services What the customer owns Product/workflow specific sales plays,

correct if wrong product sold into

account, etc

R1 Score Drivers

© 2015 Autodesk

Acme Architects 110123123123

© 2015 Autodesk

Continuing to iterate by asking why

Obtain feedback from sales reps and from

customers to understand underlying

reason for their decision, and iterate the

model to improve accuracy

5

4

© 2015 Autodesk

Surprising Renewals

Key reasons for renewing include:

Use product regularly (regardless of how frequently they actually update it)

Are ensured access to latest release, if needed

Believe it is less expensive than lapsing and repurchasing

It’s company practice to renew subscriptions/it’s in-budget

Miscellaneous (e.g., need latest rev. for compatibility testing, for their new PC)

Enterprise accounts (including government, military, and gov’t contractors) comprise a disproportionate share of

surprising renewals (in budget/company practice to renew); of 17 ENT interviews where the risk factor was not

predicted correctly, 15 were for surprising renewals.

Surprising Non-renewals

Key reasons for non-renewing include:

No need for updates (esp. LT users)

License consolidation/users no longer with company/eliminating excess licenses

Financial constraints/no budget

Out-of-business

Miscellaneous (e.g., current release won’t run on old PC, have reason to run down-rev version)

VSBs comprise a disproportionate share of surprising LT non-renewals

Grace period often used for cash flow management; high incidence of renewals subsequent to end-date [esp. 2+

year contracts]

Slide from vendor who investigated (called up) why the actual renewal behavior of certain agreements differed so

much from what we predicted; falls under the test/iterate/improve theme

© 2015 Autodesk

EWS & the Subscriber Lifecycle

365

RENEWAL

ADOPTION

90 Days In Day 1 90 Days Left

ONBOARDING

Signed In,

Downloaded,

Installed

Communities, Autodesk Knowledge

Network, Tailored Content, Support

Renewal

reminders,

Renewal

transactions

Last 90 Days

Retention Score

Low

Risk

High

Risk

First 90 Days

Onboarding Score

Low

Risk

High

Risk

Entire Term

Adoption Score

Low

Risk

High

Risk

© 2015 Autodesk

Onboarding Objective

Identify segments of new customers in the “at-risk”

category for Onboarding health that are suitable for

proactive callout programs from Autodesk

© 2013 Autodesk

Vendor research/solution/approaches

© 2013 Autodesk

Understanding the customer onboarding experience and mapping datasets

against the checklist

Data Domain

Siebel (assets, entitlements, account info, activations, licenses…)

SFDC (leads, oppties, support cases…) access via ODSG

Aprimo (marketing data)

Marketo (marketing data)

SAP / RevMart (orders, billings, revs)

DAP / CIP (desktop usage)

Cloud Platform (O2, services), shares UCP hadoop

GDW / ODSG (most is in UCP)

Lithium / Communities

AKN (some may be in UCP)

Convergent Charging – Cloud Credits (also in ODSG)

MICA Stage Servers

MIDM (source data, MIDM data)

Site Catalyst / Web

Customer Sat (corp survey)

Subs Center

MyAccount

SubAware

Flex usage data (pseb)

Rover (app dl mgr)

BIC / CLIC

UCM (person master)

Allocadia (mtkg spend data)

D&B enrichment

Other enrichment

EIDM (subs center, partner center sso system)

Checklist

1 Order Confirmation Email

2 Welcome Emai

3 Log into Subscription Center

4 Confirm account details

5 Create SCs

6 Assign SCs

7 Create Users

8 Assign Users

9 Download Software

10 Download Product Enhancements

11 Activate Software

12 Get Started Email

13 Using Your Sub Benefits Email (partner)

14 Managing Your Sub Benefits Email

15 Misc. about new releases, extensions,

upgrades

© 2015 Autodesk

Opportunity Alerts

Product / Sales Play Score

PrDS 74

Revit 64

BIM in VSB’s 34

InfraWorks 22

BIM for Construction 16

Etc…

84

Recommender Buying Readiness

Analytics produced for existing customers only

Score at the Account (aka Site) level

Combines empirical

observations with business

logic overrides and cold start

recommendations

Uses predictive

models based on

internal and external

customer activity and

behaviors

Opportunity Size

Compares an account’s

actuals to a similar “best-in-

class” account’s

performance to quantify

gap

Actual prior 12mo $ $15k

Best-in-Class 12mo

$ $55k

Opportunity Size $40k Probability of purchase in

next 3 months

2 1 3

Account UUID 123-123-123123123

Account Name VOA Associates Inc.

Analytically Driven Lead Alerts

Readiness Score 84 Readiness Trend

Readiness Drivers

On-line Activity

Mktg

Engagement

Hiring Activity

…

LOW

MED

HIGH

…

Recommender

Product Design Suite 85

Inventor 75

BIM 360 74

SIM 360 67

InfraWorks 66

Opportunity Size*

Actual prior 12mo $ $15k

Best-in-Class 12mo $ $55k

Opportunity Size $40k

Sales Plays

BIM for AEC Construction Sales Strategy

Dynamo studio Treasure Map OTS url

Structural concrete fab Treasure Map OTS url

Expand BIM for Service Providers Sales

Strategy

Structural concrete fab Treasure Map OTS url

Revit Collaboration OTS url

© 2015 Autodesk

The three pillars to ensuring successful projects when

collaborating

Support a strategic business initiative with senior level sponsorship

Clearly define roles and responsibilities

Recognize the work of all teams involved

Autodesk is a registered trademark of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA and/or other countries. All other brand names, product names, or trademarks belong to their

respective holders. Autodesk reserves the right to alter product and services offerings, and specifications and pricing at any time without notice, and is not responsible for typographical or graphical

errors that may appear in this document.

© 2013 Autodesk, Inc. All rights reserved.