automated modeling use case

8/9/2019 Automated Modeling Use Case

1/14

DEPLOYING AN AUTOMATED MODELING PROCESSFOR SALES TARGETING

A Case Study for AT&T Advertising & Publishing

Gene EdmistonSenior Manager, Marketing Analysis and Reporting

AT&T Advertising & Publishing

Don KridelAssociate Professor of EconomicsUniversity of Missouri-St. Louis

ABSTRACT

A web-based service that uses advanced dynamic model building

techniques to conduct intelligent profiling and modeling is utilized for salestargeting within AT&T Advertising and Publishing. This automatedmodelingapproach is designed to cost-effectively assist businesses in theirtargeting activities. In particular, many companies with a direct sales forcerequire a timely methodology that can be used to more effectively targetsales force activity. In many cases, the relative paucity of internal datamakes a purely internal modeling effort ineffective. In the current paper,we investigate alternative approaches to acquisition modeling for salestargeting; the output of the models will be ranked lists that will be used forsales force targeting. The process described involves a wide-variety of

alternative model specifications to be considered. The results in this caseindicate that the automated modeling approach outperforms moretraditional approaches by a substantial marginboth in terms of time andcost.

Parts of this paper were presented at DMA*07, Chicago, IL. Oct. 17, 2007.

DEPLOYING AN AUTOMATED MODELING PROCESS


2/14

FOR SALES TARGETINGA Case Study for AT&T Advertising & Publishing1

INTRODUCTIONKridel and Dolk [7] describe automated dynamic model-building techniquesderived from research in the areas of active decision support and automaticmodel generation ([3], [4], [5]) to generate context-specific customer listsfor B2C small and medium businesses (SMBs).2 Here we investigate asimilar approach for a much larger B2B company (AT&T Advertising &Publishing, henceforth AT&T-AP). In this approach, the general objectiveis to automate parts of the econometric modeling process. Specifically, theautomated process determines the appropriate independent variables, theirassociated functional form, and uses the final model for scoring (the scores

are then used to provide a smart or targeted list). Heuristics combined withdata analysis and statistical testing are used to generate a knowledge basethat is used to evaluate candidate models; the process iteratively refinesthe estimated equations until the process converges to a final model; fromthe estimated model a prospect list is generated and provided to the salesforce. In this paper, we seek to describe the process in some detail.3

In particular, we discuss:

Modeling scope, e.g., national, regional, or local models;

Independent variable scope, e.g., the inclusion of internal (AT&T-AP), external (national database) variables or both;

Project cost (in $ and time) of an automated vs. a manual process;

Results, i.e., calculating performance metrics (lift and ROI) thatdisplay overall effectiveness of the project.

INTRODUCTION

1 When the work began, AT&T Advertising & Publishing (AT&T-AP) was still SouthwesternBell Directory Operations (SBCDO). SBCDO had three regions: West (CA), Southwest (TX,AR, KS, MO, OK) and Midwest (MI, OH, WI, MI). Recently, AT&T has acquired Bell Southand the regions have been re-aligned. No Bell South directories are included in this analysis.Bell South books were added to the modeling project in 2008.

2 Kridel and Dolk [8] describe a case study for an SMB using automating modeling for a smallscale direct mail (DM) campaign. In this case, modeled prospects outperform ad hoc targeting( a list based on simple selects derived from a customer profile) by almost six to one (7.9%response rate vs. 1.4% response rate).

3 Initial modeling and analysis was performed in the Fall of 2004. The benchmark analyseswere performed for selected markets during the 2005 and 2006 contract periods.


3/14

Yellow Pages have been a resource for shoppers for nearly 130 years andtoday spans multiple platforms - print, Internet and mobile devices.Currently there more than 200 directory publishers, both telephone

company-affiliated and independents; these publishers issue more than7,000 Yellow Pages directories in the U.S. and Canada. Most major citieshave multiple directories that directly compete. Like other media, theproliferation of advertising choices has strained the economicsperformance of most Yellow Page Publishers. Historically, Yellow Pagescompanies were associated with the local telephone companies.4 In recentyears, this association has changed as a result of

spin-offs (e.g., RHD/DEX and Idearc),

increased head-to-head entry (Verizon providing Yellow Pages inAT&T service areas, e.g., St Louis), and

competitors entering various markets (e.g., Yellow Book).

AT&T-AP is the largest directory publisher in terms of revenue. AT&Tscore print product is under pressure as consumer usage of the YellowPages has begun to shift to the other media (in particular, the Internet).With the increased penetration of broadband access, the Internet provides

a richer medium for information search than traditional print search. Inparticular, web-based search provides

The ability to search using a wider range of attributes;

The ability to drill down on specific topics for more detailedinformation and cross-comparisons ;

Supports direct transactional capabilities.

4 Indeed, the initial AT&T Consent Degree had Yellow Pages going to AT&T (with the othercompetitive services like long-distance and equipment) but it has shifted (along withintraLATA toll) back to the Bell Operating Companies (BOCs) as there was widespread fearthat the BOCs were not sustainable as local-only companies.


4/14

This trend has significantly impacted Yellow Pages revenue and retention.Thus ATT-AP will face increasing challenges as usage and circulationdeclines and/or flatten. Historically, the AT&T Sales Force has focused on

increasing the spending of current customers. As penetration rates havefallen, more attention must be directed at non-customers (OPs).

As in many businesses with a large direct sales force, the issue ofproviding good leads to that sales force is important. Without a targetedlist, the sales force has to either develop its own ad hoc targets or diluteitself by attempting to contact all (too many) prospects.5 While goodsalesmen generally have effective informal targeting methods, these do notperform as well as formal targeting models.

The difficulty with formal modeling is that it is typically expensive--both interms of $ and time. AT&T-AP is interested in targeting OPs or oldprospects (businesses listed in the Yellow Pages that do not purchase anyadvertising); there are several million OPs throughout the AT&T-APterritory spread over several hundred markets (or books).

Effective targeting of OPs is essential since these prospects outnumber

paying customers by a significant factor (between five and ten to one). Asa result, the sales force effort is a binding constraint; better targeting willallow better utilization of sales force time and effort. Finally, as AT&Texpands its business lines from an advertising perspective (e.g., searchengine marketing and IYP--Internet Yellow Pages), there is yet anotherdrain on marketing and sales resources.

MODELING ANALYSIS

Since there is was a dearth of in-house modeling resources, the original

plan was for the project to follow other analysis tasks at AT&T-AP. That isto say, develop models at the region (or company level) utilizing an outsideconsultant to supplement in-house expertise.

5 The internal folklore was that all OPs were contacted; as the project unfolded, it becameclear that the sales offices were utilizing ad hoc targeting based on heading and length of timeas an OP.


5/14

During the evaluation phase, three primary questions were to beaddressed:

(1) Would modeling improve upon the existing ad hoc targetingprocess; and, if so, by how much?

(2) What independent variables (and associated data cost) would berequired to effectively build targeting models, i.e., would effectivemodels require the use of:

a. internal (AT&T) data;6

b. external (national database) customer data;

7

c. or both.

(3) What modeling scope would be most effective, e.g., (onecompany-wide model, three regional (Southwest, West,Midwest) models, or local models (one for each directory)?

Ten trial markets were chosen for this analysis.8 These markets wereanalyzed and alternative model specifications were tested. Models weredeveloped utilizing only internal (AT&T-AP) data as explanatory variables,onlyexternal (Experian national B2B) data as explanatory variables, andboth internal and external data as explanatory variables.

6 Internal AT&T data available were: account age, whether the OP had ever advertised before(and when), measure of risk (bad debt or delinquent), channel (premise, t-sales, other), andheadings information. Availability differed slightly by AT&T region.

7 The Experian NBD was utilized for the analysis; virtually the entire set of variables wasanalyzed. Variables of particularly interest were number of employees, sales, SIC code, age ofbusiness, Oxxford life-cycle, etc.

8 These were large (annual revenue > $5M) directories spread across the three AT&T APregions.


6/14

A representative model (internal & external model) is presented in Table 1.

Table 1: Representative Internal & External Model

VariableCategory

Transform

Coefficient Mean

T-Statistic

Number ofEmployees X -0.00111 14.74 -1.62Number ofEmployees n/a X2 0.00000

13065.52 1.37

Year in Business

Code C 0.48052 0.06 6.16Primary 2-digit SIC 50 -0.14676 0.05 -1.20Primary 2-digit SIC 81 0.76671 0.02 6.43Primary 2-digit SIC 80 0.63941 0.05 7.60Primary 2-digit SIC 7 0.86068 0.02 6.99Primary 2-digit SIC 65 -0.32440 0.04 -2.22Primary 2-digit SIC 64 0.36049 0.02 2.49Primary 2-digit SIC 17 0.74982 0.04 8.99Primary 2-digit SIC 76 0.74604 0.01 5.24Primary 2-digit SIC 75 0.47330 0.03 4.18

Primary 2-digit SIC 42 0.28364 0.01 1.48Primary 2-digit SIC 59 0.34603 0.06 4.17Primary 2-digit SIC 87 -0.68559 0.06 -4.68Primary 2-digit SIC 55 0.42118 0.02 3.21Primary 2-digit SIC 86 -0.52404 0.06 -3.80Primary 2-digit SIC 52 0.58078 0.01 3.86Primary 2-digit SIC 82 0.50863 0.01 2.79Legal BusinessStructure S 0.38866 0.01 2.50Prospect Age LOG -2.85276 0.87 -48.71

Constant -2.18715 1.00 -55.12


7/14

Figure 1 displays the lift-chart results (by decile, for one of the ten initial testmarkets) for each of the three modeling alternatives (scopes).

As can be seen in the figure, internal-data only models did not performany better than did random assignment (there is very slight lift in the firstthree deciles).

External-data only models performed considerably betterofferingsizable lift in the first two deciles.

The combined (internal-data and external-data) model performed the best.

The lift is considerably larger in the first 4 deciles than are the alternatives.

Using the results displayed above, it became clear that modeling wouldpay (assuming models could be developed at a reasonable cost) and thatthe use of external data was critical to the success of the project. Indeed,the estimated ROI for internal data only models was slightly negative.


8/14

It became clear very early that a company model was not feasible as therewere too few common variables across the regions. Further, it quicklybecame apparent that local market conditions were more important than

had been initially suspected. Using an aggregated or regional modelfrom the ten test markets produced results that were approximately 10-15%worse (than using the local models).9

Extrapolating to a wider roll-out and more aggregate regional models(where there would be 30 to 35 markets per region), it was clear that localmodels would significantly out-perform regional models. Clearly, giveninternal resources and limited budgets for external consulting, alternativemethods were required. In this case, AT&T evaluated a few different

providers of automated services.10

THE AUTOMATED PROCESS

Kridel and Dolk [7] describes the data model underlying automated processat a relatively high-level. To summarize here, the process begins withcustomer list from AT&T-AP. The list is then matched (and geo-coded) to anational database to obtain firmographics. In this case, the prospects (non-

customers) are also provided by AT&T-AP.

The modeling process (driven by the knowledge base) develops a logisticregression model which is used for scoring. The ranked prospect list, alongwith profiles and maps, are available for downloading and/or on-lineviewing and analysis.

9 The resulting models were quite different by market (book). In particular, the SIC codes ofimportance were quite different across books.

10 AT&T-AP selected CopperKey (KAST) as its vendor of choice in December 2004.


9/14

EVALUATION

Since early 2005, selected books have been tracked and analyzed. A

sample of books for 2005 and 2006 are displayed in Figure 2 and 3 below.While there is some variability across books and years, the results aresimilar in the sense that response rates have increased dramatically.

Figure 3 displays lift for selected books that were tracked in the first half of2005.11

Response rates increased on average approximately 285%. Typically, thismeant OP conversion rates increasing from the 0.25% - 0.5% range to the0.75% - 1.5% range.

11 Lift is calculated as: (Model Response Base Response) / Base Response. Generally, thetop-four deciles were used for the targeted lists; non-matched or random selections were usedfor base response rates.


10/14

Figure 4 displays lift for selected books that closed in the first quarter of2006; results are summarized to the region.

As can clearly be seen, the tested markets from Regions One and Threeoutperform the Region Two test markets (which nonetheless exhibited lift ofslightly over 100%).12

12 There are a variety of reasons that suggest why the Region 2 performed more poorly. Inparticular, the way prospects/customers are assigned differs by region. In addition, Region 2has more aggressively used discounting strategies to increase revenue among existingadvertisersinternal studies suggest that the resulting over-development of large ads causesa barrier-to-entry which leads to slower conversion of OPs.


11/14

Figure 5 displays the impact on response rates for the same selectedbooks that closed in the first quarter of 2006.

The business-as-usual response rates varied slightly by region (more bybook-size) and increase in response rates varied considerably by region(and less-so by book size).


12/14

Figure 6 compares the actual response rates to the predicted responserates (from the model) by decile for a selected book in 2006.

Using the same nine books that are summarized in Figures 4 and 5, we

also note that average spending (per converted OP) has increasedapproximately 50%. Further, using only the direct cost of the vendorscontract, ROI is calculated to be approximately 3000%.13

Of the increased response, the authors estimate that slightly less than one-half of the improved performance is due to local or book-specific models(as opposed to more aggregate region-specific models).

This suggests that using local models (via automated analytics)significantly outperforms regional models (built manually) with anestimated ROI of slightly over 3000%.

13 The average cost per local model (book) for the contract is slightly above $2000; incrementalrevenue is calculated to be approximately $600,000 for the nine books summarized in Figures4 and 5. In other words, incremental revenue in only these nine books that were trackedexceeds the cost of the totalcontract by approximately a factor of 3.


13/14

CONCLUSIONS

There is no surprise that targeting is shown to be effective in increasing

response rates. Targeting through automated analytics is shown (in thiscase) to outperform more traditional methods. This improved performancerelates primarily to the ability to model at a much-lower geographic level.

The automation of the analytical process outlined in this paper is not aperfect substitute for full-time modeling but it does significantly reduce theamount of time required from a full-time analyst to target market effectively.Because of this internal analyst can devote much more time to work onhigher priority (and/or less standard) projects that require less standardized

analytical methods and/or process, as well as higher revenue impactingmodels or analyses that require more time and attention.


14/14

REFERENCES

[1] Ben-Akiva, M. and S. Lerman (1985), Discrete Choice Analysis, MIT Press.

[2] Bhargava, H. K., R. Krishnan, and R. Muller, "Decision Support on Demand:Emerging Electronic Markets for Decision Technologies" Decision Support Systems,Vol. 19, pp. 193-214, 1997.

[3] Castillo, D.G., D.R. Dolk, and D.J. Kridel, GOST: An Active Modeling System forCosting and Planning NASA Space Programs, Journal of Management InformationSystems, Winter 1991-92, Vol. 8, No. 3, pp. 151-169.

[4] Dolk, D.R., and D.J. Kridel, An Active Decision Support System for Econometrics,Decision Support Systems, 7, 1991, pp. 315-328.

[5] Dolk, D.R., and D.J. Kridel, Towards a Symbiotic Expert System for Econometric

Modeling, Current Research in Decision Support Technology, edited by R.W. Blanningand D.R. King, IEEE Computer Society Press, 1993, Chapter 7.

[6] Gupta, S. and P. Chintagunta, "On Using Demographic Variables to DetermineSegment Membership in Logit Mixture Models," Journal of Marketing Research, 31,1994, pp. 128-136.

[7] Kridel, D.J. and D.R. Dolk, An On-line Marketing Consultant for Small and MediumBusinesses, Proceedings of the 32nd WDSI Conference, Lihue, HI, April 2003.

[8] Kridel, D.J. and D.R. Dolk, Using Intelligent Profiling to Generate Smart Lists: An

Empirical Test, Proceedings of the 33rd WDSI Conference, Manzanillo, Mexico, April2004.

[9] Rossi, P., R. McCulloch, and G. Allenby, The Value of Purchase History Data inTarget Marketing, Marketing Science, Vol. 15, No. 4, 1996, pp. 321-340.

[10] Shaffer, G. and Z. Zhang, Competitive Coupon Targeting, Marketing Science,Vol. 14, No. 4, 1995, pp. 395-416.

[11] Spohrer, J, P. Maglio, J. Bailey, and D. Gruhl, Towards A Science of ServiceSystems, Computer, Jan. 2007.

[12] Spohrer, J, S. Vargo, N. Caswell, and P. Maglio, The Service System is the BasicAbstraction for Service Science, Proceedings of the 41st Hawaiian InternationalConference on System Sciences, Jan. 2008, pp. 1-10.

[13] Vargo, S. L. and Lusch, R. F. Evolving to a new dominant logic for marketing.Journal of Marketing, 68, 1, 2004, 1-17.

automated modeling use case

Documents