007 www.esigmagroup com advanced site optimization
TRANSCRIPT
-
8/6/2019 007 Www.esigmaGroup Com Advanced Site Optimization
1/16
ADVANCED SITE OPTIMIZATIONSecond-Generation Digital Analytical Tools in Action
Eric T. Peterson, Web Analytics Demystified, Inc.
Sponsored by
http://www.webanalyticsdemystified.com/http://www.webanalyticsdemystified.com/http://www.tealeaf.com/ -
8/6/2019 007 Www.esigmaGroup Com Advanced Site Optimization
2/16
AdvancedSiteOptimization:SecondGenerationDigitalAnalyticalToolsinAction
1999 2010TealeafTechnology,Inc.AllRightsReserved.
TABLEOF CONTENTS
ExecutiveSummary ........................................................................................................2
TheChallengewithDigitalAnalysisToday............................................................. 3
DataVolumesCreateTheirOwnProblems..................................................... 3
WebAnalyticsApplicationsLackAnalyticalCapabilities ........................4
AnalysisStaffArentTrainedforAnalysis ......................................................5
NetResult:InsightsAreSlowtoMaterialize................................................... 5
AdvancedSiteOptimization ........................................................................................6
PuttingDatatoWorkforYou(NottheOtherWayAround)........................6
FirstStepsTowardstheComingRevolution................................................... 6
SecondGenerationDigitalAnalysisTools ...............................................................7
MultivariateandA/BTesting............................................................................. 8
AnalyticsIntelligence ..........................................................................................9Tealeaf8............................................................................................................... 10
Conclusions.................................................................................................................... 14
AbouttheAuthor ..........................................................................................................15
AboutWebAnalyticsDemystified ...........................................................................15
AboutTealeaf ................................................................................................................15
-
8/6/2019 007 Www.esigmaGroup Com Advanced Site Optimization
3/16
AdvancedSiteOptimization:SecondGenerationDigitalAnalyticalToolsinAction
1999 2010TealeafTechnology,Inc.AllRightsReserved. 2
EXECUTIVE SUMMARY
Given the increasing importance of connected channels to nearly all businesses and government
functions,itisnowonderthatanalysisofthesechannelsisgettingmoreattention.Digitalanalysisisthe
practice of making decisions using digitally collected qualitative and quantitative databy leveragingclickstream,feedback,andcustomerexperiencemanagementsolutions.Poweredbybusinessprocessand
performedbydigitalanalysts,digitalanalysisispurportedtohavetheabilitytodramaticallytransform
businessoutcomes.
Exceptthatitdoesnt.
Nearlyadecadeintothedigitalanalysissectorsgrowth,cracksinthefoundationarebeginningtoshow.
Business leaderswhohavecommittedsignificantresourcesstillcomplainaboutgetting toomuchdata,
toofewinsights,andnotnearlythereturnoninvestmentpromised.
At Web Analytics Demystified it is our belief that this situation arises from a perfect storm of
challengesrelating topeople,process,and technology.Widelyavailabledigitalmeasurementplatforms
are decreasingly able to keep up with the volume of data generatedby mobile, social, and traditional
digitalmarketingandcommunicationchannelsofeverybusiness.Whatsmore,whilethevolumeofdata
needing analysis continues to grow, the few existing dedicated resources are dragged downby aging
businessandknowledgemanagementprocesses.
Fortunately,anewbreedoftechnologies isemergingsecondgenerationdigitalanalysisapplications
within the web analytics, customer experience management, and testing and optimization sectors.
Theseapplicationsandplatformsbridgethegapbetweenbasicreportingsolutionsandthetypeofbig
ironbusinessandcustomerintelligencesolutionscommonlyfoundintheofflineenterprise.Bymoving
beyonddatafordatassakeintotherealmofstatisticalmodelsanddataforanalysissake,thesevendors
customers are pushing the limits of what is possible in digital channels and creating a competitiveadvantageintheprocess.
-
8/6/2019 007 Www.esigmaGroup Com Advanced Site Optimization
4/16
AdvancedSiteOptimization:SecondGenerationDigitalAnalyticalToolsinAction
1999 2010TealeafTechnology,Inc.AllRightsReserved. 3
THE CHALLENGE WITHDIGITAL ANALYSIS TODAY
The promise of digital analysis, using tools like web analytics, has alwaysbeen about surfacing high
value insights drivenby the tremendous volume of data produced through digital channels. These
insightswould thenbepresented tomanagement in the formofrecommendationsand,whenenacted,thevisitorexperiencewouldbecontinually improved.Thisprocesswouldberepeatedand,over time,
keymeasuresofbusinesssuccesslikerevenue,profit,andsatisfactionwouldshowsubstantialgains.
Unfortunately,thisrarelyhappens.
Mostbusiness leaders todayare frustrated with the lack of informationandrecommendations flowing
fromtheiranalyticsefforts,despitesubstantialinvestmentsintechnologyandconsultants.Accordingtoa
2009studyfromAberdeenResearch1,28%ofcompaniesdescribedasbestinclassadmittedthatthey
stillstruggletointerpretdataprovidedbytheirwebanalyticssolution.Reasonsforthisstruggleinclude
lackof insight intohow toconvertmetrics into tangibleactions, lackofresources,difficultyassociated
with data extraction, and difficulty with interpretation. The net result, according to Aberdeen, is that
nearlyonethirdofcompaniespayingforwebanalyticsdonotbelievetheyaregettingthefullvaluefrom
theirinvestment.
ThisdataiscorroboratedbyfindingsfromtheWebAnalyticsAssociationsOutlook2010:SurveyReport2where nearly 70% of respondents indicated that a key initiative in 2010 wouldbe acting on data to
improve site performance, followedby makingbusiness decisions drivenby analytics (63.5%). The
same report cited that 39% of respondents were challengedby taking action on available data, 37.6%
struggled to make business decisions driven by analytics, and 37.2% struggled to gain executive
managementawarenessandsupportforwebanalyticsefforts.
AtWebAnalyticsDemystifiedwebelievethisunfortunatesituationarisesdirectlyfromaperfectstorm
ofchallengestoomuchdatabeingfedintoinadequatesystemsandrunbyundertrainedresources.
DATAVOLUMESCREATETHEIROWNPROBLEMS
TheInternetislikenoothersourceofdatathathasbeenanalyzedbybusinessowners.Whereasoffline
effortstounderstandcustomersaretypicallylimitedtoknownindividualsorbroadsamplesbasedon
demographics or psychographics, digital channels create an almost unmanageable source of data, one
that grow with every mouse movement, click, and interaction.Every single thing each one of us does
onlineistrackedsomewherebysomeone,usuallywiththegoalofoptimizinginteractionsanddriving
conversion.
Problem is, for many sites having access to this much data creates issues rather than solving them.
Especially when dealing with multiple disparate systems such as email marketing systems, socialnetworks,adnetworks,andwebdeliveryplatforms,thehoopsthatdatamanagersneedtojumpthrough
toconnecteachindividualsinteractionarearduousatbest.Theresultisusuallyindependentreporting
silos,eachofwhichhasatendencytoproducetheirownversionofthetruth.
1http://www.aberdeen.com/AberdeenLibrary/6048/RPwebanalyticsmarketing.aspx2http://www.webanalyticsassociation.org/resource/resmgr/docs_research_committee/waaoutlooksurveyreport20.pdf
http://www.aberdeen.com/Aberdeen-Library/6048/RP-web-analytics-marketing.aspxhttp://www.aberdeen.com/Aberdeen-Library/6048/RP-web-analytics-marketing.aspxhttp://www.aberdeen.com/Aberdeen-Library/6048/RP-web-analytics-marketing.aspxhttp://www.aberdeen.com/Aberdeen-Library/6048/RP-web-analytics-marketing.aspxhttp://www.aberdeen.com/Aberdeen-Library/6048/RP-web-analytics-marketing.aspxhttp://www.aberdeen.com/Aberdeen-Library/6048/RP-web-analytics-marketing.aspxhttp://www.aberdeen.com/Aberdeen-Library/6048/RP-web-analytics-marketing.aspxhttp://www.aberdeen.com/Aberdeen-Library/6048/RP-web-analytics-marketing.aspxhttp://www.aberdeen.com/Aberdeen-Library/6048/RP-web-analytics-marketing.aspxhttp://www.aberdeen.com/Aberdeen-Library/6048/RP-web-analytics-marketing.aspxhttp://www.aberdeen.com/Aberdeen-Library/6048/RP-web-analytics-marketing.aspxhttp://www.aberdeen.com/Aberdeen-Library/6048/RP-web-analytics-marketing.aspxhttp://www.webanalyticsassociation.org/resource/resmgr/docs_research_committee/waa-outlook-survey-report-20.pdfhttp://www.webanalyticsassociation.org/resource/resmgr/docs_research_committee/waa-outlook-survey-report-20.pdfhttp://www.webanalyticsassociation.org/resource/resmgr/docs_research_committee/waa-outlook-survey-report-20.pdfhttp://www.webanalyticsassociation.org/resource/resmgr/docs_research_committee/waa-outlook-survey-report-20.pdfhttp://www.webanalyticsassociation.org/resource/resmgr/docs_research_committee/waa-outlook-survey-report-20.pdfhttp://www.webanalyticsassociation.org/resource/resmgr/docs_research_committee/waa-outlook-survey-report-20.pdfhttp://www.webanalyticsassociation.org/resource/resmgr/docs_research_committee/waa-outlook-survey-report-20.pdfhttp://www.webanalyticsassociation.org/resource/resmgr/docs_research_committee/waa-outlook-survey-report-20.pdfhttp://www.webanalyticsassociation.org/resource/resmgr/docs_research_committee/waa-outlook-survey-report-20.pdfhttp://www.webanalyticsassociation.org/resource/resmgr/docs_research_committee/waa-outlook-survey-report-20.pdfhttp://www.webanalyticsassociation.org/resource/resmgr/docs_research_committee/waa-outlook-survey-report-20.pdfhttp://www.webanalyticsassociation.org/resource/resmgr/docs_research_committee/waa-outlook-survey-report-20.pdfhttp://www.aberdeen.com/Aberdeen-Library/6048/RP-web-analytics-marketing.aspx -
8/6/2019 007 Www.esigmaGroup Com Advanced Site Optimization
5/16
AdvancedSiteOptimization:SecondGenerationDigitalAnalyticalToolsinAction
1999 2010TealeafTechnology,Inc.AllRightsReserved. 4
Highlymotivatedorganizationshavehadsomesuccessdevelopingintegrateddatamodels,oftenbuilton
topofcostlydatawarehouses.Mostbusinessessimplytryandmakedowiththeirdatasilos,hopingthat
analysisresourceswillreconcilethedifferencesandprovideaconsistentandbelievableassessmentofthe
online opportunity using the technology they have at their disposal. Unfortunately, collecting and
keeping data for datas sake nearly always fails to produce the type and level of insights most
companiesexpect.
WEBANALYTICSAPPLICATIONSLACKANALYTICAL CAPABILITIES
Despiteassurances to thecontrary, themajorityof technologiesdeployedforwebanalytics todaydo
very little analysis of data. While certainly improving, most are little more than glorified reporting
interfacesallowingforasmallamountofcustomizationandsegmentation.Themostpowerfulsolutions
advanceddatavisualizationandsegmentationsuiteslikeOmnitureInsights,CoremetricsExplore,and
Unicas NetInsights do an amazingjob of summarizing and packaging the databut stop short of
puttingthatdatatousealgorithmicallyandmathematically.
When pressed to explain why their analytics applications dont do any real analysis eitherby
leveraging statistical models, supporting predictive analytics, or simply providing confidence intervals
andqualityoffitscoresforsegmenteddatatheanswerinevitablycomestowedidntthinkouruserswouldunderstandthat typeofoutput.While thismayhavebeentrue in thepast,and tosomeextent
remainstruetoday,theresultisaprofoundgapbetweenwhatthesystemscoulddoforthebusinessand
whattheyactuallydo.
Thisgapmanifestsinavarietyofways:
Insteadofhighlightingchangesinratesandmeasuresbasedonstatisticalanomaliesdesignedto
call attention to unexpected changes, most systems simply present line graphs and require
analyststoexportthedataintoExcel;
Insteadofprovidingongoingrealtimeanalysispotentiallyacrossmultipledimensionsofdata
thatallowsanalystsandbusinessleaderstorespondinrealtimetoemergingchallenges,most
systems expect that the business cant respond quickly and assume that data latency is
acceptable;
Instead of extrapolating from existing data to make even basic predictions about potential
changes thereby enabling business owners to respond proactively, revise forecasts, and
otherwisebepreparedmostsystemsfocusonproviding informationaboutevents thathave
alreadyoccurred.
Expedia.com has longbeen known for their aggressive approach towards site testing and conversion
optimization. The company regularly conducts Opportunity Cost Analysis essentially an analytic
that uses statistical modeling to differentiate the likelihood to convert visitors experiencing problemsversusthosewhodonot.UsingTealeaf,Expediafocusesonspecificstepsinabusinessprocessflowand
developsdifferentsegmentsbasedoncustomerbehaviors.Forexampleconversionrateforcustomers
thattriggeredformvalidationmessageswhenenteringpersonalinformationversusthosewhodidnot.
Expedia then quantifies the opportunity cost for customers not convertingby applying an average
transactionvalueforsimilartypesoftransactions.
WhilethiseffortcurrentlyrequiresthatExpediaexportdatafromtheircustomerexperiencemanagement
environments forofflineanalysis, thevalue to thebusiness isclear:onecorrectedproblemreturnedan
-
8/6/2019 007 Www.esigmaGroup Com Advanced Site Optimization
6/16
AdvancedSiteOptimization:SecondGenerationDigitalAnalyticalToolsinAction
1999 2010TealeafTechnology,Inc.AllRightsReserved. 5
estimated $2 million to $3 million this year and, given that the problem had existed for some time, is
likelytoleadtotensofmillionsofdollarsinincrementalrevenueovertime.ByusingthisOpportunity
CostAnalysisanalytic,Expediaisabletoprioritizeprojectsbythebenefittotheirbusiness.Thisanalytic
has also enabled them to manage staffing levels focused on site optimization versus other competing
internalprojects,therebyensuringthatthebusinessisworkingonprojectswiththegreatestreturn.
ANALYSISSTAFFARENTTRAINEDFORANALYSIS
Given the limitationsof themajorityofwebanalyticsapplicationssold today, theresponsefrom the
business hasbeen to apply human resources to fill the gaps. Problem is, these operators web
analystsbydesignationandtradearerarelytrainedasanalystsintheclassicalsense,mostentering
the fieldby accident rather than intent. Even worse,because vendors have done such a goodjob of
convincingtheircustomersthatwebanalyticsiseasyandthetechnologypracticallyrunsitself,too
feworganizationshaveevensearchedforandhiredwebanalystresources,regardlessoftheirexperience
andeducation.
According to the Web Analytics Associations Survey 2010 report, 47% of respondents indicated threeyearsor lessexperienceinthefieldand76%hadsixyearsor lessexperiencein thefield.Comparedtomore traditional analyst roles in business intelligence and financial analysis, where training and
traditional education are firm requirements for employment, a somewhat disturbing pattern emerges:
business leaders are forced to assign increasingly highvalue analysis projects to immature technology
managedbypotentiallyinexperiencedhumanresources.
Wedontblametheanalystsorthebusinessforthissituationeveryoneissimplydoingwhattheycan
ina rapidly changingenvironment.Remember, five yearsago therewas no Facebook; three years ago
therewasnoTwitter;twoyearsagotherewasnoiPhone;lastyeartabletcomputingwassomethingthat
hadbeentriedandhadfailed.Theproblemisnotasmuchthepeople,data,ortechnology;itishowthe
technologyleveragesthedatasothatthepeoplecandothejobtheyvebeenaskedtodo.
NETRESULT:INSIGHTSARESLOWTO MATERIALIZE
Thenetresultof thecurrentstate is thatmostbusinessesstruggle to translate thevolumesofdigitally
collecteddataintomeaningfulinsights,muchlesssolidbusinessrecommendations.Whatsworseisthat
we at Web Analytics Demystified see the potential for a perfect storm in digital analytics: senior
leadershipisincreasinglyaskingformoreadvanced,morerobust,andmorevaluableanalysis,butdata
volumesareincreasingexponentially,inmanycaseswithoutacommensurateincreaseinthecapabilities
ofmostavailabledataanalysisapplications.
Fortunately,thereishopeonthehorizoncomingintheformofsystemsandmethodologiesdesignedto
accelerate time to insightanddramatically improvethevalueexistingdataprovides.Thesesystemsdo
morewith
the
available
data,
ranging
from
rapid,
statistically
powered
design
iteration
support
to
full
blown,modelbasedanalysistomoreeffectivelysurfaceinsights.
-
8/6/2019 007 Www.esigmaGroup Com Advanced Site Optimization
7/16
AdvancedSiteOptimization:SecondGenerationDigitalAnalyticalToolsinAction
1999 2010TealeafTechnology,Inc.AllRightsReserved. 6
ADVANCEDSITEOPTIMIZATION
AsdiscussedinlengthinourSASsponsoredwhitepaperTheComingRevolutioninWebAnalytics3,WebAnalyticsDemystifiedbelievesweareon the cuspofa paradigm shift in howcompanies analyze and
value digitally collected data. Increasingly we see movement towards what we described as second
generation digital analysis tools applications and platforms that allow forboth greater depth ofexplorationofexistingdataandbegintoautomatethesurfacingofinsightsinwaysthatfirstgeneration
toolssimplydonot.
PUTTINGDATATOWORKFOR YOU(NOTTHEOTHERWAYAROUND)
Fundamentally,advancedsiteoptimizationisaboutgettingyourdatatoworkforyou,notrequiringyou
tocontinue toworkforyourdata.Again, thestatusquo inwebanalytics todayisatworstwhatmy
friendDennisMortensenofYahoo!hascoinedreportsurfing4andatbest,hypothesisdrivenresearch
that requires tremendous attention to detail to account for myriad nuances in data collection, data
processing, and systems management. In either situation the analyst tasked with converting data into
information, information into insights, and insights into recommendations is required to spend hours
siftingthroughdatatoensurethatA)achangeismanifestandB)thechangemanifestissignificant.
Thisistremendouslywasteful,especiallyinthecontextofthelimitednumberoftrulyexperienceddigital
analystsworking in themarket todayand thefact that theofflineworldhasdoneaprettygoodjobof
leveraging data, statistical models, and predictive systems to focus analysis. In fact, when it comes to
timetoinsightsdespitethecommoncomplaintaboutbusinessandcustomerintelligenceinitiatives
onlineanalystshavealottolearnfromtheirofflinecounterparts.
FIRSTSTEPSTOWARDSTHECOMINGREVOLUTION
Theobservationthatmostwebanalyticseffortstodayarenotparticularlyefficientjuxtaposedagainstthe
demonstrablevalueofmodelbasedsystemshaslongprovidedWebAnalyticsDemystifiedwithonlyone
possibleoutcome:webanalyticspractitionersanddigitalmarketersneedbebegintoadoptmorerobust
andcapabletools,technologies,andprocesses.AswedescribedindetailundertheheaderofHow[web
analytics] Technologies Fail Us Today in The Coming Revolution in WebAnalytics,business owners,marketers,andanalystsalikeneedmorepowerfultoolstoaugmenttheirreactivereportingeffortstoday.
Referencing Tom Davenport andJeanne Harris in Competing onAnalytics, we agree that a sustainablecompetitiveadvantagebuiltaroundanalyticswillmanifestonlywhenthebusinesshasaddressedbothits
needforongoingreportingandadvancedanalytics.InFigure1,thetechnologiesDavenportandHarris
proposetocreatethistypeofadvantageareoutlined:
3http://www.webanalyticsdemystified.com/content/whitepapers.asp4http://visualrevenue.com/blog/2007/09/webanalyticsreportsurfingandhowto.html
http://www.webanalyticsdemystified.com/content/white-papers.asphttp://www.webanalyticsdemystified.com/content/white-papers.asphttp://www.webanalyticsdemystified.com/content/white-papers.asphttp://www.webanalyticsdemystified.com/content/white-papers.asphttp://www.webanalyticsdemystified.com/content/white-papers.asphttp://visualrevenue.com/blog/2007/09/web-analytics-report-surfing-and-how-to.htmlhttp://visualrevenue.com/blog/2007/09/web-analytics-report-surfing-and-how-to.htmlhttp://visualrevenue.com/blog/2007/09/web-analytics-report-surfing-and-how-to.htmlhttp://visualrevenue.com/blog/2007/09/web-analytics-report-surfing-and-how-to.htmlhttp://visualrevenue.com/blog/2007/09/web-analytics-report-surfing-and-how-to.htmlhttp://visualrevenue.com/blog/2007/09/web-analytics-report-surfing-and-how-to.htmlhttp://visualrevenue.com/blog/2007/09/web-analytics-report-surfing-and-how-to.htmlhttp://visualrevenue.com/blog/2007/09/web-analytics-report-surfing-and-how-to.htmlhttp://visualrevenue.com/blog/2007/09/web-analytics-report-surfing-and-how-to.htmlhttp://visualrevenue.com/blog/2007/09/web-analytics-report-surfing-and-how-to.htmlhttp://visualrevenue.com/blog/2007/09/web-analytics-report-surfing-and-how-to.htmlhttp://visualrevenue.com/blog/2007/09/web-analytics-report-surfing-and-how-to.htmlhttp://visualrevenue.com/blog/2007/09/web-analytics-report-surfing-and-how-to.htmlhttp://visualrevenue.com/blog/2007/09/web-analytics-report-surfing-and-how-to.htmlhttp://visualrevenue.com/blog/2007/09/web-analytics-report-surfing-and-how-to.htmlhttp://visualrevenue.com/blog/2007/09/web-analytics-report-surfing-and-how-to.htmlhttp://www.webanalyticsdemystified.com/content/white-papers.asp -
8/6/2019 007 Www.esigmaGroup Com Advanced Site Optimization
8/16
AdvancedSiteOptimization:SecondGenerationDigitalAnalyticalToolsinAction
1999 2010TealeafTechnology,Inc.AllRightsReserved. 7
Figure1:ReportingandAnalyticsToolsandCapabilities,takenfromCompetingonAnalytics(DavenportandHarris,2007.)
The red line is our addition, indicating the break between firstgeneration and second and third
generationdigitalanalytics tools.Everythingbelowthelineisessentiallywebanalyticsasweknow it
todayandeverythingabovethelineisdigitalanalyticsaswewilldoittomorrow.
When we published this white paper, many in the community responded negatively, essentially
concerned that the technology and processes above the line were difficult to use, expensive, and
fundamentallyoutsidetheirareaofexpertise.Whilethismaybetrue,itisdifficulttoreconcilecomplaints
thatareprimarilypersonalagainsttheneedsofthebroaderbusiness.Thelastconcern,cost,isreasonable,
althoughinourresearchwediscoveredthatmostenterpriseclasscompaniesactuallyhavemanyofthe
necessary technologies already in place. All that is required is to bridge the gap between online
analystsandbusinessanalystswithinthebusiness.
Thatsaid,weatWebAnalyticsDemystifiedprideourselvesongivingpractical,realworldadviceand
guidance and, given this response, we sought examples in the market today of the type of second
generation toolsand technologies likely tocatalyze theparadigmshiftwehavedescribed.Fortunately,
we foundagoodpartner for thiswork inTealeaf, thispaperssponsor,whohaverecentlyreleasedan
excellentexampleoftechnologythatscapableofputtingthedatatoworkforthebusinessbyleveraging
welldesigned statistical models applied to digitally collected data. We also found examples in Google
Analytics Analytics Intelligence feature and the increasingly adopted Multivariate and A/B testing
platformsprovidedbyvendorslikeGoogle,OmnitureandAutonomy.
SECONDGENERATION DIGITAL ANALYSIS TOOLS
Businessownerswishing togetaheadstarton transforming theirdigitalanalyticaleffortsand leading
the revolution are in luck. The past year has seen two new secondgeneration tools to complement
existing testing platforms,both taking good advantage of the available data to surface otherwise non
obviousinsightsandevidenceofconsumerstruggleonline.
-
8/6/2019 007 Www.esigmaGroup Com Advanced Site Optimization
9/16
AdvancedSiteOptimization:SecondGenerationDigitalAnalyticalToolsinAction
1999 2010TealeafTechnology,Inc.AllRightsReserved. 8
MULTIVARIATEANDA/BTESTING
As discussed in our paper on The Coming Revolution in Web Analytics, we believe that A/B andMultivariate testing tools provide an excellent example of analytical systems leveraging digitally
collected data to focusbusiness decisions. Without going into detail, these systems essentially power
controlled experiments, exposing a sample of site visitors to alternative messages in an attempt to
determinewhichmessageismostlikelytodrivethedesiredoutcome.
The important thing in the context of secondgeneration tools is that these systems rely entirely on
statistical modeling to determine the winner in an experiment. The model usually depends onboth
participationandoutcomedataforexample,peoplebeingexposedtotheexperimentandthengoing
on tocomplete whatevermeasurablesuccessevent isbeing tracked. In this context, the resultsarenot
presentedasthismanypeopledidXbutratherwitha95%confidencethemodelfindsthatXis10%
betterthanY.
Figure2:Exampleresults fromamultivariate test inGoogleWebsiteOptimizershowing thesystemspredictionandconfidence
thatCombination2 OwnaBusinesswilloutperformthecontrolgroup(47.8%improvementwith96.5%confidence)
Web Analytics Demystifieds point of view is that A/B and Multivariate testing systems are incredible
secondgenerationdigitalmeasurementsystemsforthreemainreasons:
1. They shift expectations away from whole numbers and raw counts towards samples and
confidenceintervals;
2. They are able to dramatically simplify reporting, focusing on winners and losers without
needinghugespreadsheets;
3. They explicitly point the business toward logical actions for example, widely deploy
combinationnumbertwoandincreaseyourconversionrate.
While testing continues to gain significant traction within the enterprise, these systems are oftentimes
difficult to deploy and require a specific set of expertise in order to reap their fullbenefits. The same
cannotbesaidforthenextnewestsecondgenerationdigitalanalyticstoolprovidedbyGoogle.
-
8/6/2019 007 Www.esigmaGroup Com Advanced Site Optimization
10/16
AdvancedSiteOptimization:SecondGenerationDigitalAnalyticalToolsinAction
1999 2010TealeafTechnology,Inc.AllRightsReserved. 9
ANALYTICSINTELLIGENCE
GooglesGoogleAnalyticsofferinginitiallygainedtractioninthemarketplaceasanexcellententrylevel
tool,capableofmeasuringareasonablenumberof common digitalactivitiesandavailableat thebest
possiblepricefree.However,intheyearsfollowingtheapplicationslaunch,theteamatGooglehas
builtouttheproducttocreateoneofthemostuserfriendlyandusefulclickstreammeasurementtoolsin
thewebanalyticsindustry.Theycurrentlysetthebarforavarietyofanalyticalneeds,includingcustom
datacollection,segmentation,andcustomreporting.
InOctober2009,GoogletookGoogleAnalyticstoanentirelynewlevelbyintroducingsecondgeneration
reportingcapabilities,anindustryfirst,intotheapplicationwiththeirAnalyticsIntelligencefeature.Still
inbeta,accordingtoGoogle:
[AnalyticsIntelligenceis]theinitialphaseofanalgorithmicdrivenIntelligenceenginetoGoogleAnalytics.AnalyticsIntelligencewillprovideautomaticalertsofsignificantchanges inthedatapatternsofyoursitemetricsanddimensionsoverdaily,weeklyandmonthlyperiods.
Figure3:Exampleof Analytics Intelligence in Google Analytics. Thebars at thebottom indicate the numberof alerts generated
basedonthealertsensitivity(aproxyfornumberofstandarddeviations)
Whilehelpingtoidentifythedaysthatstatisticallysignificantchangesareoccurringisonething,going
onestepdeeperandhighlightingthebasisforthosechangesissomethingelsecompletely.
-
8/6/2019 007 Www.esigmaGroup Com Advanced Site Optimization
11/16
AdvancedSiteOptimization:SecondGenerationDigitalAnalyticalToolsinAction
1999 2010TealeafTechnology,Inc.AllRightsReserved. 10
Figure4:Exampleofdrilldown inAnalyticsIntelligence inGoogleAnalyticshighlightingthedimensionsofdataprompting the
alerttobegenerated,theexpectedrangeandobservedvalues,andtherelativesignificance(graybars).
WhileGooglesofferingisanexcellentexampleofsecondgenerationfunctionality,criticsaptlypointout
thatGooglesuseofstatisticsissomewhatrudimentary.Moreimportantly,Googlesdataisoftendelayedbyseveralhoursormore, thus limiting theapplicationsrealutilityasa insightdiscoveryengine.This
problemhasbeensolvedbythispaperssponsor,Tealeaf,intheirmostrecentofferingTealeaf8.
TEALEAF8
Tealeafhas longbeenknown for their enterpriseclass capabilities formonitoring visitorandcustomer
behaviorinthewebandmobilechannels.TheircustomersalistthatincludesExpedia.com,BestBuy,
Walmart.com, Comcast, Wells Fargo, and hundreds of other name brands across all sectors and
geographic segments have long used the companys applications to help identify the points where
visitors struggle and fail. Through integrations with popular clickstream, feedback, and customer
relationshipmanagementapplications,Tealeafhasbecomean integralcomponent in theenterprisesite
optimizationstack.
Because Tealeaf is physically installed in the network architecture for a site, the application is able to
provide true realtime feedback on visitor behavior and has tremendous oversight into online
interactions. This capability imparts a variety of advantages to Tealeaf customers, including uptothe
momentkeyperformanceindicators,richsegmentationcapabilitiesandbrowserreplaythefeaturefor
whichthecompanyhashistoricallybeenknown.
However,inthecontextofadvancedsiteoptimization, thecompanysmostrecentapplicationupgrade
(Tealeaf8)takesautomateddiscoverytoanentirelynewlevel.SimilartoGooglesAnalyticsIntelligence,
Tealeafleveragesstatisticalmodelstoevaluatewhetherchangesfoundinthedataaresignificant.Unlike
GoogleAnalytics,
Tealeaf
8tracks
customer
behaviors
within
segments,
in
real
time,
and
gives
system
usersalmostunlimiteddrilldowncapabilitiestoquantifytheactualrevenueimpactandexploretheroot
causeofobservedchanges.
-
8/6/2019 007 Www.esigmaGroup Com Advanced Site Optimization
12/16
AdvancedSiteOptimization:SecondGenerationDigitalAnalyticalToolsinAction
1999 2010TealeafTechnology,Inc.AllRightsReserved. 11
Figure5:ExampleofTopMoversandDriversdashboardprovidedinTealeaf8illustratinghowTealeafusesstatisticalmodelingto
automaticallyuncoverstruggleandothercustomerbehaviorissueswithinspecificcustomersegments,inrealtime.
While there hasbeen a longstanding debate over the value of realtime analytics within the digital
measurementcommunity,WebAnalyticsDemystifiedsopinionhasalwaysbeen that there isacertain
class ofbusinesses that require immediate reporting. Highvolume retail, financial services, travel and
transportation, and organizations that depend as much on customer support as sales (e.g., Comcast,
DirectTV, Sky) all have sound financial and satisfactionrelated drivers to uncover challenges and
opportunitiesasquicklyaspossible,especiallyinthesocialera.
Case in point: Matt Raines,Vice President of Technology at online retailer BlueFly, notes thatbecauseBlueFly focuses on ontrend and inseason clothing, any service outage or inability to transact is
unacceptable.Recently,oneofthecompanyscreditcardprocessingpartnersmadeaseeminglyharmless
change that stopped BlueFlys ability to transact using a handful of credit cards. Because this change
occurredatmidnight,andbecausecustomersweregettingonlygenericerrors,theproblemwasslowto
surface.
WithTealeaf8inplace,BlueFlyhastheabilitytosurfaceobscure,highimpactissueslikethisonebefore
theybecomesignificantpointsofstrugglefortheircustomers.AccordingtoMr.Raines,Tealeafprovides
my team the ability to identify, diagnose, and resolve problems like thisbefore theybecome an issue
across the entire company. The realtime, automated insights providedby Tealeaf are the difference
betweenaresponse
measured
in
hours
and
one
measured
in
days
for
me,
and
that
is
huge.
-
8/6/2019 007 Www.esigmaGroup Com Advanced Site Optimization
13/16
AdvancedSiteOptimization:SecondGenerationDigitalAnalyticalToolsinAction
1999 2010TealeafTechnology,Inc.AllRightsReserved. 12
Figure6:ExampleofrevenueimpactanalysisanddrilldowntovisualsessionreplayusingTealeaf8.
AnotherTealeaf
customer,
ING
DIRECT
USA,
the
nations
largest
direct
bank,
has
been
using
Tealeafs
analytical capabilities to quickly identify problems on the site. Once identified, analysts work to
understandthenatureandseverityofthechallengetobettercommunicatewithintheorganization.This
effort,poweredbyTealeaf,reducestheamountofuncertaintywhenproblemsemerge,allowingING
DIRECTUSAtomakethebestdecisionspossiblebasedondataandintuition,notoneortheother.
WerelyheavilyonTealeaffortrackingcriticalprocessesonoursites,saysEthanJohn,ManageratING
DIRECT USA. Because we are an agile development shop, realtime monitoring of our critical online
business and customer processes gives us the ability to identify, size, and proactively respond to
customerneeds.OutputfromtheanalyticalcapabilitiesinTealeafhasbecomepartofourdailyscorecard,
andasaresultTealeafdriveninsightshavewidespreadacceptancewithinthebusiness.
-
8/6/2019 007 Www.esigmaGroup Com Advanced Site Optimization
14/16
AdvancedSiteOptimization:SecondGenerationDigitalAnalyticalToolsinAction
1999 2010TealeafTechnology,Inc.AllRightsReserved. 13
Figure7:ExampleofTealeaf8managementdashboardprovidingasnapshotofkeycustomerexperienceindicators.
AnotherTealeafcustomer,Expedia.com,identifiesissuesimpactingconversionbymakingextensiveuse
ofstatisticalregressiontoidentifyPopsandDropsofkeyindicatorsandcustomerbehaviorsgenerated
byTealeaf.GiventhatthecompanyhasthousandsofTealeafeventstheyareactivelytracking,thereisnt
anoptiontousebasicdashboardorspreadsheetreporting.Insteadtheyrelyonabasicstatisticalanalysis
oftheexporteddatatomineforeventsthathavechangedmorethanexpecteddayoverdayandweek
overweek.
Figure8:Exampleofdrilldownintorealtimerevenueopportunitylossanalysistriggeredbyasignificantdeviationinnumberof
customersstrugglingtocheckoutdetectedbyTealeaf8.
-
8/6/2019 007 Www.esigmaGroup Com Advanced Site Optimization
15/16
AdvancedSiteOptimization:SecondGenerationDigitalAnalyticalToolsinAction
1999 2010TealeafTechnology,Inc.AllRightsReserved. 14
Usingpopsanddropsservesasavaluableaddition to theotheranalysiswedoacrossExpedia.com,
saysMichaelGulmann,SeniorDirectorofGlobalSiteConversion.Tohaveanylevelofefficiencyatall
weneedourdatatoworkforus,nottheotherwayaround.Byleveragingverybasicstatisticalregression
functionality thatshouldbe ineverywebsiteoptimizationpackage inmyopinionweareable to
quicklydiscoveropportunitiesforoptimization.Byleveragingthissimpledatareportingstrategy,Mr.
Gulmann estimates that nearly $10 million hasbeen either saved or added to Expedia.coms FY 10
revenuestream.
WhileTealeaf8isnotthecompanysfirstproductusingstatisticalmodelingtosurfaceinsights,inWeb
AnalyticsDemystifiedsopiniontherecentupgradeisasignificantimprovementinhowthatinformation
is communicated. Even more crucial, Tealeaf is creating a new expectation of how enterpriseclass
analyticsapplicationswillleveragethedatatheycollecttoexpeditetheanalysisprocess.
CONCLUSIONS
Giventhatthevolumeofinformationcollectedindigitalchannelsisunlikelytodecreaseanytimesoon,
wehopethatreadersofthispapercanappreciatethenecessityoflookingforinnovativewaystomanage
data and surface insights. More importantly, we hope that readers appreciate the value second
generationwebanalyticsarelikelytoprovide.Bystreamliningthepresentationofdatabasedoneventhe
most simple data mining and modeling, these applications will give analysts and organizations a
potentially dramatic competitive advantage over companies that must continue to manually mine for
insightsusingfirstgenerationtoolsandtechnology.
Whilewestillbelieve thatdigitalchannelswillultimatelybemeasuredandmanagedusingevenmore
powerful thirdgenerationtechnologies thatarealreadywidelydeployedacross theenterprise, it isour
opinionthattheadoptionanduseoftestingplatformsandrealtimemonitoringserviceslikeTealeaf8are
a critical step towards developing internal comfort and competencies with samples, statistical models,andalgorithmicallyderived insights.Thosecompanieswishing toexplore thepotential impactsecond
generation technology might have on their businesses are encouraged to reach out to the vendors
describedinthisdocumentortoWebAnalyticsDemystifieddirectlyforguidanceandsupport.
Attheendoftheday,businessesarerunonacombinationofdata,gutinstinct,andsoundjudgment.Our
hopeisthatthiswhitepaperhasclarifiedhowWebAnalyticsDemystifiedseestheuseofdataevolving
over time tobecome more timely, relevant, and valuable as part of decisionmaking processes. As
businesses increasingly depend onconnectedchannels,adeptanalysisof data from thesechannelshas
thepotentialtocreatecompetitiveadvantagesthathaveneverexisted.Yourbusinesshasanopportunity
toleveragethoseadvantagestogrowandthriveinyourmarketplace.
Wewelcomeyourfeedbackandcomments.
-
8/6/2019 007 Www.esigmaGroup Com Advanced Site Optimization
16/16
AdvancedSiteOptimization:SecondGenerationDigitalAnalyticalToolsinAction
1999 2010TealeafTechnology,Inc.AllRightsReserved. 15
ABOUTTHE AUTHOR
Eric T. Peterson, CEO and Principal Consultant at Web Analytics Demystified, has worked in web
analytics since the late 1990s in a variety of roles including practitioner, consultant, and analyst for
several marketleading companies. He is the author of threebestsellingbooks on the subject, WebAnalyticsDemystified,WebSiteMeasurementHacks,andTheBigBookofKeyPerformanceIndicators,aswellasoneofthemostpopularwebanalyticsbloggersatwww.webanalyticsdemystified.com.Mr.Petersonhas
committedmuchofhislifetothebettermentofthewebanalyticscommunity,somuchsothatJimSterne,
President and cofounder of the Web Analytics Association says Erics leadership in the industry in
unparalleled, his devotion to the community is legendary, and his years of experience translate
immediatelyintostrategicandtacticalcompetitiveadvantageforeverybodywhoworkswithhim.
ABOUTWEBANALYTICS DEMYSTIFIED
Web Analytics Demystified, founded in 2007 by internationally known author and former Jupiter
ResearchanalystEricT.Peterson,providesobjectivestrategicguidancetocompaniesstrivingtorealize
the full potential of their investment in web analytics. Bybridging the gap between measurement
technology andbusiness strategy, Web Analytics Demystified has provided guidance to hundreds of
companiesaroundtheworld,includingmanyofthebestknownretailers,financialservicesinstitutions,
andmediapropertiesontheInternet.
FormoreinformationonEricT.PetersonandWebAnalyticsDemystified,pleasevisit
www.webanalyticsdemystified.com,[email protected],or
call(503)2822601.
ABOUTTEALEAF
This whitepaper is sponsored by Tealeaf, the leading provider of online customer experience
management solutions. Tealeafs CEM solutions includeboth a customerbehavioranalysis suite anda
customer service optimization suite. For organizations that are making customer experience a top
priority, these solutions provide unprecedented enterprisewide visibility into every visitors unique
online interactions forongoinganalysisandwebsiteoptimization.Onlineexecutivestakeholders from
ebusiness and IT to customer service and compliance are leveraging Tealeaf to build a customer
experiencemanagementcompetencyacrosstheorganization.Foundedin1999,Tealeafisheadquartered
inSanFrancisco,California,andisprivatelyheld.Formoreinformation,visitwww.tealeaf.com.
http://www.webanalyticsdemystified.com/http://www.tealeaf.com/http://www.tealeaf.com/http://www.webanalyticsdemystified.com/