january 2020 research paper · 2020. 2. 26. · unstructured data, the obstacles many organisations...

RESEARCH PAPER

Differentiate Your Product With Unstructured Data Analytic Capabilities

January 2020

Sponsored by

Differentiate Your Product With Unstructured Data Analytic Capabilities

2 Computing | research paper | sponsored by OpenText

This document is property of Incisive Media. Reproduction and distribution of this publication in any form without prior written permission is forbidden.

ContEntS• Introduction p3

• Keyfindings p4

• Unstructureddata–howmuchisthereandwhydoes it matter? p4

• Obstaclestoinsight p8

• Unstructureddata’sstructuralchallenges p9

• Newambitions,oldtechnology p10

• Conclusion p13

• Aboutthesponsor,OpenText p14

Differentiate Your Product With Unstructured Data Analytic Capabilities

Computing | research paper | sponsored by OpenText 3

IntroductionThedigitaleconomyisgeneratingunprecedentedquantitiesofdata.Businesseshavemoredataat their disposal than ever before. Some of the most valuable data is unstructured and many businessesarestrugglingtounlockitsfullpotential.Withoutit,organisationsaren’tgeneratinga complete picture of their operations. Software providers and channel partners can help their customerstoextractinsight–andsubsequentlyvalue–fromthismountainofunstructureddata,and differentiate themselves from their competitors.

Computingsurveyed150individualsactivelyinvolvedinusing,testing,evaluatingorprocuringdataanalyticstoolsattheirorganisation.Theyrepresentcompaniesfromawidevarietyofindustries,includingbankingandfinance,logistics,manufacturing,retailandeducationtoexplorethestateofdataanalyticsinenduserorganisations,opinionsandusecasesaroundstructuredversusunstructureddata,theobstaclesmanyorganisationsarefacinginextractingvalueandinsightsfrom their unstructured data and how software providers and channel partners can respond to thesechallenges,addvalueandbecomekeyplayersintheshapingofdataanalyticsstrategyandsoftware.

Differentiate Your Product With Unstructured Data Analytic Capabilities

4 Computing | research paper | sponsored by OpenText

Key findings• Sixtypercentofrespondentsstatedthatamajority–between51and100percentoftheirdata–wasunstructured.

• Thetopmotivatorsforunstructureddataanalysisweregainingprocessefficiencies,includingautomation,andthedesireforgreatercustomerinsight.

• The top three use cases for unstructured data analysis were all very much customer focused –trendtracking,sentimentanalysisandbusinesscommunicationanalysis.

• Despitethecompetitivebenefitsthatunstructureddataanalysiscanconfer,only50percentofthosewesurveyedwereactuallydoingit.

• Sixty-ninepercentofrespondentssaidtheirorganisationsfounditquiteorextremelydifficulttogatherandanalyseunstructureddata.

• Fewerthanaquarterofrespondentssaidtheirorganisationhadbeensuccessfulingatheringandanalysingunstructureddata.

• Theprimaryobstaclestoderivinginsightfromunstructureddatawerecostofimplementation,ashortageoftechnicalexpertise,andthetechnologyitselffallingshortandbeingdifficulttointegrate.

• Only36percentofrespondentswereemployedinorganisationswhereaccesstodataanalysis was democratized with the empowerment of individual users.

• There is a mismatch between reported levels of data democratization (36 percent) and the proportionofrespondentsreportingtheuseofdatavisualisationdashboardswhicharesupposed to enable this democratization (53 percent).

• Only30percentofourrespondentswereusingArtificialIntelligence/MachineLearningaspart of their data analytics platforms.

Unstructured data – how much is there and why does it matter?Thefirststeptounderstandingwhyunstructureddataissoimportant,isunderstandingjusthowmuch of it exists in relation to more structured data.

Organisationshaveatorrentofdatapouringineachdayanditisbothincreasinginvolumeanddiversifyinginformat.Afewyearsago,mostunstructureddatausedtoconsistofmainlytext–emails,textfilesandsoon.Agreatdealofunstructureddataisnowintheformofvideoorphotosfromsocialmedia,audiofiles,andwebsitecontent.

There is also a considerable amount of data already in the public domain that can be accessed by businesscustomers–suchasgovernmentorcensusdataandmarketdatafromorganisationslikethe IMF. This public data is often in unstructured or semi-structured formats.

Differentiate Your Product With Unstructured Data Analytic Capabilities

Computing | research paper | sponsored by OpenText 5

Computingaskedrespondentstooursurveytosharewithusjusthowmuchoftheirorganisation’sdataisunstructured.Wecanseefromtheillustrationbelowthat60percentofthoseansweringthisquestionsaidthata majority of their data–between51and100percent–wasunstructured.Thisfindinganswersthefirstpartofthequestionofwhyunstructureddataisimportant.Unstructureddatamatterspartlybecauseitsimplycomprisesamajorityofthedataavailable for analysis.

Fig. 1 : What proportion of your organisation’s data is unstructured (such as emails and other communications, contracts, documents, and social media posts)?

Don’t know (1%)

0 - 25% (11%)

26 - 50% (27%)

51 - 75% (45%)

76 - 100% (15%)

Unstructureddataalsomattersbecauseoftheinsightthatcanbedrawnfromitsanalysis.Computingaskedtherespondentswhoseemployerswereanalysingunstructureddatawhattheyhopedtoachievefromdoingso(Fig.2,see next page).Thetopmotivator,citedby43percent,wasthescopeforreducedspendthroughgreaterefficiency/automation.Certainly,thepotentialforproductivitygainsbythebusinessprocessanalysisandoptimisationthatunstructureddataanalysiscaninformishuge.Businessprocessesofallkindsinvolveincreasingvolumesandvarietyof data and more applications are involved in business processes than in the past.

Manybusinessesarelookingtooptimisebusinessprocessesviagreaterautomationinordertobecomefasterandmoreflexible,andultimatelydeliverthespeedandpersonalisationofservicesthatconsumersareincreasinglydemanding.Replacingslow,manual,error-proneprocesseswithautomated,integratedworkflowscanbringabouthugeefficiencygains.However,inordertobesuccessful,themovetoautomationmustbedata-informedtoensurethatoptimisationisappliedwhereitcandeliverthebiggestgains.

Differentiate Your Product With Unstructured Data Analytic Capabilities

6 Computing | research paper | sponsored by OpenText

Fig. 2 : What are the main motivations for your organisation’s decision to gather and analyse unstructured data? [three maximum]

Reduced spend through efficiency/automation

Customer insights

Productivity gains

Competitive advantage

Improved working environment

New business models

Company reputation

Technology upgrades

New products

43%

42%

36%

26%

22%

21%

21%

15%

12%

Onlymarginallylessamotivatorwascustomerinsight,whichwascitedby42percent.Customerservicedataistypicallyunstructured–phonecalls,email,socialmediaposts,reviewsitesandevenchatbotinteractions.Gainingvisibilityofthisdataandanabilitytovisualise,sliceanddiceitenablesbusinessestogaininsightsintowhattheircustomerswantnow–andpredictwhattheyarelikelytowantinthefuture.Businessescangettoknowtheircustomersonalevelthatisimpossible when structured data is the only source of information.

Inarelatedquestion,wealsoaskedsurveyrespondentsexactlywhattheywereusingtheirunstructureddatafor.Thetopthreeusecaseswereallverymuchfocusedoncustomers,asthediagrambeneathillustrates.Themostpopularusecasewastrendtracking.Therearetwolevelsoftrendtracking.Manyofourrespondentswillbeanalysinghistoricalunstructureddatatolookfortrendsintheircustomerbehaviourwhichtheycanutiliseinthefuture,perhapsbysegmentingtheircustomersandtargetingthemaccordingly.

Differentiate Your Product With Unstructured Data Analytic Capabilities

Computing | research paper | sponsored by OpenText 7

Fig. 3 : What does your organisation currently use unstructured data analysis for? [select all that apply]

Trend tracking

Customer sentiment tracking

Business communications analysis

Intelligent recommendations

Paperwork processing

Predictive maintenance

AI-augmented document digitisation

Product recommendation

Other

44%

39%

39%

29%

28%

25%

19%

16%

4%

UsingAI-basedtextminingtoolsallowsbusinessesnottojustseewhathasoccurredbuttominehugedatasetsfrommanysources–blogs,socialmedia,reviewsitesarejustafewexamples–tomakepredictionsaboutmarkettrendsforkeytopics,productsorservices.Itisinterestingthatthesecond and third most popular use cases of unstructured data analysis were customer sentiment trackingandbusinesscommunicationsanalysis.Sentimenttrackingisamachinelearning-ledapplicationofAItopredictwhatcustomerswillwantanddeliveragainstexpectationsitbymeansof hyper personalised services and recommendations.

Lessloyalcustomerscanbeidentifiedandtargeted,whichmeansthatadvertisingcanbedeliveredwithlaser-likeaccuracy.BusinessCommunicationsAnalysisalsofocusesoncustomersto ensure that they are happy with the type of communications they have with companies rather thanbeingdissatisfiedwithinteractionstheydeemimpersonalorfaceless.

Differentiate Your Product With Unstructured Data Analytic Capabilities

8 Computing | research paper | sponsored by OpenText

obstacles to insightTheinsightthatunstructureddataanalysiscanprovideisunparalleled.Theusecasessetoutaboveillustratethispotential.Giventhesebenefitsitisreasonabletoexpectthatthemajority ofrespondentswouldbeanalysingthisdata.Theproblemis,asrevealedbythisresearch, that50percentofthemaren’t.

Thisposesanobviousquestion.Why?

Computingaskedaseriesofquestionsabouttheday-to-daytechnicalandorganisationalaspectsofunstructureddataanalysisanditistheanswerstothesequestionswhichprovideahintastowhybusinessesaren’tmakingbetteruseofthispreciousresource.Thefirstisshownbelowin Fig.4.Theanswerstothisquestionmakeitimmediatelyapparentthatmanymoreorganisationsthannotarefindingiteitherdifficultorverydifficulttogatherandanalysetherelevantdata.

Fig. 4 : on a scale of 1 to 5, ‘1’ being ‘extremely difficult’ and ‘5’ being ‘extremely easy’, how easy would it be for your organisation to gather and analyse its unstructured data at present?

Arelatedquestionaskingrespondentstoratethesuccessoftheirorganisationsatgatheringandanalysingunstructureddateyieldedaresultwhichcanprobablybebestdescribedasmediocre.Twenty-fourpercentawardedtheirbusinessesamoderatelyorhighlysuccessfulrating.Eighteenpercentdidtheopposite.Theremaining58percentchosefaintpraiseandwentwithamiddleratingofneitherparticularlysuccessfulnorunsuccessful.

Extremely difficult

23% 46% 22% 5% 4%

Extremely easy

1 2 3 4 5

Differentiate Your Product With Unstructured Data Analytic Capabilities

Computing | research paper | sponsored by OpenText 9

Unstructured data’s structural challengesWhat’sgoingwrong?Computingaskedrespondentstochoosethreemainobstaclestotheirorganisationsabilitytogatherandanalyseunstructureddata.Comingasasurprisetoabsolutelynoonewasthemostfrequentlycitedissue–costofimplementation(45percent.)Sofarthismakessense,butit’sworthsteppingbacktoviewthisfromapartnerperspectivetoaskwhy costs are so steep.

Fig. 5 : What are the main obstacles to your organisation’s ability to gather and analyse unstructured data? [three maximum]

Cost of implementation

Lack of expertise

Technology falling short

Technology poorly integrated

Security

Identifying possible applications

Privacy

Resistance on the ‘shop-floor’

Compliance

Governance

Building a business case

Don’t see the benefit

Board-level resistance

Ethical issues

Other

45%

41%

25%

25%

19%

17%

17%

15%

14%

13%

13%

10%

9%

8%

0%

Differentiate Your Product With Unstructured Data Analytic Capabilities

10 Computing | research paper | sponsored by OpenText

Publiccloudeconomicswereabeguilingprospecttobusinessesdesperatetofindcostsavingsintheharshbusinessclimateprevailinginthewakeofthefinancialcrisis.However,therealityhasn’tquitematchedexpectationsbecauseofthecomplexitythatmulti-cloudandhybridcloudinfrastructureshaveengendered.

It’simpossibletoanalysedataifyoucan’tseeit,andthisgrowingcomplexityhaslimitedvisibilityofdataandmadeitmuchhardertointegratedataandapplications.Itisnotacoincidencethatthesecondmostfrequentlycitedobstaclewaslackofexpertise,which41percentofrespondentsmentioned.Theexpertiseneededtointegrateandmanagethisfragilewebofapplicationsandunderlyingdataisinshortsupply–andthat’sbeforeyouevengettothedatascienceandanalysisskills.Seventy-eightpercentagreed,albeittovaryingdegrees,that,“itisdifficulttosuccessfullyintegratedataanalyticstoolsintoourtechnologystack.”

Thesefindingsrepresentarealopportunityforchannelpartnerslookingtoaddgreatervaluetofillskillsgapswheretheyexistonashort-termbasis.Employingstafffull-timeonpermanentcontractsisnotnecessarilyanapproachwellsuitedtobusinessestryingtomovetoamoreagilewayofworking–whichmanyaretryingtodo.WorkingAgileinvolvessmallteamstodeliverprojectswhichthenbreakandreassembleasrequired.Ideallyyoubringpeopletotheproject,ratherthantheotherwayaround.Byreducingthecostsandrisksassociatedwithpermanentemployment,theuseofcontracthiresfromchannelpartnersforprojectmanagement,consultancyandtechnicalskillsatalllevelscouldhelpbusinessestoshowsomequickwinsfordataanalysisprojects. It pays to lean on those that have been there and done it before.

new ambitions, old technologyAftercostandalackofexpertise,thenextmostcommonlyraisedobstaclestounstructureddataanalysisbothrelatedtotechnology.Twenty-sevenpercentofrespondentsraisedtechnologyfallingshortandpoorlyintegratedtechnologyasaproblem.Wehavediscussedthedifficultiesofintegrationalreadybutthesubjectoftechnologyfailingtokeeppacewithrequirementsisonethatrequiresfurtherexamination.

Therearelikelytobetwodimensionstothetechnologyproblem.Thefirstrelatestoorganisationalstructures–whoisresponsiblefordataanalysis?Thesecondismoreaboutthetoolsthemselves.Thesetwodimensionsarecloselyrelated–theeasierusersfindittovisualiseandcutdata,themoredemocratisedyourdataanalysiscanbe.Organisationalandprocessbarriersaresignificant.Eighty-fourpercentofrespondentsagreedtoatleastsomeextentthat,“itisdifficulttosuccessfullyintegratedataanalyticstoolsintoourbusinessprocesses.”

Ofcourse,mostorganisationshaveamixtureofanalysismethodsbeingapplied.Forty-twopercentofourrespondent’semployershadadedicatedcentralteamand44percenthadlocalspecialistteamsthroughouttheirorganisations.In36percentofcases,respondentsstatethatthey had a more democratised systems where individual employees were empowered.

Differentiate Your Product With Unstructured Data Analytic Capabilities

Computing | research paper | sponsored by OpenText 11

Fig. 6 : What organisational structures do you have in place to deliver data, insights and business value from data? [select all that apply]

26%

We have a dedicated central team

We have local specialist teams across the business

29%

19%

4%

4%

5%

7%

6%

Other

We practice data-democratisation (where individual employees use data analytics to inform their day-to-day decision making)

Fig.6indicatessomeoverlapwithorganisationsrunningmultiplescenarios.Forexample,it’squitepossibletohavealimitednumberofusersanalysingtheirowndataalongsideacentralresourcetohelpwithmoreintricateanalysis.Thelackofhybridapproacheshereshowsthelackofmaturityinmostorganisationsdataanalysisstrategy–particularlyatthoselargerorganisationswhostandtobenefitfromacombinationofmethods.

Theactualtoolsandtechnologiesinusevarysignificantly,asFig.7shows(see over).

Differentiate Your Product With Unstructured Data Analytic Capabilities

12 Computing | research paper | sponsored by OpenText

Fig. 7 : How does your organisation analyse its data? [select all that apply]

Thefindingstothisquestionprovideaseriesofcluesintowhyunstructureddataanalysisisprovidingsolittlecollectiveinsight.Thefactthatthereisamismatchbetweenreportedlevelsofdatademocratisationandtheproportionofrespondentsusingdatavisualisationdashboards,whicharesupposedtoenablethisdemocratisation,indicatesthatthevisualisationdashboardsaren’tasuserfriendly,orsuitableforbroaduse,astheyshouldbe.

The essence of self-service dashboards is that they enable the user to slice and dice data with ease butthecombinedfindingsheresuggestthattheystillneedconsiderableinputfromdataexpertstogettotheinsight.Thistimelagoninsightisprofoundlyunhelpfulforbusinessestryingtomakemoreinformeddecisionsfaster–afactnotlostonthe90percentofrespondentswhoagreedto atleastsomeextentwiththestatement,“datademocratisationisthebestapproachtobuilding adata-guidedorganisation.”

ThehighshowingofBusinessIntelligencetoolsalsohintsatpartofthereasonwhyunstructureddataanalysisisprovingsodifficultforsomanybusinesses.BItoolsjustweren’tdesignedwiththehugequantitiesofunstructureddatathatbusinessesneedtoanalyseinmind.ManyBIdashboardsonlyidentifyeventsaftertheyhaveoccurred.Thisisnotinsightful–iteffectivelyamountstomonitoring.Itcantellyouaboutaneventinthepastbutthereisoftenlimitedcontextarounddatapoints,sounderstandingandinsightisequallylimited.

ThirtypercentofourrespondentswereusingAIormachinelearning,andwhilstthesealgorithmscanderiveinsightsfromhugedatasets,theygenerallyrequiredatascienceskillstooperate,andwehavealreadyestablishedthattheseareinveryshortsupply.Itisalsolikelythatthelargeamountofbespokeandthird-partytoolswhichourrespondentsreportarecreatingfurthercomplexityandslowingdowntheprocessofderivinginsight.

Ourresearchsuggeststhataconsiderableproportionofbusinesseswhoaretryingtobringunstructureddataintotheiranalysisaredoingsobytryingtoadaptrelativelyoldanalysistoolstoaverynewchallenge.Thereportedsuccessratessuggestthisisunlikelytoproveawinningstrategy.

Business intelligence tools

Data visualisation dashboards

Third-party tools

Bespoke tools

AI/Machine learning

Third-party service provider(s)

Other

57%

53%

45%

30%

30%

16%

0%

Differentiate Your Product With Unstructured Data Analytic Capabilities

Computing | research paper | sponsored by OpenText 13

ConclusionIthasbecomeacommonclichéincommerceandtechnologytoassertthatdataisthenewoil.Thisisalimitedanalogybecause,unlikeoil,dataisincreasinginaggregatevolumeeverysecondofeveryday.Datascarcityisnotanissuethatbusinesscustomersarestrugglingwith.Whattheyarebattlingwithishowtorealisevaluefromthemountainofdataavailabletothem–particularlyunstructured data. Sixty percent of participants in our research told us that most of their data was unstructured.

Whatwererespondentshopingtogainfromunstructureddataanalysis?Reducedcostsviagreaterefficienciesandautomationandcustomerinsightwerethetopmotivators.Interestingly,themostpopularusecaseswereallverymuchfocusedoncustomers–trendtracking,sentimenttrackingandbusinesscommunicationsanalysis.Thedesiretounderstandthemotivationsandthoughtsofcustomersaboutvariousproducts,servicesorbrandsiswidespread,asisthedesiretopredict what customers will want.

Unstructureddataanalysissoundslikeanessentialcapabilityforbusinesseskeentodigitisetheirgoodsandservicepropositions.However,halfoftheorganisationswespoketowerenotdoingso.Almost70percentsaidthattheirorganisationsfounditdifficultorextremelydifficulttogatherandanalyseunstructureddataandfewerthanaquartersaidtheyhadbeensuccessfulindoingso. Themostcommonlyencounteredobstaclestogainingvaluefromunstructureddatawasthecostofimplementationandalackoftechnicalexpertise.Thesecostsandtechnicaldifficultiesarise,inpart,fromtheheadspinningcomplexityofhybridcloudinfrastructure.Thiscomplexitygivesrisetoseveralissuesbutrelevantherearethedifficultiesofdatavisibilityandintegration.Ahuge78percentofourrespondentsagreedthatitwasdifficulttosuccessfullyintegratedataanalyticstoolsintotheirtechnologystacks.

Analyticstechnologyisalsofallingshort.Thesefailuresareinpartcausedbyamismatchbetweenorganisationalstructuresandthetechnology,and,inpart,becausealotofthetechnologywasdesignedforabusinessenvironmentthatwasnotawashwithunstructureddata.

Perhapsthemosttellingfindingisthatonly30percentofrespondentswereusingAIormachinelearningtoanalysetheirunstructureddata.Businessesarelookingforhelpindefiningstrategiesandimplementingtherighttechnologytohelpthemextractinsight,andsubsequentvalue,fromtheir data mountains.

Softwarevendorsthatcanhelporganisationsovercomethesechallengesandtapintothebenefitsonoffer–leveragingtheclearhungerinthemarketforcapablesolutions–standtoflourishasunstructureddataanalysisbecomesubiquitous.

Often,thebestwaytodosoisbyembeddingalreadyavailablewhite-labelsolutions–bypassingthemassiveinvestment,timeandotherresourcesrequiredtodevelopAI-basedanalyticstools.Thisleavesvendorstofocusonwhattheydobest,andallowsthemtocreateproductsthatcanmakeunstructureddataanalysismoredemocratised.Whensimpletousebuthighlycapabletoolsareinthehandsofmoreendusers,informingtheirday-to-daydecisionmaking,unstructureddataanalysis reaches its potential.

83percentofthosewhosaiditisextremelyeasyfortheirorganisationtogatherandanalyseitsunstructureddata,revealedthattheanalysisofunstructureddatahadbeenextremelysuccessfulattheircompany.Softwarevendorsandotherpartnerscanhelpmakethisarealityformorebusinesses.

Differentiate Your Product With Unstructured Data Analytic Capabilities

14 Computing | research paper | sponsored by OpenText

About the sponsor, opentextOpenText,TheInformationCompany,enablesorganizationstogaininsightthroughmarketleadinginformationmanagementsolutions,on-premisesorinthecloud.

OpenTextprovidesorganisationswithkeycapabilitiesformanaginginformationatanystageoftheinformationlifecycle,andtheOpenTextOEMProgrammakesthissametechavailabletoothertechnologyvendorstobecustomized,extended,embeddedandwhite-labelledforuseintheirownproducts and services.

AkeyareaoffocusfortheOpenTextOEMProgramistheanalyticsspace.OpenText’sanalyse,predictandreportsolutionsprovideOEMpartnerswiththeabilitytoreadilyintegrateahostofanalyticcapabilities,includingunstructureddataanalytics.

FormoreontheOpenTextOEMProgram,pleasevisitourwebsiteorcontactustogetstartedtoday. You can also stay up to date on trends and opportunities that tech vendors care about by readingtheOpenTextOEMblog.

OEM Program: www.opentext.com/products-and-solutions

Contact: www.opentext.com/about/contact-us

Blog: blogs.opentext.com