building business service intelligence with itsi

SetupBeforeYouCanPlay1. Downloadthispresentationslidedeck:https://splunk.box.com/v/Chicago-HandsOn-Workshop

2.Ifyouhavenotdonesoalready,SignupfortheFREESplunkITSIOnlineSandbox:• http://splunk.com/itsi• Select"FreeOnlineSandbox"

3.Pleasetestaccesstoyoursandbox;• Chrome,Firefox,Safari

arerecommended;• IEisNOTrecommended

4.Afterloggingin,selectITServiceIntelligencefromthelistofappsattheleft

1

Copyright©2016SplunkInc.

BuildingBusinessServiceIntelligencewith

SplunkITServiceIntelligence

TomHarropITOperationsSpecialist

ThursdayOctober27,2016

MichaelDonnellyITOAArchitect

Agenda

3

u IntroductionsandSetUpu Splundamentals – ITTroubleshootingwithSplunku WhatisITServiceIntelligence?u ServiceIntelligenceDesignPracticesu Let'sPlay!u What'sNext?u HappyHour!

SafeHarborStatementDuring the course of this presentation, wemaymake forward looking statements regarding future eventsor the expected performance of the company. We caution you that such statements reflect our currentexpectations and estimates based on factors currently known to us and that actual events or results coulddiffermaterially. For important factors that may cause actual results to differ from those contained in ourforward-looking statements, please review our filings with the SEC. The forward-looking statementsmade in this presentation are being made as of the time and date of its live presentation. If reviewedafter its live presentation, this presentationmay not contain current or accurate information. We do notassume any obligation to update any forward looking statements we may make. In addition, anyinformation about our roadmap outlines our general product direction and is subject to change at anytime without notice. It is for informational purposes only and shall not be incorporated into any contractor other commitment. Splunk undertakes no obligation either to develop the features or functionalitydescribed or to include any such feature or functionality in a future release.

4

DefiningServiceIntelligence

Enablingabusiness-awareITMeasuringandreportingonindicatorsthatmatter

UnlockingoperationalefficienciesCollaboratingacrosssilostoimproveserviceoperations

Data-baseddecisionmakingSolvingproblemsandanticipatingpitfallswithsophisticatedanalyticsandpowerfulinsights

Key Takeaways

1 BuildonwhatyouarealreadydoingwithSplunk

ServiceIntelligencedesignandconfigurationpractices

3 WhatispossiblewithSplunk ITServiceIntelligence

Splundamentals – ITTroubleshootingwithSplunk

Challenging Traditional Methods

Network Infra

structureLayer

Storage

Server

74%

-36%

Applica

tionLayerSyntheticAPM

ByteCodeInstrumentation

AdaptiveThresholding

HPRun-TimeServiceModelCAServiceOperationsInsight

IBMNetCool/OmnibusServiceModeldefinition&CorrelationEngine

Business Layer

Aggregation/Correlation/Visualization

Service Layer

Challenges• Toomanydisparatecomponents• DifficulttodefineServiceModel• Laborintensive• Mostimplementationsfail• Veryimportantsourceis

missing!(machinedata)

Data-Defined & Driven Service Insights

Infra

structureLayer

Applica

tionLayer

Splunk> isthemissinglink• DataFidelity• SingleRepositoryforALLdata• EasiertoManageServices• ReducedIntegrations• ReducedPointSolutions• CollaborativeApproach• Quicktimetovalue

Data Fabric Platform

Service Intelligence

NetworkPacket,Payload,Traffic,Utilization,Perf

SyntheticAPMAvailability,Capacity,UserExperience

ByteCodeInstrumentationUsage,Experience,Performance,Quality

AdaptiveThresholdingApps,Services,Systems74%

-36%

ServerPerformance,Usage,Dependency

StorageUtilization,Capacity,Performance

MACHINEDATA

SplunkApproachtoMachineData

10

StructuredRDBMS

SQL

SchemaonWrite

Traditional

ETL

Search

SchemaonRead

Splunk

UniversalIndexing

Volume Velocity Variety

Unstructured

• DefineStaticschema• ETLintoSchema• Enrichatwrite• Newdata=newcolumns• Newquestions=newcolumns• “Dataatrest”(delayedinfo)• LaborIntensive&timeconsuming

IdealforReporting

• “Schema-on-the-Fly”• Datainnativeformat• Enrichatread• Newdata=nochangesneeded• Newquestions=nochangesneeded• “Datainmotion”(Realtime)• Fasttimetovalue

IdealforInvestigation

ListentoyourdataLet’stakeacloserlookatITtroubleshootingwithSplunk

11

Machinelearning-poweredanalyticsforreal-timeserviceinsights,simplifiedoperationsandroot-causeisolation

ITServiceIntelligenceValueStack§ AdaptiveThreshold§ BehaviorAnomaly§ CorrelatesDataintoKnowledge

§ Visualizesentirestack§ ViewtheentireEcosystem§ 3clickstogettheanswerversus10

§ TimeSeriesIndex§ SchemaonRead§ DataModel

ServiceModel

ML

§ Accelerators§ Trendaggregation§ MultiKPIAlerts

ITSI

ThepossibilitiesforBusiness…

ThepossibilitiesforITOperations…

ServiceHealth

Buttercup Games Example

WhatisaService?

Service RequestsResponses

InITSI,aService isalogicalgroupoftechnologycomponentsthatauserdeemsneedtobemonitoredtogether.

Itcanoftenbegeneralizedasa“blackbox”whichwesendrequests,andexpectresponses

17

WhatisaService?

DNS RequestsResponses

TechnicalServices

Auth RequestsResponses

Web RequestsResponses

Servicescanbelowerlevel(technical)…

18

WhatisaService?

DNS RequestsResponses

TechnicalServices

OrderEntry VolumeRevenue

BusinessServices

Auth RequestsResponses

Web RequestsResponses

CustomerCare

RequestsSLACompliance

Servicescanalsobehigherlevel(business)…

19

WhatisaService?

PacketNetwork

HypervisorandHosts

RBMDBs

StorageTier

APIServices

WebServices

CustomerTransactions

Mobile

API/Middlew

are

BusinessFunction

DNS

ServicescanencompassmultipletiersoftheITdomain.Servicesmayalsodependuponotherservices

20

WhatisaKPI?

DNS

KPI:RequestvolumeKPI:ErrorrateKPI:AverageresponsetimeKPI:ServerCPUloadKPI:Configurationchanges

CustomerTransactions

KPI:TransactionvolumeKPI:ErrorrateKPI:AverageresponsetimeKPI:MaxresponsetimeKPI:CountofChangerecords

KPIsandHealthscoresconstitutethemeansbywhichServicesaremonitored.

21

BusinessFunction

KPI:BusinessvolumeKPI:ErrorrateKPI:RevenuerateKPI:ConversionrateKPI:CountofIncidenttickets

KeyPerformanceIndicators(KPIs)

22

AKeyPerformanceIndicator(KPI)ispoweredbyaSplunk searchinITSIthatmonitorsaspecificattributelikeCPUutilization,ResponseTime,NumberofErrorsandsoon.KPIsarecontainedwithinServicestomeasuretheirhealth.

ServiceHealthScores

23

AHealthscoreisascoreform0-100(0beingcriticaland100beingnormal)thatmeasuresthehealthofaService.ItiscalculatedbasedonallKPIsimportanceanditsstatus(e.g.green,orange,red),onceeveryminute.

Splunk ITServiceIntelligenceLet’stakeacloserlookatServiceIntelligencewithSplunk

24

ServiceIntelligenceDesignPractices

25

BringSubjectExpertsTogether

DesignBeforeConfiguring

BestPracticesforServiceIntelligence

StartWithaProblemWorth

Solving

StartWithAProblemWorthSolving

Reviewyourorganization’scriticalservices

Identifyaservicethathasimpactfulandmeasurablechallenges

ButtercupGames– HowCanWeHelp?

Manufactureroftoysandgames

Desiretoimprovesupplychainefficiencyandcustomersatisfaction

Newonlinestorehasissuesthatimpactcustomerexperienceandrevenue

TheBusinessProblemforButtercupGames

SupplyChain

LimitedVisibility

FrequentBottlenecks

ERPSystems

BusinessImpact

$48,000/wkinrevenue

loss

Warrooms32hrs/wk

???

FailedInteractions

OnlineStore

PoorCustomerSatisfaction

BringSubjectExpertsTogether

Identifystakeholdersandsupportpersonnelfortheselectedservice

Createawarenessandinvitetheircollaborationtosolvethebusinesschallenge

31

YourServiceIntelligenceCollaboratorsServiceOwners

• Businessfunctions

• Performanceindicators

• Commonbusinessissues

• Frequencyofissues

• Businessimpactofissues

OperationsandSupport

• Commonissues• Performanceindicators

• Resolutionprocesses

• Toolsusedforresolvingissues

• Frequencyofissues

• ITimpactofissues

EnterpriseArchitecture

• Businessprocesses

• Keyinputsandoutputs

• Technologyarchitecture

• Dataarchitecture

• Commonissues

Administrators

• Currenttoolsandusage,andadoptionlevels

• Splunkexpertise• Environmentexpertise

• Personalpain

DesignBeforeConfiguring

Identifypains,performanceindicatorsandmeasurementgoalsfortheservice

Identifycomponentsanddataneededtodriveserviceinsights

Consolidatethemappingsintoanenterpriseprocess/ITservicesmap

ServiceIntelligenceGoalsforButtercupGames

SupplyChain

LimitedVisibility

FrequentBottlenecks

ERPSystems

BusinessImpact

$48,000/wkinrevenue

loss

Warrooms32hrs/wk

???

FailedInteractions

OnlineStore

PoorCustomerSatisfaction

GOAL1Continuousimprovementthroughvisibilitytokeyindicatorsofsupplychain

performance

GOAL2Increasecustomersatisfactionandreducecostthroughfewerfailuresandrestoration

activities

ServiceIntelligenceDesign– ButtercupGames

Infrastructure Layer

Application Layer

Business Layer

Service Layer

OrderEntry Manufacturing Shipping Fulfillment

SupplyChain

OnlineStore EDI

WebTier Middleware

• TotalOrders• TotalRevenue

• UnitCount• UnitFailures

• ServiceLevel • DeliveryTime

• OnlineOrders• OnlineRevenue• ResponseTime

• ServiceHealth• Incidents/Changes• CustomerSatisfaction

• HTTPHits• ErrorRate

• CPULoad•MemoryUsed• DiskUsed• IOLatency


• ResponseTime• ErrorRate

• ResponseTime• StorageFree

ServiceDecomposition

InfrastructureLayerPower/Cooling/FacilitiesServer–Networking–Storage

ServiceLayer BusinessService

ApplicationLayerMiddleware–ApplicationServer-DatabaseCustomApps

BusinessLayerMailTransport-OrderProcessingE-Commerce-Financials

ServiceIntelligenceDesigninITSI1. High-valuebusinessservices

• ButtercupGamesOnlineStoreandSupplyChain

2. Majorbusinessfunctions• OrderEntry,Manufacturing,ShippingFulfillment

3. Supportingservices• Web,Middleware,Database

4. RelevantKPIsforeachservice• Database:,errors,SQLhits,…)

5. Splunk searchforeachKPI• (index=DB(warn*ORerror*)|statscount)

36

ServiceDecomposition– ButtercupGames


Application Layer

Business Layer

Service Layer


SupplyChain

OnlineStore EDI

WebTier Middleware

PuttingItAllTogether


Application Layer

Business Layer

Service Layer


SupplyChain

OnlineStore EDI

WebTier Middleware

• TotalOrders• TotalRevenue

• UnitCount• UnitFailures

• ServiceLevel • DeliveryTime

• OnlineOrders• OnlineRevenue• ResponseTime

• ServiceHealth• Incidents/Changes• CustomerSatisfaction

• HTTPHits• ErrorRate



• ResponseTime• ErrorRate

• ResponseTime• StorageFree

TypicalDataSources


Application Layer

Business Layer

Service Layer


SupplyChain

OnlineStore EDI

WebTier Middleware

• ApplicationLogs• CorporateDatabases• ServiceManagement

• ApplicationLogs•WebserverLogs• DBPerf Counters•Wiredata

• Perf Counters• AccessLogs• NetworkLogs


Let’sPlay!

SettingupServiceIntelligence

ServiceVisibilityinITSI

41

CLICK“GlassTables”


42

CLICK(openinnewtab)“ButtercupGamesBusinessProcess(INPROGRESS)”


43

CLICK(openinnewtab)“ButtercupGamesOnlineStore”

Goal1:SupplyChainVisibility

44

Goal2:OnlineStoreProcessFlow

45

NewRequirements!

46

● CreateanewKPIfortheDBService:● NetworkUtilization

● ModifytheExecutiveGlassTableinordertoshowofftheservicesyouslaveover

“WEonlyhaveabout15minTODOWHAT???!!???”

Thinkabouthowlongthiswouldtakeyoutoday?

47

ConfigurationofDBService

Click Configure >Click Services

Let’sTalkEntities

48

● Select DBService

● Entitiesaretherelevantthingswhichsupportthisservice(usuallyhosts)

● Selecttherightentrieswithfilters,ANDs,ORs● OriginalEntitylistcancomefromCMDB,

spreadsheet,Splunksearch,others

AKPIin5minutes?Absolutely!

49

ClickNew– GenericKPI

Select DataModel● HostOperatingSystem● Network● #bytes● Next

Callit“NetworkUtilization”,withyourusername upfront

KPIsContinued….

50

SplunkBuildsSearchesforyou–OhYeah,that’shappeningJ

● Select Yesfor Splitby& Filteroptions● Select hostfor EntityLookup& Aliasoptions● Click Next

AlmostThere…

51

Select● KPISearchSchedule:EveryMinute● EntityCalculation:Average● Service/AggCalculation:Average● CalculationWindow:LastMinute● Click Next

● Unit:Bps● Click Next

FinalSteps…

52

Setyourthresholds:● Aggregate(All)● PerEntity

● Click “AddThreshold”TWICE● MaketheNeapolitanicecreamcolors

Yellow,Green,Yellow● Dragtheslidersaroundinordertoget

thecurrentdatagraphentirelyinsidetheGreen(normal) band

● Click Finish● Otheroptionsarealsoavailable,

includingadaptivethresholdsandanomalydetection

AdaptiveThresholds

53

WhatifyourKPIdatalookslikethis?

54

AdaptiveThresholdsStaticthresholdswillnotwork…

55

AdaptiveThresholdsAdaptiveThresholdingworksbeautifullywithcyclical(andotherdynamic)data

AnomalyDetection

56

● MachineLearning

● Workswellfordatawithpatterns

● Requiressome“training”(trial&error)tozeroinonbestsensitivity

● Moresophisticatedcapabilitiescoming!(multivariate,morealgorithms,etc)

Let’sFixthatGlassTable

57

ClonetheGlassTable

58

ReturntoSavedGlassTablespage(click onGlassTablesintheuppermenubar)

CLICKEdit for“ButtercupGamesBusinessProcess(INPROGRESS)”• Select Clone• Title:Add yourusername

tothefront• Permissions:SharedinApp• Click ClonePage

• Click onyournewGlassTablefromthelist,toviewit

Edit&HaveFun!

59

ClickonEdit intheupperrightcornerofyourGlassTable

Usethe“Services”panelonthelefttoselectIndividualKPIs,or AggregateServiceHealthScores• Choose2KPIsfromOnlineStore thatwouldbeusefulin

the“OrderProcess”section• Dragtheselectedwidgetsontothecanvas,positioningin

thegrayoval

• What’sthedifferencebetweenthe

and toolsatthetopleft?

MoreFunwiththeGlassTableEditor…

60

UsetheConfigurations panelontherighttoeditaselectedwidget• Canchangethevisualizationtype,drilldown

behavior,andothersettings

• YoushouldhitSave frequently• RevertAllChangescanbehelpful,occasionally

Finishingup…

61

• AddaServiceHealthScore widgetforOnlineStoreunderButtercup

• ChooseaVizTypewithasparklinegraph,thenresizetomakeitlookpretty

• ModifytheCustomDrilldownactiontogotothesavedglasstable,ButtercupGamesOnlineStore

• BonusPoints:Makethelabelbigger,morereadable

• Click Save• View whendone


Let’sPlay!

ATroubleshootingExercise

ATroubleshootingExercise

63

Let’suseITSItotroubleshootanoutage● StartatyourGlassTable,“<UserName>ButtercupBusinessProcess”● CustomerCarereportsthatunhappycustomersarecomplainingoffailures

andlongdelayswhentryingtopurchase● Thecallsbegancominginataroundthetopofthelasthour.● IntheupperrightcorneroftheGlassTable,changethetimepickerfromNow

toXX:00:00.0,whereXXistheprevioushour.Forexample,ifitiscurrently14:05,setthetimepickerto13:00:00.0,thenApply

● Thisishowwecan“timetravel”backtoseeconditionsataparticularoutage– ohyeah!

ATroubleshootingExercise,cont’d

64

● TheOnlineStoreseemstobedegraded,justasCustomerCarereported.ClickonthewidgetunderButtercuptodrilldownfurther

ATroubleshootingExercise,cont’d.

65

● TheOnlineStoreGlassTableshowsamuchmoredetailedview,includingtheimpactedcustomer-facingKPIsatthefarleft(Revenue,etc)

● Basedonthisviewofalltherelevantservices,wheredoyouthinktherootcauselies?

● Whichserviceshouldwetroubleshootfirst?● ClickonHealthwidgetforthatservice,to

drilldowntoaDeepDive

DeepDive

66

● DeepDiveshowsmultipleKPIsandHealthScoresinparallel“swimlanes”.

● TheHealthScoreforthisServiceisthetopswimlane.Canyouseewhenitbeginstodegradefrom100%?

● Mousingoverthispointintime,canyouspottheKPIwiththeleadingfaultindication,i.e.,whatfailedfirst?

● Toimprovereadability,makesurethePrimaryTimeRange(lowerleftcorner)issettoPresets >Last60minutes

Multi-KPIAlertsandNotableEvents

67

● Click onNotableEventsReview● MultipleKPIsandHealthscorescan

becombinedinsophisticatedwaystocreateMulti-KPIalerts

● WhenaMulti-KPIalertfires,oneoftheoutcomesisthecreationofaNotableEvent

● NotableEventsallowNOCpersonnelandotherstotriageandcoordinateeventmanagementefforts

ServiceAnalyzer

68

● Click onServiceAnalyzer> DefaultServiceAnalyzer

● Backwherewestarted!● Thisviewshowsa“no-frills”listof

services(top)andhottestKPIs(bottom)

● ProvidesaccessintoServiceDetails● ItisusefulforNOCsandothers

whoneedahigh-levelsituationalview


Let’sPlay!

AdvancedExercises

Summary

70

● High-valueservicescanbedecomposedandmodeledinITSI,usingmachinedatafromtherelevantsystems

● Services andKPIs canbecreatedinminutes,withsophisticatedthresholdingtechniquestodistinguish“normal”from“notnormal”

● GlassTablesallowservicehealthandKPImetricstobedisplayedinawaythatmakessensetospecificgroups,suchasExecutiveLeadership,BusinessServiceOwners,theNOC,DevOps&Others

● DeepDivesallowKPIstobecomparedside-by-sideacrossanytimerange,acceleratingrootcauseanalysisandsignificantlyreducingMTTR

● Multi-KPIAlertsandNotableEventsreducealertnoise,producingactionableeventsandameanstomanagethem

● …andit’sfast+fun tobuild!

WhatourITSICustomersaredoing

Splunk ITServiceIntelligenceMachineLearning-Powered,Analytics-DrivenITOperations

Simplifyserviceoperations

Prioritizeincidentswithcontext RedefinetheroleofIT

Combineevents&metricsacrosssiloswithease,flexibility&scaleindays

Unifysiloed monitoringLeveragemachinelearningtodetectanomalies&highlight

eventsthatmatter

Deliverbusiness&servicecontexttoprioritizeincidentinvestigation&action

Supportdecisions&communicateresultswithpowerfulservice-levelinsights

Copyright©2016Splunk,Inc.

Splunk’sSolution:Alenscouldbemultipleprocesses…

AllthescoresaretimebasedKPI’sornestedsubprocessesthataresearchinginrealtimeforsomerelevantconditionofinterest.

TheseareHeathScores– ahighlevelaggregationofthehealthoftheunderlyingprocesses.

Allthescoresarecolorcodedtoconveyiftheyare“normal”or“abnormal”basedonyourcriteriaORSplunk’s PackagedMachineLearning,enabledwithanON/OFFswitch.

Thisshowshow‘GlassTables’canvisualizekeyperformanceindicatorsandhealthscoresthatcombinedatafromdiversesources.

Thisexampleisanabbreviated‘BooktoBill’,orsometimescalled‘OrdertoCash’businessprocess.

Call Center Service

Service Health Transactions

ACD Analysis – Core SplunkCall Wait History

Inbound Analysis

Social Media

Online Msg

Social Media

Mail SupportVOIP Service

Inbound Calls

Online Transactions Services

Internal Transfer Service

External Wire ServiceMoney Exchange Service

Money Transfer Services

Service Health Corporate

Reconciliation Service

Fed Exchange Service

Core Splunk SearchesTransaction History

System Investigation

Heat Map Analysis

CIO ScorecardEnterprise Service Status Major Incidents

Service Health

Continuous Operational Visibility

Volume Revenue Incidents Changes

Major Changes

Service Health Volume Revenue Incidents Changes

Service Health Volume Ontime DeliveryIncidents Changes Service Health VolumeRevenue Incidents Changes

Service Health Volume Revenue Incidents Changes Container UtilService Health Throughput Incidents Changes

TheVision- BusinessOperationsCenter

• SplunkITSIhasthefundamentalstodeliveronthepromiseofrealtimebusinessvisualizations• ModeledafteryourSecurity,Network,andITOperationsCenters• Monitoringanddiagnosisofimportantecommerceandbrickandmortaroperations

• Enhancedwithprocessinsightfromend-to-end,alerts,machinelearningandreal-timeresponse

NOC

SOC

BOC

Sign Up Now – We’re here to help!

Harness the creativity and domain knowledge of your organization to unlock the value of data and solve an

important Business Service problem through a joint service intelligence workshop with key stakeholders

Define methods for:› Proactive service monitoring› Reduced risk and failures› Faster issue resolution› Increased business performance

What is it? › 1 Day Onsite Workshop› Tightly linked with value› Collaborative approach› Build your own Glass

Table

Our Workshop In Action

Bringyoursubjectexpertstogether

ConductaServiceIntelligenceworkshop

YourMission,shouldyouchoosetoacceptit…

Findaproblemworthsolvinginyourenterprise

ReferenceStuff

81

● ITSIGuidebook:InyourITSIinstance:Search->Dashboards->ITSISandboxGuide

● ITSIDocumentation:http://docs.splunk.com/Documentation/ITSI

ThankYouPleasefillouttheSurveyhttps://www.surveymonkey.com/r/NBXBYCG