building business service intelligence with itsi
Post on 20-Jan-2017
136 Views
Preview:
TRANSCRIPT
Copyright©2016SplunkInc.
BuildingBusinessServiceIntelligencewith
SplunkITServiceIntelligence
DavidMillisStaffArchitect,ITOperaDonsAnalyDcs
Tuesday,October4,2016Calgary
WiFi:MarrioH_CONFERENCE/splunk
SetupBeforeYouCanPlay1.Downloadthispresenta9onslidedeck:h;ps://splunk.box.com/v/ITSI-HandsOn-Calgary2.Ifyouhavenotdonesoalready,SignupfortheFREESplunkITSIOnlineSandbox:• hHp://splunk.com/itsi• Select"FreeOnlineSandbox"
3.Pleasetestaccesstoyoursandbox;• Chrome,Firefox,Safari
arerecommended;• IEisNOTrecommended
4.AYerloggingin,selectITServiceIntelligencefromthelistofappsattheleY
2
3
Agenda• IntroducDonsandSetUp• WhatisServiceIntelligence?• Splundamentals--CoreSplunkinITOps• WhatisITServiceIntelligence(ITSI)?• Let'sPlay!• What'sNext?• HappyHour!
SafeHarborStatementDuring the course of this presentaDon, wemaymake forward looking statements regarding futureeventsortheexpectedperformanceofthecompany.WecauDonyouthatsuchstatementsreflectourcurrentexpectaDonsandesDmatesbasedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.Forimportantfactorsthatmaycauseactualresultstodifferfromthosecontainedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-lookingstatementsmadeinthispresentaDonarebeingmadeasoftheDmeanddateofitslivepresentaDon.IfreviewedaYeritslivepresentaDon,thispresentaDonmaynotcontaincurrentoraccurateinformaDon.WedonotassumeanyobligaDontoupdateanyforwardlookingstatementswemaymake.InaddiDon,anyinformaDonaboutourroadmapoutlinesourgeneralproductdirecDonandissubjecttochangeatanyDmewithoutnoDce. It is for informaDonalpurposesonlyandshallnotbe incorporated intoanycontract or other commitment. Splunk undertakes no obligaDon either to develop the features orfuncDonalitydescribedortoincludeanysuchfeatureorfuncDonalityinafuturerelease.
4
WhatisServiceIntelligence?
Enablingabusiness-awareITMeasuringandrepor<ngonindicatorsthatma?er
Unlockingopera9onalefficienciesCollabora<ngacrosssilostoimproveserviceopera<ons
Usingdata-drivendecisionmakingSolvingproblemsandan<cipa<ngpiBallswithsophis<catedanaly<csandpowerfulinsightsfrommachinedata
ServiceModelingThere’saBeHerWay…
Key Takeaways
1 TradiDonalMethodsareoutdated
ServiceMappingisCriDcal
3 Data-DrivenServiceInsights
Challenging Traditional Methods
Network
InfrastructureLayer
HPNNMi,HPNA,SolarWinds,CASpectrum,
StorageHPStorageOperaDons,NetApp,EMC
ServerHPOV/Sitescope,SCOM,Nagios,Tivoli,BMCPatrol,CAUIM
74%
-36%
ApplicaD
onLayer
SyntheDcAPMAppD,NewRelic,Dynatrace,HPAPM,CA,IBM,Appica
ByteCodeInstrumentaDon
AppD,NewRelic,Dynatrace,HPDiag,CAWily
AdapDveThresholding
HPSHA,BMCProacDveNet,NetuiDve,Preelert
HPRun-TimeServiceModelCAServiceOperaDonsInsight
IBMNetCool/OmnibusServiceModeldefiniDon&CorrelaDonEngine
Business Layer
Aggregation/Correlation/Visualization
Service Layer
Challenges• Toomanydisparatecomponents• DifficulttodefineServiceModel• Laborintensive• MostimplementaDonsfail• Veryimportantsourceis
missing!(machinedata)
Data Approach With Splunk>
Network
InfrastructureLayer
Packet,Payload,Traffic,UDlizaDon,Perf
StorageUDlizaDon,Capacity,Performance
ServerPerformance,Usage,Dependency
74%
-36%
ApplicaD
onLayer
SyntheDcAPMAvailability,Capacity,UserExperience
ByteCodeInstrumentaDonUsage,Experience,Performance,Quality
AdapDveThresholdingApps,Services,Systems
Splunk>isthemissinglink• DataFidelity• SingleRepositoryforALLdata• EasiertoManageServices• ReducedIntegraDons• ReducedPointSoluDons• CollaboraDveApproach• QuickDmetovalue
MACHINEDATA
Data Fabric Platform
Service Intelligence
BringSubjectExpertsTogether
DesignBeforeConfiguring
BestPracDcesforServiceModeling
StartWithaProblemWorth
Solving
BestPracDcesforServiceModeling
StartWithaProblemWorth
Solving
• Whatarethetopbusinessservicesinyourenterprise?
• Howdoyoumeasurethecustomerexperiencewiththeseservices?
• Whatisthecustomerexperiencewiththeseservices?
CriDcalServices
BestPracDcesforServiceModeling
StartWithaProblemWorth
Solving
• HowoYendocustomersexperienceissueswiththeservice?
• Whenissuesarise,whogetsinvolvedinresolvingthem?
• Howdoteamsworktogethertoresolveissues?
IssueFrequency
BestPracDcesforServiceModeling
StartWithaProblemWorth
Solving
• What’stheaverageDmetoissueresoluDon?
• What’stheimpactwhencustomershaveabadexperiencewithyourservices?
Impact
BestPracDcesforServiceModeling
BringSubjectExpertsTogether
• Business&ServiceOwners,meetIT• IT,meetBusiness&ServiceOwners
BestPracDcesforServiceModeling
DesignBeforeConfiguring
• DefineYourServices–TechnicalANDBusiness• DefineYourKPI’sforeachservice• Understandservicedependencies&interacDons
COLLECTDATAFROMANYWHERE
SEARCHANDANALYZE
EVERYTHING
DELIVERREAL-TIMEOPERATIONALINTELLIGENCETOITANDTHEBUSINESS
THEPOWEROF
Splundamentals--CoreSplunkinITOps
Data-drivenserviceinsightsforroot-causeisolaDonandimprovedserviceoperaDons
withamarkeDngcatchphrasethatistoolong
INTRODUCING
SplunkITServiceIntelligence
SPLUNKITSERVICEINTELLIGENCE
Time-SeriesIndex
PlarormforMachineData
DynamicServiceModels
Schema-on-Read DataModel CommonInforma9onModel
At-a-GlanceProblemAnalysis
EarlyWarningonDeviaDons
SimplifiedIncidentWorkflows
SplunkITServiceIntelligence
ITSICoreConcepts
21
WhatisaService?
ServiceRequestsResponses
InITSI,aServiceisalogicalgroupoftechnologycomponentsthatauserdeemsneedtobemonitoredtogether.
ItcanoYenbegeneralizedasa“blackbox”whichwesendrequests,andexpectresponses
22
WhatisaService?
DNS RequestsResponses
TechnicalServices
AuthRequestsResponses
WebRequestsResponses
Servicescanbelowerlevel(technical)…
23
WhatisaService?
DNS RequestsResponses
TechnicalServices
CustomerTransac9ons
RequestsResponses
BusinessServices
AuthRequestsResponses
WebRequestsResponses
SupportDesk RequestsResponses
Servicescanalsobehigherlevel(business)…
24
WhatisaService?
PacketNetwork
HypervisorandHosts
RBMDBs
StorageTier
APIServices
WebServices
CustomerTransac9ons
MobileAPI/
Middlew
are
PartnerPortal
DNS
ServicescanencompassmulDpleDersoftheITdomain.Servicesmayalsodependuponotherservices
25
WhatisaKPI?
DNS RequestsResponses
KPI:NumberofrequestsKPI:ErrorrateKPI:AverageresponseDmeKPI:ServerCPUloadKPI:ServernetworkI/Ferrors
CustomerTransac9ons
RequestsResponses
KPI:NumberoftransacDonsKPI:ErrorrateKPI:AverageresponseDmeKPI:CountofIncidentTicketsKPI:SyntheDcTransxHealth
KPIsandHealthscoresconsDtutethemeansbywhichServicesaremonitored.
26
KeyPerformanceIndicators(KPIs)
27
AKeyPerformanceIndicator(KPI)isaSplunksavedsearchcreatedwithintheITSIUIthathelpsmonitoraspecificfieldlikeCPU,Memory,NumberofErrors
andsoon.KPIsarecontainedwithinServices.
ServiceHealthScores
28
AHealthscoreisascoreform0-100(0beingcriDcaland100beingnormal)thathelpsdeterminethehealthofaService.ItiscalculatedbasedonallKPIs
importanceanditsstatus(e.g.green,orange,red),onceeveryminute.
ITSITour
29
ServiceDecomposiDoninITSI
30
1-Whatisahigh-valuebusinessservice?(“OnlineStore”inBuHercupGames)
ServiceDecomposiDoninITSI
31
1-Whatisahigh-valuebusinessservice?(OnlineStore)
2-Processflow,andunderlyingsub-services?(Web->Middleware->DB->Middleware->Web)
ServiceDecomposiDoninITSI
32
1-Whatisahigh-valuebusinessservice?(OnlineStore)
2-Processflow,andunderlyingsub-services?(Web->Middleware…)
3-Foreach(sub)service:KPIstoshowhealth&status?(Database:errors,SQLhits,responseDme,…)
ServiceDecomposiDoninITSI
33
1-Whatisahigh-valuebusinessservice?(OnlineStore)
2-Processflow&underlyingsub-services?(Web->Middleware…)
3-Foreach(sub)service:KPIs?(Database:errors,SQLhits,…)
4-ForeachKPI:NeedaSplunksearch(index=DB(warn*ORerror*)|statscount)
ServiceDecomposiDoninITSI
34
1-Whatisahigh-valuebusinessservice?(OnlineStore)
2-Processflow&underlyingsub-services?(Web->Middleware…)
3-Foreach(sub)service:KPIs?(Database:errors,SQLhits,…)
4-ForeachKPI:NeedaSplunksearch(index=DB(warn*ORerror*)|statscount)
Let’sPlay!
ServiceDecomposiDoninITSI
36
CLICK“GlassTables”
ServiceDecomposiDoninITSI
37
CLICK(openinnewtab)“Bu?ercupGamesBusinessProcess(INPROGRESS)”
ServiceDecomposiDoninITSI
38
CLICK(openinnewtab)“Bu?ercupGamesOnlineStore”
ServiceDecomp:TheBusinessProcesses
39
ServiceDecomp:End-To-EndProcessFlow
40
NewRequirements!
41
● CreateanewKPIfortheDBService:● NetworkUDlizaDon
● ModifytheExecuDveGlassTableinordertoshowofftheservicesyouslaveover
“WEonlyhaveabout15minTODOWHAT???!!???”Thinkabouthowlongthiswouldtakeyoutoday?
42
ConfiguraDonofDBService
ClickConfigure>ClickServices
Let’sTalkEnDDes
43
● SelectDBService
● EnDDesaretherelevantthingswhichsupportthisservice(usuallyhosts)
● Selecttherightentrieswithfilters,ANDs,ORs● OriginalEnDtylistcancomefromCMDB,
spreadsheet,Splunksearch,others
AKPIin5minutes?Absolutely!
44
ClickNew–GenericKPI
SelectDataModel● HostOpera<ngSystem● Network● #bytes● Next
Callit“NetworkU>liza>on”,withyourusernameupfront
KPIsConDnued….
45
SplunkBuildsSearchesforyou–OhYeah,that’shappeningJ
● SelectYesforSplitby&Filterop<ons● SelecthostforEn>tyLookup&Aliasop<ons● ClickNext
AlmostThere…
46
Select● KPISearchSchedule:EveryMinute● En>tyCalcula>on:Average● Service/AggCalcula>on:Average● Calcula>onWindow:LastMinute● ClickNext
● Unit:Bps● ClickNext
FinalSteps…
47
Setyourthresholds:● Aggregate(All)● PerEn>ty
● Click“AddThreshold”TWICE● MaketheNeapolitanicecreamcolors
Yellow,Green,Yellow● Dragtheslidersaroundinordertoget
thecurrentdatagraphenDrelyinsidetheGreen(normal)band
● ClickFinish● OtheropDonsarealsoavailable,
includingadapDvethresholdsandanomalydetecDon
AdapDveThresholds
48
WhatifyourKPIdatalookslikethis?
49
AdapDveThresholdsStaDcthresholdswillnotwork…
50
AdapDveThresholdsAdapDveThresholdingworksbeauDfullywithcyclical(andotherdynamic)data
AnomalyDetecDon
51
● MachineLearning
● WorkswellfordatawithpaHerns
● Requiressome“training”(trial&error)tozeroinonbestsensiDvity
● MoresophisDcatedcapabiliDescoming!(mulDvariate,morealgorithms,etc)
Let’sFixthatGlassTable
52
ClonetheGlassTable
53
ReturntoSavedGlassTablespage(clickonGlassTablesintheuppermenubar)CLICKEditfor“BuHercupGamesBusinessProcess(INPROGRESS)”• SelectClone• Title:Addyourusername
tothefront• Permissions:SharedinApp• ClickClonePage• ClickonyournewGlassTable
fromthelist,toviewit
Edit&HaveFun!
54
ClickonEditintheupperrightcornerofyourGlassTableUsethe“Services”panelontheleYtoselectIndividualKPIs,orAggregateServiceHealthScores• Choose2KPIsfromOnlineStorethatwouldbeusefulin
the“OrderProcess”secDon• Dragtheselectedwidgetsontothecanvas,posiDoningin
thegrayoval• What’sthedifferencebetweenthe
andtoolsatthetopleY?
MoreFunwiththeGlassTableEditor…
55
UsetheConfigura>onspanelontherighttoeditaselectedwidget• CanchangethevisualizaDontype,drilldown
behavior,andotherse}ngs• YoushouldhitSavefrequently• RevertAllChangescanbehelpful,occasionally
Finishingup…
56
• AddaServiceHealthScorewidgetforOnlineStoreunderBuHercup
• ChooseaVizTypewithasparklinegraph,thenresizetomakeitlookpreHy
• ModifytheCustomDrilldownacDontogotothesavedglasstable,BuVercupGamesOnlineStore
• BonusPoints:Makethelabelbigger,morereadable
• ClickSave• Viewwhendone
ATroubleshooDngExercise
57
Let’suseITSItotroubleshootanoutage● StartatyourGlassTable,“<UserName>BuHercupBusinessProcess”● CustomerCarereportsthatunhappycustomersarecomplainingoffailures
andlongdelayswhentryingtopurchase● Thecallsbegancominginataroundthetopofthelasthour.● IntheupperrightcorneroftheGlassTable,changetheDmepickerfromNow
toXX:00:00.0,whereXXistheprevioushour.Forexample,ifitiscurrently14:05,settheDmepickerto13:00:00.0,thenApply
● Thisishowwecan“Dmetravel”backtoseecondiDonsataparDcularoutage–ohyeah!
ATroubleshooDngExercise,cont’d
58
● TheOnlineStoreseemstobedegraded,justasCustomerCarereported.ClickonthewidgetunderBuHercuptodrilldownfurther
ATroubleshooDngExercise,cont’d.
59
● TheOnlineStoreGlassTableshowsamuchmoredetailedview,includingtheimpactedcustomer-facingKPIsatthefarleY(Revenue,etc)
● Basedonthisviewofalltherelevantservices,wheredoyouthinktherootcauselies?
● Whichserviceshouldwetroubleshootfirst?● ClickonHealthwidgetforthatservice,to
drilldowntoaDeepDive
DeepDive
60
● DeepDiveshowsmulDpleKPIsandHealthScoresinparallel“swimlanes”.
● TheHealthScoreforthisServiceisthetopswimlane.Canyouseewhenitbeginstodegradefrom100%?
● MousingoverthispointinDme,canyouspottheKPIwiththeleadingfaultindicaDon,i.e.,whatfailedfirst?
● Toimprovereadability,makesurethePrimaryTimeRange(lowerleYcorner)issettoPresets>Last60minutes
MulD-KPIAlertsandNotableEvents
61
● ClickonNotableEventsReview● MulDpleKPIsandHealthscorescan
becombinedinsophisDcatedwaystocreateMulD-KPIalerts
● WhenaMulD-KPIalertfires,oneoftheoutcomesisthecreaDonofaNotableEvent
● NotableEventsallowNOCpersonnelandotherstotriageandcoordinateeventmanagementefforts
ServiceAnalyzer
62
● ClickonServiceAnalyzer>DefaultServiceAnalyzer
● Backwherewestarted!● Thisviewshowsa“no-frills”listof
services(top)andhoHestKPIs(boHom)
● ProvidesaccessintoServiceDetails● ItisusefulforNOCsandothers
whoneedahigh-levelsituaDonalview
AdvancedExercises
63
Summary
64
● High-valueservicescanbedecomposedandmodeledinITSI,usingmachinedatafromtherelevantsystems
● ServicesandKPIscanbecreatedinminutes,withsophisDcatedthresholdingtechniquestodisDnguish“normal”from“notnormal”
● GlassTablesallowservicehealthandKPImetricstobedisplayedinawaythatmakessensetospecificgroups,suchasExecuDveLeadership,BusinessServiceOwners,theNOC,DevOps&Others
● DeepDivesallowKPIstobecomparedside-by-sideacrossanyDmerange,acceleraDngrootcauseanalysisandsignificantlyreducingMTTR
● Mul9-KPIAlertsandNotableEventsreducealertnoise,producingacDonableeventsandameanstomanagethem
● …andit’sfuntobuild!
Possibili9es
Order Analysis – Core Splunk Record Systems
Payment Processing
Patient Queue Analysis
Patient Processing Service Health Transactions
Co-Pay
Insurance
MHR
Check-In Procedures
Pharmacy
Assessment
Call Center Service
Service Health Transactions
ACD Analysis – Core Splunk Call Wait History
Inbound Analysis
Social Media
Online Msg
Social Media
Mail Support VOIP Service
Inbound Calls
Banking
68
69
HealthcareOperations
70
End User Experience for Streaming Video
Whatever This Is
71
Hospital Emergency Rooms
72
Sign Up Now – We’re here to help!
Harness the creativity and domain knowledge of your organization to unlock the value of data and solve an important Business Service problem through a joint service intelligence
workshop with key stakeholders
Define methods for: › Proactive service monitoring › Reduced risk and failures › Faster issue resolution › Increased business performance
What is it? › 1 Day Onsite Workshop › Tightly linked with value › Collaborative approach › Build your own Glass Table
ReferenceStuff
74
● ITSIGuidebook:InyourITSIinstance:Search->Dashboards->ITSISandboxGuide
● ITSIDocumentaDon:hHp://docs.splunk.com/DocumentaDon/ITSI
ThankYou
DavidMillisdmillis@splunk.com
top related