dhruva kumar bhagi sr. software engineer · 2017-10-08 · sr. software engineer splunk inc....
TRANSCRIPT
Copyright©2016SplunkInc.
DhruvaKumarBhagiSr.SoftwareengineerSplunkInc.
Replicationofsummarydatainindexercluster
Disclaimer
2
Duringthecourseofthispresentation,wemaymakeforwardlookingstatementsregardingfutureeventsortheexpectedperformanceofthecompany.Wecautionyouthatsuchstatementsreflectourcurrentexpectationsandestimatesbasedonfactorscurrentlyknowntousandthatactualeventsorresultscoulddiffermaterially.Forimportantfactorsthatmaycauseactualresultstodifferfromthose
containedinourforward-lookingstatements,pleasereviewourfilingswiththeSEC.Theforward-lookingstatementsmadeinthethispresentationarebeingmadeasofthetimeanddateofitslivepresentation.Ifreviewedafteritslivepresentation,thispresentationmaynotcontaincurrentoraccurateinformation.Wedonotassumeanyobligationtoupdateanyforwardlookingstatementswemaymake.Inaddition,anyinformationaboutourroadmapoutlinesourgeneralproductdirectionandissubjecttochangeatanytimewithoutnotice.Itisforinformationalpurposesonlyandshallnot,beincorporatedintoanycontractorothercommitment.Splunkundertakesnoobligationeithertodevelopthefeaturesor
functionalitydescribedortoincludeanysuchfeatureorfunctionalityinafuturerelease.
Indexerclustertopology
3
MasterSearchhead
FORWARDERS
Indexer Indexer IndexerForReplication
Forgenerationinfo
ForSearch
Forwardingdatatoindexers
Master-slavecommunication
Forreplication
Searchhead-idx
Summaries
4
Summaries arehigherlevelrepresentationofthesearchresultsResideonindexertohelpacceleratetheexecutionoflongrunningsearchesoriginatingfromsearchheadsApplicableforstatisticalsearcheslike‘index=main|statscount’
Summaries
5
Twotypesofsummaries:• Reportaccelerationsummaries– Generatedbyacceleratedreports/searches• Datamodelsummaries– Generatedbyaccelerateddatamodelpivotsearches
ResideunderINDEX/{datamodel_summary|summary}/BID/SH_ID/summary_name/…Note:summariesaredifferentfromsummaryindex
Problem
6
SummarieswerepreviouslyneverreplicatedIfanindexerwithasummarygoesdown,bucketprimariesmovetoanothersearchablecopy,andsearcheswillnothaveaccesstothesummaries(untiltheygetregenerated),therebysearchesrunslow.Regeneratingsummariescantakelotoftime
Replicatingsummaries
7
Fromversion6.4,SplunkcanreplicateDA&RMsummariesinindexerclusterenvironmentPeersmaintainthelistoftheirsummaries(justlikebuckets)andreportsthemtoCMduringregistration(or)onanystatechangeIfanypeerismissingasummaryforabucket,masterschedulesasummaryreplicationwiththispeerasareplicationtargetOnlyreplicatesummariesforwarm/coldbuckets(nothot)services/cluster/master/buckets/BUKCETlistssummariesofeachbucket
Summariesforhotbuckets
8
Forhotbuckets,summarysearchesfromsearchheadshitallthesearchablecopies(insteadofjustprimariesfornormalsearches)therebylettingindexersbuildthesummariesindividuallyforhotbuckets.Noneedforsummaryregenerationevenonindividualnodefailures.Runningsummarygeneratingsearchesonallsearchablecopiesofahotbucketintroducesmoreresourceusage(longersearchprocess,extramemoryusage)!
VolumeRetention
9
Summariescanbesetupinavolumetomanagespaceusage
Ifindexerexceedsthevolumespace(maxVolumeDataSizeMB),splunkstartstrimmingsummaries– Atrimmedsummaryleavestombstoneindicatingthetrimmedstate– Tombstoneavoidsregeneratingsummariesinthesubsequentsearchruns– Oncewetrimasummary,wepropagatethisinfototheCM,andtheCMwill
issueatrimtoallothercopiesofthesummary.
Configuration
10
Toturnonsummaryreplication,makesummary_replication=trueunderclusteringstanzaonclustermaster.Bydefaultsummaryreplicationisturnedoff.max_peer_sum_rep_load (defaultsto5)configureshowmanymaximumsummaryreplicationsperpeerConfigchangesarereloadable(i.e.doesnotrequireasplunkrestart)
THANKYOU
Copyright©2016SplunkInc.
````
Wednesday,Sep283:00PM- 3:15PM