open location platform - logs, monitoring, and alerts user ... · your processes. olp provides a...

26
Open Location Platform - Logs, Monitoring, and Alerts User Guide Version 1.5.2

Upload: others

Post on 14-Jul-2020

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

OpenLocationPlatform-Logs,Monitoring,andAlertsUserGuide

Version1.5.2

Page 2: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

LegalNotices©2018HEREGlobalB.V.anditsAffiliate(s).Allrightsreserved.

Thismaterial,includingdocumentationandanyrelatedcomputerprograms,isprotectedbycopyrightcontrolledbyHERE.Allrightsarereserved.Copying,includingreproducing,storing,adaptingortranslating,anyorallofthismaterialrequiresthepriorwrittenconsentofHERE.Thismaterialalsocontainsconfidentialinformation,whichmaynotbedisclosedtootherswithoutthepriorwrittenconsentofHERE.

TrademarkAcknowledgements

HEREistrademarkorregisteredtrademarkofHEREGlobalB.V.Otherproductandcompanynamesmentionedhereinmaybetrademarksortradenamesoftheirrespectiveowners.

Disclaimer

Thiscontentisprovided"as-is"andwithoutwarrantiesofanykind,eitherexpressorimplied,including,butnotlimitedto,theimpliedwarrantiesofmerchantability,fitnessforaparticularpurpose,satisfactoryqualityandnon-infringement.HEREdoesnotwarrantthatthecontentiserrorfreeandHEREdoesnotwarrantormakeanyrepresentationsregardingthequality,correctness,accuracy,orreliabilityofthecontent.Youshouldthereforeverifyanyinformationcontainedinthecontentbeforeactingonit.

Tothefurthestextentpermittedbylaw,undernocircumstances,includingwithoutlimitationthenegligenceofHERE,shallHEREbeliableforanydamages,including,withoutlimitation,direct,special,indirect,punitive,consequential,exemplaryand/orincidentaldamagesthatresultfromtheuseorapplicationofthiscontent,evenifHEREoranauthorizedrepresentativehasbeenadvisedofthepossibilityofsuchdamages.

1

Page 3: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

DocumentInformation

Product

Name: OpenLocationPlatform-Logs,Monitoring,andAlerts

Version: Version1.5.2

Document

Name: OpenLocationPlatform-Logs,Monitoring,andAlertsUser’sGuide

ID: 548cbb8-1536627726-41db9917

Status: FINAL

Date: 2018-09-11T01:02:33.317Z

2

Page 4: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

TableofContents

Introduction

HowTo

UsePipelineLogging

ChangeLoggingLevel

RetrieveLoggingLevel

FindPipelineLogs

SearchApplicationLogs

MonitorYourMetrics

Data,Catalog,andLayerMetrics

IngestionMetrics

FlinkMetrics

SparkMetrics

PipelineMetrics

CurrentUsageMetrics

ZeppelinNotebookMetrics

CreateReports

CreateandManageAlerts

DuplicateaDashboard

NeedHelp?

3

Page 5: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

Introduction

WhyUseLogs,Metrics,andAlerts

YourusageofOLPgeneratesoperationallogsandmetricsthatyoucanusetomakesenseofwhat'shappeningwithyourprocesses.OLPprovidesamanagedinstanceoftheopensourceGrafanatoolformanagingyourmetricsdashboardandalerts.Inaddition,youcanusethemanagedinstanceoftheSplunkloganalyticsenvironmenttosearchyourapplciationlogsandsavetheresultsasreports.Dashboards,alerts,andlogsarevisibletoallusersinyouraccount.

Introduction

4

Page 6: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

UsePipelineLogging

PipelineLoggingBasics

OLPPipelinesuseloggingtoprovidemoredetailsduringtheiroperation.Differentlevelsofloggingareavailablefordifferentpurposes.OLPpipelinessupportthefollowinglevelsoflogging:

Debug—Includesfine-grainedinformationaleventsthataremostusefultotroubleshootapipeline.Info—Includesinformationalmessagesthathighlighttheprogressofthepipelineatacoarse-grainedlevel.Warn—Includesinformationonpotentiallyharmfulsituations;includingotherrun-timesituationsthatareundesirableorunexpected,butnotnecessarily"wrong".Error—Includesotherrun-timeerrorsorunexpectedconditionssuchaserroreventsthatmightstillallowthepipelinetocontinuerunning.

Note

Pipelineshaveadefaultlogginglevelofwarn.

Tools

EventLoggingishandledbyanembeddedversionofSplunk.Ingeneral,youwillnotneedtobeconcernedbecauseSplunkrunsinthebackground.However,youcancreateanewSplunkdashboardorrunreportsonthePortal.

Seealso

PipelineAPIReferencePipelineLoggingviaCLIGrafanaUserDocumentationSplunkEnterprise6.5.2UserDocumentation

UsePipelineLogging

5

Page 7: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

ChangetheLoggingLevel

TheLoggingLevelissetforaPipelineVersion.AlltheJobsusethelogginglevelassociatedwiththecorrespondingPipelineVersion.

TheLoggingLevelcanbesetattherootlevelfortheentirepipelineand/orattheindividualloggerlevelforapipelineclass.Duetooperationallatency,ittakesafewminutesforthechangestotakeeffect.ThismaydelaytheavailabilityofthelogsatthenewlevelinSplunk.

Note

ThedisplayedPipelineVersiondetailsincludeeitherthelastlogginglevelmodifiedbytheuserorthedefaultlogginglevel(i.e.warn).

ThelogginglevelforaPipelineVersioncanbechangedonlywhenitisinoneofthefollowingstates:

Running—ForaPipelineVersioninaRunningstate,whenthelogging-levelischanged,thesystemwillchangethelogging-levelofthecurrentlyrunningjob.ReadyorScheduled—ForaPipelineVersioninReadyorScheduledstate,whenthelogging-levelischanged,thesystemwillrunthefuturejobsusingthenewlogging-level.Paused—ForaPipelineVersioninthePausedstate,whenthelogging-levelischangedandPipelineVersionisresumed,thesystemwillrunthefuturejobusingthenewlogging-level.

APISolution

TochangeaPipelineVersion'slog-leveltoDEBUGattheroot-level,usethefollowingcommandfromtheRESTAPI:

PUT/v2/pipelines/{pipelineId}/versions/{versionId}/logging-configuration

withthefollowingbody:

{"configuration":{"loggers":{"root":{"level":"debug"}}}}

Formoreinformation,seethePipelineAPIReference.

CLISolution

TochangeaPipelineVersion'slog-leveltoDEBUGattheroot-level,usethefollowingcommandfromtheOLPCLI:

olppipelineversionlogset-level<pipeline-id><pipeline-version-id>[commandparameters]

ChangeLoggingLevel

6

Page 8: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

Example

TochangethelogginglevelofapipelineversiontoDEBUGattheroot-level:

olppipelineversionlogset-levelf2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96d77f288e-2c89-4c94-b4ba-79fbd1e26e79--rootDEBUG--json

Onsuccess,thecommandreturnssomethinglikethefollowing.

{"configuration":{"loggers":{"root":{"level":"debug"}}}}

Inthiscasethepipelineserviceconfirmsthatthelog-levelhasbeensettodebug.

ChangeLoggingLevel

7

Page 9: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

RetrievetheLoggingLevel

Tocheckthecurrentlog-levelconfiguration,youneedtoretrievethelog-levelsettingforaspecificPipelineVersion.

APISolution

ToretrieveCurrentLoggingLevelviaAPI,usethefollowingcommandfromtheRESTAPI:

GET/v2/pipelines/{pipelineId}/versions/{versionId}/logging-configuration

Formoreinformation,seethePipelineAPIReference.

CLISolution

TheretrieveCurrentLoggingLevelusingtheOLPCLI,usethefollowingcommand:

olppipelineversionlogget-level<pipeline-id><pipeline-version-id>[commandparameters]

Example

Toretrievethecurrentlogginglevel.

olppipelineversionlogget-levelf2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96d77f288e-2c89-4c94-b4ba-79fbd1e26e79--json

Onsuccess,thiscommandreturnssomethinglikethefollowing:

{"configuration":{"loggers":{"logger":[{"level":"info","name":"com.example.someLoggingClass"},{"level":"info","name":"com.example.examplepkg.otherClass"}],"root":{"level":"error"}}}}

RetrieveLoggingLevel

8

Page 10: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

FindPipelineLogs

EachrunningPipelineVersionhasauniqueURLwherethelogsforthatpipelinearestored.WhenusingtheCLI,thisURLinformationissuppliedbythepipelineservicewheneverapipelineversionisactivated,upgraded,orwhenastatusrequestissubmittedtotheservice.ButifyouarerunningyourpipelinebyinterfacingdirectlywiththePipelineService'sRESTAPI,youwillneedtomakeastatusrequesttodiscovertheloggingURL.

Example

TheCLIshowfunctionisonewayyoucancheckstatus.Inthecaseofaspecificjob,youcanusethefollowingCLIcommand.

olppipelineversionjobshowf2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96d77f288e-2c89-4c94-b4ba-79fbd1e26e79e8e31070-4f4b-4e4b-a8e0-40425844cb75

Successfuloutputreturns:

{"catalogVersions":null,"created":"2018-03-01T15:23:04.618Z","id":"e8e31070-4f4b-4e4b-a8e0-40425844cb75","state":"failed","updated":"2018-03-01T15:23:32.497Z","loggingUrl":null}

Inthiscase,theloggingUrlparameterhasanullvalue.Thisiswhatyouwouldexpecttofindforajobthatisnotyetrunning.Oncethejobruns,theloggingURLwouldbeshown.

Note

UntilascheduledPipelineruns,theloggingURLwillremainnull.

FindPipelineLogs

9

Page 11: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

SearchApplicationLogs

ApplicationlogscanbeaccessedthroughSplunk.Tostartanewsearch,clickTools>Logs(PoweredbySplunk).ThenfromtheSplunkHomepageinterface,clickSearch.

FormoreinformationonusingSplunk,seethetutorialsintheofficialdocumentation.

PipelineLogIndex

Pipelinelogsarestoredinthe<realm>_commonindex.Forexample,ifyouraccountisintheolp-examplerealm,yourindexwouldbeolp-example_common.YoucansearchforthisbyaddingthefollowingstringtoyourSplunksearchquery:

index="olp-example_common"

TroubleshootPipelineIssues

Youcanuseapplicationlogstodebugandtroubleshootpipelineissues.

Duringthepipelinesetup(beforeitissubmittedtoberun),thepipelineservicedoesnotproduceanylogs.AnyfeedbackisprovidedonlythroughthePipelineserviceAPIs.

Onceapipelinejobissubmitted,theerrorscanhappenintwodifferentscnearios:beforeandafterthepipelinehasstartedrunning.

BeforethePipelineStartsRunning

Thereareanumberofstepsthatthepipelinegoesthroughbeforeitactuallystartsrunning.Anyerrorsatthispointarenotattachedtoajob.TheseerrorsarefirstcapturedinternallybyOLP,andthenpushedouttoyourappropriateSplunkindex.OnlyerrorlogsrelatedtoyourpipelinecanappearinyourspecificSplunkindex.

TosearchyourSplunkindexfortheselogs,usethepipelineorpipelineversionUUID.

OncethePipelineStartsRunning

Oncethepipelinestartsrunning,thereisacorrespondingjobreportedinPipelineVersionDetailsontheOLPwebportalandPipelinesAPIresponses.Eachpipelinejobisassignedalinktothelogsforthatjobrun.

Note:TimeWindowExpiration

Thedefaulttimewindowbuiltintothelinkexpiresaftersometime.Forexample,thedefaultlinkmaybepullinglogsfromthelast15minutes.Thislinkdoesnotworkonthenextday.

Tosearchfortheselogsagain,selectatimewindowusingSplunk.Youcandothisthroughadrop-downlistoptionatthetoprightoftheSplunkUI.

SearchApplicationLogs

10

Page 12: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

SearchApplicationLogs

11

Page 13: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

MonitorYourMetrics

OLPprovidesamanagedinstanceoftheopensourceGrafanatoolformanagingyourmetricsdashboardsandalerts.Toaccessthesedashboards,clickontheMonitoringandAlertslinkundertheToolsmenu.ThenintheGrafanatool,clickontheHomebuttonontoplefttoviewthemetricdashboardsofferedbyOLP.

TimeRangesandIntervals

Youcanselectthedashboardtimerangebyusingthedropdownmenuintheupper-rightcornerofaGrafanadashboard.Thiscontrolstherangeoftimeoverwhichyourdashboardwidgetsandgraphsqueryfortheirdata.

Figure:Grafanamenutoselectatimerange

Whenanewtimerangeisselected,Grafanaautomaticallycalculatestheoptimaldisplaytimeinterval(suchas1dayor8hours)foryou.Thisisbecausewhenthereismoredatathancanbeshown,Grafanacanmakeamoreefficientdisplaybygroupingdatapointsinlargerintervals.

Thisauto-calculatedintervalmaynotbedesiredforalldashboards.Fordashboardsthatarelabeled"w/CustomTimeInterval",youcanusetheCustomTimeIntervaldropdownmenuintheupper-leftcornertoselectyourown.

Figure:Grafanamenutoselectatimeinterval

MonitorYourMetrics

12

Page 14: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

Data,Catalog,andLayerMetrics

DataMetrics

METRIC DESCRIPTION

TotalNumberOfCatalogs SumOfCurrentExistingCatalogs

TotalNumberofLayers SumOfCurrentExistingLayers(AllLayerTypes)AcrossallCatalogs

TotalNumberofVersionedLayers SumOfCurrentVersionedLayersAcrossallCatalogs

TotalNumberofVolatileLayers SumOfCurrentVolatileLayersAcrossallCatalogs

TotalNumberofStreamLayers SumOfCurrentVolatileLayersAcrossallCatalogs

VersionedLayers-TotalStorageVolume(Usage)

SumOfDataVolumeAcrossallVersionedLayers(allCatalogs)

VolatileLayers-TotalCapacity SumOfReservedCapacityAcrossAllVolatileLayers(allCatalogs)

VolatileLayers-UsedCapacity SumOfUsedCapacityAcrossAllVolatileLayers(allCatalogs)

StreamLayers-TotalThroughputCapacity SumOfThroughputCapacityAcrossallStreamLayers(allCatalogs)

CatalogMetrics

METRIC DESCRIPTION

LayerCountPerCatalog TotalLayerCountPerCatalog

VersionedLayerStorageVolumePerCatalog TotalVolume(inGB)OfallVersionedLayersinthisCatalog

VolatileLayerUniqueCapacityPerCatalog

TotalUniqueCapacity(AmountofStoredDataWithoutReplication)OfallVolatileLayersinthisCatalog

AttemptedCommitsPerCatalog(VersionedStorageOnly)

Anychangetoaversionedcatalogwhichtriggersaversionincrement.Attemptedmeansatleaststarted.

CommitMetrics

METRIC DESCRIPTION

SuccessfullyCompletedCommitsPerCatalog(VersionedStorageOnly)

SuccessfulcommitstotheCatalogwhereanewversionispublishedintheMetadataservice

CanceledCommitsPerCatalog(VersionedStorageOnly)

UserinitiatedcanceledcommitsperCatalog

Data,Catalog,andLayerMetrics

13

Page 15: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

FailedCommitsPerCatalog(VersionedStorageOnly)

IncompletecommitstoaCatalog-Changebatchcommittedbutjobnotfinished.Failurereasonexamples:UserErrors,500Errors,Etc.Doesnotincludecanceledcommits.

VolatileLayerMetrics

METRIC DESCRIPTION

CapacityAllocatedforVolatileStorage TotalcapacityconfiguredinGB

CapacityUsedforVolatileStorage TotalcapacityusedinGB

UniqueCapacityUsedforVolatileStorage Amountofstoreddatawithoutreplication

StreamLayerMetrics

METRIC DESCRIPTION

TTLperStreamLayer "TimeToLive(TTL)"userconfigurationsettingperStreamLayer

ThroughputperStreamLayer IN/OUTthroughput(Bytes/Second)perStreamLayer

VersionedLayerMetrics

METRIC DESCRIPTION

DurableStorageUsedforVersionedLayer TotalcapacityusedforVersionedLayerinthecatalogs

Data,Catalog,andLayerMetrics

14

Page 16: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

IngestionMetrics

METRIC DESCRIPTION

ReceivedSingleRequests Numberofrequestswithasinglemessagerequestbody.

FailedSingleRequests Outof"TotalReceivedSingleMessageRequests",thenumberofrequestswhichreturnedanon-successstatuscode.

Multi-MessageRequests Numberofrequestsamultimessagerequestbody.

MessagessuccessfullyprocessedfromReceivedMulti-MessageRequests

Numberofmessagescontainedinmulti-messagerequestswhichhavebeensuccessfullyreceivedandprocessed.

FailedMulti-MessageRequests Outof"TotalReceivedMulti-messageRequests",thenumberofrequestswhichreturnedanon-successstatuscode.

MeanIngestionRequests Averagerateofrequestssenttotheingestionservicepersecond.

TotalBytesINperProvider(validmessagesonly) TotalGBIngested(inclusiveofallmessagetypes)

IngestionMetrics

15

Page 17: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

FlinkMetrics

ThefollowingmetricsareavailableforFlinkpipelines.ChecktheofficialFlinkdocumentationformoreinformationaboutFlinkmetrics.

CPU/MemoryMetrics

NAME METRIC DESCRIPTION

MaxStatus_JVM_CPU_Load cpu_load TherecentCPUusageoftheJVM

MaxStatus_JVM_Memory_Heap_Used heap_usedTheamountofheapmemorycurrentlyused.

MinStatus_JVM_Memory_Heap_Committed heap_committedTheamountofheapmemoryguaranteedtobeavailabletotheJVM.

MinStatus_JVM_Memory_Heap_Max heap_maxThemaximumamountofheapmemorythatcanbeusedformemorymanagement.

MaxStatus_JVM_Memory_NonHeap_Used nonheap_usedTheamountofnon-heapmemorycurrentlyused.

MaxStatus_JVM_Memory_NonHeap_Committed

nonheap_committedTheamountofnon-heapmemoryguaranteedtobeavailabletotheJVM.

MaxStatus_JVM_Memory_NonHeap_Max nonheap_maxThemaximumamountofnon-heapmemorythatcanbeusedformemorymanagement

CurrentStatus_JVM_Memory_Direct_Count direct_countThenumberofbuffersinthedirectbufferpool.

MaxStatus_JVM_Memory_Direct_MemoryUsed

direct_memoryusedTheamountofmemoryusedbytheJVMforthedirectbufferpool.

CurrentStatus_JVM_Memory_Direct_TotalCapacity

direct_totalcapacityThetotalcapacityofallbuffersinthedirectbufferpool.

CurrentStatus_JVM_Memory_Mapped_Count mapped_countThenumberofbuffersinthemappedbufferpool.

MaxStatus_JVM_Memory_Mapped_MemoryUsed

mapped_memoryusedTheamountofmemoryusedbytheJVMforthemappedbufferpool.

MaxStatus_JVM_Memory_Mapped_TotalCapacity

mapped_totalcapacityThenumberofbuffersinthemappedbufferpool.

FlinkClusterMetrics

METRICNAME UNIT DESCRIPTION

numRegisteredTaskManagers Count Thenumberofregisteredtaskmanagers.

numRunningJobs Count Thenumberofrunningjobs.

FlinkMetrics

16

Page 18: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

taskSlotsAvailable Count Thenumberofavailabletaskslots.

taskSlotsTotal Count Thetotalnumberoftaskslots.

FlinkI/OMetrics

METRIC UNIT DESCRIPTION

currentLowWatermark Count Thelowestwatermarkthistaskhasreceived.

numSplitsProcessed Count ThetotalnumberofInputSplitsthisdatasourcehasprocessed(iftheoperatorisadatasource).

numBytesInLocal Count Thetotalnumberofbytesthistaskhasreadfromalocalsource.

numBytesInLocalPerSecond Count Thenumberofbytesthistaskreadsfromalocalsourcepersecond.

numBytesInRemote Count Thetotalnumberofbytesthistaskhasreadfromaremotesource.

numBytesInRemotePerSecond Count Thenumberofbytesthistaskreadsfromaremotesourcepersecond.

numBytesOut Count Thetotalnumberofbytesthistaskhasemitted.

numBytesOutPerSecond Count Thenumberofbytesthistaskemitspersecond.

numRecordsIn Count Thetotalnumberofrecordsthisoperator/taskhasreceived.

numRecordsInPerSecond Count Thenumberofrecordsthisoperator/taskreceivespersecond.

numRecordsOut Count Thetotalnumberofrecordsthisoperator/taskhasemitted.

numRecordsOutPerSecond Count Thenumberofrecordsthisoperator/tasksendspersecond.

FlinkMetrics

17

Page 19: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

SparkMetrics

ThefollowingmetricsareavailableforSparkpipelines.

ChecktheSparkaccumulatordocumentationformoreinformationaboutthesemetrics.

SparkMetricsforPipelines

METRIC UNIT DESCRIPTION

PipelineStatus Count

NumberofPipelineExecutions Count

CustomAccumulators Key:Value

SparkMetricsforNotebooks

METRIC DESCRIPTION

AverageMemoryperExecutor AveragememoryperexecutorandSparkdriver

AverageandTotalSparkMemoryUsageforAllUnits

Aggregateofaveragememoryperexecutoranddriver.Alsoaggregatesallmemoryofthecluster

ActiveCores Numberofactivecores

Stages Stages,suchasrunning,pendingandfailed

TasksbyAllExecutors Tasksbyexecutors,active,andpool.Thisisanotherwaytoobservetheactiveandavailablecores

MessageProcessingTime Averagemessageprocessingtime

CompletedTasksbyEachExecuter Completedtasksbyexecutorsandcounters

FileSystemReads/WritesbyExecutors

Filesystemreadandwritesinbytes(whenthefilesystemisusedwithinjobsonly)

SparkMetrics

18

Page 20: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

PipelineMetrics

Youhaveaccesstothefollowingfivemetricstounderstandthestatusofyourpipelinejobs.

Formoreinformationonmonitoringandtroubleshootingpipelines,seePipelineMonitoringinthePipelineDeveloperGuide.

METRIC VALUE DESCRIPTION

pipeline_jobs_failed 0or1

Valueof1indicatesthatthepipelinejobhasfailed.

pipeline_jobs_canceled 0or1

Valueof1indicatesthatthepipelinejobhasbeencanceled.

pipeline_jobs_running 0or1

Valueof1indicatesthatthepipelinejobisrunning.

pipeline_jobs_completed 0or1

Valueof1indicatesthatthepipelinejobhascompleted.

pipeline_jobs_submitted 0or1

Valueof1indicatesthatthepipelinejobhasbeensubmittedtotheservice.

PipelineMetrics

19

Page 21: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

CurrentUsageMetrics

ThefollowingmetricsareavailableforyoutomonitoryourcurrentusageofOLP.Themetricinthesingle-statpaneltotherightofthedashboardshowsthecurrentvaluewithinthelastonehour,whilethegraphpaneltotheleftshowsthetimeseriespresentationofthisvalue.

StorageUsageMetrics

METRIC DESCRIPTION

Versioned Storageofversionedcontrolled,Geo-Indexeddata

Volatile Storageinvolatilememoryformoreperformancewithsmallwritesandreads

Stream CapacityallocatedtoqueueStreamdata

StreamTTL

Capacityallocatedtostorelonglivedstreamingjobs.ThiscoverstheEBSvolumesassociatedwiththeStreamNodes

MetaData Storageofmeta-dataaboutthedatayoustore

MetricsTimeSeries

Storageofindexedmetricsdataforquicknumerictimeseriesdashboards.ToknowhowmuchisallocatedtoDashboards,andifmoreshouldbeinvested.

TransferUsageMetrics

METRIC DESCRIPTION

LogSearchIO Storageofindexedlogdatafordebuggingsearch

PipelineIO NetworkI/OtrafficgeneratedbyPipelines

ComputeUsageMetrics

METRIC DESCRIPTION

ComputeCore CPUCoreHoursconsumedbyBatchorStreamPipelines

ComputeRAM RAMHoursconsumedbyBatchorStreamPipelines

CurrentUsageMetrics

20

Page 22: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

ZeppelinNotebookMetrics

METRIC DESCRIPTION

HTTPResponseCodes HTTPmetrics,responsecodes,andrequest/responsetime

ZeppelinMemory ZeppelinserverJVMmemoryusage

ZeppelinThreads Zeppelinserverthreadstates

Zeppelinnotebooksstatus Numberofnotebooksrunningandfailing

ZeppelinNotebookMetrics

21

Page 23: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

CreateReports

InSplunk,youcandevelopyourownreportbycreatingasearchandsavingitasareport.Savedreportsareavailablewithinthe"SavedReports"tab.Foradditionaldetails,seetheSplunkdocumentationonsavingandsharingreports.

CreateReports

22

Page 24: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

CreateandManageAlerts

Youcansetupalertsandrequestemailnotificationswhenaconditionorthresholdismet.Addingametriccanbesummarizedbythefollowingtwosetsofsteps.

CreateaNotificationChannel

AnalertinGrafanahastwocomponents—anotificationchannelandanalerttrigger.AnotificationchannelisdefinedasawaythatyoucanbenotifiedbyGrafana,suchasemail.

Tocreateanotificationchannel,followthesesteps:

1. OpentheGrafanatoolfromhttps://platform.here.com/byclickingonTools>Monitoringandalerts.2. InGrafana,clickthedropdownmenuintheupperleft,selectAlerting>NotificationChannels.3. Onthisscreen,youcancreateanewchannelandspecifythenotificationmethodbyselectingthetypefrom

thedropdownmenu.Weonlysupportemailandwebhooknotificationtypes.

Note:GrafanaDocumentation

Formoredetailsandoptions,seethetheGrafanaDocumentationonNotifications.

CreatetheAlert

Therearetwoimportantthingstokeepinmindbeforecreatinganalert:

AlertscannotbecreatedonthestandardmetricsdashboardsthataresuppliedwithOLP.Alertscanonlybesetupona'Graph'panelinadashboard.

Tocreatethealert,followthesesteps:

1. Createanewdashboard,useanexistingoneyou'vecreated,orduplicateanexistingdashboard.Formoreinformation,seetheGrafanaGettingStarteddocumentation

2. SelectanewpaneloftypeGraph.AlertscanonlybesetuponGraphPanels.3. EditthegraphbyclickingonthePanelTitleandchoosingEdit.4. AddyourmetrictotheMetricstab.YoucanfindmoreinformationonhowtodothisintheGrafana

documentationonAlertingRules.5. SelecttheAlerttabtoaddvaluestothealert.6. SelectNotificationsontheleftsidemenuandaddyourpreviouslycreatednotificationchannel.

Formoreinformation,seetheGrafanaUserGuide.

CreateandManageAlerts

23

Page 25: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

DuplicateaDashboard

Youcancreateanidenticalcopyofthedashboardbyperformingthefollowing:

1. FromtheGrafanatool,clickontheGeariconatthetop.2. Select'SaveAs'andprovideanewdashboardname.

Figure:Grafanamenutoduplicateadashboard

DuplicateaDashboard

24

Page 26: Open Location Platform - Logs, Monitoring, and Alerts User ... · your processes. OLP provides a managed instance of the open source Grafana tool for managing your metrics dashboard

NeedHelp?

IfyouneedhelpwiththisoranyotherHEREOpenLocationPlatformServices,visitplatform.here.comforsupport.

NeedHelp?

25