howtoactuallyuse splunkdatamodels · disclaimer* 2...

Copyright © 2014 Splunk Inc.

David Clawson SplunkYoda

How to actually use Splunk Data Models

Disclaimer

2

During the course of this presentaDon, we may make forward-‐looking statements regarding future events or the expected performance of the company. We cauDon you that such statements reflect our current expectaDons and

esDmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements,

please review our filings with the SEC. The forward-‐looking statements made in the this presentaDon are being made as of the Dme and date of its live presentaDon. If reviewed aPer its live presentaDon, this presentaDon may not contain current or accurate informaDon. We do not assume any obligaDon to update any forward-‐looking statements we may make. In addiDon, any informaDon about our roadmap outlines our general product direcDon and is subject to change at any Dme without noDce. It is for informaDonal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligaDon either to develop the features or funcDonality described or to

include any such feature or funcDonality in a future release.

Agenda

!   The Big Picture !   Data Models 101 !   How do we model informaDon in Splunk !   A real world example

How to actually use Splunk Data Models

Problem

!   I have data that contains business criDcal indicators !   The data is machine generated and mulD layered !   The data is oPen organized into non-‐simple structures !   The users are comparaDvely non-‐technical

!   I don’t want to be their commissioner of data forever

4

Splunk Machine AnalyDcs To the Rescue

Build complex reports without the search language

Provides more meaningful representaDon of underlying raw machine data

AcceleraDon technology delivers up to 1000x faster analyDcs over Splunk 5

5

Pivot

Data Model

Analy0cs Store

OperaDonal Intelligence Across the Enterprise

IT professional !  Create and share data models !  Accelerate data models and custom searches with the analy0cs store

!  Create reports with pivot

Developer Analyst !  Leverage data models to abstract data

!  Leverage pivot in custom apps

!  Create reports using pivot based on data models created by IT

Pivot Data Model

Raw Data

AnalyDcs Store

[ 1 0 / 1 1 / 1 2 18 :57 :04 UTC] 0 0 0 0 0 0 b 0

Splunk Late Binding Schema

As Seen Through

Data DefiniDon and PresentaDon Layers

PresentaDon

Data Models

Data Enrichment

Fielded Data

Data Models 101

Be[er Living Through Models

9

How do I make it easier to understand my world?

•  Data Models are a way to make raw data easier to use

•  Models to clarify meaning •  Models to simplify complex data

You make things simple when you bring people to understand them.

Making things simple is complicated!

The one able to translate misty complexity into familiar simplicity has therefore power. And responsibility.

Data set

Source

Source

Source

The Simplest of Splunk Models

Sourcetype

Success

Failure

Warning

Convey More Meaning with Logical OrganizaDon

Data set

Business division Source

Source

Business division Source

Source

But Can be Used to Simply Segregate Business Divisions

Common model

Technology 1

Technology 2

Technology 3

Or Make Data From Different Origins Appear to Carry the Same Meaning

Customer

Order History

Web History

Problem Tickets

Or, Break Down the Complexity Into Understandable Pieces

How do we Model InformaDon in Splunk?

Hurdles

•  Simple searches easy… mulD-‐stage munging/reporDng is hard! •  Need to understand data’s structure to construct search •  Non-‐technical users may not have data source domain knowledge •  Splunk admins may not understand end-‐user search context

index=main source=*/banner_access* uri_path=/js/*/*/login/* guid=* useragent!=*KTXN* useragent!=*GomezAgent* clientip!=206.80.3.67 clientip!=198.144.207.62 clientip!=97.65.63.66 clientip!=175.45.37.78 clientip!=209.119.210.194 clientip!=212.36.37.138 clientip!=204.156.84.0/24 clientip!=216.221.226.0/24 clientip!=207.87.200.162 | rex field=uri_path "/js/(?<t>[^/]*)/(?<v>[^/]*)/login/(?<l>[^/]*)” | eval license = case(l LIKE "prod%" AND t="pro", "enterprise", l LIKE "trial%" AND t="pro", "trial", t="free", "free”) | rex field=v "^(?<vers>\d\.\d)” | bin span=1d _time as day | stats values(vers) as vers min(day) as min_day min(eval(if(vers=="5.0", _time, null()))) as min_day_50 dc(day) as days values(license) as license by guid | eval type = if(match(vers,"4.*"), "upgrade", "not upgrade") + "/" + if(days > 1, "repeat", "not repeat")| search license=enterprise | eval _time = min_day_50| timechart count by type| streamstats sum(*) as *"

Enter the Splunk Data Model

18

Data models enable users to create compelling reports and dashboards

without having to write the searches that generate them.

What is a Data Model?

19

!   A data model is a hierarchically structured search-‐Dme mapping of knowledge about one or more datasets

!   Data models make what appear to be complex, simple !   Data models encode the domain knowledge necessary to build a variety of specialized searches of those datasets

If you are familiar with relaDonal database design, think of data models as analogs to database schemas.

How Can I Develop a Splunk Data Model?

!   Manage Knowledge Objects –  Data interpretaDon:

Fields and Field ExtracDons –  Data classificaDon:

Event Types and TransacDons –  Data enrichment:

Lookups and Workflow AcDons –  Data normalizaDon:

Tags and Aliases

!   Build a Data Model

Combine together into a

Data Model

Fields and Field

ExtracDons

Lookups and Workflow AcDons

Event Types and

TransacDons

Tags and Aliases

Data InterpretaDon: Fields and Field ExtracDons

!   Data models can get their fields from extracDons that you set up

This is the starDng point. You must create field extracDons

In data model terminology, the fields that data models use are called ”a[ributes”.

GeneralSuccess=“Most Assured” and Be[erKnowledge=“Almost Certain”

Data ClassificaDon: Event Types and TransacDons

Event types let you classify events that have common characterisDcs !   When you search your event data, you're essenDally weeding out all unwanted

events. Therefore, the results of your search are events that share common characterisDcs, and you can give them a collecDve name

Use the power of Splunk to make the data richer

"failed login" OR "FAILED LOGIN” OR "AuthenDcaDon failure" OR "Failed to authenDcate user"

Data ClassificaDon: Event Types and TransacDons

A transacDon is a group of conceptually-‐related events that spans Dme For example, a customer purchase in an online store could generate a transacDon that Des together events from several sources:


message ID transacDon ID message ID

session ID transacDon ID session ID

web access events

applicaDon server log

message queue event

purchase fulfillment event

Data Enrichment: Lookups and Workflow AcDons

! Lookup tables use informaDon in your events to determine how to add other fields from external data sources such as staDc tables (CSV files) and scripts


h[p_status = 503

status_descripDon = “Service Unavailable”

In your event

Would add…..

Data Enrichment: Lookups and Workflow AcDons

! Workflow AcDons enable you to set up interacDons between specific fields in your data and other applicaDons or web resources

A really simple workflow acDon would be one that is associated with an IP_address field,

which when launched, opens an external WHOIS search in a separate browser window based on the IP_address value


Data NormalizaDon: Tags and Aliases

!   Help you track abstract field values, like IP addresses or ID numbers !   For example, you could have an IP address related to your main office with the value 192.168.1.2.

!   Tag that IPaddress value as main_office, and then search on that tag to find events with that IP address


> tag:main_office

Build a Data Model

Combine together into a

Data Model

Fields and Field

ExtracDons

Lookups and Workflow AcDons

Event Types and

TransacDons

Tags and Aliases

27

All that is leP is to combine all of these together to create a

Data Model

How do I do this?

Building Data Models in Splunk

What is a Data Model? A data model is a search-‐0me mapping of data onto a hierarchical structure !   Encapsulate the knowledge needed to build a search

!   Pivot reports are build on top of data models

!   Data-‐independent Screenshot here

A Data Model is a CollecDon of Objects

Screenshot here

Objects Have Constraints and AUributes

Screenshot here

Child Objects Inherit Constraints and A[ributes

Screenshot here

Child Objects Inherit Constraints and A[ributes

Field Case #1 -‐ SyntheDc Performance Monitoring

34

Simple data collected in a complicated structure to answer a simple quesDon.

How are my transacDons running today?

Business Use Case

!   Fortune 100 Manufacturer !   Based in USA but with faciliDes in 64 countries !   Currently capturing Gomez data via an API to use in central performance dashboards

!   Want to combine Gomez data with other performance data to gain complete knowledge of network performance data.

!   Examining current Web strategy to ensure that user experience is the same globally

35

Wants to Use True Performance Data

1.  Be[er understand performance from an end user perspecDve 2.  Understand end-‐user performance in key global markets before

they go-‐live in those regions 3.  Monitor global end-‐user performance on an ongoing basis 4.  Measure and idenDfy problems in criDcal customer groups 5.  Combine the data with applicaDon monitoring data to know when

outages are network vs applicaDon stack related 6.  Large number of users and uses for the resulDng data

36

But the Data Can be Difficult to Work With

37

Monitor id 4043050 test node <![CDATA[Minneapolis, MN PA (277)]]> test_rt 17939 0mestamp <![CDATA[2014-‐09-‐03 14:50:18.697]]> step seqno 0 url <![CDATA[hUps://intra3.work.com/enl/]]> step_rt 293 status 0

}

} } Monitored Step Results

* up to 10 results

Monitor ID

Test ID * n tests

Demo Building a Data Model

Manage Knowledge Objects

39

!   Curate Splunk Enterprise knowledge !   Develop naming convenDons for knowledge objects !   Understand and use the “Common InformaDon Model” !   Manage knowledge object permissions

Who will be the: Data Architect? Knowledge Manager? Informa0on Specialists? Splunk Dude?

Some Best PracDces for Data Model Design

Use root event objects whenever possible

To take advantage of the benefits of

data model acceleraDon

Minimize object hierarchy depth

whenever possible

Constraint-‐based filtering is less efficient deeper down the tree

When possible include the index or indexes it

is selecDng from

Data model acceleraDon efficiency is

improved when the data model isn't

searching across all of your indexes

40

Some Best PracDces for Data Model Design

Use a[ribute flags to selecDvely expose small

groups of a[ributes for each object

You can expose and hide different a[ributes for different objects

A child a[ribute can expose an enDrely different set of a[ributes than those

exposed by its parent

Your Pivot users will benefit from this selecDon by not having to deal with a

bewildering array of a[ributes whenever they set out to make a pivot chart or table

Instead they'll see only those a[ributes that make sense in the context of

the object they've chosen

Reverse-‐engineer your exisDng dashboards and searches into data models

This can be a way to quickly get started with data models

Dashboards built with pivot-‐derived panels are

easier to maintain

Start from understanding what your Pivot users hope to

be able to do.

The structure of your model should be determined by your users' needs and expectaDons

Work backwards from there

41

Other Data Model PresentaDons

!   Technical ImplementaDon Guidance from Archna Ganapathi !   Managing non-‐IT data (Data Mart techniques) from Pete Sicilia

42

THANK YOU Be a Model

how*to*actually*use* splunk*datamodels* · disclaimer* 2...

Documents

howtoactuallyuse splunkdatamodels · disclaimer* 2...