datamodelsfordevelopers - conf.splunk.com · legalnokces...

Copyright © 2013 Splunk Inc.

Alice Neels So<ware Engineer, Splunk

Brian Bingham So<ware Engineer, Splunk – Content #splunkconf

Data Models for Developers

Legal NoKces During the course of this presentaKon, we may make forward-‐looking statements regarding future events or the expected performance of the company. We cauKon you that such statements reflect our current expectaKons and esKmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements, please review our filings with the SEC. The forward-‐looking statements made in this presentaKon are being made as of the Kme and date of its live presentaKon. If reviewed a<er its live presentaKon, this presentaKon may not contain current or accurate informaKon. We do not assume any obligaKon to update any forward-‐looking statements we may make. In addiKon, any informaKon about our roadmap outlines our general product direcKon and is subject to change at any Kme without noKce. It is for informaKonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligaKon either to develop the features or funcKonality described or to include any such feature or funcKonality in a future release.

Splunk, Splunk>, Splunk Storm, Listen to Your Data, SPL and The Engine for Machine Data are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respecCve

owners.

©2013 Splunk Inc. All rights reserved.

2

About Us Alice: !   At Splunk since 2011 !   Before this, UW CSE, then worked on iOS at Apple !   On the core search team !   Backend architect for Data Model

Brian: !   Splunk since 2012 !   Past Life -‐ 15 years as an SA !   Lead Dev for Infra-‐Ops Content Team !   Major Apps: VMware, ES, and several others

3

Agenda

!   What is data model, and why do I care? !   Data models 101 !   Building a data model !   AcceleraKon and management !   Using data models !   Q&A

4

What is a Data Model?

What is a Data Model? A data model is a search-‐Cme mapping of data onto a hierarchical structure •  Encapsulate the knowledge needed

to build a search •  Pivot reports are build on top of

Data Models •  Data-‐independent Screenshot here

7

Who is it for? !   Admins/power users build data models !   Business users use data models via Pivot UI !   Data models can be used in apps to organize and generate searches

8

Why do I Care? !   Search is hard !   Admins and power users know how their data works !   Non-‐technical users (usually) don’t !   A data model makes it easy to share and organize that knowledge

9

But Really !   This search

( sourcetype="access_*" OR sourcetype="iis*" ) ( uri="*" ) uri=* uri_path=* status=* clientip=* referer=* useragent=* ( status=2* ) ( uri_path!=*.php OR uri_path!=*.html OR uri_path!=*.shtml OR uri_path!=*.rhtml OR uri_path!=*.asp ) ( uri_path=*.avi OR uri_path=*.swf ) ( uri_path=*.itpc OR uri_path=*.xml ) | litsearch ( sourcetype=access_* OR sourcetype=iis* ) ( uri="*" ) uri=* uri_path=* status=* clientip=* referer=* useragent=* ( status=2* ) ( uri_path!=*.php OR uri_path!=*.html OR uri_path!=*.shtml OR uri_path!=*.rhtml OR uri_path!=*.asp ) ( uri_path=*.avi OR uri_path=*.swf ) ( uri_path=*.itpc OR uri_path=*.xml ) | eval newX = " " | eval "useragent ::: status"='useragent'+" ::: "+'status' | addinfo type=count label=prereport_events | fields keepcolorder=t "newX" "prestats_reserved_*" "psrsvd_*" "useragent ::: status" | fillnull value=NULL "useragent ::: status" | prestats count by newX "useragent ::: status"!

!   Becomes this search | pivot WebIntelligence PodcastDownload count(PodcastDownload) AS "Count of PodcastDownload" SPLITCOL useragent SPLITCOL status FILTER uri isNotNull NUMCOLS 100!

And that’s cool!

10

How Can I Use It? Three ways 1.  Use the pivot UI to build dashboards with simple XML 2.  Use the search commands (data model, pivot, tstats) to simplify

building searches 3.  Use data model rest endpoints

11

Data Models 101

A Data Model is a CollecKon of Objects

Screenshot here

13

Objects Have Constraints and A/ributes

Screenshot here

14

Child Objects Inherit Constraints and Asributes

Screenshot here

15

Child Objects Inherit Constraints and Asributes

16

Pivot UI Subhead

Count of http_success events, split by

useragent events

fields

17

More Info !   See slides from other data models talks: “AnalyKcs with Splunk Enterprise 1 & 2”

18

Building Data Models

Three Root Object Types !   Event

–  maps to Splunk events –  requires constraints and asributes

•  Search –  maps to arbitrary Splunk search (may

include generaKng, transforming and reporKng search commands)

–  requires search string and asributes

•  TransacKon –  maps to groups of Splunk events or

groups of Splunk search results –  requires objects to group, fields/

condiKons to group by, and asributes



!   Search –  maps to arbitrary Splunk search (may



•  TransacKon –  maps to groups of Splunk events or





!   Search –  maps to arbitrary Splunk search (may



!   TransacKon –  maps to groups of Splunk events or



Child Object Facts !   A child object is a type of its parent object: e.g. An HTTP_Success object is a type of HTTP_Access

!   Adding a child object is essenKally a way of adding a filter on the parents

!   A parent-‐child relaKonship makes it easy to do queries like “What percentage of my HTTP_Access events are HTTP_Success events?”

23

Object Asributes !   Auto-‐Extracted – default and pre-‐defined fields

!   Eval Expression – a new field based on an expression that you define

!   Lookup – leverage an exisKng lookup table

!   Regular Expression – extract a new field based on regex

!   Geo IP – add geographical fields such as lat / lon, country, etc.

Object Asributes !   Set field types

!   Configure various flags Note: Child object configuraKon can differ from parent

Display Names !   Models have a “modelName” and a “displayName” !   Objects have an “objectName” and a “displayName” !   They’re usually the same !   “modelName” and “objectName” are used internally and in search, must be unique, and can only contain

!   “displayName” is what’s displayed to label charts and graphs

26

Other Stuff !   Search and transacKon objects can have children too –it works the same way (they’re filters)

!   Only event-‐based objects are accelerated (more on that later) !   Performance degrades with the depth of the hierarchy

27

Other Splunk Knowledge Objects

How it Works with… !   Various Splunk knowledge objects can help make your data model more powerful

!   If you already have event types/tags etc., you can absolutely use these in your object constraints and asributes

!   When starKng from scratch, consider using your data model to do the same thing

29

Event Types: Background !   Event types in Splunk Enterprise are a way of separaKng out a single source file, into separate classificaKons. –  Example: Apache access logs: 400’s and 200’s are in the same file, these

can be broken out and used in search –  E.g. “sourcetype=apache_access hsp_code=* NOT hsp_code=4*”

becomes: “evensype=apache_success” –  Can be used for “tags”

30

Event Types With Data Model !   Before: set up event types via Manager

–  E.g. Two event types: apache_success and apache_failure !   With data models: in most cases, use a data model object

–  E.g. One data model “ apache “, with 2 objects, “ success ” and “ failure “.

!   Objects provide extra power and flexibility!

31

Tags !   A tag is associated with a parKcular field/value combinaKon, or with whether a field is present on an event

!   In apps, the best use case for tags, is to link up mulKple data sources together based on a common “goal”

!   Most o<en based on event types. –  Example: tag=AuthenKcaKon tag=Success could be used to Ke together

evensypes from LDAP, MySQL, Unix, MS AcKve Directory or VPN where a user successfully logged in, like evensype = “ldap_auth_success” or “vpn_auth_success”

!   Add through Manager

32

Tags with Data Model !   Data model may be used with tags in several ways:

–  To create fields that make it easier to define objects ê  Usually event type tags used to make several different sources “fit” a common model

–  An asribute of an object: ê  Frequently a way of finding out “field tagging” differences in events. Great for finding out “missing” fields or extracKons for creaKng objects that are based on a field value

ê  Can also use calculated fields for this

33

Macros, Saved Searches !   Data model does not replace the common use cases for macros or saved searches, and it’s important to understand when to use what

!   For dashboards, saving a pivot report may be preferable to saving a search (easier to modify and maintain and share)

!   Macros happen at a lower level of the search than data model – best pracKce is to NOT mix and match

34

Knowledge Objects Summary !   There are a lot of Splunk Enterprise features that can be used with data models

!   By combining these features, we get lots of power and flexibility !   Data model helps you manage complex data !   Which features you should use will depend on your needs and data, but keep it simple!

35

AcceleraKon and Management

What is a Data Model Really? !   Data models are stored as JSON files on disk (spec in docs) !   They live in <myapp>/local/data/models (or <myapp>/default/data/models for pre-‐installed models)

!   They also have associated conf stanzas and metadata

It Looks Like this

EdiKng JSON

39

!   EdiKng data models by hand: NOT SUPPORTED !   Very easy to shoot yourself in the foot, hard to recover !   When you edit models via the UI, we validate them !   ExcepKon: Installing a model by adding the file to <myapp>/<local OR default>/data/models is probably okay

At your own risk!

DeleKng a Model !   Use the UI !   This will do appropriate cleanup for you !   If you go deleKng files on disk, it’s easy to break things

Permissions !   Data models have permissions just like other splunk knowledge objects

!   Permissions are per model, NOT per object

!   Edit permissions through the UI

Permissions cont. !   Data models exist in a parKcular app !   Different user roles may or may not have read or write access !   If your model relies on lookups, etc., they need to also be available in that app

Data Model AcceleraKon

43

Run a pivot report

Polling: is are there new accelerated

models?

Turn on acceleration via UI

Setting written to conf file

Kick off collection

acceleraKon

Kick off ad-‐hoc acceleraKon and run search

Run search using on-‐disk acceleraKon

Admin or Power User

Backend Magic

Non-‐technical User no acceleraKon

AcceleraKon Facts !   Works with search-‐head pooling – we collect on indexers !   Only the first event-‐based object and its children are accelerated !   No acceleraKon for search and transacKon-‐based objects !   No ediKng accelerated models

Using Data Models

Pivot Interface !   Build a data model !   Build a report with Pivot !   Embed report in a dashboard

47

Demo

Search Commands !   You can use data models in the search language !   They’re basically just macros

| datamodel !   Look at models

–  “| datamodel” –  Returns JSON model descripKons as separate events

!   Look at just one model –  “| datamodel mymodel” –  Returns JSON model descripKon for just one model

!   Look at an object –  “| datamodel mymodel myobject” –  Returns JSON object descripKon

!   Run the search for an object –  “| datamodel mymodel myobject search” –  Runs the search

| datamodel cont. !   Model name and object name must be the internal names, not the display names

!   No acceleraKon

| pivot !   Any table you can build in the pivot UI can be expressed with the pivot command

!   Syntax is to complex to fully cover here – see docs !   “Open in search” from pivot UI uses the pivot command !   The pivot command will take advantage of acceleraKon where available !   Example:

| pivot WebIntelligence HTTP_Request count(is_HTTP_Success) AS "Count of is_HTTP_Success" count(is_HTTP_Error) AS "Count of is_HTTP_Error" count(is_HTTP_Redirect) AS "Count of is_HTTP_Redirect" FILTER status != 404!

52

Demo

REST Endpoints !   See docs for details !   Two main endpoints:

–  servicesNS/<user>/<app>/datamodel/model – examine models and change se~ngs

–  servicesNS/<user>/<app>/datamodel/pivot – get the search for a pivot report

54

Demo

Next Steps

55

Download the .conf2013 Mobile App If not iPhone, iPad or Android, use the Web App

Take the survey & WIN A PASS FOR .CONF2014… Or one of these bags!

1

2

THANK YOU

datamodels*for*developers - conf.splunk.com · legal*nokces*...

Documents

datamodelsfordevelopers - conf.splunk.com · legalnokces...