splunklive london 2014 developer presentation

Copyright © 2014 Splunk Inc.

Developer PlatformJon RooneyDirector of Developer Marketing

Damien DallimoreDeveloper Evangelist

Who is Jon Rooney?Director of Developer Marketing Splunker since April 2012 Based in Splunk’s Seattle OfficeVeteran of Microsoft, start-ups, Accenture

2

What Does Splunk Have to Do with Developers?

3

Application Development Challenges

4

Build DeploymentServer

QA / Test

Staging Environment

Continuous Integration / Build Server

SourceRepository

Task Tracking

Local Build

Application Development Challenges

5

Build DeploymentServer

QA / Test

Staging Environment

Continuous Integration / Build Server

SourceRepository

Task Tracking

Local Build

Lack of visibility across the product development lifecycle

Slows down the ability to detect and troubleshoot errors

Limited visibility into application usage and performance

6

CI / Build Servers

Project and Issue Tracking

Code Repository

QA / Testing Tools

Splunk can solve these problems

Deployment Servers

Splunk for Application Lifecycle Intelligence

7

Reduce Time to Market

Resolve issues faster

Gain Agility

Improve Code Quality and Lifecycle Visibility

Generate Application Insights

Instrument Analytics

Demo

Customer Successes

9

Touring the Splunk Developer Platform

Evolving the Splunk Platform

Collection

Indexing

Search Processing Language

Core Functions

Inputs, Apps, Other Content

SDKs

Operational Intelligence Platform

Content

Core Engine

User and Developer Interfaces

Web Framework

REST API

Powerful Platform for Enterprise Developers

12

REST API

Build Splunk Apps Extend and Integrate Splunk

Simple XML

JavaScript

Django

Web Framework

JavaJavaScriptPython

RubyC#PHP

Data Models

Search Extensibility

Modular Inputs

SDKs

Splunk Web Framework

13

Familiar Technologies- Code with JavaScript & Django/Python- HTML5/CSS/JS Support - Built on JQuery & Backbone.js

Flexible and Extensible- Create custom layouts- Integrate visualizations like Sankey

charts, heat maps and bubble charts - SimpleXML to HTML Conversion

Simple XML

JavaScript

Django

Web Framework

Log directly to Splunk via TCP,

UDP, HTTP

Integrate search results with other applications using

custom visualizations

Create and run searches from

other applications

The REST API and SDKs

14

VisualizeSearch Manage

Add/Delete Users

Manage Inputs

Index

Let’s dig a bit deeper

Who is Damien Dallimore

16

Worldwide Developer Evangelist @ SplunkI developI talk about developingHelping to build the Splunk developer ecosystemCame from the Splunk CommunityOnce was a customerCoder

I develop

17

All 100% free and open sourced , published to Splunk Apps , source code on Github

An Open Platform for Developers

18

Splunk is an open and extensible platform at numerous different touchpoints for developers

Extensibility creates ecosystems

A Developer’s Smorgasboard

19

CLIREST API6 language SDKsSplunkbase Apps / Add-onsCustom search commandsScripted InputsModular InputsThe Web Framework Standard HTML/Javascript/CSSData Models

External scripted lookupsCustom REST EndpointsTools , utilitys and librarysIntegrations with other software frameworks (Spring)Hadoop dev with HUNKCustom user interfaces / visualizationsMobile with BugsenseCustom data connectors (ODBC)Custom authentication handlers

Which Splunk Product for Devs ?

20

Splunk> Enterprise : Free to download and use. Index 500 MB/day.

Splunk> Cloud : Premium, cloud hosted. Full Enterprise stack.

Splunk> AMIs : BYOL versions for Amazon AWS Cloud.

Hunk> : Splunk for data in Hadoop clusters. Same platform, same easy apps.

Splunk> Storm : Free to use, cloud hosted. 20GB/30days.

Modular Inputs

22

Modular Inputs• Extend the Splunk framework to define a custom input capability, just like the standard inputs you are

familiar with (TCP/UDP/File etc…)

• Splunk treats your custom input definitions as if they were part of Splunk's native inputs, totally integrated first class citizen objects in Splunk

• Users interactively create and update your custom inputs using Splunk manager, just as they do for native inputs.

• When deploying without a UI , you push out the inputs.conf file.

• All the properties are fully manageable via the REST API

• Version 5.0 +

23

Developing

24

• My preference is to use Python, however any language can be used.

• http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModInputsIntro

• There is a certain amount of “plumbing” to put in place , so I like utilities that take care of this for you, so you can just focus on the business logic.

• Java,Python,C# SDKs also have Modular Input APIs

• Eclipse plugin has a wizard for creating Modular Inputs in Java

REST : The Data Potential

25

Twitter FoursquareLinkedIn Facebook Fitbit Amazon Yahoo Reddit YouTube Flickr Wikipedia GNIP Box

Okta Datasift Google APIs Weather Services Seismic monitoringPublicly available socio-economic dataTraffic data Stock monitoring Security service providers Proprietary systems and platforms Other “data related” software productsThe REST “dataverse” is vast , but I think you get the point.

There is a world of data out there available via REST that can be brought into Splunk, correlated and enriched against your existing data, or used for entirely new uses cases that you might conceive of once you see what is available and where your data might take you.

You are only limited by your own “data imagination”

26

Demo

Custom Search Commands

Splunk Search

29

Splunk’s search and querying language is called SPLAllows you to search, analyze and manipulate your data.Designed with the “unix pipeline” in mind – a “search pipeline”

From a (simplistic) mental point of view:– Series of commands– Each commands takes the input of the previous one– Each command outputs a sparse table

Splunk Search - Example 1:

30

“index=_internal foo | eval bar=... | stats count by bar”Initial command fetches result from index/raw data store, and outputs a table, with a row for each event, and a column for each field (not all rows have all columns)Second command adds a new column to each row, “bar”Third command looks at all the unique values of “bar”, and counts how many rows each value has.Note that the last command completely transformed the table, hence it is a “transforming” or “non-streaming” command.The second command was merely additive, known as a “streaming” command.You can also have “generating” commands ie: inputlookup

Custom Search Commands

31

Just like the “eval” or “stats” commands, you can write your own “search commands”.Python scripts which take data on stdin, and output data on stdoutData comes in/out as CSV (with special handling for MV fields)Many of Splunk’s builtin commands are written as Python scripts (e.g. head, return, transpose)

Custom Search Commands: Building Blocks

32

Custom search commands are built in two parts:

A Python script containing the implementation

An entry in commands.conf declaring configuring the command

Splunk SDK for Python has librarys and examples for creating custom search commands

Source Code !

Web Framework

Splunk Web Framework

35

Familiar Technologies- Code with JavaScript & Django/Python- HTML5/CSS/JS Support - Built on JQuery & Backbone.js

Flexible and Extensible- Create custom layouts- Integrate visualizations like Sankey

charts, heat maps and bubble charts - SimpleXML to HTML Conversion

Simple XML

JavaScript

Django

Web Framework

Splunk JS Stack & Django Bindings Concepts

36

Managers• Search Query Wrappers• SearchManager and SaveSearch

Manager• Include Search Parameters• Available within Django or

JavaScript

Splunk JS Stack & Django Bindings Concepts

37

Splunk Views• UI widgets• Designed to work with Search

Managers• Charts, Maps, Inputs, Timeline,

TimePicker, etc.

Splunk JS Stack & Django Bindings Concepts

38

URL Maps & Django View• URL Maps enable custom routes• Django Views, enable custom logic

to provide to templates

Splunk JS Stack & Django Bindings Concepts

39

Templates & Template Tags• Templates enable quick layout

options • Templates support inheritance of

other templates• Tags call a Python function can be

used for:– Text manipulation– Flow control– Load external information– … and more

Splunk JS Stack & Django Bindings Concepts

40

Data Binding using Tokens• Token based data-binding

mechanism that keep shared data in sync.

• Enables in-page interactivity

Web Framework Toolkit App

41

• Improved productivity for developing your own Web Framework based apps

• Reusable Visualization• Packaged to lower the effort of adding cool

visualizations• Improved Developer focused Command Line

Tools• Getting started templates• Automates common tasks

• Includes examples pages demonstrating advanced concepts.

SimpleXML converted to HTML

42

Transition a Simple XML page to HTML/JS Option to overwrite/edit existing dashboard or create newFull Splunk JS Stack is available (Add custom viz, tokens, etc.)Does not include Django backendNo support for visual dashboard editor or PDF printing

Simple XML with JS Import

43

Splunk 6 Dashboard Examples App

44

Custom Visualizations

50

My Guiding Viz PrincipleThe visualization must be simple and intuitive to understand and derive meaning from at a glance.

Cool viz , but what are you telling me ?

So many options , which one for me ?Splunk Web Form Editor– If you are not a coder , not familiar with Simple XML

Edit Simple XML– Familiar with Simple XML , what to customize more

Convert Simple XML to HTML/JS – Coder , want to see the underlying JS/HTML , want custom UI behavior above Simple

XML, want to use some other JS/CSS

Simple XML JS Import– Closely related to the above , perhaps you still want dashboard editing / PDF export

Django– Previous benefits + want to leverage Django tags , want custom server side processing in

Django views51

Splunk 6.1 Features For Building Apps

52

Mobile Developers

Company Overview

The right toolset for analyzing and troubleshooting mobile apps in real-time

54

Deliver high quality, engaging apps.Splunk BugSense

55

Capabilities

Mobile Data

HTML5 dashboard• Actionable reports• Easy to use

How It Works

Cross Platform SDKs • Install in < 5 min• Crashes/Errors• Events• Sessions• Transactions

Bug Sense Cloud

• Highly Scalable• Secure• Cloud Service• Highly available• Cost-effective

Integration (Android)1. Add the .jar (download or use Maven/Gradle) & import

2. Add the permissions

3. Initialize:

57

58

59

Why Develop

WHY should you develop ?

61

Make money , Promote your company, Make sales !Community and collaborationShare / Give BackGet a job / Build a careerLearn new skills / Educate yourself and othersHadoop productivityDo goodOpen up new data sources for others to collaborate on

We talk a lot about the how , what , where and who ….. but what about the WHY

apps.splunk.com

62

Wrap Up and Questions

Splunk Developer License

64

Where to go for More InfoTutorials, Code Samples, Getting Started, Downloads– http://dev.splunk.com/

Splunk Apps– https://apps.splunk.com

GitHub– https://github.com/splunk/

Twitter– https://twitter.com/splunkdev

Blogs– http://blogs.splunk.com/dev/

65

The 5th Annual Splunk WWUC

• 50+ Customer speakers• 30+ Apps in Splunk Apps

Showcase• 30+ Technology Partners• Ask The Experts• Sales Meetings• Business Value ROI booth

conf.splunk.com

Las Vegas: Oct 6-9, 2014 The MGM Grand Hotel4000+ IT and Business Professionals3 days of content, 130+ sessions3 days of Splunk UniversityGet Certified!

Thank you