splunklive london 2014 developer presentation
DESCRIPTION
Developer Presentation by Jon Rooney and Damien Dallimore at SplunkLive London 2014TRANSCRIPT
Copyright © 2014 Splunk Inc.
Developer PlatformJon RooneyDirector of Developer Marketing
Damien DallimoreDeveloper Evangelist
Who is Jon Rooney?Director of Developer Marketing Splunker since April 2012 Based in Splunk’s Seattle OfficeVeteran of Microsoft, start-ups, Accenture
2
What Does Splunk Have to Do with Developers?
3
Application Development Challenges
4
Build DeploymentServer
QA / Test
Staging Environment
Continuous Integration / Build Server
SourceRepository
Task Tracking
Local Build
Application Development Challenges
5
Build DeploymentServer
QA / Test
Staging Environment
Continuous Integration / Build Server
SourceRepository
Task Tracking
Local Build
Lack of visibility across the product development lifecycle
Slows down the ability to detect and troubleshoot errors
Limited visibility into application usage and performance
6
CI / Build Servers
Project and Issue Tracking
Code Repository
QA / Testing Tools
Splunk can solve these problems
Deployment Servers
Splunk for Application Lifecycle Intelligence
7
Reduce Time to Market
Resolve issues faster
Gain Agility
Improve Code Quality and Lifecycle Visibility
Generate Application Insights
Instrument Analytics
Demo
Customer Successes
9
Touring the Splunk Developer Platform
Evolving the Splunk Platform
Collection
Indexing
Search Processing Language
Core Functions
Inputs, Apps, Other Content
SDKs
Operational Intelligence Platform
Content
Core Engine
User and Developer Interfaces
Web Framework
REST API
Powerful Platform for Enterprise Developers
12
REST API
Build Splunk Apps Extend and Integrate Splunk
Simple XML
JavaScript
Django
Web Framework
JavaJavaScriptPython
RubyC#PHP
Data Models
Search Extensibility
Modular Inputs
SDKs
Splunk Web Framework
13
Familiar Technologies- Code with JavaScript & Django/Python- HTML5/CSS/JS Support - Built on JQuery & Backbone.js
Flexible and Extensible- Create custom layouts- Integrate visualizations like Sankey
charts, heat maps and bubble charts - SimpleXML to HTML Conversion
Simple XML
JavaScript
Django
Web Framework
Log directly to Splunk via TCP,
UDP, HTTP
Integrate search results with other applications using
custom visualizations
Create and run searches from
other applications
The REST API and SDKs
14
VisualizeSearch Manage
Add/Delete Users
Manage Inputs
Index
Let’s dig a bit deeper
Who is Damien Dallimore
16
Worldwide Developer Evangelist @ SplunkI developI talk about developingHelping to build the Splunk developer ecosystemCame from the Splunk CommunityOnce was a customerCoder
I develop
17
All 100% free and open sourced , published to Splunk Apps , source code on Github
An Open Platform for Developers
18
Splunk is an open and extensible platform at numerous different touchpoints for developers
Extensibility creates ecosystems
A Developer’s Smorgasboard
19
CLIREST API6 language SDKsSplunkbase Apps / Add-onsCustom search commandsScripted InputsModular InputsThe Web Framework Standard HTML/Javascript/CSSData Models
External scripted lookupsCustom REST EndpointsTools , utilitys and librarysIntegrations with other software frameworks (Spring)Hadoop dev with HUNKCustom user interfaces / visualizationsMobile with BugsenseCustom data connectors (ODBC)Custom authentication handlers
Which Splunk Product for Devs ?
20
Splunk> Enterprise : Free to download and use. Index 500 MB/day.
Splunk> Cloud : Premium, cloud hosted. Full Enterprise stack.
Splunk> AMIs : BYOL versions for Amazon AWS Cloud.
Hunk> : Splunk for data in Hadoop clusters. Same platform, same easy apps.
Splunk> Storm : Free to use, cloud hosted. 20GB/30days.
Modular Inputs
22
Modular Inputs• Extend the Splunk framework to define a custom input capability, just like the standard inputs you are
familiar with (TCP/UDP/File etc…)
• Splunk treats your custom input definitions as if they were part of Splunk's native inputs, totally integrated first class citizen objects in Splunk
• Users interactively create and update your custom inputs using Splunk manager, just as they do for native inputs.
• When deploying without a UI , you push out the inputs.conf file.
• All the properties are fully manageable via the REST API
• Version 5.0 +
23
Developing
24
• My preference is to use Python, however any language can be used.
• http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModInputsIntro
• There is a certain amount of “plumbing” to put in place , so I like utilities that take care of this for you, so you can just focus on the business logic.
• Java,Python,C# SDKs also have Modular Input APIs
• Eclipse plugin has a wizard for creating Modular Inputs in Java
REST : The Data Potential
25
Twitter FoursquareLinkedIn Facebook Fitbit Amazon Yahoo Reddit YouTube Flickr Wikipedia GNIP Box
Okta Datasift Google APIs Weather Services Seismic monitoringPublicly available socio-economic dataTraffic data Stock monitoring Security service providers Proprietary systems and platforms Other “data related” software productsThe REST “dataverse” is vast , but I think you get the point.
There is a world of data out there available via REST that can be brought into Splunk, correlated and enriched against your existing data, or used for entirely new uses cases that you might conceive of once you see what is available and where your data might take you.
You are only limited by your own “data imagination”
26
Demo
Custom Search Commands
Splunk Search
29
Splunk’s search and querying language is called SPLAllows you to search, analyze and manipulate your data.Designed with the “unix pipeline” in mind – a “search pipeline”
From a (simplistic) mental point of view:– Series of commands– Each commands takes the input of the previous one– Each command outputs a sparse table
Splunk Search - Example 1:
30
“index=_internal foo | eval bar=... | stats count by bar”Initial command fetches result from index/raw data store, and outputs a table, with a row for each event, and a column for each field (not all rows have all columns)Second command adds a new column to each row, “bar”Third command looks at all the unique values of “bar”, and counts how many rows each value has.Note that the last command completely transformed the table, hence it is a “transforming” or “non-streaming” command.The second command was merely additive, known as a “streaming” command.You can also have “generating” commands ie: inputlookup
Custom Search Commands
31
Just like the “eval” or “stats” commands, you can write your own “search commands”.Python scripts which take data on stdin, and output data on stdoutData comes in/out as CSV (with special handling for MV fields)Many of Splunk’s builtin commands are written as Python scripts (e.g. head, return, transpose)
Custom Search Commands: Building Blocks
32
Custom search commands are built in two parts:
A Python script containing the implementation
An entry in commands.conf declaring configuring the command
Splunk SDK for Python has librarys and examples for creating custom search commands
Source Code !
Web Framework
Splunk Web Framework
35
Familiar Technologies- Code with JavaScript & Django/Python- HTML5/CSS/JS Support - Built on JQuery & Backbone.js
Flexible and Extensible- Create custom layouts- Integrate visualizations like Sankey
charts, heat maps and bubble charts - SimpleXML to HTML Conversion
Simple XML
JavaScript
Django
Web Framework
Splunk JS Stack & Django Bindings Concepts
36
Managers• Search Query Wrappers• SearchManager and SaveSearch
Manager• Include Search Parameters• Available within Django or
JavaScript
Splunk JS Stack & Django Bindings Concepts
37
Splunk Views• UI widgets• Designed to work with Search
Managers• Charts, Maps, Inputs, Timeline,
TimePicker, etc.
Splunk JS Stack & Django Bindings Concepts
38
URL Maps & Django View• URL Maps enable custom routes• Django Views, enable custom logic
to provide to templates
Splunk JS Stack & Django Bindings Concepts
39
Templates & Template Tags• Templates enable quick layout
options • Templates support inheritance of
other templates• Tags call a Python function can be
used for:– Text manipulation– Flow control– Load external information– … and more
Splunk JS Stack & Django Bindings Concepts
40
Data Binding using Tokens• Token based data-binding
mechanism that keep shared data in sync.
• Enables in-page interactivity
Web Framework Toolkit App
41
• Improved productivity for developing your own Web Framework based apps
• Reusable Visualization• Packaged to lower the effort of adding cool
visualizations• Improved Developer focused Command Line
Tools• Getting started templates• Automates common tasks
• Includes examples pages demonstrating advanced concepts.
SimpleXML converted to HTML
42
Transition a Simple XML page to HTML/JS Option to overwrite/edit existing dashboard or create newFull Splunk JS Stack is available (Add custom viz, tokens, etc.)Does not include Django backendNo support for visual dashboard editor or PDF printing
Simple XML with JS Import
43
Splunk 6 Dashboard Examples App
44
Custom Visualizations
50
My Guiding Viz PrincipleThe visualization must be simple and intuitive to understand and derive meaning from at a glance.
Cool viz , but what are you telling me ?
So many options , which one for me ?Splunk Web Form Editor– If you are not a coder , not familiar with Simple XML
Edit Simple XML– Familiar with Simple XML , what to customize more
Convert Simple XML to HTML/JS – Coder , want to see the underlying JS/HTML , want custom UI behavior above Simple
XML, want to use some other JS/CSS
Simple XML JS Import– Closely related to the above , perhaps you still want dashboard editing / PDF export
Django– Previous benefits + want to leverage Django tags , want custom server side processing in
Django views51
Splunk 6.1 Features For Building Apps
52
Mobile Developers
Company Overview
The right toolset for analyzing and troubleshooting mobile apps in real-time
54
Deliver high quality, engaging apps.Splunk BugSense
55
Capabilities
Mobile Data
HTML5 dashboard• Actionable reports• Easy to use
How It Works
Cross Platform SDKs • Install in < 5 min• Crashes/Errors• Events• Sessions• Transactions
Bug Sense Cloud
• Highly Scalable• Secure• Cloud Service• Highly available• Cost-effective
Integration (Android)1. Add the .jar (download or use Maven/Gradle) & import
2. Add the permissions
3. Initialize:
57
58
59
Why Develop
WHY should you develop ?
61
Make money , Promote your company, Make sales !Community and collaborationShare / Give BackGet a job / Build a careerLearn new skills / Educate yourself and othersHadoop productivityDo goodOpen up new data sources for others to collaborate on
We talk a lot about the how , what , where and who ….. but what about the WHY
apps.splunk.com
62
Wrap Up and Questions
Splunk Developer License
64
Where to go for More InfoTutorials, Code Samples, Getting Started, Downloads– http://dev.splunk.com/
Splunk Apps– https://apps.splunk.com
GitHub– https://github.com/splunk/
Twitter– https://twitter.com/splunkdev
Blogs– http://blogs.splunk.com/dev/
65
The 5th Annual Splunk WWUC
• 50+ Customer speakers• 30+ Apps in Splunk Apps
Showcase• 30+ Technology Partners• Ask The Experts• Sales Meetings• Business Value ROI booth
conf.splunk.com
Las Vegas: Oct 6-9, 2014 The MGM Grand Hotel4000+ IT and Business Professionals3 days of content, 130+ sessions3 days of Splunk UniversityGet Certified!
Thank you