splunk's hunk: a powerful way to visualize your data stored in mongodb
DESCRIPTION
TRANSCRIPT
![Page 1: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/1.jpg)
Copyright © 2014 Splunk Inc.
Spunk HunkA Powerful Way to Visualize Your Data Stored in MongoDB
Mark Groves
Sr. Director, Product Management
Splunk Developer Platform
![Page 2: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/2.jpg)
2
The Accelerating Pace of DataVolume | Velocity | Variety | Variability
GPS,RFID,
Hypervisor,Web Servers,
Email, Messaging,Clickstreams, Mobile,
Telephony, IVR, Databases,Sensors, Telematics, Storage,
Servers, Security Devices, Desktops
Machine data is the fastest growing, most complex, most valuable area of big data
![Page 3: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/3.jpg)
Platform for Machine DataAny Machine Data
Online Services Web
Services
ServersSecurity GPS
Location
StorageDesktops
Networks
Packaged Applications
CustomApplicationsMessaging
TelecomsOnline
Shopping Cart
Web Clickstreams
Databases
Energy Meters
Call Detail Records
Smartphones and Devices
RFID
Datacenter
Private Cloud
Public Cloud
Enterprise Scalability
Search and Investigation
Proactive Monitoring
Operational Visibility
Real-time Business Insights
Operational Intelligence
![Page 4: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/4.jpg)
4
What Does Machine Data Look Like?
4
Sources
Care IVR
Middleware Error
Order Processing
![Page 5: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/5.jpg)
5 5
Customer ID Order ID
Customer’s Tweet
Time Waiting On Hold
Twitter ID
Product ID
Company’s Twitter ID
Sources
Care IVR
Middleware Error
Order Processing
Customer IDOrder ID
Customer ID
What Does Machine Data Look Like?
![Page 6: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/6.jpg)
6 6
Order ID
Customer’s Tweet
Time Waiting On Hold
Product ID
Company’s Twitter ID
Sources
Care IVR
Middleware Error
Order Processing
Order ID
Customer ID
Twitter ID
Customer ID
Customer ID
What Does Machine Data Look Like?
![Page 7: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/7.jpg)
7
How does this relate to MongoDB?
Hunk…Enables you to combine Time Series event data with leading Big Data StoresWhat does this look like? Demo…
![Page 8: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/8.jpg)
Copyright © 2014 Splunk Inc.
Cell Tower Monitoring App
Merging Machine Data with MongoDB
![Page 9: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/9.jpg)
10
Concepts
Splunk Index != Database IndexSchema on ReadTime is a first class citizen in Splunk
![Page 10: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/10.jpg)
11
Components of Hunk Server
64-bit Linux OS
REST API COMMAND LINE
Explore Analyze Visualize Dashboards Share
ODBC
splunkd
Hadoop Interface• Hadoop Client Libraries• JAVA
Streaming Resource Libraries• NoSQL & Other Stores
splunkweb
Web and Application server
Virtual Indexes
Python, AJAX, CSS, XSLT, XML
Search Head C++, Web Services
![Page 11: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/11.jpg)
12
Powerful Platform for Enterprise Developers
12
REST API
Build Splunk Apps Extend and Integrate Splunk
Simple XML
JavaScript
Django
Web Framework
SDKsJava
JavaScript
Python
C#
Ruby
PHP
![Page 12: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/12.jpg)
13
Virtual Indexes – Connector into MongoDB
• Enables seamless use of almost the entire Splunk stack on data• Automatically handles query execution to Mongo, Hadoop, etc
![Page 13: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/13.jpg)
14
HunkSearch Head >
Examples of Virtual Indexes
External System 1
External System 2
External System 3
index = syslog (/home/syslog/…)
index = apache_logsindex = sensor_data
index = twitter
![Page 14: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/14.jpg)
15
Hunk Search Architecture
Query per Index/Virtual Index
Search Processor
HunkSearch Head >
1.3.
4.
2.
Splunk Distributed
Search
Hadoop External Results Provider
MongoDB Streaming
Resource LibraryMongoDBProvider
MongoDB
MongoDB
MongoDB
JSON Config
Results Reduction
![Page 15: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/15.jpg)
16
Hunk applies schema for all fields – including transactions – at search time
Hunk Applies Schema on the Fly
• Structure applied at search time
• No brittle schema to work around
• Automatically find patterns and trends
![Page 16: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/16.jpg)
Integration
![Page 17: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/17.jpg)
18
Install via GUI
18
1. 2.
3.
![Page 18: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/18.jpg)
19
Install via Command LineGo to <apps.splunk.com URL>Download MongoDBProvider.splEither:– Copy MongoDBProvider.spl to $SPLUNK_HOME/etc/apps– tar –zxvf MongoDBProvider.spl
19
![Page 19: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/19.jpg)
20
Configure Indexes.conf - OverviewIndexes.conf defines indexes, physical and virtualNeed to two configuration items, a provider and a virtual index– Provider should be 1:1 to your MongoDB Server– There can be multiple virtual indexes per Provider
Indexes.conf can be in any Splunk App, probably easiest to put it in MongoDBProvider folder
20
![Page 20: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/20.jpg)
21
Configure Indexes.conf
21
[wocorders]vix.provider = local-mongodbvix.mongodb.db = demovix.mongodb.collection = wocordersvix.mongodb.field.time = timestampvix.mongodb.field.time.format = date
[provider:local-mongodb]vix.family = mongodb_erp_familyvix.splunk.search.debug = 0vix.mongodb.host = localhost:27017
Provider Name (referenced in Virtual Indexes)FamilyDisable DebuggingHostname:Port
Provider
[mongodb_vix]vix.provider = local-mongodbvix.mongodb.db = hunkvix.mongodb.collection = testvix.mongodb.field.time = _idvix.mongodb.field.time.format = ObjectId
Name of the Virtual Index (used by users)Provider Name (matches earlier stanza)MongoDB DB NameMongoDB Collection NameField to extract time fromFormat of the Field to Extract Time From (Valid Options are ObjectID, Date, or Epoch)
Virtual Index 1
![Page 21: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/21.jpg)
22
Configure Indexes.conf
22
[wocorders]vix.provider = local-mongodbvix.mongodb.db = demovix.mongodb.collection = wocordersvix.mongodb.field.time = timestampvix.mongodb.field.time.format = date
Name of the Virtual Index (used by users)Provider Name (matches earlier stanza)MongoDB DB NameMongoDB Collection NameField to extract time fromFormat of the Field to Extract Time From (Valid Options are ObjectID, Date, or Epoch)
Virtual Index 2
![Page 22: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/22.jpg)
23
How to query Mongo
23
index=mongodb (foo=xyz OR other=val) | fields foo, bar, baz
Query your MongoDB Virtual
Index
Match any fields by specifying the field name and matching
parameters
Minimize results returned by
projecting down only the fields you
want returned
![Page 23: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/23.jpg)
24
Mongo Specific Integration Highlights
24
index=mongodb foo=xyz | timechart avg(bar) by baz
Predicate Pushdown Projections
Filtering terms are processed on the MongoDB side, so only results where the
field foo matches xyz are returned
We only return back fields which are mentioned in the particular search, in this
case _time, bar and baz
![Page 24: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/24.jpg)
25
Roadmap for the Future
Full text search engineBSON support
25
![Page 25: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/25.jpg)
26
Get The Bits!
Hunk– http://splunk.com/download
MongoDB App– http://apps.splunk.com/app/1810/– Or search for “MongoDB” on apps.splunk.com
26
![Page 26: Splunk's Hunk: A Powerful Way to Visualize Your Data Stored in MongoDB](https://reader033.vdocuments.us/reader033/viewer/2022061211/549201c4b47959a11b8b45ab/html5/thumbnails/26.jpg)
27
Where to go for More Info
Contact Me: [email protected] - @markgrovsSplunkDev - http://dev.splunk.com/Splunk Apps - https://apps.splunk.com GitHub - https://github.com/splunk/Twitter - https://twitter.com/splunkdev Blogs - http://blogs.splunk.com/dev/
27