Download - Log Analytics Optimization
Optimizing Log Analytics from the Edge
April 2016
© Hortonworks Inc. 2011 – 2015. All Rights Reserved
2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
About HortonworksCustomer Momentum
~800 customers (as of Feb 10, 2016)Publicly traded on NASDAQ: HDP
Hortonworks Data PlatformCompletely open multi-tenant platform for any app and any dataConsistent enterprise services for security, operations, and governance
Partner for Customer SuccessLeader in open-source community, focused on innovation to meet enterprise needsUnrivaled Hadoop support subscriptions
Founded in 2011
Original 24 architects, developers, operators of Hadoop from Yahoo!
800+E M P L O Y E E S
1500+E C O S Y S T E M
PA R T N E R S
3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
EMBRACE AN OPEN APPROACH
MASTER THE VALUE OF DATA
EVERY BUSINESS IS A DATA BUSINESS
4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DATA AT REST
DATA IN MOTION
ACTIONABLEINTELLIGENCE
MODERN DATA APPLICATIONS
Actionable Intelligence from Connected Data Platforms
Capturing perishable insights from data in motion
Ensuring rich, historical insights on data at rest
Necessary for modern data applications
Hortonworks DataFlow
Hortonworks Data Platform
5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Optimizing Log Ingest with Hortonworks DataFlow
6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Why Hortonworks DataFlow?
Because even the best data scientists and most powerful platforms need the right data to analyze
7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Store Data
Process and Analyze Data
Acquire Data
Perception of DataFlows: Easy, Definitive
Dataflow
8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Reality of Dataflows: Complex, Convoluted
Store Data
Process and Analyze Data
Acquire Data
Store DataStore Data
Store Data
Store Data
Acquire Data
Acquire Data
Acquire Data
Dataflow
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
HDF has 130+ Processors - Multiple for Log Analytics
HTTP
Syslog
HTML
Image
Hash Encrypt
Extract
TailMerge
Evaluate
Duplicate Execute
Scan
GeoEnrich
Replace
ConvertSplit
Translate
HL7
FTP
UDP
XML
SFTP
Route Content
Route Context
Route Text
Control Rate
Distribute LoadAMQP
10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Log Analytics Systems Today
LOG ANALYTICS PLATFORMNetwork
Device Logs
• Not all data can be captured• Not all captured data is valuable• Transport all data
11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Cost Effectively Expand Storage Options of Log Data
LOG ANALYTICS PLATFORM
Network Device Logs
HDPHDF
3. Cost effectively expand collection and grow timescale of logs collected
2. Content-based routing based on dynamic evaluation of content, attributes, priority
1. Integrate and enrich logs across data centers and security zones
12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Efficiently Expand Log Ingestion from the Edge
LOG ANALYTICS PLATFORM
Network Device Logs
HDF
HDF
HDF
HDPHDF
• Expand collection to new sources of machine data• Edge analytics to transform, enrich and prioritize content based routing• Capture and transport only valuable data
13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Expand Analytics and Reporting Options with HDP
LOG ANALYTICS PLATFORM
Network Device Logs
HDF
HDF
HDF
HDPHDF
ODBC interface traditional BI tools
Easy access to log analytics data through traditional BI tools
Give data scientists better tooling – Spark, Storm etc
14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Expand to small scale, remote systems
LOG ANALYTICS PLATFORM
Network Device Logs
HDF
HDF
HDF
HDPHDF
15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Optimize Log Analytics with Content Based Routing
LOG ANALYTICS PLATFORM
Edge analytics for cost-effective and efficient movement of
machine data
HDF
Intelligent, content based routing, transformation
and enrichment
Send data to alternative systems based on value,
content, priority
HDP
HDF
HDF
HDF
16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Splunk Optimization:Using HDP as Data Refinery
17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Splunk Hadoop Connect
17
Reliable bi-directional integration
ImportBrowseExport
Splunk Hadoop Connect
>2000 downloads
HA Indexes and Storage
CommodityServers
Hadoop (MapReduce &
HDFS)
Report & analyze
Custom dashboards
Monitor and alert
Ad hoc search
18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Splunk, Hunk & Hortonworks
YARN Ready Partner
Certified on Hortonworks Data Platform
Existing Sandbox tutorial
19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Splunk, Part of the Modern Data Architecture
• Bi-directional data integration between Splunk & HDP
• Collect data from across the organization, deliver it to Hadoop for refining data and batch analytics
• Output of Hadoop jobs can be imported into Splunk Enterprise for rapid analysis and visualization
• Archiving from Splunk Enterprise to Hadoop
20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Splunk, Part of the Modern Data Architecture
• Bi-directional data integration between Splunk & HDP
• Collect data from across the organization, deliver it to Hadoop for refining data and batch analytics
• Output of Hadoop jobs can be imported into Splunk Enterprise for rapid analysis and visualization
• Archiving from Splunk Enterprise to Hadoop
21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hunk + Hortonworks
21
Explore, analyze and visualize data in HDP from one integrated platform
Simply point Hunk at your HDP cluster(s) and start exploring data immediately
Search data, change perspectives and preview results as MapReduce jobs run
INTERACTIVE EXPLORATION
RICH DEVELOPER ENVIRONMENT
Build big data apps on data in HDP using standard web languages and frameworks
FULL-FEATUREDANALYTICS
FAST TO DEPLOY AND DRIVE VALUE
22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Augment Splunk Deployment with Hortonworks Data Platform
Heavy Indexer
Universal Forwarders
HDP Enables
Splunk Storage
• Expansion to more data than previously feasible• Archive data from Splunk into Hadoop• Query archived Splunk data in Hadoop• Focus Splunk infrastructure on what really matters
23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Find out how much you can optimize your log analytics infrastructure today.
Contact [email protected]