using big data tools to analyze log files, event logs and performance metrics
TRANSCRIPT
Using Big Data Tools to Analyze Log Files, Event Logs and Performance Metrics
Hal RottenbergDeveloper Evangelist, Splunk
Agenda
Talk.Do.
Talk.Do.
Talk.Do.
Talk.Do.Talk.Do.Do.Talk.
Do.Talk.
Do.Talk.
Do.Talk.Do.Talk.Do.
Talk.Do.
Talk.
* Repeat as needed.
THREE HOURS LATER
ObjectivesLearning• What is Big Data and
Machine Data?• Use cases?• Tools?
Doing• Install the tools• Use them!
Who is Hal?
A Disclaimer• I work for Splunk• This will not be a sales pitch--but• Pardon the bias– Feel free to call me out on the bias! (Or anything
else.)
One more thing
INTERACTIVITY DISCUSSION LEARNING
LET’S DO THIS!
What is Big Data?“Big data usually includes data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process data within a tolerable elapsed time.”
-Wikipedia says so
What is Big Data?
high volume
high velocity
What is Big Data?high volume
high velocity
high variability
What is Big Data?
high volume
high velocity
high variability
What is Machine Data?“Machine data contains a definitive record of all the activity and behavior of your customers, users, transactions, applications, servers, networks and mobile devices.”
- Splunk says so
Yeah, but what IS it?• Log files
Yeah, but what IS it?• Audit / change records
Yeah, but what IS it?• Performance Metrics
Yeah, but what IS it?• Configuration Items
Yeah, but what IS it?• Diagnostic Output
…even more• API output• Message queues• Call detail records• Sensor readings
USE CASES
Big Data Use Cases in IT Ops• Troubleshooting• Security• Visibility• Agility
Troubleshooting
Security
Visibility
Agility
THE TOOLBOX
Big Data Toolbox Requirements• Getting Data In• Scalability• Flexibility
Getting Data In• Aggregation• Indexing• Parsing• Normalization?
Scalability• Scaling up versus out• In-memory versus on-disk• Map reduce• Bloom filters
Flexibility• Relational DB versus NoSQL– Schemas
• Architecture
Free Tools
Free Tools• Elastic Search, Logstash, Kibana
Commercial Tools• VMware Log Insight
Commercial Tools• Splunk
LET’S DO THIS!
Lab Objectives1. Install Splunk2. Get Data3. Perform use case labs
1. Troubleshooting2. Security3. More!