instrumentation with splunk

Instrumentation with Splunk

11/17/2016 2www.datavail.com

About Datavail

13+ Years delivering data services

300+ customers with average client retention of 7 years

Managed services, projects, and staffing


The Problem…

Oracle Applications have allot of moving pieces…


The Problem…

Oracle Applications can have allot of moving pieces…

In allot of different locations…


The Problem…

Root causes are quite often buried and difficult to find.


The Problem…


What if there was a neat way to federate the telemetry from all your key pieces?


The Problem…


What if there was a neat way to federate the telemetry from all your key pieces? Including the contents of your critical logs?


A Solution…

Web Based GUI Search Engine

Filtered Reports on Key Perf Indicators

Raw Log File Repository


A Solution…


What is Splunk?

Splunk is an enterprise application.

Splunk was made to monitor.

Splunk is agnostic.

Splunk is a framework!


What Splunk isn’t

Not a collection of purchased lock-downed modules (Solar Winds etc.)

It’s not an application that lives on a database.

It’s not an out of the box solution.


Splunk’s Pieces Forwarder

• Installed on the server

• Light weight

• Has a watch list

• Basic filtering

• Basic classification

• Looks for file changes

• Looks for new files

• Sends it all to the Indexer


Splunk’s Pieces

Indexer

• Transforms the raw data into events.

• An event is like a line in your log file.

• Can be set up in HA configs.


Splunk’s Pieces

Search Head

• The GUI interface to your data.

• Provides API and CLI support.

• Runs on *NIX, OSX and Windows.

Splunking Your DataWhat can I do with Splunk?


Splunk’s Features

Interactive Searching• Has robust SPL to search with.

• Combination of grep, regex, and custom functions like eval, average,

• Has grouping and de-dupe functions

• Searches can be saved as carts or reports.


Splunk’s FeaturesData is organized as Events.

Events assign metadata to the raw data.


Splunk’s FeaturesMetadata is automatically aggregated and displayed as dynamic ‘tool tips’.


Splunk’s Features

REGEX Field Extractions

Splunk extracts your KPIs using REGEX quickly parsing through your collected data to identify the information you are most interested in.

(?i) ORA-(?P<OracleAlertError>.+)


Splunk’s Features

Automated Field Extractions

Splunk can also automatically generate regex extractions for many common patterns of data.


Splunk’s Features

Powerful Data Classification Tools

Data Models provide easy access to extracted data that can be used to easily create pivot charts and tables.


Splunk’s Features

Data Models

Select FieldsSelect Pivots


Splunk’s Features

Interactive Charts

Customer Use Case


Scripted Inputs For GC Monitoring

Use Cases• Access data that is not available as an ordinary file.

• Access data that cannot be sent using TCP or UDP.

• Stream data from command-line tools, such as SQLPlus.

• Reformat complex data so you can more easily parse the data into events and fields.

• Attach a timestamp to transient data such as iostat.

Methods of Implementation• Shell Scripts

• Batch Files

• Python/Perl Scripts

• Command Line Output

• Anything that writes to STDOUT

Methods of Capture• Direct from STDOUT

• Write a file to be indexed


Forwarder Recap

The forwarder is the beginning of classification.

Source, source types, hosts, and filenames are collected and sent to the indexer.

Forwarders are self contained. They do not have to be ‘installed’. They can run as any OS user.

Forwarders run on Windows, *NIX, and OSX

[monitor:///.../opmn]disabled = falsesourcetype = OPMNLogsindex=euebpignoreOlderThan = 7d

[monitor:///.../Apache/access_log*]disabled = falsesourcetype = ApacheAccessLogsindex=euebpignoreOlderThan = 7d

[script://$SPLUNK_HOME/etc/apps/<appName>/bin/rcat.sh] disabled = false host = rmanhostindex = main interval = 30 #frequency to run the script, in seconds sourcetype = RMAN


Garbage Collection Monitor Use Case

Based On jstat

jstat –gcutil <ospid>

Use perl to parse for pid and format into this.

11-03-2016 07:05 OSPid=28733 Proc=forms-c4ws_server1 S0=0.00 S1=8.99 E=56.21 O=66.80 P=85.68 YGC=83

YGCT=13.092 FGC=21 FGCT=80.713 GCT=93.80511-03-2016 07:05


Implementing the Script

Create the perl script

Create the shell script

Move scripts into place

Add script stanza to the input


Activate the Script

Inputs.conf Stanza

[script://$SPLUNK_HOME/etc/apps/<appName>/bin/jstat.sh]

disabled = false

host = rmanhost

index = main interval = 30 #frequency to run the script, in seconds

sourcetype = RMAN


Graph The Results

How to Get StartedIf I Don’t Have Splunk?


Splunk Installation Reqs

Splunk can be downloaded for free. You will be limited to 500mb of ingested data a day. This is actually can be sufficient for one environment if you are judicious about what you log.

Splunk for *NIX can be downloaded in tar ball format. This can be unpacked in any directory and does not have to be installed by or run by root.

Splunk stores everything in its self contained path, so you just have to delete the directory to remove it.

You can install Splunk directly on the system you wish to monitor (not always a good idea)

Splunk uses ports 8000 and higher for the browser and the forwarders so again, no root user is needed.


Existing Splunk?

If your company already uses Splunk you can ask your admin for the following.

• Your own index (The index is actually the directory structure and files Splunk uses to store the data).

• Additional forwarders, or…

• If your server already has a forwarder on it, you just need to get your log locations added to inputs.conf and sent to your index.

• Your own Application. An application is just a collection of settings like searches, chart and report descriptions and etc. This way you won’t interfere with the network and security guys.

Amazon Echo Giveaway

instrumentation with splunk

Data & Analytics