overview of the log correlation engine splunk...

Overview of the Log Correlation Engine Splunk Client November 20, 2014

(Revision 3)

Copyright © 2014. Tenable Network Security, Inc. All rights reserved. Tenable Network Security and Nessus are registered trademarks of Tenable Network Security, Inc. 2

Table of Contents

Introduction ......................................................................................................................................... 3

Standards and Conventions........................................................................................................................... 3

Configuring Log Correlation Engine Splunk Client ......................................................................... 3

Importing the Edited Policy File ........................................................................................................ 7

Configuring Splunk Indexer to Forward Data ................................................................................... 9

Data Comparison .............................................................................................................................. 11

For More Information ........................................................................................................................ 14

About Tenable Network Security ..................................................................................................... 15


Introduction

Tenable’s Log Correlation Engine unifies vulnerability collection and event analysis data through SecurityCenter, which provides easy-to-use dashboards to display multiple data points in a centralized view. Organizations that choose to send Splunk logs to the LCE have a unique advantage in that Splunk data is normalized by LCE and can be included for automatic anomaly detection, discovering assets, and additional vulnerability information including botnet and malware detection.

The Log Correlation Engine Splunk Client has one purpose: to forward data that Splunk collects to the LCE server. Once the data reaches the LCE server, the data is reviewed and normalized so it can be queried in SecurityCenter. The scope of this client can vary depending on what data is being forwarded from Splunk to the LCE Splunk Client.

This document is intended to assist with basic configuration of the LCE Splunk Client. It will also briefly cover some configuration steps that are required on the Splunk Indexer.

A working knowledge of Splunk, SecurityCenter, and LCE operation and architecture is assumed. Familiarity with general log formats from various operating systems, network devices, and applications, as well as a basic understanding of Linux/Unix, is also assumed.

Standards and Conventions

Throughout the documentation, filenames, daemons, and executables are indicated with a courier bold font such as

gunzip, httpd, and /etc/passwd.

Command line options and keywords are also indicated with the courier bold font. Command line examples may or

may not include the command line prompt and output text from the results of the command. Command line examples will display the command being run in courier bold to indicate what the user typed while the sample output generated by

the system will be indicated in courier (not bold). Following is an example running of the Unix pwd command:

# pwd

/opt/local/lce

#

Important notes and considerations are highlighted with this symbol and grey text boxes.

Tips, examples, and best practices are highlighted with this symbol and white on blue text.

Configuring Log Correlation Engine Splunk Client

The first step is to install and then configure the LCE Splunk Client on a RHEL 5 or 6 32/64-bit host. The client can be installed on the same server as Splunk, if desired. Throughout this example, the LCE Splunk Client resides on the same server as Splunk. Choose the correct installation package for the server OS and install the package. An example of the installation is shown below:

[root@LCE_splunk tmp]# rpm -ivh lce_splunk-4.2.0-el6.x86_64.rpm

Preparing... ########################################### [100%]

1:lce_splunk ########################################### [100%]

Please run /opt/lce_splunk/set-server-ip.sh to configure your LCE server's IP and

port.


After the installation is complete, execute the set-server-ip.sh script. The LCE server IP address and port that the

client communicates with the LCE server will be required. An example is shown below:

[root@LCE_splunk lce_splunk]# /opt/lce_splunk/set-server-ip.sh

Enter the new desired LCE server IP or hostname.

>>

192.168.1.14

Enter the new desired LCE server port [31300].

>>

31300

Updating LCE Server IP from 192.168.1.140 to 192.168.1.14... Updating LCE Server Port

from 31300 to 31300...

Done

Stopping LCE Splunk Client daemon [ OK ]

Starting LCE Splunk Client daemon [ OK ]

After the LCE Splunk Client configuration is complete, access SecurityCenter as the admin user and choose “Resources” from the menu options, followed by “LCE Clients”.


Next, select the host that has the LCE Splunk Client installed, and choose “Assign Policy”:

Once the “Assign Policy” window opens, scroll down, select the “default_rhel_lcesplunk.lcp” file, and choose “Export”.

Save the file and then open it in a text editor.

The default policy downloaded will be similar to the one shown below:

<?xml version="1.0" encoding="UTF-8" standalone="no" ?>

<options xmlns:xi="http://www.w3.org/2003/XInclude">

<log-directory>/opt/lce_splunk/</log-

directory>



<listen-port>9800</listen-port>



<splunk-server>127.0.0.1</splunk-server>



 <heartbeat-

frequency>300</heartbeat-frequency>



<statistics-frequency>60</statistics-frequency>



<compress-events>1</compress-events>

</options>

To configure the LCE Splunk Client to receive messages from the Splunk Indexer, a “listen” port must be chosen. The default listen port is 9800, and it will be left unchanged from the default in this example.

<listen-port>9800</listen-port>

If the splunk-server IP address is different from the local host IP address that is in the default policy, it can be changed

by editing the splunk-server tag. The local host IP address works for this example since the LCE Splunk Client is

located on the Splunk Indexer.


<splunk-server>127.0.0.1</splunk-server>

If you install the LCE Splunk Client on the Splunk Indexer, changes to the LCE Splunk Client policy file will not be required.

Importing the Edited Policy File

If the LCE Splunk Client policy file required modification, follow the steps below to upload and apply the modified policy. Select the “lcesplunk” client from the list of clients, and choose “Assign Policy” from the “LCE Clients” menu.

The “Import LCE Client Policy” menu will be displayed. Select “Browse” to start the policy upload process.


The next menu for file upload will be displayed. Select “Browse” to locate the policy file, and after the file has been chosen, click “Upload”.

Finally, select “Import” to complete the policy upload.

Choose the policy from the “Assign Policy” menu, and select “Assign”.

The new policy will now be in effect on the chosen LCE Splunk Client.


Configuring Splunk Indexer to Forward Data

To allow the Splunk Indexer to forward data to the LCE Splunk Client, first choose “Settings” from the main menu of the Splunk Indexer, and select “Forwarding and receiving”.

Next, select “Add new” from the “Configure forwarding” option.


Enter the IP address of the Splunk client along with the port number as shown below and select “Save”.

Once complete, the entry will be displayed as shown below in the “Forward data” section of Splunk.


To complete the configuration of Splunk, access the Splunk Indexer as the root user and edit the file “/opt/splunk/etc/system/local/outputs.conf”. The lines added to the file are highlighted below:

[tcpout]

defaultGroup = default

disabled = 0

indexAndForward = 1

[tcpout-server://LCE_IP_OR_Hostname:9800]

[tcpout:default]

disabled = 0

server = LCE_IP_OR_Hostname:9800

sendCookedData = false

A restart of the Splunk services will be required after the “outputs.conf” file has been edited.

Data Comparison

An example of the data shown in Splunk is shown below. The example shown contains search results for a Cisco ASA firewall. The exact search used narrowed the results to “sourcetype=syslog”, and matched the text string “%ASA”.

The same type of log information is available in SecurityCenter. When a user logs into SecurityCenter, there can be multiple dashboards available that display pertinent information for that user. It is possible to set a specific collection of dashboards as the default view in SecurityCenter. Examples of dashboards that can be created for events that are collected by the LCE Splunk client are shown below.


The dashboard shown above contains the dashboard named “NormalizedEvent Types Collected by Splunk”. Select the

arrow “>” beside that dashboard to view all the information available.


The dashboard above includes the Cisco ASA Firewall events and all event types in a normalized format that is easy to interpret. There are several views that you can select from the “Event Analysis” window that can be displayed by selecting “Normalized Event Summary”. A similar view can be displayed in Splunk by selecting the “Raw Syslog Events” option.

It is also possible to filter the “Normalized Event Summary” along with any other summary view by selecting the double Arrow “>>” at the top left of the window. The text string “%ASA” used in the Splunk search above could be entered under “Syslog Text”.


For More Information

Tenable has produced a variety of documents detailing the LCE’s deployment, configuration, user operation, and overall testing. These documents are listed here:

Log Correlation Engine 4.2 Architecture Guide – provides a high-level view of LCE architecture and supported platforms/environments.

Log Correlation Engine 4.4 Administrator and User Guide – describes installation, configuration, and operation of the LCE.

Log Correlation Engine 4.4 Quick Start Guide – provides basic instructions to quickly install and configure an LCE server. A more detailed description of configuration and management of an LCE server is provided in the “LCE Administration and User Guide” document.

Log Correlation Engine 4.2 Client Guide – how to configure, operate, and manage the various Linux, Unix, Windows, NetFlow, OPSEC, and other clients.

LCE 4.4 High Availability Large Scale Deployment Guide – details various configuration methods, architecture examples, and hardware specifications for performance and high availability of large scale deployments of Tenable's Log Correlation Engine (LCE).

LCE Best Practices – Learn how to best leverage the Log Correlation Engine in your enterprise.

Tenable Event Correlation – outlines various methods of event correlation provided by Tenable products and describes the type of information leveraged by the correlation, and how this can be used to monitor security and compliance on enterprise networks.

Tenable Products Plugin Families – provides a description and summary of the plugin families for Nessus, Log Correlation Engine, and the Passive Vulnerability Scanner.

Log Correlation Engine 4.2 Log Normalization Guide – explanation of the LCE’s log parsing syntax with extensive examples of log parsing and manipulating the LCE’s .prm libraries.

Log Correlation Engine 4.4 TASL Reference Guide – explanation of the Tenable Application Scripting Language with extensive examples of a variety of correlation rules.

Log Correlation Engine 4.0 Statistics Daemon Guide – configuration, operation, and theory of the LCE’s statistic daemon used to discover behavioral anomalies.

Log Correlation Engine 3.6 Large Disk Array Install Guide – configuration, operation, and theory for using the LCE in large disk array environments.

Example Custom LCE Log Parsing - Minecraft Server Logs – describes how to create a custom log parser using Minecraft as an example.

Documentation is also available for Nessus, the Passive Vulnerability Scanner, and SecurityCenter through the Tenable Support Portal located at https://support.tenable.com/.

There are also some relevant postings at Tenable’s blog located at http://www.tenable.com/blog and at the Tenable Discussion Forums located at https://discussions.nessus.org/community/lce.

For further information, please contact Tenable at [email protected], [email protected], or visit our web site at http://www.tenable.com/.

http://static.tenable.com/prod_docs/LCE_4.2_architecture.pdf

http://static.tenable.com/prod_docs/LCE_4.4_admin_user.pdf

http://static.tenable.com/prod_docs/LCE_4.2_Quick_Start_Guide.pdf

http://static.tenable.com/prod_docs/LCE_4.2_clients.pdf

https://static.tenable.com/prod_docs/LCE_4.4_Large_Scale_Deployment_Guide.pdf

http://www.tenable.com/sites/drupal.dmz.tenablesecurity.com/files/uploads/documents/whitepapers/LCE_best_practice.pdf

http://static.tenable.com/prod_docs/Tenable_Event_Correlation.pdf

http://static.tenable.com/documentation/Tenable_Products_Plugin_Families.pdf

http://static.tenable.com/prod_docs/LCE_4.2_log_analysis.pdf

http://static.tenable.com/prod_docs/LCE_4.4_TASL_reference_guide.pdf

http://static.tenable.com/prod_docs/LCE_4.0_stats.pdf

http://static.tenable.com/prod_docs/LCE_3.6_SAN-DAS_guide.pdf

http://static.tenable.com/prod_docs/Example_Custom_LCE_Log_Parsing_Minecraft_Server_Logs.pdf

https://support.tenable.com/

http://www.tenable.com/blog

https://discussions.nessus.org/community/lce

mailto:[email protected]

mailto:[email protected]

http://www.tenable.com/


About Tenable Network Security

Tenable Network Security provides continuous network monitoring to identify vulnerabilities, reduce risk, and ensure compliance. Our family of products includes SecurityCenter Continuous View™, which provides the most comprehensive and integrated view of network health, and Nessus®, the global standard in detecting and assessing network data.

Tenable is relied upon by more than 24,000 organizations, including the entire U.S. Department of Defense and many of the world’s largest companies and governments. We offer customers peace of mind thanks to the largest install base, the best expertise, and the ability to identify their biggest threats and enable them to respond quickly.

For more information, please visit tenable.com.

http://www.tenable.com/

overview of the log correlation engine splunk...

Documents