syslogng and splunk
TRANSCRIPT
Building Centralized Logging: Syslog
Steven “Maniac” McGrath
Syslog?
• logging service
• UNIX based
• Networkable
Wait a Sec...Network?
• UDP port 514
• Typically limited to 1024bytes
One more thing...
• FIFO Buffers
• First In First Out
• Rolling View of Logs
• Type of Named Pipe
FIFO...Tasty *chomp*
Item 5
Item 4Item 3Item 2
Item 1
3 Line FIFO Buffer
Getting Started...
• Ubuntu 6.06 Server
• Base Install
Installing Syslog...
• Update The Repository
Upgrade the OS
• We need to upgrade the OS to current.
Install Syslog-NG
• Syslog-NG will remove klogd, this is normal.
Reconfiguring Syslog-ng
• Configuration depends on network environment.
• Windows Hosts
• Cisco Devices
• Linux Hosts
• Other Devices and Gear
First off...Global!/etc/syslog-ng/syslog-ng.confoptions { chain_hostnames(0); time_reopen(10); time_reap(360); log_fifo_size(2048); create_dirs(yes); group(admin); perm(0640); dir_perm(0755); use_dns(no); stats_freq(0);};
• Disable Hostname Chaining• Time to wait before re-establishing a dead connection• Time to wait before an idle file is closed• FIFO Buffer size• Create Directories• Permissions• Disable DNS• Disable Statistic Logging
Next, The Source
source s_all { internal(); unix-stream("/dev/log"); file("/proc/kmsg" log_prefix("kernel: ")); udp();};
/etc/syslog-ng/syslog-ng.conf
Defining Filters
• Windows Filter
• Cisco Filter
Windows Filter
filter f_windows { program(MSWinEventLog);};
/etc/syslog-ng/syslog-ng.conf
Cisco Filter
filter f_cisco_pix {host(IP.OF.PIX.DEVICE);
};
/etc/syslog-ng/syslog-ng.conf
General Filter
filter f_not_others {not host(IP.OF.PIX.DEVICE)and not program(MSWinEventLog);
};
/etc/syslog-ng/syslog-ng.conf
Destinations
• FIFO Buffers
• One Large File
Windows FIFO
destination d_windows {pipe(“/var/log/buffers/windows”);
};
/etc/syslog-ng/syslog-ng.conf
Cisco FIFO
destination d_cisco {pipe(“/var/log/buffers/cisco”);
};
/etc/syslog-ng/syslog-ng.conf
General FIFO/etc/syslog-ng/syslog-ng.conf
destination d_gen_fifo {pipe(“/var/log/buffers/syslog”);
};
...And the Archive
destination d_all {file(“/var/log/arch/$MONTH$DAY$YEAR”);
};
/etc/syslog-ng/syslog-ng.conf
Tying it all Together!
• Now we tell syslog to handle the configs. ;)
Windows Log
log { source(s_all); filter(f_windows);destination(d_windows);
};
/etc/syslog-ng/syslog-ng.conf
Cisco Log
log { source(s_all); filter(f_cisco_pix);destination(d_cisco);
};
/etc/syslog-ng/syslog-ng.conf
General FIFO
log { source(s_all); filter(f_not_others);destination(d_gen_fifo);
};
/etc/syslog-ng/syslog-ng.conf
Archive Log
log { source(s_all); destination(d_all);
};
/etc/syslog-ng/syslog-ng.conf
Finishing up...
• Making the FIFO buffers
• Creating the directory structure
Run me :)
$ sudo mkdir /var/log/arch$ sudo mkdir /var/log/buffers
$ sudo mkfifo /var/log/buffers/windows$ sudo mkfifo /var/log/buffers/cisco$ sudo mkfifo /var/log/buffers/syslog
Restart Syslog-ng
$ sudo /etc/init.d/syslog-ng restart
Is it working?
• Check your Logfiles (/var/log/arch/*)
• Check your FIFO Buffers
• cat /var/log/buffers/windows
• cat /var/log/buffers/cisco
• cat /var/log/buffers/syslog
Awsome! Wait....
• How are we gonna view this data?
splunk
• Web-based Interface
• Indexes arbitrary data
• Searchable
• Reporting
>
• No, I don’t work for them...I just really like their product.
splunk>
• Download The latest version (3.0b3 as of writing)
• Extract the tarball
• Run the application
• Make it startup with a system boot
Installing splunk>
$ wget 'http://www.splunk.com/index.php/download_track?file=/3.0b3/linux/splunk-3.0b3-20872-Linux-i686.tgz&ac=&wget=true&name=wget'
$ sudo mkdir /opt;cd /opt
$ sudo tar xzvf ~/splunk-3.0b3-20872-Linux-i686.tgz
$ sudo /opt/splunk/bin
Installing splunk>
Configuring splunk>
Configuring splunk>
Configuring splunk>
Configuring splunk>
Configuring splunk>
splunk>
Syslog Agents
• Windows Agents
• UNIX Agents
• Other Devices
Windows Logs?
• SNARE Agent
• Converts Event Logs to Syslog
• Free
UNIX Agents
• Use the syslog service!
• *.* @Syslog Server
Other Devices
• Various systems can be configured
• Cisco, Juniper, Lotus Domino, Apache, IIS, etc. are just a few examples.
Recap
• What is Syslog
• What is FIFO
• Installing and Configuring Syslog-NG
• Installing and Configuring Splunk
• Agents
Questions?