Lecture 16: UNIX Forensics

6/26/2003

CSCE 590

Summer 2003

Syslog

• A standard system logging facility– Unix, Windows, routers, switches, blenders, etc

• On UNIX, configuration in /etc/syslog.conf

• Daemon called syslogd

• Can syslog over the network to a dedicated syslog server

• Targeted by intruders

Syslog.conf

• Which messages are sent to which logs

• Each line contains:– Facility field – subsystem that produces the log

file• Auth(security), authpriv, cron, daemon, kern, lpr,

mail, ftp, news, syslog, user, uucp, local0-local7

– Priority field – severity of log (8 levels)• Debug, info, notice, warning, err, crit, alert, emerg

– Action field – name of log file, IP or remote syslog server

Syslog Priority Field

• Debug - all occurrences, everything• Info – usual occurrences (like fyi’s)• Notice – unusual occurrences, investigate• Warning – warning messages• Err – other error conditions• Crit – critical condition or failure• Alert – urgent situation• Emerg (panic) – panic situation (warp core

breach)

Programmer’s interface

• #include <syslog.h>• void openlog(const char *ident, int option,

int facility);– Opens a connection to the system logger for a program

• void syslog(int priority, const char *format, ...);– Generates a log message to be distributed by syslogd

• void closelog(void);– Closes the descriptor to the system logger for a

program

Sample syslog.conf

Shell Histories

• History of all commands you type• In each user’s home directory

– .history– .bash_history– .sh_history– .ksh_history

• Commonly targeted by intruders– Delete it, recreated as directory– Delete it, link it to /dev/null (bit bucket)– Just turn off history function in your shell, delete it

The grep Family• grep – search for string in file

– bzgrep - in a bzip2 compressed file

– zgrep – search possibly compressed files

– zipgrep - search files in a ZIP archive

– grepjar - search files in a jar file for a pattern

• fgrep – search for strings identified within a given file, one pattern per line– bzfgrep - in a bzip2 compressed file

• Egrep – search using extended regular expressions– bzegrep - in a bzip2 compressed file

grep Options

• -r – recursion

• -i – case insensitive

• -a – handle binary files (kind of like piping to strings)

• -v – NOT this string

find

• grep looks in files, find searches other attributes of files (metadata)– File name, including regular expressions, case

insensitive– Time periods for MAC– Belongs to GID or group’s name– Belongs to a UID or user name– Nouser and nogroup – doesn’t have a user or

group defined for its GID or UID

find

– Is on file system of type xxxx– Has a particular inode number– Has a particular number of links to it– Is a symbolic link– Search on permission bits– File size– File type

find Actions

• -print – print what you find

• -printf

• -exec xxx – execute xxx command on a hit

• -ls – list it in “ls –dils” format

• Much more stuff! Good man page to read.

Hiding in the File System

• Hide in a rarely visited or ‘busy’ directory– /dev

• Look for regular files, should be too many

– Font directories– OS source code directories– Man page directories

• Creative naming– …– “. “– “.. “– “ “

Hiding in the File System• Slack space• Deleted files• Unlinked open files• Trojaned system files• Decoy file system mounts

– Mount a file system over existing data in a current file system

– Existing data becomes hidden, could hide an executable being run or a file being written to

– df may show a lot more space used in a file system that you can account for with du

Checking RPMs

• RPM are applications packages (Linux)• Compares info about files in an installed

package with info stored about themin the RPM database

• Simple integrity check– # for i in `rpm –qa`; do rpm –V $i; done

• Error prone and can be subverted• Catches less skilled intruders

Output of Verify RPMs

• S - file Size differs• M - Mode differs, includes permissions, file type• 5 - MD5 sum differs• D - Device major/minor number mis-match• L – (readlink(2)) path mis-match• U - User ownership differs• G - Group ownership differs• T - mtime differs• c – configuration file (expected to change)

Rpm Verify Example

Inode “Timelines”

• ls –lit | sort |more

• List all inodes

• Looking for entries that seem out of place, very high or very low

• If you find any out of place, look for other inodes around that number to find possible related files

Inode “Timelines” Example

Signals

• Simple interprocess communications– One program sends a message to another– Pre-defined messages– 16 or 32 depending on platform

• Some are useful for terminating a program gracefully

• Might be able to freeze it in memory so as not to lose evidence

Useful Signals

• HUP (1) – Hangup• INT (2) – Interrupt, stop running <ctrl>C• KILL (9) – Stop unconditionally and immediately• TERM (15) – Terminate gracefully if possible• STOP (17) – Stop unconditionally; continue with

CONT• TSTP (18) – Stop executing, ready to continue• CONT (19) – Continue executing after STOP

or TSTP• USR1 (30) – A user defined signal

Startup and Shutdown Scripts

• Usually found in /etc• Can be files like rc.local and rc.shutdown• Can be directories of scripts or links to scripts like

rc0.d-rc6.d, rc.d, and init.d• The kernel boots and first loads

– init – process control initialization– If init dies, the system reboots– Makes sure the system enters the correct run level

(single user, multi-user, etc)

BSD-Like RC Scripts

• Simpler scripts:– rc.conf: configuration variables for what to start,

included in other startup scripts– Rc: starts up a bunch of system services that must be

run before securelevel changes– rc.securelevel: levels –1 through 2– rc.local: run next, local services, network, system

daemons– rc.shutdown: clean up commands when system is going

down• Ex. Gracefully stopping a databse

rc.securelevel

• Run after rc script

• Level –1: Permanently insecure– Init can’t raise securelevel but sysctl can

• Level 0: Insecure mode– During bootstrapping, single user– all devices may be read/written subject to

permissions– system file flags may be cleared

rc.securelevel• Level 1: Secure mode (default multi-user)

– Only init may lower securelevel– /dev/mem and /dev/kmem may not be written to– raw disk devices of mounted file systems are read-only– Can’t remove system immutable and append-only file

flags– kernel modules may not be loaded or unloaded

• Level 2: Highly secure mode (Level 1 still applies)– raw disk devices are always read-only, mounted or not– settimeofday(2) may not set the time backwards– ipf(8) and ipnat(8) rules may not be altered– the ddb.console and ddb.panic sysctl(8) variables may

not be raised (keeps people from using in-kernel debugger ddb(4) to modify securelevel)

System V-ish RC Scripts

• On a Solaris machine:– 8 different run levels, 0-6 and s and S (same thing)– Default runlevel in /etc/inittab

• Level s or S: single user state • Level 0: firmware mode• Level 1: sys admin mode, single user, all

filesystems mounted, limited processes running• Level 2: multi-user mode, all multiuser processes

running

Init Levels (cont.)

• Level 3: extended multiuser mode, level 2 + local resources are available over the network

• Level 4: usually not used, can ber defined as alternative multiuser environment

• Level 5: Shut the machine down, safe to power off• Level 6: stop the OS and reboot to default state

level

Startup Scripts

• There is a directory for each of the 0-6 runlevels.• /etc/rc.d/rc0.d -> /etc/rc.d/rc0.d • Also /etc/rc.d/init.d

– Contains the actual startup/shutdown scripts

– Are shell scripts that take as arguments• start – start up the process

• stop – stop the process

• restart – sometimes a restart

Startup Scripts

• Each of the rcX.d directories contain symbolic links to scripts in the init.d directory

• Format of name of link determines argument to start up script and when it is started– K03nfs

• run script pointed to by this link with the stop option (K=Kill)• Run it “third” in the order of scripts

– S75ntpd• run script pointed to by this link with the start option (S=Start)• Run it “75th” in the order of scripts

References

• Chapters 11,12