THERE’S GOLD IN THEM THAR PACKAGE MANAGEMENT DATABASES!

Phil Hagen phil@lewestech.com @PhilHagen plus.google.com/+PhilHagen

Image: flickr.com/matthigh

WHY ARE WE HERE?

Goals:

Describe the value of package management databases during the course of a Linux system forensic examination

Provide information with immediate benefit!

���2

WHO IS THIS GUY?

Forensic/infosec consultant

Former DoD/IC contractor, USAF Comm Officer (USAFA CompSci)

Course lead, FOR572: Advanced Network Forensics & Analysis

Linux guy since Slackware needed a stack of floppies and an unsupported SCSI controller meant bootstrapping a kernel compilation

���3

OPEN SOURCE: INFINITE POWER (ITTY BITTY LIVING SPACE)

Downloading and compiling source code is an amazingly powerful aspect of most open-source software

Not viable for the large scale of a production environment

Does not address dependencies

No common install/uninstall process

Fun for the lab or a hobbyist environment, but a headache in an operational environment

���4

PACKAGE MANAGEMENT SOFTWARE TO THE RESCUE!

Dependencies, file manifests, install/uninstall/upgrade scripts

Generally makes software management less of a headache!

Many solutions in widespread use among various distributions:

opkg (fork of ipkg): Embedded devices like QNAP NAS, etc.

dpkg: Debian and Ubuntu

tgz: Slackware

RPM: RedHat, CentOS, Fedora,many more(part of Linux Standard Base)

���5

…STILL NOT END-ALL/BE-ALL

Most incorporate higher-layer software to handle automatic inclusion of dependencies and other “meta” functions and avoid the much-feared “dependency hell” or “RPM hell”

apt-get / aptitude

YUM

RedHat Network

Not focusing on this higher-layer software

Some useful artifacts available - bad guys known to “yum install nmap”

���6

SCOPE FOR THIS PRESENTATION

Just looking at RPM

Most concepts apply to other package management standards

Notably dpkg

All examples created and tested on CentOS 6.5

Ideally: Same OS as subject (RPM library versions, etc)

Technically: Same version of RPM, BDB, related libraries

May have some success with unmatched versions, but beware!!

Presentation notes (including all commands) published soon

http://stuffphilwrites.com���7

SOURCES FOR RPM EVIDENCE

Populates Berkeley database

Lives in /var/lib/rpm/

Includes metadata for every RPM-controlled file

User/Group ownership

Mode (aka permissions)

MD5 (er… SHA256) checksum

File size

Major/minor number (For entries in /dev/)

Symbolic link string (aka “target”) for symlinks

Modification time /var/log/yum.log

May have useful timestamped history of install/remove/upgrade actions

/var/log/rpmpkgs*���8

PRACTICAL USAGE

1. Mount all partitions from subject filesystem under /mnt/subject/

$ mount | grep subject /dev/mapper/vg_centos6vm-lv_root on /mnt/subject type ext4 (ro,noload) /dev/mapper/loop0p1 on /mnt/subject/boot type ext4 (ro,noload)

2. Run RPM commands with “--root /mnt/subject” option

Don’t trust the rpm(1) binary from a suspect system!

Note: This performs a chroot(2) for all operations - needs root user privileges for validation (not query) actions

���9

USE CASE:FILE ORIGIN$ rpm -qf <filename>

Identifies what package owns the specified file

Useful to answer “where did this file come from?” or to identify a file as package-less

$ rpm --root /mnt/subject -qf /usr/sbin/sshdopenssh-server-5.3p1-94.el6.x86_64

$ rpm --root /mnt/subject -qf /etc/mail.rcmailx-12.4-7.el6.x86_64

$ rpm --root /mnt/subject -qf /etc/crypttabfile /etc/crypttab is not owned by any package

���10

Remember chroot!

USE CASE:PACKAGE VALIDATION (1)$ rpm -V <packagename>

Verifies contents of specified package

Compares expected (database) to actual (filesystem)

Displays files that failed >=1 check, noting which checks failed

SM5DLUGT (Size, mode, MD5 (or SHA256!), major/minor nos, link target, user, group, mtime)

Shows “?” in output if user running command lacks permission to check (e.g. read access to generate checksums)

���11

USE CASE:PACKAGE VALIDATION (2)

WARNING!!! Packages can include “verification scripts” which will execute when the “-V” option is used!

You’re not planning to run arbitrary, unknown code on your forensic workstation/VM, are you?

The chroot action needs root - these will execute as a child to the sudo process!

Use the “--noscripts" option with “-V” to prevent this

���12

USE CASE: PACKAGE VALIDATION (3)

Some files are expected to change after installation: config files!

Still show changes after installation, but denoted with a “c” character

Missing files also noted in output as such

$ sudo rpm --root /mnt/subject \ -V openssh-server --noscripts <no output>

$ sudo rpm --root /mnt/subject \ -V sudo --noscripts S.5....T. c /etc/sudoers

���13

USE CASE: PACKAGE VALIDATION (4)$ sudo rpm --root /mnt/subject -Va --noscriptsS.5....T. c /etc/sudoers .M....... /proc S.5....T. c /etc/maven/maven2-depmap.xml ....L.... c /etc/pam.d/fingerprint-auth ....L.... c /etc/pam.d/password-auth ....L.... c /etc/pam.d/smartcard-auth ....L.... c /etc/pam.d/system-auth ..5....T. c /usr/lib64/security/ classpath.security

$ rpm -root /mnt/subject —V postfix --noscriptsmissing c /etc/postfix/master.cf

���14

COOL FEATURE ALERT!

Many packages are GPG-signed

Independently verify package without using compromised or untrusted system

However: RPM database contents not signed: Trojaned RPM package reports no anomalies!

How can we use this great feature to our advantage?

���15

VALIDATE FILESYSTEM AGAINST A PACKAGE FILE (1)

The “-p” option runs validation checks between filesystem contents and RPM package file contents

Signed package files can be GPG-verified!

Avoids an untrusted RPM database entirely

Prevents false negative validation from compromised RPM installations

Relatively simple process:

1. Download trusted binary RPM file

2. Validate RPM file using GPG

3. Validate filesystem contents against package contents���16

VALIDATE FILESYSTEM AGAINST A PACKAGE FILE (2)

Consider a system with a compromised RPM database, or a trojaned installation of the Apache web server software

$ rpm --root /mnt/subject -V httpd --noscripts S.5....T c /etc/httpd/conf/httpd.conf

$ wget "http://mirror.centos.org/centos-5/5.10/os/x86_64/CentOS/httpd-2.2.3-82.el5.centos.x86_64.rpm"$ rpm -K httpd-2.2.3-82.el5.centos.x86_64.rpm httpd-2.2.3-82.el5.centos.x86_64.rpm: (sha1) dsa sha1 md5 gpg OK $ rpm --root /mnt/subject -Vp httpd-2.2.3-82.el5.centos.x86_64.rpm --noscriptsS.5....T c /etc/httpd/conf/httpd.conf S.5....T /usr/sbin/httpd

���17

REAL-WORLD USE CASES (1)

Find all non-config files owned by an RPM that fails a verification check

$ sudo rpm --root /mnt/subject -Va --noscripts | \ grep -v " c " ... S.5....T /var/www/awstats/lang/awstats-tt-tr.txtS.5....T /var/www/awstats/lang/awstats-tt-tw.txtS.5....T /var/www/awstats/lang/awstats-tt-ua.txt.......T /var/www/awstats/lib/blacklist.txtS.5....T /var/www/awstats/lib/browsers.pmS.5....T /var/www/awstats/lib/browsers_phone.pm...

���18

REAL-WORLD USE CASES (2)

File only config files owned by an RPM, which fail checksum verification

$ sudo rpm --root /mnt/subject -Va --noscripts | \ grep "^..5..... c " S.5....T c /etc/pam.d/sshd S.5....T c /etc/ssh/sshd_config S.5....T c /etc/openldap/slapd.conf S.5....T c /etc/sysconfig/ldap S.5..... c /etc/sysconfig/saslauthd S.5..... c /etc/security/limits.conf S.5....T c /etc/logrotate.conf SM5....T c /etc/snmp/snmpd.conf S.5....T c /etc/sysconfig/snmpd.options ...

���19

REAL-WORLD USE CASES (3A)

Find all files not owned by an RPM (This is going to be slow!)

$ cat find_orphans.sh for file in $( sudo find /mnt/subject/etc -type f ); do file=$( echo "$file" | sed -e 's/\/mnt\/subject//' ) rpm --root /mnt/subject -qf "$file" 2>&1 | \ grep 'package$\|directory$' | \ sed -E 's/^(error: )?file (.*)(: No such file or directory| is not owned by any package)/\2/'done

$ ./find_orphans.sh /etc/crypttab /etc/sysconfig/network /etc/sysconfig/keyboard /etc/sysconfig/iptables ...

���20

REAL-WORLD USE CASES (3B)

Find all files not owned by an RPM (This is going to be slow!)

$ ./find_orphans.sh /etc/crypttab /etc/sysconfig/network /etc/sysconfig/keyboard /etc/sysconfig/iptables ...

���21

COOL PARTY TRICK (1)

!

Use the “--queryformat” option to output only relevant/useful fields from the RPM database

Provides >150 different tags that can be output for package or for each file in a package

Available tags vary by version - online documentation is terrible Use “--querytags” for listing specific to your version of RPM

!

Consider “RPMDBtoTimeline”…���22

COOL PARTY TRICK (2)*

$ for pkg in $( rpm --root /mnt/subject -qa ) ; do rpm --root /mnt/subject -q $pkg --queryformat \ "[%{FILEDIGESTS}|%{FILENAMES}|0|%{FILEMODES:perms}| %{FILEUSERNAME}|%{FILEGROUPNAME}|%{FILESIZES}|0| %{FILEMTIMES}|0|0\n]" | sed -e 's/^|/0|/' \ -e 's/|0|d/|0|d\/d/' -e ’s/|0|-/|0|r\/r/' \ done ... 4398551f...a13988cf|/usr/share/doc/gamin-0.1.10/callbacks.gif|0|r/rrw-r—r—|root|root|4514|0|1183556209|0|00|/usr/lib64/libmenuw.so.5|0|lrwxrwxrwx|root|root|15|0|1282146079|0|0 0|/usr/share/cracklib|0|d/drwxr-xr-x|root|root|4096|0|1308983949|0|0 ...

���23

* “Cool” claim not valid at all parties. YMMV.

IN CONCLUSION

RPM is a pretty cool way to eliminate known files from a Linux system examination

Know the shortcomings in the RPM package database so you can mitigate them

With a little shell scripting, you can develop useful tools toquickly and consistentlyminimize input data