Chapter 7

Securing Commercial Operating Systems

Chapter Overview

• Retrofitting Security into a Commercial OS

• History of Retrofitting Commercial OS's

• Commercial Era

• Microkernel Era

• UNIX Era

– IX

– Domain and Type Enforcement

– Recent Unix Systems

• Summary

Retrofitting Security into a Commercial OS

• Requires reference Monitor Concept

– Complete Mediation

– Tamperproofing

– Verifiability

Complete Mediation Challenges

• Identify all security-sensitive operations

– Some embedded deep inside the kernel code.

– Examples:

• Open

• Sockets

• Shared memory, etc.

– Covert channel identification is usually not even attempted

Tamperproofing Challenges

• Obvious: place in ring 0, but

• Kernel is often updated.

• /dev/kmem, /proc, Sysfs, netlink sockets

• → Every root process must STILL be part of the UNIX TCB

Verification Challenges

• Musts:

– Mediation is implemented correctly, but

• Mediation interface designed manually

• Implemented in unsafe languages

– Policy enforces required security goals

• Large number of queries and processes.

• Complicate policies.

– Reference monitor implementation is correct

• Rest of TCB is huge.

– Rest of the TCB behaves correctly.

History of Retrofitting Commercial OS's

• Commercial Era

– Emulate system on security kernel

– Retrofit security into OS

– → UNIX MLS

• Microkernel Era

– Independent Server Processes → went to kernel

– New security models addressing both confidentiality and integrity

• Unix Era

– Composed solutions from the two eras with focus on system integrity.

Commercial Era

• Emulated Systems

– Data Secure UNIX

– KSOS

• KVM/370 – 25% Performance overhead

• VAX/VMS DEC/Sandia Labs: MLS

• Secure Xenix (IBM) Access control and auditing

– Added Compatibility

• Retrofitted Unix services

• Hidden subdirectories – Polyinstantiated file systems

– Trusted Path (Secure attention sequence)

• 1990 saw many secure Unix variants

Microkernel Era

• Goal: minimal size kernel emphasizing system abstractions; no emphasis on security per se.

• Source: Mach (1980's)

– Trusted Mach (Tmach)

– Distributed Trusted Mach (DTMach)

– Distributed Trusted OS (DTOS)

– Flask

Trusted Mach

• Built by Trusted Information Systems (TIS)

• Added MLS for files, memory.

• Aim was to provide function for other systems like Unix and Windows. (Single level)

Distributed Trusted Mach

• Secure Computing Corporation and NSA

• Hybrid access control model:

– MLS labels for confidentiality

– Type Enforcement labels for integrity (TE)

• Similar architecture to Tmach + servers for networking and general security policy server.

DTMach II

• DTMach = Mach + security server

– Security server = reference monitor outside the kernel

• Each port access implies an authorization query

• For example, opening a file opens a port to the file server, etc.

– Security server used both MLS and TE rules.

• TE rules:

– code could only be modified by administrators– Limited code that could be executed

• There were limitations:

– For example, there was an arbitrary send right...

Distributed Trusted OS (DTOS)

• AIM: True reference monitor in the Mach microkernel.

• Richer set of operations for ports than just send.

• Microkernel:

– Managed labeling of subjects and kernel objects.

– Mediated each kernel operation by querying security server.

• Focus on verifiability of microkernel and TCB.

Flask

• Fluke was a second generation microkernel developed at University of Utah, better than Mach.

• Flask = DTOS – Mach / Fluke

• More emphasis on TE.

UNIX Era

• By early 1990's, many Unices had MLS support.

• Search for adding integrity (very ad-hoc at this point).

• Cover two systems:

– IX

– DTE

IX

• AT&T prototype, enforces MLS and integrity.

• Includes a reference monitor over file access

• Mandatory access control policy providing both confidentiality and integrity protections.

• Care has been taken to prevent tampering in the TCB.

• Verification not a goal.

• MLS was high water mark, for files and processes. However processes could not go beyond a certain ceiling.

IX (2)

• Integrity was LoMac, with floors.

• Since levels are dynamic, each data transfer must be checked/mediated.

• No memory-mapped files.

• Trusted paths/pipes between processes (pex); a pex includes a label for the process at each end so that only that process may work with it.

An assured pipeline in IX

Domain and Type Enforcement

• Trusted Information Systems:

• Problem: protecting TCB from vulnerable root processes

• Runs on Tmach system, but reference monitor added to OSF/1.

DTE Policy Model

• Subject types are now called Domains, object types are still types.

• Each domain is a triple (access rights to objects, access rights to subjects in other domains (signals), entry point program)

• A domain describes how a process accesses files, signals other processes and creates processes.

• DTE Unix defines limited protection domains for root processes. Key point is “least privilege”.

• Domain transitions are limited and their execution is limited also.

• Labeled Networking.

Recent Unix Systems

• BSD variants

– Trusted BSD

• MAC, auditing, authentication

• Reference monitor interface similar to LSM

• SEBSD is a version of SELinux for BSD

– FreeBsd Jail

– OpenBSD emphasizes correct coding and configuration

• Code separation

• Buffer overflow protection

• Least privilege configurations

– NetBSD

• In-kernel authentication and verification of file execution

• Veriexec

Summary

Retrofitting Security into a Commercial OS

- Requirements and Challenges

- History

• Commercial Era• Microkernel Era• Unix Era – recent Unix variants