A Dynamic Visualization of Core-2 Duo Interrupts

Allan B. Cruse

University of San Francisco

17 September 2009

Organization of this Talk

• This talk is about “using the computer to study the computer”

• The focus here will be on using a recent version of the Linux operating system on your personal computer, where you can use “free” utilities -- and can exercise a few “root” privileges

Dynamic VisualizationIts General Thesis

Core-2 Duo InterruptsIts Specific Application

‘Dynamic Visualization’

• This refers to a type of computer program which can show us some volatile aspects of a machine’s inner workings in real time

• Without the ability to “see” these activities we are left simply to try and imagine what they are like -- never quite feeling certain

• Such visualizations often reveal aspects which our textbooks forgot to mention!

UNIX’s ‘top’ utility

This program lets a user see some volatile information from inside the kernel; however, it’s not really being displayed in real time since it only gets updated about once every 3 seconds – still it does help us to understand ‘timesharing’.

The software ‘architecture’

application program(e.g., ‘top’)

shared functionlibraries

(e.g., ‘printf()’)

call

ret

user space(restricted privileges)

kernel space(unrestricted privileges)

Linux operating system

syscall

sysret

hardware devices

in out

The software ‘architecture’

application program(e.g., ‘top’)

shared functionlibraries

(e.g., ‘printf()’)

call

ret

user space(restricted privileges)

kernel space(unrestricted privileges)

Linux operating system

syscall

sysret

hardware devices

in out

LKM

LKM = Linux Kernel Module

Formatting screen-output

• The Linux operating system is written in C -- plus some “inline” assembly language, and a large portion of the shared libraries and standard system utilities are in C/C++

• Using functions like ‘printf()’ you can very quickly write C/C++ code that will output data values in a humanly-readable form

• There’s similar kernel-function: ‘printk()’

The ‘teletype’ model

• But the usual way C/C++ programs output text to consoles (or to desktop windows) is unsuitable for doing dynamic visualizations

• It’s based on a software emulation of early teletype terminals, where a new character gets added at the end of earlier text, until the screen fills and is then scrolled upward

• A cursor blinks where the next text will go

Non-canonical I/O

• Our ‘dynamic’ visualizations will require us to employ ‘full-screen’ terminal-output and ‘un-buffered’ terminal-input, with no ‘echo’ of keystrokes, and with no flashing cursor or upward-scrolling screen

Hello, world_

ANSI ‘escape’ codes

• There are standard control-codes we can use in ‘printf()’ statements to achieve our ‘draw anywhere’ and ‘hide cursor’ goals:

printf( “\e[H\e[J” ); // erase the entire displayprintf( “\e[?25l” ); // make the cursor disappear

// draw a textstring at the center of the 80-by-25 screenint row = 12, column = 40;printf( “\e[%d;%dH%s”, row, column, “Hello” );

fflush( stdout ); // flush the output buffer

‘struct termios’

• We can use the ‘tcgetattr()’ and ‘tcsetattr()’ library functions to install changes to the data that controls our terminal’s behavior

kernel space

struct termios

user space

struct termios

struct termios tcgetattr()

tcsetattr()

Our ‘visualization’ application

• It consists of two ‘pieces’ of code:– a user-program, written in C++ – a kernel-module, written in C

// activity.cpp

#include <stdio.h>#include <termios.h>

int main( void ){

}

// activity.c

#include <linux/module.h>#include <linux/proc_fs.h>#include <asm/uaccess.h>

int init_module( void ){

}

void cleanup_module( void ){

}

compiled using g++ compiled using mmake

What are ‘interrupts’?

• Normally a CPU inside our computer is fetching and executing our sequence of program-instructions from a contiguous region of the system’s main memory

• But occasionally some peripheral device undergoes a change in its state, and our system needs to take note of it promptly

• An ‘interrupt’ is the signaling of that event to the CPU so it’s dealt with appropriately

Core-2 Duo

System-component overview

CPU 0 CPU 1

MCHDRAM

Graphics

ICH

Ethernet

Disk Drives

CD/DVD

Keyboard

USB Hub

Real-Time Clock

Audio

Camera

Mouse Printer

MCH = Memory Controller Hub ICH = I/O Controller Hub

Some examples

• The system needs to take note promptly when:– your keyboard has a key that’s been pressed– your mouse has been moved, or ‘clicked’– your network controller has received a packet– your internal clock’s time has been advanced – your disk controller has finished saving a file– your printer’s running low on paper or toner– your application’s ‘time-slice’ has expired

• Each of these needs a different CPU response

Interrupt ‘handlers’

• Your operating system includes all of the code-fragments for responding to any of the events that could ‘interrupt’ the CPU

• They’re called ‘Interrupt Service Routines’, and the addresses of their entry-points are stored within an array, known as the IDT, whose location and size are held in a CPU register that’s dedicated to that purpose

isr_kbd: ... iret

isr_prn: ... iret

isr_rtc: ... iret

isr_dvd: ... iret. . .

Interrupt Service Routines (aka ‘interrupt handlers’)

Interrupt Descriptor Table

Main Memory

IDTR

CentralProcessing Unit

ISRs, IDT and the IDTR

EIP

ESP

EBP

EAX

EBX

EFLAGS

. . .

‘Gate’ descriptors

• The Interrupt Descriptor Table (IDT) has room for up to 256 entries (called ‘gates’):

offset[31..16] reserved(=0)

code-segment selector offset[15..0]

type andaccess

attributes

32-bits

bytes 3,2,1,0

bytes 7,6,5,4

The peculiar arrangement of this information, in which the 32-bit offset’s value is split into a pair of non-adjacent 16-bit fields, is due to the history of Intel’s earlier processor-architecture and its commitment to ‘backward compatibility’

Some questions…

• A typical PC doesn’t have 256 peripherals attached to it! So, is the IDT-array larger than is really necessary?

• The interruption-requests coming from the various devices will be occurring at times that application-programs cannot predict! So how often will this be happening, and how likely is it to ‘degrade’ performance?

Let’s take a look

• Our ‘dynamic visualization’ will let us view all the various interrupt occurrences -- as they are happening (i.e., in ‘real time’)

This is a static screenshot of our interrupt activity ‘visualization’

Our source-code (6 pages) activity.c activity.cpp

Our SMP version

The ‘enhanced’ version of our visualization shows separate interrupt-counters for each of the two ‘logical’ processors inside our Core-2 Duo Linux platform

Core-2 Duo

APIC components

CPU 0 CPU 1

MCHDRAM

Graphics

ICHEthernet

Disk Drives

CD/DVD

Keyboard

USB Hub

Real-Time Clock

Audio

Camera

Mouse Printer

I/O-APIC

The I/O Controller Hub contains the so-called I/O-APIC whose registers control routing of Interrupt Request signals (IRQs) to specific interrupts (INTs) on one or more of the logical CPUs that are present in a system

Each CPU contains its own Local-APIC with a ‘processor-ID’ register Local-APIC Local-APIC

Linux’s interrupt assignments

0 1 2 3 4 5 6 7 8 9 A B C D E F

0x000x10 0x20 0x30 0x40 0x50 0x60 0x70 0x80 0x90 0xA0 0xB0 0xC0 0xD0 0xE0 0xF0

Reserved by Intel

Legacy PIC interrupts

Local-APIC interrupts

Assigned for use by IO-APIC interrupts or ‘Message-Signaled’ interrupts (or unused)

…but usage is usually documented in Linux’s <asm/irq_vectors.h> kernel header

The kernel developers frequently make changes to their interrupt routing for uniprocessor versus multiprocessor platforms, or 32-bit versus 64-bit

Not in use with SMP

Writing LKM code

• The first two chapters of this O’Reilly book teaches how to write Linux kernel modules

• You can use kernel ‘helper’ functions --– to allocate kernel memory: kmalloc()– to create ‘pseudo’ files: create_proc_entry()– to execute code on another CPU: smp_call_function()– to insert “inline” assembly language statements: asm()

Our module’s organization

my_read()

init_module()

cleanup_module()

Our module’s two required administrative functions

Our module’s ‘payload’of file-operation methods

my_open()

my_write()

my_fops{ } The pseudo-file’s struct of method-pointers

n_interrupts[ 256 ]

original_isr[ 256 ]

*oldidt

Our module’s ‘global’ data

*newidt . . .

several module helper-functions…

includes 256 ISR “hooks” written in in assembly language

We ‘hook’ every ISR

• We build a new Interrupt Descriptor Table whose entries point to our own set of very short interrupt-handler routines: each will simply increment a counter, then transfer control to the usual Linux interrupt-handler

• This is NOT a new idea! It’s been used in PCs for at least thirty years, although only one or two interrupts got ‘hooked’ typically

How ‘hooking’ works

original IDT

isr: ... iret

original ISR

substitute IDT

isr: ... ret

substitute ISR

IDTR

…then the substitute ISR puts the address of the original ISR on top of its stack, so that it can transfer control there merely by executing a ‘ret’ instruction

Our substitute ISR will increment the count of its previous interrupts…

n_interrupts[ … ]

…so, use a ‘repeat-macro’.text.type isr_entry, @function.align 16

isr_entry:n = 0.rept 256pushl $njmp ahead.align 16n = n+1.endr

ahead:push %ebpmov %esp, %ebppush %eaxpush %ebx

mov 4(%ebp), %ebxincl n_interrupts(, %ebx, 4)mov original_isr(, %ebx, 4), %eaxmov %eax, 4(%ebp)

pop %ebxpop %eaxpop %ebpret

We really didn’t want to type in the code for over two-hundred separate assembly language routines…

☻

Compiling and Installing

• Compiling a kernel module for Linux 2.6.x is inherently complicated, so we wrote a utility (‘mmake.cpp’) that does it easily:

$ ./mmake activity.c

• To install the compiled ‘kernel object’ in a running kernel is a step that normally will require ‘root’ privileges:

# /sbin/insmod activity.ko

Some exercises to try

• Can you modify our LKMs (‘activity.c’ and ‘smpwatch.c’) to use with a 64-bit kernel?

• Can you see what changes will be needed if you want a ‘dynamic visualization’ of all the interrupts on a multiprocessor platform with more than two CPUs? (Core-2 Quad)

• Can you imagine other kinds of ‘dynamic visualizations’ that would be enlightening?

Website resources

• You can download the source-code for all the demos discussed during this talk from this website:

<http://cs.usfca.edu/~cruse>

• You can obtain the newest versions of the Linux kernel source-code from this site:

<http://www.kernel.org>