Analyzing Malicious Windows Programs

Malicious Windows Programs

Windows API: a broad set of functionality -> governs the way malware interacts with Microsoft libraries.

Handles: items opened or created by the OS (e.g. window, process, module, menu, file) a value of some opaque type that identifies an object (the type of the

handle is unrelated to the element referenced) Can be implemented by a pointer or other than a pointer Can be thought of a “handle’’ to a door (a door has only one handle

for it to open). E.g. CreateWindowEx function returns HWND -> handle to a

window; whenever want to do anything with that window, need to use that handle.

File System Functions Important: file activity can hint what the malware does. CreateFile, ReadFile, WriteFile CreateFileMapping -> loads a file from disk into memory This function is used to create a handle to a file mapping that loads a file into memory and makes it

accessible via memory addresses. Launchers, loaders, and injectors use this function to read and modify PE files.

MapViewofFile -> returns a pointer to the base address of the mapping.

Windows Registry

Store configuration information (settings and options –networking, driver, startup, user account)

Malware often hits registry Root Keys (HKLM, HKCU, etc) HKEY_LOCAL_MACHINE (HKLM) – Settings global to the

machine HKEY_CURRENT_USER (HKCU) – Settings for current user Regedit tool for examining values

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run -> determines the executables to start when a user logs in

Registry functions Common registry functions (malware use them to modify registry) RegOpenKeyEx: open a registry for editing and querying RegSetValueEx: adds a new value and set data RegGetValue: return data for value entry.

Once clicked, it will do

Networking APIs Berkeley sockets API

socket, bind, listen, accept, connect, recv, send

WinINet API Higher-level API (stored in Wininet.dll) InternetOpen: initiate connection to the Internet InternetOpenURL: connect to a URL InternetReadFile: read data from a downloaded file

Dynamic link libraries (DLL) DLL use libraries to share code among multiple applications (code re-use

among apps including malware) Memory shared among running processes (static libs are loaded n

times for n programs, n times of memory) Minimize software distributions

Malware: hides malicious code in DLL rather than .exe uses standard Windows DLLs to interact with OS Uses third-party DLLs (e.g. Firefox DLL) to avoid re-implementing

functions

No real difference between DLL and .exe (except a flag)

Processes

Windows uses processes as containers to manage resources/resource sharing.

Processes can have the same memory address as 0x004000, but physical memory stores the data should be different.(street address without zip code)

Malware can execute code outside of current process as part of another process. Create a process to execute its malicious code (bypass firewall);

create an instance of Internet Explorer and use that program to access malicious code

Threads A process contains one or more threads Share the same memory space Each thread has its own registers and stack Complete control of the CPU (core), other threads cannot affect. Before switching between threads, all values are saved in structure

called thread context. (Possibly other threads are modifying the register)

If another thread runs in between

1. Stores EDX in thread context

2. After the other thread finishes, restore EDX from thread context

Threads Malware can Use CreateThread to load a new malicious DLL into a process

(CreateThread with address of LoadLibrary as start address)

Also used to remotely control a process Two threads created (input and output) One takes network input and sends to process stdin (via WriteFile) One takes process stdout (via ReadFile) and sends to network Send all information to a single socket to communicate seamlessly with the

running application

Mutex Mutexes: Only one thread can own a mutex at a time -> used to

coordinate multiple processes and threads Malware: often use hard-coded names Mutex name must be consistent if used by two processes Thread gains access of mutex WaitForSingleObject Subsequent threads must wait A thread finishes: ReleaseMutex CreateMutex, OpenMutex (get a handle to another process’s

mutex)

Malware will commonly create a mutex and open an existing mutex with the same name to ensure only one version of the malware is running at a time

Mutex Example (Listing 7-9)

Mutex Name

Try to OpenMutex an existing name HGL345

If NULL -> Create a Mutex Name HGL345

If NOT NULL -> Exit

Services Malware installing as services

Processes run in the background Scheduled and run by Windows service manager without user input Starts automatically with OS, may not even show up in Task Manager

as process (Malware isn’t running a separate process)• OpenSCManager(returns a handle to the service control

manager), CreateService, StartService

• Service Types: WIN32_SHARE_PROCESS = allows multiple processes to share service (e.g.

svchost.exe) or dll. WIN32_OWN_PROCESS= independent process KERNEL_DRIVER= loads code into kernel

Kernel mode/User Mode – Kernel mode direct access to I/O, hardware, all memory address, interrupt

Microsoft Component Object Model(COM)

• Interface standard that allows software components to call each other• Implemented as a client/server framework. Clients: programs making

use of COM objects; server: resusable software components. • If uses COM, thread must call OleInitialize, CoInitializeEx (need more identifiers)

• COM objects access via class identifiers (CLSIDs) and interface identifers (IIDs).

• Common function used by malware is Navigate in IWebBrowser2 interface (launch IE and access a web address) IWebBrowser2 implemented by Internet Explorer Details see Listing 7-11, 7-12 (Read p.155) –

IWebBrowser2/IE have their unique identifiers.

Com Server Microsoft Component Object Model Malware implemented as COM server within web browser via Browser Helper Objects (3rd

party plug-ins) Used to monitor traffic going through browser without creating a separate process that

can be detected (run inside IE) Can be detected via export functions (features) DllCanUnloadNow, DllGetClassObject, DllInstall, DllRegisterServer, DllUnregisterServer

Exceptions

• Allow program to handle exceptional conditions during program execution

• Transfer to a special routine to resolve the exception– Windows Structured Exception Handling

• Exception handling information stored on stack before function invocation

• Not all handlers respond to all exceptions• Thrown to caller's frame if not handled• If none responds, top-level handler crashes

– Used by malware to hijack execution• pointer to exception is stored on stack• attacker can overwrite the pointer in stack overflow• Attacker then triggers exception to gain execution

Kernel Mode Two privilege levels: kernel mode and user modeNearly all code (applications) runs in user mode, except driversUse Windows API to manipulate kernel structures Kernel modeShared resources/memory addressesFewer security checks Invalid -> blue screenAntivirus/firewall->kernel modeOnly sophisticaed malware runs in kernel

Native API Popular among malware – bypass normal Windows API

Ntdll.dll: manages interactions between user space and kernel

Processor then switches to kernel mode

Ntdll uses the Native API

Separation allows Microsoft to changekernel without affecting applications

Native APIMalware often call ntdlldirectly to avoid detection of security programs between kernel32.dll and ntdll.dll

Malware not calling Windows API (ReadFile, WriteFile) but calling Native API (NtReadFile, NtWriteFile) instead

Well-designed security program will monitor calls at all levels including the kernel