storage and file systems data timeline and recovery · 2019-11-04 · ntfs partition 2 t deleted...
TRANSCRIPT
4. 11. 2019
1
Storage and file systems
Data timeline and recovery
GOPAS: [email protected] | www.gopas.cz | www.facebook.com/P.S.GOPAS
Ing. Ondřej Ševeček | GOPAS a.s. |
MCM:Directory | MVP:Security | CEH | CHFI | CISA | CISM | CISSP |
[email protected] | www.sevecek.com |
Recap
4 B ~ max 2^32
128 b = 16 B
low-endian
• 658026814 = 0x2738b13e => 3E B1 38 27
big-endian
• 658026814 = 0x2738b13e => 27 38 B1 3E
base64
• 41-68-6F-6A-20-4F-6E-64-C5-99-65-6A-69
• QWhvaiBPbmTFmWVqaQ==
4. 11. 2019
2
PowerShell brief
250MB
3221225472 / 3GB
0x1C6EE88F
'{0:X8}' -f 477030543
Storage and disk formats
4. 11. 2019
3
Storage
JBoD (just a bunch of disks)
• single physical disk
disk array + RAID
• usually redundancy
• many disks virtualized as several independent disks
slices, RAIDs, virtual disks, ...
Windows: LDM, Storage Spaces
addressing
• cylinder - head - sector (CHS)
• logical block address (LBA)
Disk array 1 Disk array 2
RA
ID m
n
RA
ID x
y
Disk arrays
RA
ID a
b
4. 11. 2019
4
RA
ID m
n
RA
ID x
y
Disk arrays
RA
ID a
bslice
disk 2
slice
disk 1
disk 3
disk 4
Windows file systems
FAT
FAT32
ExFAT
CDFS
UDF
NTFS
ReFS
gwmi Win32_Volume
4. 11. 2019
5
Windows and Hyper-V virtual hard disks
VHD
• <= 2 TB
• 512 B sector
VHDX
• newer format
• <= 64 TB
• 512 B or 4096 B sector
• TRIM/UNMAP
partition driver, format, NTFS, ReFS reclaiming and zeroing
freed storage
Disk formats
diskette or USB flash by default
• single partition boot sector
MBR
• 4 partitions, <= 2 TB
• BIOS boot
GPT
• 128 partitions, <= 18446744073709551616 sectors
• UEFI boot
MBR + LDM
GPT + LDM
4. 11. 2019
6
GPT disk
Disk formats
MBR disk
Partition 1
BOOTMBR
Partition 2
BOOT
Partition 1
BOOT
MBR
protec
tive
Partition 2
BOOTGPT
diskette, flash, CD/DVD
Partition 1
BOOT
Master Boot Record (MBR)
0th sector
4 byte random signature
4 primary partitions• 32bit = 4 bytes = offset = start at most at 2 TB
• 32bit = 4 bytes = length = max 2 TB length
bootstrap assembler code
sig 55AA
active partition (0x80)
standard data partition NTFS, ReFS (0x07)• even the System Reserved is a normal partition formatted with NTFS
standard data partition FAT32 (0x0C)
extending partition (0x0F)
GPT protective partition (0xEE)
LDM protective partition (0x42)
4. 11. 2019
7
MBR boot with the old BIOS
MBR disk
active partitionBOOTMBR
OS partition any FSBOOT
boot code
boot code bootmgr
ntldr
BCD
boot.ini
C:\Windows\winload.exe
C:\Windows\ntoskrnl.exe
BIOS
Globally Unique Identifier (GUID)
128bit =16 bytes
randomly generated, not registered
example:{185656d3-32af-43ee-a28c-281e0efd2201}
D3-56-56-18-AF-32-EE-43-A2-8C-28-1E-0E-FD-22-01
LDAP, PKI, GPT, volume, shadow copy, ...
4. 11. 2019
8
GUID in PowerShell
$ng = [Guid]::NewGuid()
$ng
[BitConverter]::ToString($ng.ToByteArray())
$rxGUID = '[\dA-Fa-f]{8}(?:-[\dA-Fa-f]{4}){3}-[\dA-Fa-f]{12}'
$ng.ToString('B') -match "\A\{$rxGUID\}\Z«
# from byte[] sequentially as in memory
New-Object guid @(,([byte[]] (0x33, 0x20, 0xA8, ...)))
GUID Partition Table (GPT)
0th sector protective MBR partition type 0xEE• signature + 2 TB
1th sector GPT
128 partitions• 64bit = 8 byte = start offset in LBA => 8.5 MLD TB
• 64bit = 8 byte = end offset in LBA => 8.5 MLD TB
{EBD0A0A2-B9E5-4433-87C0-68B6B72699C7}• Microsoft Basic Data Partition (BDP)
• normal formatted data FAT/NTFS
{E3C9E316-0B5C-4DB8-817D-F92DF00215AE}• Microsoft Reserved Partition (MSR)
• space reservation, no format, will be replaced later if necessary
• invisible from diskmgmt.msc, visible from diskpart
{DE94BBA4-06D1-4D40-A16A-BFD50179D6AC}• Microsoft WinRE Recovery Partition
• bootable Windows Recovery Environment
{C12A7328-F81F-11D2-BA4B-00A0C93EC93B}• EFI System Partition
• boot-loader on UEFI firmware platforms
• must be FAT32 formatted
• cannot assign letter
4. 11. 2019
9
GPT boot with the new UEFI firmware
GPT disk
EFI system partition FAT32BOOTGPT
OS partition any FSBOOT
bootmgr
BCD
C:\Windows\winload.exe
UEFI
protec
tive
MBR
signature
checks with
Secure Boot
FS and boot sector notes
(ex)FAT(32)• 1x boot sector
• 1x directory (names)
• 2x FAT (chained allocations) sector offset of FAT1 in field called reserved sectors
• volume signature
• volume name <= 10 chars ASCII
• dirty bit
NTFS• 2x boot sector (represented as $boot in MFT)
• 2x MFT (names + allocated data runs)
• volume signature
• volume name in $volume
• dirty bit in $volume
• permissions in $secure
• transaction log $logfile boot recovery
• free clusters in $bitmap
• USN journal in $extend
4. 11. 2019
10
7 7
Cluster vs. sector
1 2 3 4 5 6 7 87 8
legitimate data
NTFS offset 32bit
maximum cluster size 64kB
=>
2^32 * 64kB = 256 TB volume
NTFS offset 32bit
default cluster size 4kB
=>
2^32 * 4kB = 16 TB volume
Windows Mount Manager
HKLM\System\MountedDevices
• no Last Known Good backup
letter ~ partition mapping
• MBR - disk signature + partition start [B]
• GPT - partition GUID
• removable USB flash, CD, DVD - PNP device ID
4. 11. 2019
11
Disk, volume and partition serial numbers found and used
in system
Disk hardware serial numbergwmi Win32_DiskDrive | select SerialNumber
• not guaranteed to be unique
• not guaranteed to be present at all
Volume serial numberdir
gwmi Win32_Volume
MBR• disk signature 4B
gwmi Win32_DiskDrive | select Signature
GUID• disk GUID not used
• partition GUIDs
Flush file system volume caches
sync tool from sysinternals
disk.sys
FS
IO manager
kernel
cache
manager
write
always async
thread until 1809
flash drives
user
4. 11. 2019
12
disk
Deleted partition example
(UNMAP does not hit yet)
partition 1
visible
partition 3
visible
NTFS
partition 2
deleted
MB
R o
r G
PT
boot
boot
back
MFT
MFT2
data
data
data
disk
visible overlapping partition
Deleted partition example
(UNMAP hits during the red format)
partition 1
visible
partition 3
visible
NTFS
partition 2
deleted
MB
R o
r G
PT
boot
boot
back
MFT
MFT2
data
data
data
boot
back
boot
MFT
MFT2
data
data
data
data
4. 11. 2019
13
disk
visible overlapping partition
Deleted partition example
partition 1
visible
partition 3
visible
NTFS
partition 2
deleted
MB
R o
r G
PT
boot
boot
back
MFT
MFT2
data
data
data
boot
back
boot
MFT
MFT2
data
data
data
data
Dynamic Disks aka Logical Disk Manager (LDM)
on both MBR and GPT disks
on MBR disks
• a single protective MBR partition type 0x42
• last 7 MB of the disk contains LDM tables in un-partitioned
space
on GPT disks
• LDM Metadata Partition (1 MB)
{5808C8AA-7E8F-42E0-85D2-E1E90434CFB3}
• Microsoft Reserved Partition
• LDM Data Partition (the rest of the disk)
{AF9B60A0-1431-4F62-BC68-3311714A69AD}
4. 11. 2019
14
Storage Spaces
Lower-level implementation of software RAID
Windows Server 2012+
File systems
4. 11. 2019
15
NTFS notes
timestamps
Get-Item c:\theFile.txt | select *time | fl *
fsutil behavior query disablelastaccess
resident vs. non-resident data
• whole allocation in MFT
• allocation in MFT in the form of data runs
• TRIM/UNMAP on supporting storage
Common application write handling
original.docx
savedIntoTemp~.do~
backupedUpOriginal~.do~
Application
original.docx
read
write
rename
rename
delete
4. 11. 2019
16
Files in PowerShell
$file = New-Object System.IO.FileStream 'size.txt', 'Create',
'Write', 'ReadWrite'
$file.SetLength(800)
$file.Seek(0, 'Begin')
$bytes = [System.Text.Encoding]::ASCII.GetBytes('some text')
$file.Write($bytes, 0, $bytes.Length)
$file.Close()
# Make it smaller without zeroing the data manually
$file = New-Object System.IO.FileStream 'size.txt', 'Open',
'Write', 'ReadWrite'
$file.SetLength(100)
7 7
Slack space - statically hidden data
1 2 3 4 5 6 7 87 8
legitimate data
never read/copied by FS
properties of slack-space:
on TRIM/UNMAP zeroed by storage when cluster released
file length shortening inside a cluster may leave content
other hiding possibilities:
non-allocated space (may contain some old data)
falsely marked bad clusters
inter-partition space (may contain some old data)
zeroed on write operations
deleted file portions on
nonUNMAP media
4. 11. 2019
17
NTFS deleted files
MFT FILE record marked as free
• reused as soon as possible
resident data remains until FILE reused
non-resident data
• allocation remains in MFT
• marked as free in $BitMap
• TRIM/UNMAP immediately
(ex)FAT(32) notes
no internal record change date
• only CreationTime, LastAccessTime, LastWriteTime
deleting from Directory with E5
• short names always created
• long names
if present split on multiple entries
E5 does not overwrite the first letter
chained allocation in FAT
4. 11. 2019
18
(ex)FAT(32) chained allocation
next is 38theFile.txt
next is 39
next is 40
next is 41
next is 42
next is 2085
next is 2086
next is 2087
next is 2088
last
37
(ex)FAT(32) chained allocation
next is 38E5heFile.txt
next is 39
next is 40
last
next is 42
next is 2085
next is 2086
next is 2087
next is 2088
last
37
next is 37
4. 11. 2019
19
NTFS special objects
hard link• more names for a single "file object"
• used for short names if generated
junction (mount point)• special "file object" reparse point containing a link reference as data
• cannot point to any other system than local volumes
• R2L processed by the local system when accessed remotely
symbolic (soft) link• special "file object" containing reparse point information
• can point to any other system namespace object including SMB etc.
• R2L parsed by the remote system when accessed remotelyfsutil behavior query symlinkevaluation
fsutil behavior set symlinkevaluation R2L:1
fsutil behavior set symlinkevaluation R2R:1
sparse files• zeroed sections are not allocated and have 0 bits in $bitmap
• summing file sizes may yield more than is volume size
• NTFS is guaranteed to never return the real data regardless (non)sparse nature
alternative data streams
NTFS in CMD
fsutil hardlink list c:\windows\system32\kernel32.dll
fsutil reparsepoint query "%userprofile%\application data"
mklink /D c:\symlink c:\windows
fsutil reparsepoint query c:\symlink
mklink /J c:\junction c:\windows
fsutil reparsepoint query c:\junction
echo some visible text > alternate.txt
echo some hidden text > alternate.txt:HiddenData
more < alternate.txt:HiddenData
sysinternals\streams.exe
fsutil file createnew sp.txt 0x100000000
fsutil sparse setflag sp.txt 1
# Note: only marks an existing data range as sparse
fsutil sparse setrange sp.txt 0 0x100000000
4. 11. 2019
20
Junctions vs. symbolic links
local systemremote system
c:\junction
c:\windows
c:\symlink
C:\Windows
\\system\c$\junction
\\system\c$\symlink
C:\Windows
Symbolic links resolved locally in kernel
local systemremote system
c:\symlink
c:\windows
c:\UNCsymlink
\\data\Doc
\\system\c$\junction
\\system\c$\UNCsymlink
c:\windows
4. 11. 2019
21
volume D:\NTFS "FILE"
FILE record
timestamps
attributes
name1.txt folder1
name2.txt folder2
name3.txt folder3
security descriptor
data DEFAULT
data NAMED1
data NAMED2
volume C:\
volume D:\
symlink/junction
FILE record
timestamps
NTFS "FILE"
FILE record
timestamps
attributes
name1.txt folder1
name2.txt folder2
name3.txt folder3
security descriptor
data DEFAULT
data NAMED1
data NAMED2
security descriptor
4. 11. 2019
22
Start process from NTFS ADS
$exePath = 'T:\sevecek\TestWinApp-40-any.exe'
$hiddenExe = 'C:\temp\calc.txt'
$hiddenExeADS = '{0}:EXEfile' -f $hiddenExe
if (Test-Path $hiddenExe) { Remove-Item $hiddenExe -Force }
New-Item $hiddenExe -ItemType File -Force | Out-Null
$sourceBytes = [System.IO.File]::ReadAllBytes($exePath)
Set-Content -Path $hiddenExe -Value 'Calculate something' -Encoding
UTF8 -Force
Set-Content -Path $hiddenExeADS -Value $sourceBytes -Encoding byte
-Force
(gwmi -List Win32_Process).Create($hiddenExeADS)
Folder with data ADS
$theFolder = 'C:\TEMP\folderWithADS'
$theFolderADS = '{0}:SECRET' -f $theFolder
if (Test-Path $theFolder) { Remove-Item $theFolder -Recurse
-Force }
New-Item $theFolder -ItemType Directory -Force | Out-Null
Set-Content $theFolderADS -Value 'Hello' -Encoding utf8
4. 11. 2019
23
Start process from folder with ADS
$exePath = 'T:\sevecek\TestWinApp-40-any.exe'
$theFolder = 'C:\TEMP\folderWithADS'
$theFolderADS = '{0}:SECRET' -f $theFolder
if (Test-Path $theFolder) { Remove-Item $theFolder -Recurse -Force }
New-Item $theFolder -ItemType Directory -Force | Out-Null
$sourceBytes = [System.IO.File]::ReadAllBytes($exePath)
Set-Content -Path $theFolderADS -Value $sourceBytes -Encoding byte -Force
(gwmi -List Win32_Process).Create($theFolderADS)
NTFS transaction log
low-level metadata changes
• no file name usually
circular logging
• short time only until commit
4. 11. 2019
24
NTFS change journal
USN change journal
• file ref#
• enabled on system volumes by default
• file names
• growths with new allocations, older records left on
nonTRIM/UNMAP media
fsutil usn queryjournal c:\
fsutil usn enumdata
fsutil file queryfilenamebyid c:\ 0x...
(ex)FAT(32) notes
short names vs. long names
• 8.3
• short names "deleted" with 0xE5, long name remains if
present
• long name trick - 0x0F volume label, system, R/O, hidden
no transaction log
no journal
longer file name life than NTFS
4. 11. 2019
25
Mounted VHDs or ISOs or volume mount points
NTFS junction
when removed, may remain as deleted FILE records
in MFT
Volume Shadow Copy (VSS)
Copy-on-write partition block backup
System Volume Information• can be file imaged to another computer
• can be on another volume or even disk
• machine independent
• stores copies up to a limit or free disk space
• automatic copy deletion when necessary
Used by• backup software, Windows Server Backup (wbadmin)
• System Restore, System Protection Windows Installer, updates, ...
Some files excluded• HKLM:\System\CCS\Control\BackupRestore\FilesNotToSnapshot
• DisableUserProfileExpansion
4. 11. 2019
26
VSS in CMD
vssadmin
diskshadow
list shadows all
set context persistent
begin backup
add volume C:
add volume E:
create
end backup
expose {guidguid-guid-guid-guid-guidguidguid} X:
delete shadows all
mklink /J C:\mountedVSS
\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy35\
VSS in PowerShell
(gwmi -List Win32_ShadowCopy).Create('C:\', 'ClientAccessible')
gwmi Win32_ShadowStorage |
select *,
@{ n = 'DriveLetter' ; e = { ([wmi] $_.Volume).DriveLetter
} },
@{ n = 'DiffVolumeLetter' ; e = { ([wmi]
$_.DiffVolume).DriveLetter } }
gwmi Win32_ShadowCopy |
select *,
@{ n = 'DriveLetter' ; e = { (gwmi -Query ('SELECT * FROM
Win32_Volume WHERE DeviceId = "{0}"' -f $_.VolumeName.Replace('\',
'\\'))).DriveLetter } }
4. 11. 2019
27
DefineDosDevice() kernel symbolic link
$src = @'
namespace Sevecek {
using System.Runtime.InteropServices;
public class Win32Api {
[DllImport("kernel32.dll", SetLastError = true, PreserveSig = true,
CharSet = CharSet.Auto)]
public static extern bool DefineDosDevice(uint dwFlags, string
lpDeviceName, string lpTargetPath);
}
}
'@
if (-not ('Sevecek.Win32Api' -as [Type])) { Add-Type $src }
[Sevecek.Win32Api]::DefineDosDevice(0, 'S:',
'\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy38')
System Namespace
Named kernel objects• managed by a kernel driver .SYS
SysInternals WINOBJ
\\?\GLOBALROOT\• \GLOBAL??\C:\test.txt => \Device\HarddiskVolume3\test.txt
• \GLOBAL??\Volume{GUIDGUID-GUID-GUID-GUID-GUIDGUIDGUID}\test.txt => \Device\HarddiskVolume4\test.txt
• \GLOBAL??\UNC\gps-data\Doc\test.txt = \Device\MUP\gps-data\Doc\test.txt
• \SystemRoot\windowsupdate.log
• \Device\CdRom1
• \GLOBAL??\PhysicalDrive1 = \Device\Harddisk1\DR1
• \GLOBAL??\COM1 = \Device\Serial1
4. 11. 2019
28
Volume{GUID}
GPT partitions
• volatile
• \GLOBAL??\Volume{partition-GUID-from-GPT}
MBR partitions
• volatile
• \GLOBAL??\Volume{MBR-signature-0000-0000-0000-byte-
Offset}
CDROM, USB flash, floppy
• permanent by PnP Device Instance ID
• HKLM\System\MountedDevices
Security ID (SID)
EMPTY• S-1-0-0
BUILTIN• S-1-5-32-544 = Administrators
• S-1-5-32-545 = Users
NT AUTHORITY• S-1-5-18 = SYSTEM
• S-1-5-20 = Network Service
• S-1-5-11 = Authenticated Users
account database SIDs• S-1-5-21-numberA-numberB-numberC-RID
local SAM• HKLM\SAM\SAM\Domains\Account - V=REG_BINARY
• 12B at the end
domain NTDS.DIT Active Directory database
4. 11. 2019
29
SID in PowerShell
'S-1-5-18', 'S-1-5-32-544' | Select @{ n = 'SID' ; e = { $_ } }, @{
n = 'Name' ; e = { (New-Object
System.Security.Principal.SecurityIdentifier
$_).Translate([System.Type]::GetType('System.Security.Principal.NTA
ccount')).Value } }
'Administrators', 'NT AUTHORITY\Network Service' | Select @{ n =
'Name' ; e = { $_ } }, @{ n = 'SID' ; e = { (New-Object
Security.Principal.NTAccount
$_).Translate([Security.Principal.SecurityIdentifier]).Value } }
$rxSID = '[Ss]-1(?:-\d+){1,}'
[regex]::Match('This SID S-1-5-80-3964583643-2633443559-2834438935-
3739664028-1580655619 has been detected', $rxSID).Value
64bit vs. 32bit
System32 vs. SysWow64
SysWow64 vs. SysNative
Program Files vs. Program Files (x86)
• %ProgramFiles% vs. %ProgramFiles(x86)%
HKLM:\Software\Wow6432Node
no easy way to navigate 64bit registry from 32bit process$hklm = [Microsoft.Win32.RegistryKey]::OpenBaseKey('LocalMachine',
'Registry64')
$key64 = $hklm.OpenSubKey('SOFTWARE\XX64')
$key64.GetValue('theValue')
4. 11. 2019
30
64bit vs. 32bit EXE/DLL/SYS/EFI (portable executable)
MZ ... DOS header
0x3C offset PE header 64bit
P E 00 00 D P E 00 00 L
MZ ... DOS header
0x3C offset PE header 32bit
Explorer desktop.ini
False folder names and icons
$Recycle.Bin
%userprofile%
C:\Windows\Assembly
4. 11. 2019
31
Other clues of deleted files and usage
timestamps
executables
• Security event log - Process Tracking
• Prefetcher (SuperFetch, sysmain)
• MRU, bags, MUICache, ...
• Compatibility Assistant
• BAM
documents
• Windows Search (Indexing Service)
• MRU, bags, MUICache, ...