windows file systems and improving performance
DESCRIPTION
Windows File Systems and Improving Performance. APCUG 1/5/2008 Greg Hayes Raxco Software [email protected]. Introduction. Company Background Raxco founded 1978 Largest provider of OpenVMS system management software Spun off Axent Technologies (now part of Symantec) in 1996 - PowerPoint PPT PresentationTRANSCRIPT
![Page 2: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/2.jpg)
Introduction
• Company Background– Raxco founded 1978– Largest provider of OpenVMS system
management software– Spun off Axent Technologies (now part of
Symantec) in 1996– First Windows product released in 1998
![Page 3: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/3.jpg)
Introduction
• Company Background– Microsoft Gold Certified Partner– Microsoft ISV– Member of Microsoft Development Network– Software used by some of the largest
companies and government organizations in the world
![Page 4: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/4.jpg)
Introduction
• Speaker background– System manager for 10+ years
– Raxco customer for 10 years
– Started at Raxco in 1996
– Manager, Technical Solutions
– Microsoft MVP 2003-2007 - Windows File Systems
– Online communities
– Speaking/Education
![Page 5: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/5.jpg)
Topic Background
• Who remembers their first PC?
• PC hardware performance continues to rise– CPU clock speeds > 2Ghz– 2GB RAM– 500GB+ capacity hard drives
• Limiting factor is still disk I/O• Anything that speeds up access to disk
improves performance
![Page 6: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/6.jpg)
Windows File Systems
• FAT16
• FAT32
• exFAT (new)
• NTFS
![Page 7: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/7.jpg)
Windows File Systems
• FAT16/32
The File Allocation Table is created when you format a FAT drive and contains information about all of the files found on the drive and where they logically reside.
There are actually 2 FATs per partition.
![Page 8: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/8.jpg)
Windows File Systems – FAT16
• Benefits:– Faster performance on smaller partitions - less
than 4GB– Best Performance for Pagefile
• Drawbacks:– Little or no security– High rate of file system corruption– Limits on partition size (FAT16 – 4GB)
![Page 9: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/9.jpg)
Windows File Systems – FAT16
• FAT16– 4GB FAT16 partition has a cluster size of 64k.– Wastes space as even the smallest files take up
a full cluster.– Maximum file size is 2GB– Files per volume 65,536
![Page 10: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/10.jpg)
Windows File Systems – FAT32
• FAT32– Maximum file size is 4GB– Allows smaller cluster sizes to be used –
resulting in less space “waste”– Maximum partition size is 32GB (as formatted
by XP/Vista/2K3)– Up to 2TB can be written/read (XP/Vista/2K3)– Files per volume ~4,177,920
![Page 11: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/11.jpg)
Windows File Systems – exFAT
• exFAT– “Extended FAT”– Designed for flash media – exchange media
with PCs.– Free space bitmap for faster allocation– Maximum theoretical file size is 16EB– More than 1000 files per folder
![Page 12: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/12.jpg)
Windows File Systems - NTFS
• Journaling/Transaction Based File System
• “Self Describing” File System
• Metadata - data that describes data
![Page 13: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/13.jpg)
Windows File Systems - NTFS
• Benefits:– Better security– Better performance with large partitions –
greater than 4GB– Better resistance to file system corruption
• Drawbacks:– Can’t natively boot to MSDOS and get access
to partition
![Page 14: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/14.jpg)
Windows File Systems - NTFS
• NTFS– Maximum file size is approx 16TB– Maximum partition size is 2TB – 256TB
(depends on cluster size used)– Files per volume 4,294,967,295
![Page 15: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/15.jpg)
Windows File Systems - NTFS
• NTFS– Disk Quotas (Win2K and newer)– File/Folder compression built into the file
system– Encryption built into the file system (Win2K
and newer)– Volume Shadow Copy (VSS – WinXP and
newer)– txNTFS
![Page 16: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/16.jpg)
NTFS Metadata
• $MFT - Master File Table (0)– The $MFT contains at least 1 record for each
file that exists on the partition.
• $MFTMirr – Master File Table Mirror (1)– Duplicate of the 1st 4 records of the $MFT– Used for redundancy/recoverability
![Page 17: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/17.jpg)
NTFS Metadata
• $LogFil – NTFS Transaction Log (2)– Updates to the file system first get posted to
the transaction log– Gives NTFS its “self-healing” abilities
• $Volume – Volume Information (3)– Volume Label/Version
![Page 18: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/18.jpg)
NTFS Metadata
• $AttrDef – Attribute Definitions (4)– Names, Numbers, Descriptions
• $Root – (5)– Root folder on the drive. (I.e. C:\)
![Page 19: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/19.jpg)
NTFS Metadata
• $Bitmap – Cluster Bitmap (6)– Indicates whether a cluster is used or empty– Uses this to quickly find free space
• $Boot – Partition Boot Sector (7)– Location of the partition boot sector– Bootstrap loader code if bootable drive
![Page 20: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/20.jpg)
NTFS Metadata
• $BadClus – Bad Cluster File (8)– Clusters the file system has reported as bad
– Updated by CHKDSK
• $Secure – Security Descriptors (9)
• $Upcase – Upcase Table (10)– Translation of lowercase characters to their equivalent
Unicode upper case characters
![Page 21: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/21.jpg)
NTFS Metadata
• $Extend – Optional Extensions (11)– User Change Journal– Quotas
![Page 22: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/22.jpg)
Anatomy of a $MFT Record
• Fixed size records – 1K• Attribute Records
– $FILE_NAME• 8.3 file name and parent directory• Long File Name
– $DATA• Extent information for file. Includes Logical
Cluster Number (LCN) and Run Length• $DATA record contains information on file size
![Page 23: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/23.jpg)
Anatomy of a $MFT Record
• DiskEdit
![Page 24: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/24.jpg)
MFT Reserved Zone
• Created when partition is formatted
• $MFT will “grow into” as needed
• File system will keep free if possible
• Defragmenters can not use free space inside of MFT Reserved Zone (NT4/Win2K)
![Page 25: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/25.jpg)
MFT Reserved Zone
• NT4– Fixed Size– 12.5% by default– Can be increased up to 50% via registry key
![Page 26: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/26.jpg)
MFT Reserved Zone
• Windows 2003/2000/XP/Vista/2008– Dynamically created every time partition is
mounted– First record of $MFT to first non-free cluster -
up to a default max of 12.5%
![Page 27: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/27.jpg)
Logical vs Physical Clusters
• Logical Clusters– File system level– Every partition starts at logical cluster 0– No idea of hard drive technology in use
• IDE
• SCSI
• RAIDx
• # platters or read/write heads
![Page 28: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/28.jpg)
Logical vs Physical Clusters
• Physical Clusters– Hard drive level– Hard drive controller translates logical-to-
physical and positions heads
![Page 29: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/29.jpg)
Disk Disk Disk
Disk Controller
O S
File System
ApplicationRequests
File
Logical ClusterView
Physical ClusterView
![Page 30: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/30.jpg)
Cluster Size and Performance
• Smaller clusters– less wasted space– Worse performance – especially large files
• Larger clusters– more wasted space– Better performance – especially large files
![Page 31: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/31.jpg)
Conversion from FATx to NTFS
• NT4/Win2K– Results in 512byte clusters– Not “the best” for performance - especially
with video/image applications
• WinXP/Vista– Will try to convert using 4k clusters.– Best for general file system performance.
![Page 32: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/32.jpg)
OEM Vendors and Cluster Size
• IBM/Dell/HP– Provide Software to install/configure Windows
Server– Formats system drive with 512byte cluster– Absolute worse for system drive performance– Only way to convert is 3rd party tools and have
server unavailable for extended period
(HP SmartStart, Dell OpenManage, IBM ServerGuide)
![Page 33: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/33.jpg)
Biggest Cause of Poor File System Performance
• Fragmentation!
![Page 34: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/34.jpg)
Fragmentation Causes
• What causes fragmentation?– Occurs when files are created, extended or
deleted– Happens regardless of how much free space is
available (After XP/SP2 installation – 944 files/2943 fragments)
– More than one Logical I/O request has to be made to the hard drive controller to access a file
![Page 35: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/35.jpg)
Fragmentation Impacts
• What does fragmentation do to my system?– Slows down access to files– Extra CPU/Memory/Disk resource usage– Some applications may not run– Slow system boot/shutdown– Audio/Video record/playback drops frames or
“skips”
![Page 36: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/36.jpg)
Measuring Impact of Fragmentation
• Measuring the performance loss in reading a fragmented file
![Page 37: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/37.jpg)
What Can I Do About Fragmentation?
• You can’t stop fragmentation from happening (you can only slow it down)!
• What you CAN do is to defragment
![Page 38: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/38.jpg)
Defragmenting - Results
• What does defragmenting do?– Locates logical pieces of a file and brings them
together• Faster to access file and takes less resources
• Improves read performance
– Consolidates free space into larger pieces• New files get created in 1 piece
• Improves write performance
![Page 39: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/39.jpg)
Measuring Impact of Fragmentation
• Measuring the performance difference in reading a contiguous file
![Page 40: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/40.jpg)
Defragmenting - Issues to Consider
• Safety, Safety, Safety!– No loss of data– No corruption of data
![Page 41: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/41.jpg)
Defragmenting - Issues to Consider
• Free Space– How much is enough?– Where is free space located?
• Inside MFT Reserved Zone
• Outside of MFT Reserved Zone
– Consolidation of free space
![Page 42: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/42.jpg)
Safety - Microsoft’s Defrag APIs
• Provided as part of the operating system
• Defragmenters do not actually move files
• Integrated with caching system and Memory Manager
• Performs all I/O synchronization
• Allows even files in use to be defragmented
![Page 43: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/43.jpg)
Safety - Microsoft’s Defrag APIs
• Restrictions– Move Granularity– Free Space in MFT Reserved Zone– NTFS Cluster Size– Special OS files
• Pagefile
• Hibernate file
• NTFS Metadata
![Page 44: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/44.jpg)
Microsoft’s Defrag APIs
• “Quirks”– Compressed files (NT4/Win2K)– File System Cache (NT4/Win2K)– Encrypted Files (Win2K)– NTFRS/DFS (Win2K)– VSS (Vista,Win2K3)
![Page 45: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/45.jpg)
Free Space Consolidation
• Allows new files to be created contiguously– Maintains file system performance longer– Requires less frequent defrag passes
![Page 46: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/46.jpg)
Free Space Consolidation
• Defragmenting files improves read performance
• Free space consolidation improves write performance
![Page 47: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/47.jpg)
Single Pass Defragmentation
• As my dad always told me…– If you are going to do a job – do it right the first
time
• After defrag has completed, you don’t have to “wonder” if you need to run again
• Fragmentation issue is solved – done!
![Page 48: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/48.jpg)
File Placement Strategy
• When you play chess, do you have a strategy or do you just start moving pieces around?
• A good file placement strategy accomplishes several things:– Slows down re-fragmentation– Speeds up future defrag passes– Works with the operating system – not against
![Page 49: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/49.jpg)
Windows Prefetch
• What is Prefetching?– Windows monitors system boots and
application launches– Uses information gathered to perform driver
load optimization (speeds up boot process)– Uses information gathered to speed up
application launches
![Page 50: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/50.jpg)
Windows Prefetch
• Windows stores this information in the \Windows\Prefetch folder– Layout.ini– Xxxx.pf
• Every 3 days, Windows will automatically perform a “partial” defrag of the files indicated in layout.ini
![Page 51: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/51.jpg)
Windows Prefetch
• This process depends on a large enough piece of contiguous free space being available. If there isn’t, then it doesn’t happen!
![Page 52: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/52.jpg)
Windows Prefetch
• Defragmenters - working WITH the operating system NOT against. Microsoft says:– Either respect where Windows places these
files
OR– Manage these files yourself
![Page 53: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/53.jpg)
“Tweaking” - BootVis
• BootVis– One of the most mis-understood “tweaking”
tools around– Available from Microsoft– Used to profile system boots
• Optimize driver loading for faster boots
• Calls built-in defragmenter to “optimize” boot files
• Also can be used to identify drivers that load slowly
![Page 54: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/54.jpg)
“Tweaking” - BootVis
• Windows XP does this boot optimization over a period of boots (after 3 boots have been performed)
• BootVis does it “now”
• End result is the same
![Page 55: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/55.jpg)
“Tweaking” – Disable Last Access
• HKLM\System\CurrentControlSet\Control\FileSystem\Disablelastaccess =1
• Improves NTFS performance by 2%
• Last Access updating turned off by default in future versions of NTFS (2008 Server)
![Page 56: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/56.jpg)
“Tweaking” – 8.3 Name Creation
• Win2k3/XP/Vista: fsutil.exe behavior set disable8dot3 1
• Win2k:HKLM\System\CurrentControlSet\Control\FileSystem\NtfsDisable8dot3NameCreation = 1
• Provides faster directory enumeration
• Legacy apps may no longer install/work.
![Page 57: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/57.jpg)
“Tweaking” – Purging Prefetch
• Belief is that a large number of entries in layout.ini slows down system.
• Windows will manage effectively. Use of entries in layout.ini actually speeds up system boots and application launch.
![Page 58: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/58.jpg)
“Tweaking” – Disabling Vista’s SuperFetch
• Belief is that not having a large amount of free memory available is a bad thing.
• SuperFetch caches frequently accessed programs/data and uses available RAM. Unused RAM is a wasted resource.
• SuperFetch survives system restarts. XP caching doesn’t.
![Page 59: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/59.jpg)
Volume Shadow Copy (VSS)
• VSS and defragmentation
• Multiple of 16k cluster size
• Default cluster size is 4k because NTFS compression hasn’t been modified to support greater than 4k cluster size.
• BitLocker (Vista) also restricted to 4k cluster size
![Page 60: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/60.jpg)
DiskPar/DiskPart
• DiskPar - Win2k• DiskPart – Win2k3• Want to avoid crossing track boundaries• Align on 64k for best MS SQL performance• Win2k8 – default is 64k when creating
volumes• Contact storage vendor – i.e. EMC
recommends 64k
![Page 61: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/61.jpg)
Disk Technology
• SATA worse in high write environments – however cheap
• RAID1+0 best for high read/write environments AND redundancy – however expensive
• More onboard cache – better performance
![Page 62: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/62.jpg)
Performance Measuring Tools
• Windows Performance Monitor – Split I/O Count (fragmentation)– Disk Queue Length (<= 2/spindle)
• hIOMon – www.hiomon.com– Device AND File based metrics
• SQLio – Microsoft– Stress Test I/O subsystem
![Page 63: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/63.jpg)
Cluster Size Recommendations
Drive Use
File System/Cluster
Size Notes/ConsiderationsWindows OS NTFS - 4k (Default)
Pagefile (separate physical drive) FAT32
Need to also have pagefile on OS drive if you want to capture memory dump on Windows crash.
File Server - general Office data NTFS - 16k
Windows 2003/2008/Vista - minimizes shadow copy/Restore Point purges. For Windows 2000, use 4k.
Databases NTFS - 64k Windows 2000, use 4k.Exchange NTFS - 64k Windows 2000, use 4k.
Virtual - VHD/VMDK NTFS - 64k Windows 2000, use 4k.Video/Graphics NTFS - 16k Windows 2000, use 4k.
* You can’t use compression if cluster size greater than 4k
![Page 64: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/64.jpg)
Conclusion
• To improve file system/drive performance– Use appropriate disk technology– Use the most appropriate file system– Use the most appropriate cluster size– Align on cluster boundaries– Make sure free space is consolidated– When you defragment, make sure that it is
being done effectively.
![Page 65: Windows File Systems and Improving Performance](https://reader036.vdocuments.us/reader036/viewer/2022062408/56814326550346895daf9044/html5/thumbnails/65.jpg)
Conclusion
Questions/Answers