sysvol replication: frs or dfs-r??? rhonda layfield rhonda@minasi.com contents copyright rhonda j....

SYSVOL Replication: FRS or DFS-R???

Rhonda LayfieldRhonda@Minasi.com

Contents Copyright Rhonda J. Layfield 2009

Rhonda Layfield

• IT industry 25+ years• NT/2000/2003 MCSE, MCT, MCSE+Security• Contribute articles to Redmond and

Windows IT Pro magazines• Setup and Deployment MVP• Desktop Deployment Product

Specialist (DDPS)• Co-Author of 2 Mastering Windows Server

books (2003 & 2008)

SYSVOL

Uptown DC Downtown DC

LScript.vbsLScript.vbs

Prehistory to Today• Always been a need for a tool to keep two or

more server’s folders in sync• NT -> LMRepl (Directory Replication)• 2000 introduced FRS (2003 used this also)

• Sysvol• DFS

• 2003 R2: a new Replication engine• Sysvol uses “old” FRS• R2’s DFS – which is called “DFS Namespace” – uses the

“new” Replication engine called “DFS Replication” or “DFS-R”

• Server 2008 can be configured to use DFS-R for SYSVOL replication

Sysvol• Created during dcpromo on domain

controllers• Automatically shared• C:\Windows\SYSVOL\sysvol\

Bigfirm.com (DNSDomainName)• Group Policy Template (Settings)• System Policies• Scripts (NT & AD)

• Must reside on an NTFS volume

FRS Sysvol Terminology

DC 1

DC 2

Upstream Partner

DownstreamPartner

Direct Replication

DC 3

Transitive Replication

Upstream Partner

DownstreamPartner

FRS Process (10,000 foot view)• NTFS logs changes to the “NTFS Change

Journal” • FRS monitors NTFS Change Journal• FRS places changed files in staging area• Replication partners notified

“something’s changed”• Partners request replication• Files are transferred

NTFS Change Journal (aka USN Journal)

• What does it do?Logs all changes to an NTFS volumeSeparate log on each NTFS volume

• Doesn’t it take a lot of space?• Sure; in fact it would eventually fill up a

drive• So Microsoft limits its size

So how large is it, and can I change it?• W2K SP2 32 MB• W2K SP3 512 MB• 2003 pre-SP1 128 MB• 2003 Hotfix 823230 or SP1/R2 512

MB• Server 2008 512 MB• Maximum size is 2 TB• MS Recommends increasing by 128

MB for every 100,000 files/folders

What happens when the change journal fills up?• Then NTFS just goes back up to the

top and starts overwriting the oldest entries

• So… if FRS hasn’t checked in with the change journal in a while, then FRS may get lost

• This is called a “journal_wrap” Ooh No!

FRS monitors NTFS Change Journal

NTFS Change Journal

1 GUID1

2 GUID2

3 GUID3

4 GUID4

Received 1-4

NTFRS FileIDTable

1 GUID1

2 GUID2

3 GUID3

4 GUID4

FRS monitors NTFS Change Journal

NTFS Change Journal

1 GUID1

FileRef#: 0x000f000000003a6f

USN: 0x000000000034cf40

NTFRS FileIDTable

GUID1

FID4222124650674799

USN

3460928

Tying the Change journal and FRS database together• FileRef# & USN in the NTFS Change Journal – NTFS Utility:

FSUtil• FSUtil USN ReadData C:\WINDOWS\SYSVOL\sysvol\

Domain\Policies\GUID1Major Version: 0x2Minor Version: 0x0FileRef#: 0x000f000000003a6fParent FileRef#: 0x0002000000002f45Usn: 0x000000000034cf40Time Stamp: 0x0000000000000000 12:00:00 AM

1/1/1601Reason: 0x0Source Info: 0x0Security Id: 0x308File Attributes: 0x20File Name Length: 0xeFile Name Offset: 0x3cFileName: GUID1

Dump GUID1 from FRS FileIDTable

wmic /namespace:\\root\microsoftdfs path dfsridrecordinfo where filename=“GUID1"

Attributes: 32Clock: 20060906155126.906250-000CreateTime: 20060906155119.203125-000Fence: 3Fid: 4222124650674799FileHash: 0000000000000000 0000000000000000FileName: GUID1Flags: 1GVsn: {55FDBBB9-0E2C-495C-8416-

7CE2706D62C7}-v1468Index: 729ParentUid: {3AB160AD-E505-492F-9C3B-09382DDB0CCC}-v1ReplicatedFolderGuid: 3AB160AD-E505-492F-9C3B-09382DDB0CCCUid: {55FDBBB9-0E2C-495C-8416-

7CE2706D62C7}-v1467UpdateTime: 20060906155126.937500-000Usn: 3460928Volume: \\.\C:

And So The Journal Begins…

NTFS Change Journal

1 GUID1

2 GUID2

3 GUID3

4 GUID4

NTFRS FileIDTable

1 GUID1

2 GUID2

3 GUID3

4 GUID4

Journal Wraps - Good

NTFS Change Journal

5 GUID5

6 GUID6

7 GUID7

4 GUID4

NTFRS FileIDTable

1 GUID1

2 GUID2

3 GUID3

4 GUID4

Journal Wraps - Good

NTFS Change Journal

5 GUID5

6 GUID6

7 GUID7

4 GUID4

5 GUID5

6 GUID6

7 GUID7

NTFRS FileIDTable

1 GUID1 2 GUID2 3 GUID3 4 GUID4

5 GUID5 6 GUID6 7 GUID7

Journal Wraps gone Bad

NTFRS FileIDTable

1 GUID1

2 GUID2

3 GUID3

4 GUID4

NTFS Change Journal

5 GUID5

6 GUID6

7 GUID7

8 GUID8

FRS Database is lost!

This is a Journal Wrap

Error

Journal Wrap Error

• Likely Causes• Turning off FRS for an extended period

of time• Disk intensive DCs

• Fixes• Increase NTFS change journal size• Self-Healing • Non-Authoritative Restore

Non-Authoritative Restore• “Flush everything in Sysvol and ask

my upstream partner for its entire Sysvol”

• Stop ntfrs.exe• HKLM\System\CCS\Services\ntfrs\

Parameters\Backup/Restore\Process at Startup

• Modify existing REG_DWORD entry Burflags, set to hex D2

• Start ntfrs.exe

Non-Authoritative Restores Automagically • W2k or W2k SP1– only happens

“manually” – when you set BurFlags=D2• W2k SP2 – occurs automatically whenever

a journal wrap is detected, no Reg hacking required

• W2k SP3 – Back to manual-only but a Reg entry will make it happen automatically again• HKLM\System\CCS\Services\ntfrs\Parameters

• “Enable journal wrap automatic restore” key to 1

• 2003/2008: MS says not to make automatic (KB 292438)

FRS keeps track of things via a database…• Database lives in C:\Windows\ntfrs\

jet\ntfrs.jdb• Consists of 5 tables• Connection Record table• Version vector table• File ID table (“IDtable”)• Inbound log (“inlog”)• Outbound log (“Ntfrs Outlog”)

Viewed using

Ntfrsutl

How FRS Handles New or Modified Files• FRS classifies files as new, modified or

deleted• Handles new and modified files similarly• FRS creates a Change Order (CO) in the

“Inbound log” table• FRS uses the Backup API to create a

compressed copy of the file in a “staging area” folder

• FRS creates a Change Order (CO) in the “Outbound log” table

• FRS creates a new entry in the IDTable table… but only for new files

FRS Database ntfrs.exe

GPO created

In Log

CO (C)GPO

Out LogCO (C)GPO

NTFS CJ

GPO – (C)

IDTable

GPO 123456748USN

DC

GPO

Partner Notification• Come and get it!!!!• KCC AD Connection Objects & Site

Links• FRS polls AD at startup (and every 5

minutes after) to check for list of replication partners

• Ntfrsutl poll /now (forces polling)• Ntfrsutl poll /now Computer (forces

polling remotely)• Intra-Site (immediate replication)• Inter-Site (replication schedule)

FRS Database

GPO created

In Log

CO New GPO Out Log

CO New GPO

NTFS CJ

GPO - Create

Change Notification

Psstt…I have changes

GPO

FRS Database

In Log

CO New GPO Out Log

CO New GPO

GPO

DC 1 DC 2

Entire File is Copied!!

FRS Issues

• Relocating the FRS database & logs• Relocating SYSVOL• Authoritative Restore• Morphed files/folders• FRSDiag• Ultrasound• Sonar

Relocate FRS database & log files• Stop FRS (net stop ntfrs)• Copy ntfrs folder to new destination• Modify Working Directory to reflect new path

• HKLM\SYSTEM\CCS\Services\ntfrs\Parameters• Confirm Administrator/System accounts have full

control to:• ntfrs• ntfrs\Jet• ntfrs\Jet\Log• ntfrs\Jet\Sys• ntfrs\Jet\Temp

• Start FRS (net start ntfrs)• Verify Inbound and Outbound replication with

partners

Relocate Sysvol

• Confirm replication is working properly

• Dcpromo down (demoting)• Wait for the removal of the DC

from all DC’s (NTDS file system settings object deletion propagates, ADSS)

• Dcpromo back up (re-promoting)• KB 842162 (manual – may the force

be with you!)

Sysvol Authoritative Restore• When would I do this?

• When many DC’s SYSVOL are inconsistent• Multiple DCs have journal_wrap errors

• How do I do it?• Stop FRS on ALL DCs• Select one DC to be the reference machine (this copy of

SYSVOL will be copied to all other DCs)• On the reference machine copy all folders/files that should

reside in SYSVOL to a temporary folder• Restart FRS on the reference machine with the Burflag set to

D4 HKLM\SYSTEM\CCS\Services\ntfrs\Parameters\Cumulative Replica Sets\GUID

• Restart all other DCs with the Burflag set to D2• On the reference machine copy files/folders from the

temporary location into the root of SYSVOL• Monitor that replication is consistent• Detailed steps KB 315457

FRS debug logs

• C:\Windows\Debug• ntfrs_0001.log - ntfrs_0005.log

• FRS Transactions & event details• Settings range from 0-5 (5 highest – most information

logged)• Default setting is 2• Log detail controlled by HKLM\System\CCS\Services\

ntfrs\Parameters• Debug Log Severity• Debug Log Files

• Can also be set using FRSDiag• Must stop and start ntfrs.exe

FRS Conflicts!Morphed Files/Folders• File/folder created on 2 different DC’s in the

exact same folder with the same name prior to a replication cycle

• When replication occurs – the inbound file will be renamed• Logon.vbs becomes

Logon.vbs_ntfrs_0001ab39• How does this happen?

• Administrators are attempting to make data consistent with manual copies

• A server’s FRS was not stopped prior to an authoritative restore on another server

• During an authoritative restore ONE server did not have it’s BurFlag set to D2

Resolve Morphed files/folders• Rename the original file/folder and the

morphed file/folder to different names• Wait for this to propagate to all servers• After propagation – choose the

file/folder you would like to keep and rename it back to the original name

• Delete all unwanted copies of the file/folder

• KB 328492

Tools• FRSDiag• Dump event logs• Find members GUID numbers

• Ultrasound• Requires a SQL database• Installs WMI providers on each DC• Polls DCs at defined intervals on FRS status

and places information in a SQL database• Configure Ultrasound to email or log an

event in the application log whenever an error condition is detected

Sonar

• Sonar-d.htm • installs with Sonar & is a great document on

troubleshooting FRS• Specific event ids which will help in resolving issues

• Find out which servers FRS service have been disabled or are not running

• SYSVOL not shared• Staging area full• Staging files being regenerated• Burflags set• NTFS change journal size• Morphed folders/files

Replicating Sysvol via DFS-R• All DC’s must be running Server

2008 • DFS-Replication can be managed

from:• XP-Pro SP2 workstation• Server 2008• Vista

DFS-Replication Terminology• Replication group - A set of servers, called

members, that participate in replicating one or more folders.

• Replicated folder - sysvol• Connection topology - Which members replicate

with other members.• Schedule - When replication is available.• Upstream partner - The partner who sends the

notification that it has changes for a replicating partner.

• Downstream partner - The partner who received the notification from an upstream partner and initiates replication.

DFSR and Journal Wraps• DFSR also monitors the NTFS change

journal• What’s different? DFSR always heals

itself

• And here is how…

Self-Healing Journal Wraps• DFS-R stops processing changes

from the NTFS change journal• Replication Stops• All entries in the DFS-R FileIDTable

receive a JWED (Journal Wrapped) flag

• The Directory Walker thread (DIRW) compares all Update Sequence Numbers (USNs) between the NTFS change journal and the DFS-R database

Upon Comparison - 3 Possible Conditions1. If the USNs in the NTFS change journal and the

DFS-R Database are the same:• DFS-R has the latest changes and removes the JWED

flag2. If the USNs are NOT the same:

• The USN in the NTFS change journal is incremented• The JWED flag is cleared from the DFS-R Database• The file will be replicated normally

3. Once all files in the DFS-R Database have been compared to the NTFS change journal and their JWED flags are cleared• One last scan is performed to find any files still flagged

JWED• If a file was deleted from the NTFS change journal, but

still exists in the DFS-R database. The files are tombstoned and will eventually be deleted from the replication process.

Do I have all the changes?• Version Vectors (VVs) are compared

Houston

Dallas

Server VVHouston 20

Dallas 30

Server VVDallas 31

Houston 20

RDC in Action!

Data (16 Bytes)

MD4

Row Row Row 27your house 42gently dow 17

Houston DC

Dallas DC

Data (16 Bytes)

MD4

Row Row Row 27

your house 42

gently dow 17

RDC in Action!

Data (16 Bytes)

MD4

Row Row Row 27your house 42gently dow 17

Dallas DC Houston DC

Data (16 Bytes)

MD4

Row Row Row 27

your boat g 35

ently down 22

What if there is a Conflict?• What causes a conflict?• The same file/folder (same UID) being

modified on two different servers, within one replication cycle

• A file/folder is created on two different servers in the same folder with the same name (different UID), within one replication cycle

File and Folder Conflicts Resolved• File with the Same Name & UID• Last writer wins (based on time stamps)

• File with the Same Name but different UID• First created wins (based on time stamps)

• Folder with the Same Name and UID• First created wins (based on time stamps)

• Folder with the Same Name but different UID• The contents are consolidated

Performance Monitor• DFS Replicated Folders• Bandwidth Savings• Number of conflicts that have occurred

• In bytes, or number of files/folders• RDC information• Staging Files

• DFS Replication Connections• Number and size of files replicated

• DFS Replication Service Volume• Number of entries read and accepted from the

NTFS change journal (USN Journal)

Why Use DFSR vs FRS?• FRS silently fails if the volume

SYSVOL resides on < 1GB of free space

• Copies changes (RDC) not entire files

• Version Vector tables• Journal Wraps are self-healing• Morphed files and folders

automagically taken care of

Migrating Sysvol Replication to DFS-R• Pre-Migration• Migration• Demo• PDC EM - Server 2003 that’s been

upgraded to 2008• Or a 2008 / 2008-R2 Server• Domain mode = Server 2008

Pre-Migration• On 2008 Server (PDC)• Pop in the Server 2008 DVD

• Adprep /forestprep from Sources/Adprep folder

• Raise domain functional level to Server 2003• Active Directory Domains and Trusts

• From Server 2008 DVD• Adprep /domainprep

• Upgrade PDC & all DCs to Server 2008• Raise domain Functional level to Server 2008• Backup SYSVOL

Migration Process has 2 Types of States• Stable states• Processes are complete• Can Rollback to a point

• Transitional states• Processes are in a working state

Stable State

T

Migration Process

START(State 0)

PREPARED(State 1)

RE-DIRECTED(State 2)

ELIMINATED(State 3)

4 5

6

7

Meet DfsrMig.exe• DfsrMig.exe lives in Windows\

System32 on Server 2008• Run dfsrmig from the PDC Emulator• State 0 - All DCs begin here• Replicating SYSVOL via NTFRS

• State 1 “Prepared”• Dfsrmig /SetGlobalState 1

• To confirm a state has been set…• Dfsrmig /GetGlobalState

What Happens…

• New Windows\SYSVOL_DFSR on all DC’s• Contents of Windows\SYSVOL copied to new

SYSVOL_DFSR folder• Windows\SYSVOL_DFSR

• domain & sysvol folders

• Netlogon share still points to Windows\SYSVOL\sysvol\Bigfirm.Com\Scripts

• SYSVOL share still points to Windows\SYSVOL\sysvol

• Both SYSVOL and SYSVOL_DFSR are being replicated

ADUC

• Advanced View• System• DFSR-GlobalSettings is created

• Domain System Volume (Replication Group)• DFSR-Replication Group

• DFSR-Content – SYSVOL Share• DFSR-Topology – List DCs

Re-Directed State…

• Dfsrmig /SetGlobalState 2

SYSVOL_DFSR

UptownDC

DownTownDC

DFS-RSYSVOL_DFSR

• Dfsrmig /GetGlobalState• Current DFSR global state:

Redirected

SYSVOL

SYSVOLFRS

SYSVOL_DFSR & SYSVOL out of sync…• The original copy of SYSVOL to

SYSVOL_DFSR was performed by Robocopy

• This copy is only done once• If you need to run it again you’ll

have to do it manually

Copying SYSVOL

• ROBOCOPY c:\Windows\Sysvol\Domain c:\Windows\Sysvol_DFSR\Domain /Copyall /MIR /B /R:0 /XD “Do_Not_Remove_NtFrs_PreInstall_Directory” “DfsrPrivate” “NtFrs_Prexisting__See_Eventlog” “ NTFRS_CMD_FILE_MOVE_ROOT” /XF “DO_NOT_REMOVE_NtFrs_PreInstall_Directory” “DfsrPrivate” “NtFrs_PreExisting__See_Eventlog” “NTFRS_CMD_FILE_MOVE_ROOT”

• Windows\Debug\SYSVOL_DFSR-Robocopy.txt

Re-Directed State

• Redirects the SYSVOL share to the new SYSVOL_DFSR folder

• HKLM\Sys\CCS\Services\Netlogon\Params

• Sets SysvolReady to False• Sets Sysvol = C:\WINDOWS\

SYSVOL_DFSR\sysvol• Sets SysvolReady to True

Migration Process

START(State 0)

PREPARED(State 1)

RE-DIRECTED(State 2)

ELIMINATED(State 3)

4 5

6

7

X

The Eliminated State• Dfsrmig /SetGlobalState 3• Deletes the NTFRS replica set from

AD• Deletes the old SYSVOL folder• Leaves NTFRS on• There is no going back!

Health Report

Create a Diagnostic Report• Health Report• Propagation Test• __DFSR_DIAGNOSTICS_TEST_FOLDER__• FRSRIP@A13948E4-5E44-483A-B56B-

65A075C446C0@Domain System Volume-SYSVOL Share.xml

• Propagation Report

Create a Diagnostic Report using

• The following example will generate a health report that:• Gathers information ALL servers in the sysvol

replication group.• Houston is the Reference Member. • The report will be named HealthReport.html stored in

the C:\Reports folder of the local machine.

• DfsrAdmin Health New /RgName: ”Domain System Volume” /RefMemName:Bigfirm\Houston /RepName:C:\Reports\HealthReport.html /FsCount:true

More Diagnostic Reports• The following example will generate a

health report that:• Gathers information only from Dallas• Houston is the Reference Member for the

sysvol replication group. • The report will be named DallasHealth.html

stored in the C:\Reports folder of the local machine.

• DfsrAdmin Health New /RgName:”Domain System Volume” /MemName:Bigfirm\Dallas /RefMemName:Houston /RepName:C:\Reports\DallasHealth.html /FsCount:true

The last Diagnostic Report… I promise• The following example will generate a

health report that:• Gathers information from all Servers listed in

the C:\Servers.txt file for the sysvol replication group.

• Houston is the Reference Member. • The report will be named ServersHealth.html

stored in the C:\Reports folder of the local machine.

• DfsrAdmin Health New /RgName:”Domain System Volume” /MemberListFile:C:\Servers.txt /RefMemName:Bigfirm\Houston /RepName:C:\Reports\ServersHealth.html /FsCount:true

In Summary

• FRS – RIP• DFS-R Rocks!• DFSRMig Works!

Hope you Enjoyed This Session

Please Don’t forget to fill out your evaluations

• Rhonda@Minasi.Com

• Thank You!