the definitive guide to active directory troubleshooting

206

Upload: api-3774147

Post on 10-Apr-2015

842 views

Category:

Documents


6 download

TRANSCRIPT

Page 1: The Definitive Guide to Active Directory Troubleshooting
Page 2: The Definitive Guide to Active Directory Troubleshooting

Ask the average Windows system administrator what they think the most prominent feature of Windows 2000 isand they will probably say "Active Directory." Why? First, Active Directory will enable them to deploy Windows-based networks on an unprecedented scale. Second, they know that Active Directory represents a significant changein the way they will manage network entities such as users, computers, network devices and applications. Forexample, there are new user interfaces for most common management activities such as adding users and managingprinters. There are also many new system concepts, such as multi-master replication and DNS integration thatadministrators must understand in order to keep their Active Directory installations healthy.

Some might ask why there are so many new concepts to learn. The answer lies in understanding both limitations ofearlier directory services and Microsoft's design goals for Active Directory. First-generation directories demonstratedthe power of standards-based repositories, but didn't support replication. By running only on a single machine,they provided no opportunity for scale-out and became a single point of failure in the network. Second-generationtechnologies added single-master replication (with read-only replicas) that scaled much better, but supportedupdates only against the master. In practice, single-master models bound individual deployments to 'regions' ofcontinuous network connectivity. Third-generation directories added multi-master support but with importantconstraints. For example, one third-generation directory service is limited, in practice, to approximately 10 update-able replicas per partition and works best only over high-speed network connections.

Microsoft decided that, in order to scale to enterprise levels, Active Directory had to support large numbers ofgeographic locations (potentially in excess of 1,000) and not be limited by slow or intermittent network connectivity.This led to fourth-generation features in Active Directory such as sites, trees, forests, bridgehead servers, and globalcatalogs. These features have enabled Active Directory to scale to unprecedented levels while remaining manageable.

At the same time, there are, as author Sean Daily notes, a lot of moving parts in the average Active Directorydeployment. Most of the time, these parts work together just fine. There will be occasions, however, when issuesarise. For example, if a replica is unable to contact any other replica for long periods of time (usually due to net-work configuration problems) some troubleshooting will eventually be required. Then, it will be important tounderstand how Active Directory's parts fit together in order to get to the root of the issue quickly.

I believe that this innovative, on-line book from NetPro and Realtimepublishers.com will prove to be a valuableresource for any administrator who is tasked with managing an Active Directory installation. The approach toinformation delivery is ideal. The book starts with important background concepts that will enable the reader tounderstand the design of Active Directory and relationships between components, in a clear and concise way. Laterchapters build on this knowledge by providing step-by-step procedures for diagnosing common issues even whenthe root cause of an issue may not be clear.

Most important, this book provides a methodology for proactively keeping an Active Directory installation healthywhile simultaneously enabling administrators to get to the root causes of problems quickly when they do occur.Such a methodology can be 'worth its weight in gold' to today's systems administrators who are responsible forkeeping ever-more complex systems up and running with ever-less downtime!

Peter J. Houston Group Program Manager, Active Directory Microsoft CorporationRedmond, WashingtonMarch 6th, 2001

The Definitive Guide to Active Directory Troubleshooting Foreword

Foreword by Peter Houston

Page 3: The Definitive Guide to Active Directory Troubleshooting

Chapter 1

Introducing Active Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1The Importance of Directories and Directory Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2

Many Eggs, One Basket . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2New Tools for New Times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2

Meet Active Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3The AD Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4Logical Architecture of AD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4

Objects and Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4The Schema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5LDAP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6Domains, Trees and Forests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8Organizational Units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11The Global Catalog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11

Physical Structure of AD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12Domain Controllers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12Directory Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12The Operations Masters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13Sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15

AD’s Backbone: DNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16Introduction to AD and Win2K Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17

AD and Win2K Monitoring Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19Change Monitoring and Auditing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21Problem Resolution, Automation and Alerting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21Other Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22

Chapter 2

Designing an Effective Active Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24Active Directory’s Logical and Physical Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24

Logical Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24Namespace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25Naming Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25

Physical Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26Designing Active Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26Designing the Forest and Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26

Determining the Number of Forests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28Setting Up and Managing Multiple Forests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29

Determining the Number of Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31Designing the Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31

Determining the Number of Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33Choosing a Forest Root Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34

The Definitive Guide to Active Directory Troubleshooting table of contents

Table of Contents

Page 4: The Definitive Guide to Active Directory Troubleshooting

The Definitive Guide to Active Directory Troubleshooting table of contents

Using a Dedicated Root Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34Assigning a DNS Name to Each Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36

Using an Internet-Registered Name for the Top-Level Domian . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36Using Internet Standard Characters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .36Using Locations to Name Child Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37Never Using the Same Name Twice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38

Dividing the Forests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .38Placing the Domain Controller for Fault Tolerance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39Determining Trust Relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40

Using Bi-Directional Transitive Trusts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40Using One-Way Trusts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42Using Cross-Link Trusts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42

Designing Organizational Units for Each Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44Creating OUs to Delegate Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .44Creating OUs to Reflect Your Company’s Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45Creating OUs for Group Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .46Creating OUs to Restrict Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47

Designing the Sites for the Forest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47Creating Sites and Site Links Based on Network Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .47

About Sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .48About Site Links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .48Creating the Site Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .49

Using Sites to Determine the Placement of Domain Controllers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .50Using Sites to Determine the Placement of DNS Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51

Chapter 3

Monitoring and Tuning the Windows 2000 System and Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .53Monitoring Windows 2000 Domain Controllers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .53Monitoring the Overall System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .54

Using Task Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .55Using the Performance Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .56Event Viewer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .58

Events Tracked in Event Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .58Types of Event Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .59Starting Event Viewer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .59Types of Events Logged by Event Viewer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60Sorting and Filtering Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61Exporting Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61

Monitoring Memory and Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62Using Task Manager to View Memory on a Domain Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .63Using the Performance Console to Monitor Memory on a Domain Controller . . . . . . . . . . . . . . . . . . . . .65

Available Memory Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .65Page-Fault Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .66Paging File Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .68System Cache . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .69

Page 5: The Definitive Guide to Active Directory Troubleshooting

Monitoring Processors and Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .70Using Process Viewer to Monitor Processes and Threads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .71Using Task Manager to View Processes on a Domain Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72

Working with the List of Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .73Viewing Information about Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .74

Using the Performance Console to View Processes on a Domain Controller . . . . . . . . . . . . . . . . . . . . . . .75% Processor Time Counter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .75Interrupts/sec Counter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .76Processor Queue Length Counter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .77

Monitoring the Disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .78Using the Performance Console to Monitor the Disk Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79% Disk Time and % Idle Time Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .79Disk Reads/sec and Disk Writes/sec Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .80Current Disk Queue Length Counter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .81% Free Space Counter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82

Monitoring the Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82Using Network Monitor to Watch Network Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .82Using the Performance Console to Monitor Network Components on a Domain Controller . . . . . . . . . . . . .83

Domain Controller Network Throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .83Network Interface Throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .83

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .85

Chapter 4

Monitoring Active Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87Using the Monitoring Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87

Third-Party Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87DirectoryAnalyzer from NetPro . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .87AppManager from NetIQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89

Built-In Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89System Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89

Event Viewer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .90Replication Diagnostics (REPADMIN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .91

Monitoring the AD Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92Monitoring the Domain Controllers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .93

Using DirectoryAnalyzer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .94Using NTDS Performance Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .98

Monitoring the Domain Partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .101Using DirectoryAnalyzer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .101Using Domain Database Performance Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .103

Installing the Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .105Monitoring the Global Catalog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .105Monitoring Operations Masters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .107Monitoring Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .109

Using Directory Partition Replicas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .112Schema Partition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .112

The Definitive Guide to Active Directory Troubleshooting table of contents

Page 6: The Definitive Guide to Active Directory Troubleshooting

Configuration Partition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .112Domain Partition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .112

Using Directory Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .112Using Replication Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .113Using DirectoryAnalyzer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .114

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .117

Chapter 5

Troubleshooting Active Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .118Following a Specific Troubleshooting Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .118Troubleshooting Network Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .118

Testing for Network Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .118Testing the IP Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .119Testing the TCP/IP Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .121Performing Other Troubleshooting Tests Using DirectoryAnalyzer . . . . . . . . . . . . . . . . . . . . . . . . .122

Domain Controller Connectivity Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .122Domain Connectivity Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .124Site Connectivity Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .125

Troubleshooting Name Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .126Understanding Name Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .126Checking That DNS Records Are Registered . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .126

Using Event Viewer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .127Using PING . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .128Using NSLOOKUP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .128

Checking the Consistency and Properties of the DNS Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .129When the DNS Server Doesn’t Resolve Names Correctly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .129

How the Caching DNS-Resolver Service Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .130Using Other Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .130

Troubleshooting the Domain Controllers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .131Understanding the Active Directory Database and Its Associated Files . . . . . . . . . . . . . . . . . . . . . . .131Comparing Directory Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .133Analyzing the State of the Domain Controllers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .133Using NTDSUTIL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .135

Locating the Directory Database Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .136Checking for Low-Level Database Corruption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .137Checking for Inconsistencies in the Database Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .139Cleaning Up the Metadata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .140Moving the Active Directory Database or Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .142Repairing the Active Directory Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .144

Troubleshooting Secure Channels and Trust Relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .146Troubleshooting the Operations Masters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .147

When Operations Masters Fail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .148Schema Master . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .148Domain Naming Master . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .148

The Definitive Guide to Active Directory Troubleshooting table of contents

Page 7: The Definitive Guide to Active Directory Troubleshooting

Relative ID (RID) Master . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149Infrastructure Master . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149PDC Emulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .149

Determining the Operations Master Role Holders Locations . . . . . . . . . . . . . . . . . . . . . . . . . .150Using the DSA and Schema MMC Snap-Ins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .150Using NTDSUTIL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .151Using the Windows 2000 Resource Kit’s Dumpfsmos.cmd . . . . . . . . . . . . . . . . . . . . . . . .151Using DCDIAG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .151Using AD Replication Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .151Using Third-Party Utilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .152

Seizing an Operations Master Role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .152Checking for Inconsistencies among Domain-Wide Operations Masters . . . . . . . . . . . . . . . . . .153

Troubleshooting the Replication Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .154Viewing the Replication Partners for a Domain Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . .154

Using DirectoryAnalyzer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .154Using REPADMIN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .155Forcing Domain Controllers to Contact Replication Partners . . . . . . . . . . . . . . . . . . . . . . .155Tracking Replicated Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .156

Forcing Replication among Replication Partners . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .156Viewing Low-Level AD Replication Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .157Checking for KCC Replication Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .157

Troubleshooting Using Change Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .158Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .160

Chapter 6

Backing Up and Recovering Windows 2000 and Active Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161Building a Fault-Tolerant System That Includes a Backup and Restore Strategy . . . . . . . . . . . . . . . . . . . . .161Using the Windows 2000 Backup and Restore Utility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163

Using the Backup Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .164Specifying Default Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .164Specifying Advanced Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .167

Backing Up Using Manual Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .170Maintaining System State Backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .172Using the Restore Wizard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .173

Specifying Advanced Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .175Restoring Required Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .177

Windows Internet Naming Service (WINS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .178Dynamic Host Configuration Protocol (DHCP) . . . . . . . . . . . . . . . . . . . . . . . . . .179Remote Storage Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .179Certificate Server Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .179Internet Information Services (IIS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .180Active Director . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180SYSVOL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .180

Creating an Emergency Repair Disk (ERD) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .180Options to Use When Your Server or Domain Controller Won’t Start . . . . . . . . . . . . . . . . . . . . . . . . . . . .181

The Definitive Guide to Active Directory Troubleshooting table of contents

Page 8: The Definitive Guide to Active Directory Troubleshooting

Safe Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181The Recovery Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .182

Using the Recovery Console . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .183Adding the Recovery Console to the Startup Options . . . . . . . . . . . . . . . . . . . . . . . . . . .183

Windows 2000 Setup Emergency Repair Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .184Developing a Backup and Restore Strategy for Active Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .185

Backing Up Active Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .185Restoring Active Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .187Non-authoritative Restores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .188Authoritative Restores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .190Verifying Restores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .190

Advanced . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .190Basic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .190

The Active Directory Backup Bug . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .191How It Occurs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .192Working Around It . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .192

Repairing a Domain Controller in Active Directory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .192Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .193

The Definitive Guide to Active Directory Troubleshooting table of contents

Page 9: The Definitive Guide to Active Directory Troubleshooting

By Sean Daily, Series Editor

Welcome to The Definitive Guide to Active Directory Troubleshooting!

The book you now hold in your hands - or, in many cases - are reading on your screen, represents an entirely newmodality of book publishing and a major first in the publishing industry. The founding concept behindRealtimepublishers.com was the idea of providing readers with high-quality books on today's most critical IT topics-- at no cost to the reader. Although this may sound like a somewhat impossible feat to achieve, it is made possiblethrough the vision and generosity of corporate sponsors such as NetPro, who agree to bear the book's productionexpenses and host the book on their website for the benefit of their website visitors.

It should be pointed out that the free nature of these books does not in any way diminish their quality. Withoutreservation, I can tell you that the book you're about to read is the equivalent of any similar printed book youmight find your local bookstore (with the notable exception that it won't cost you $30 to $80). In addition to thefree nature of the books themselves, this publishing model also provides other significant benefits. For example, theelectronic nature of this eBook makes events such as chapter updates and additions, or the release of a new editionof the book possible to achieve in a far shorter timeframe than is possible with printed books. Because we publishour titles in "real-time" - that is, as chapters are written or revised by the author - you benefit from receiving theinformation immediately rather than having to wait months or years to receive a complete product.

Finally, I'd like to note that although it is true that the sponsor's website is the exclusive online location of thebook, this book is by no means a paid advertisement. Realtimepublishers.com is an independent publishing companyand maintains, by written agreement with the sponsor, 100% editorial control over the content of our titles.However, by hosting this information, NetPro has also set themselves apart from their competitors by providingreal value to their customers and by transforming their site into a true technical resource library - not just a place tolearn about their company and products. It is my opinion that this system of content delivery is not only ofimmeasurable value to readers, but represents the future of book publishing.

As series editor, it is my raison d'être to locate and work only with the industry's leading authors and editors, andpublish books that help IT personnel, IT managers, and users to do their everyday jobs. To that end, I encourageand welcome your feedback on this or any other book in the Realtimepublishers.com series. If you would like tosubmit a comment, question, or suggestion, please do so by sending an e-mail to [email protected],leaving feedback on our website at www.realtimepublishers.com, or calling us at (707)539-5280.

Thanks for reading, and enjoy!

Sean DailySeries Editor

The Definitive Guide to Active Directory Troubleshooting Introduction

eBook Introduction

Page 10: The Definitive Guide to Active Directory Troubleshooting

The Definitive Guide to Active Directory Troubleshooting

Chapter 1

Page 11: The Definitive Guide to Active Directory Troubleshooting

Introducing Active DirectoryAs computer networks have evolved over the years, the focus in enterprise computing has shifted away from a PCnetwork operating system-centric (NOS) model to one based on the concept of directories, or directory services.A directory service is a network service that stores information about network resources and makes those resourcesavailable to network users and applications. Directories also provide an environment that allows for the uniformnaming, location, access, management, and security of network resources. These days, nearly all companies withlarge enterprise-level networks, and even many of those with small- to medium-sized networks, employ one ormore directories within their organization. Although the concept of directories has been around for some time, itis only in recent years that the directory has moved into the limelight of network computing.

Although Microsoft’s Windows NT operating system introduced a pseudo-directory in the form of the NT DirectoryService (whose heart and soul was the Security Accounts Manager – SAM – database), this "directory" had a numberof major limitations. Among these were:

Non-hierarchical structure and namespaceNT’s directory used a flat, non-hierarchical directory structure that didn’t support the naming and structuralneeds of complex organizations.

Lack of extensibilityNT’s directory stored only basic user information and couldn’t be inherently extended.

Lack of scalabilityThe NT directory was stored inside the NT system registry database; due to this architecture, the maximumnumber of users topped out in the neighborhood of around 40,000 per domain.

Poor manageability featuresAdministration roles weren’t layered and couldn’t be natively delegated.

Poor directory replication performanceBecause NT’s architecture was bandwidth- and network topology-ignorant, the NT operating system couldn’tautomatically tune replication frequency and bandwidth usage to adapt to variable WAN link speeds between multiple physical locations within a network.

Single-master, Single point of failure architectureNT’s architecture called for a single server in each network domain – the Primary Domain Controller (PDC) – to house the "master" copy of the directory, thus making it a single point of failure for logon authentication for the entire domain.

In Windows 2000 (Win2K), NT’s successor OS, Microsoft set out to deliver a directory capable of addressing eachof these limitations. Win2K’s new directory service, dubbed Active Directory (AD), provides an industrial-strength

Chapter 1 www.netpro.com1

Page 12: The Definitive Guide to Active Directory Troubleshooting

Chapter 1 www.netpro.com2

directory service that can serve the needs of both small and very large organizations, and everyone in between.Because it stores its data outside the system registry, AD has virtually unlimited storage capacity (AD databases cancontain hundreds of millions of entries, as compared to the tens of thousands NT is capable of ). AD allowsadministrators to define physical attributes of their network, such as individual sites and their connecting WANlinks, as well as the logical layout of network resources such as computers and users. Using this information, AD isable to self-optimize its bandwidth usage in multi-site WAN environments. AD also introduces a new administrationmodel that provides a far more granular and less monolithic than was present under NT 4.0. Finally, AD alsoprovides a central point of access control for network users, which means users can log in once and gain access toall network resources.

Although other directories such as Banyan’s StreetTalk and Novell’s NDS have existed for some time, manyWindows NT-centric organizations have opted to wait and use Microsoft’s entry in the enterprise directory arenaas the foundation for their organization-wide directory environment. Consequently, Win2K’s Active Directory willrepresent the first foray into the larger world of directories and directory management for many organizations andnetwork administrators.

The Importance of Directories and Directory ManagementDirectories provide a logically centralized repository for all critical information within an enterprise network. Ratherthan spreading information around between many different databases, organizations can use a centralized directorysuch as Win2K’s Active Directory to consolidate all critical company information in a single shared networkresource. In addition to improving organizational efficiency, this move also allows for significant reductions in thetotal cost of ownership (TCO) of the corporate network. The concept of wholesale migration to Active Directoryhas also become more feasible with both existing and announced support from major application vendors, includingthose producing enterprise resource planning (ERP), groupware, human resources, and accounting packages.

Many Eggs, One BasketAlthough the large-scale centralization and consolidation of critical data is one of the most significant benefits ofmigrating to a directory-based network operating system such as Win2K and its Active Directory, this also repre-sents one of its greatest potential weaknesses. Whenever critical information is moved from a distributed model toone that is highly centralized, the tolerance for downtime and problems is greatly reduced, while at the same timethe risk of loss due to downtime is increased. Furthermore, many organizations planning Win2K migrations havechosen to focus the majority of their preparatory efforts and budgets on issues such as legacy hardware and soft-ware compatibility, and application interoperability under the Win2K environment. Although these are certainlyworthwhile and important considerations, they are by no means the only steps required to guarantee a successfulWin2K deployment. In addition to compatibility and capacity issues, IT departments within these organizationsmust also determine what additional tools, information, and training that will be required to properly supporttheir Win2K network environment on a day-to-day basis.

New Tools for New TimesTo effectively support Win2K networks, administrators need to engage in additional network management activi-ties beyond those taken with previous versions of Windows NT in order to maintain the same levels of networkavailability they had in the past. With any computer network, it is imperative that critical statistics such as serverCPU, memory, and disk utilization, as well as network connectivity statistics be monitored on an ongoing basis.However, Win2K introduces additional components, services, and dependencies that must also be regularly moni-tored alongside these other metrics. These elements, which collectively comprise Win2K’s core infrastructure,include items such as domain controllers, Active Directory databases and services, the Global Catalog, intra- andinter-site replication, site links, and DNS servers. Because Win2K and Win2K-centric applications rely heavily onthese services and components for proper network operation, network administrators must be able to guaranteenot only their general availability, but an acceptable baseline of performance as well. Failure to do so can result in

Page 13: The Definitive Guide to Active Directory Troubleshooting

severe, network-wide problems including slow or failed user logon authorizations, failed convergence of directorydata, the inability to access critical applications, printing problems, and similar maladies. These problems are ofparticular concern for IT shops that offer service-level agreements (SLAs) to their corporate parents or clients. Tobe able to properly maintain their Win2K infrastructure, IT shops will need not only Win2K-aware monitoringand management tools, but specific knowledge about what needs to be monitored, what thresholds must be set tomaintain acceptable levels of performance, and what needs to be done in the event that problems should occur.

Meet Active DirectoryOf all of the elements that comprise a Win2K network, the most important by far is the Active Directory, Win2K’scentralized directory service. However, before we delve into the specifics of Win2K and the Active Directory, let’sfirst define some of the fundamental terms and concepts related to directory-enabled networks. A directory (whichis sometimes also referred to as a data store) maintains data about objects that exist within a network, in a hierarchicalstructure, making the information easier to understand and access. These objects include traditional networkresources such as user and machine accounts, shared network resources (such as shared directories and printers), aswell as resources such as network applications and services, security policies, and virtually any other type of objectan administrator or application wishes to store within the directory data store.

As we discussed earlier, a directory service is a composite term that includes both the directory data store as well as theservices that make the information within the directory available to users and applications. Directory services are avail-able in a variety of different types and from different sources. Operating system directories, such as Microsoft’s ActiveDirectory and Novell’s NDS, are general purpose directories included with the network operating system and aredesigned to be accessible by a wide array of users, applications, and devices. There are also some applications, such asenterprise resource planning systems, human resource systems, and e-mail systems (e.g. Microsoft Exchange 5.x) thatprovide their own directories for storing data specific to the functionality of those applications.

Chapter 1 www.netpro.com3

Microsoft Exchange 2000 is a notable exception to this and is completely integrated with Active Directory.

Exchange 2000’s installation process extends Active Directory’s structure to accommodate Exchange-spe-

cific data and subsequently uses AD to store its own directory information.

Active Directory is Microsoft’s directory service implementation in the Win2K Server operating system. The ActiveDirectory is hosted by one or more Win2K domain controllers, and is replicated in a multi-master fashion betweenthose domain controllers to ensure greater availability of the directory and network as a whole.

The term multi-master indicates that multiple read/write copies of the database exist simultaneously, on

each Win2K domain controller computer. Thus, each Win2K domain controller is effectively an equal peer of

the other controllers, and any controller can write directory updates and propagate those updates to other

controllers. This is in notable contrast to NT 4.0’s single-master PDC/BDC replication topology wherein a

single domain controller, the PDC, houses a read/write copy of the database.

In addition to providing a centralized repository for network objects and a set of services for accessing those objects,Active Directory also provides security in the form of access control lists (ACLs) on directory objects that protectthose objects from being accessed by unauthorized parties.

Page 14: The Definitive Guide to Active Directory Troubleshooting

The AD DatabaseAt a file system level, the Active Directory uses Microsoft’s Extensible Storage Engine (ESE) to store the directorydatabase. Administrators familiar with Microsoft Exchange Server may recognize this as the same database technol-ogy used in that product. Like Exchange Server, Active Directory’s database employs transactional logging files tohelp ensure database integrity in the case of power outages and similar events that interfere with the successfulcompletion of database transactions. In addition, Active Directory also shares Exchange’s ability to perform on-linedatabase maintenance and defragmentation. At the file level, AD stores its database in a single database file namedNtds.dit, a copy of which can be found on every Win2K domain controller.

Although the building blocks that make up the Active Directory are largely masked by the directory’s high-levelmanagement interfaces and APIs, the physical aspects of the directory are nonetheless an important consideration forWin2K administrators. For example, it is critical that all volumes on domain controllers hosting the Active Directorydatabase and its transaction logs maintain adequate levels of free disk space at all times. For performance reasons, itis also important that the Active Directory databases on these machines not become too heavily fragmented.

Because Active Directory is a database, this effectively turns Win2K domain controllers into critical database serverson the network. These servers should therefore be treated no differently than any other important database server interms of fault tolerance preparation (e.g. disk redundancy, backups, and power protection) and capacity planning.

Logical Architecture of ADTo gain an appreciation for and understanding of AD and AD management concepts, it’s important to first under-stand AD’s logical architecture. In this section, we’ll discuss the most important concepts associated with AD, con-cepts which form the foundation of all Win2K networks.

Objects and AttributesJust as the primary item of storage in a file system is a file, the primary item of storage in the Active Directory is anobject. Objects can take many different forms; for example, users, computers, and printers all exist as objects with-in the directory. However, other items you might not immediately think of are also stored as objects; for example,policies that define which applications a particular group or user should have on their computer.

AD uses an object-oriented approach to defining directory objects. That is to say there exists a hierarchy of classes,which define the kinds of objects one can create (or instantiate, as in "creating an instance of...") within the direc-tory. Each class has a set of attributes that define the properties associated with that class. For example, ActiveDirectory has a user class with attributes like First Name, Address, etc.

Chapter 1 www.netpro.com4

There are special types of objects in AD known as container objects that you should be familiar with. Put simply,

container objects are objects that may contain other objects. This design allows you to organize a tree or hierarchy

of objects. Examples of container objects include organizational unit (OU) and domain objects. Container objects

may hold both objects and/or other container objects. For example, an OU object can contain both regular objects

such as users and computers, as well as other OU container objects.

Although it’s perfectly acceptable to say "create" in lieu of "instantiate" when referring to the generation of a

new object within the directory, we’ll use the latter more frequently in this book. The reason is that ‘instantiate’

is more appropriate when you consider the underlying event that actually occurs -- that being the "creation of

an instance of" an object. And, hey, let’s face it: saying ‘instantiate’ sounds a lot cooler and is more likely to

impress people (just kidding – if we really believed that we’d start throwing around words like orthogonal!)

Page 15: The Definitive Guide to Active Directory Troubleshooting

The SchemaAs you might imagine, all of the object classes and attributes discussed thus far have some kind of underlying refer-ence that describes them -- a sort of "dictionary" for Active Directory. In Win2K parlance, this "dictionary" isreferred to as the schema. The Active Directory schema contains the definitions of all object types that may beinstantiated within the directory.

Chapter 1 www.netpro.com5

Even the Active Directory schema itself is stored in the directory as objects. That is, AD classes are stored

as objects of the class "classSchema" and attributes are stored as objects of class "attributeSchema". The

schema, then, is just a number of instances of the classes "classSchema" and "attributeSchema", with

properties that describe the relationship between all classes in the AD schema.

To understand the relationship between object classes, objects, and the schema, let’s go back to the object-orientedmodel upon which the AD schema is based. As is the case with object-oriented development environments (e.g.C++, Java, etc.), a class is a kind of basic definition of an object. When I instantiate an object of a certain class, Icreate an instance of that particular object class. That object instance has a number of properties associated withthe class from which it was created. For example, suppose I create a class called "motorcycle", which has attributeslike "color," "year," "enginesize," etc. I can instantiate the class "motorcycle" and create a real object called"Yamaha YZF600R6" with properties like "red" (for the color attribute), 2000 (for the year), and 600 (for themotorcycle engine’s size in CCs).

Similarly, an Active Directory implementation within your enterprise is just the instantiation of the ActiveDirectory schema classes and attributes into hundreds or thousands of different object classes and their associatedattributes. For example, I might create an object of the class user called Craig Daily, who has properties like pass-word, address, home directory location, etc.

You can view the AD Schema through the AD Schema Microsoft Management Console (MMC) snap-in, which isshown in Figure 1.1.

Figure 1.1: Viewing the Active Directory Schema using the AD Schema MMC snap-in.

Page 16: The Definitive Guide to Active Directory Troubleshooting

LDAPOne of the early design decisions that Microsoft made regarding Active Directory was the use of an efficientdirectory access protocol known as the Lightweight Directory Access Protocol (LDAP). LDAP also benefits fromits compatibility with other existing directory services. This compatibility in turn provides for the interoperabilityof AD with these other directory services.

Chapter 1 www.netpro.com6

Viewing the AD SchemaSome of you may be curious at this point how to use the AD Schema MMC snap-in shown in Figure 1.1 to view the

AD schema. It’s not immediately obvious how to do this using the MMC console, since AD Schema isn’t available in

the default list of snap-ins that appears. One note of caution here: editing the schema is a potentially dangerous

activity—you need to know exactly what you’re doing and why you’re doing it. Before you make schema changes, be

sure to back up the current AD database contents and schema (e.g., by using ntbackup.exe or a third-party utility’s

System State backup option on an up-to-date domain controller).

To view the AD schema, use the Microsoft Management Console (MMC) Active Directory Schema snap-in, which

you’ll find among Win2K’s Support Tools. (You can install these tools from the Win2K CD-ROM’s \support folder.) To

use this snap-in, you need to manually register the snap-in by selecting Start, Run (or entering a command-prompt

session) and typing:

regsvr32 schmmgmt.dll

You’ll receive a message stating that the OS successfully registered the .dll file. You can now load and use the Active

Directory Schema snap-in through the MMC utility (i.e., mmc.exe). For example, you can open an MMC session and

choose Add/Remove Snap-in from the Console menu, then select Active Directory Schema from the Add Standalone

Snap-In dialog box (Figure 1.1 shows the Active Directory Schema snap-in’s view of the AD schema).

To modify the AD schema, you need to use a different utility: the MMC ADSI Edit snap-in. ADSI Edit is essentially a

low-level AD editor that lets you view, change, and delete AD objects and object attributes. In terms of usefulness and

potential danger, ADSI Edit is to AD what the regedit or regedt32 registry editors are to the system registry. To use

the ADSI Edit utility to make schema modifications, you first need to be a member of the Schema Admins group. The

Schema Admins group is a universal group in native-mode Win2K domains, and it’s a global group in mixed mode

Win2K domains (i.e. those which still are still running NT 4.0 domain controllers or which have no more NT domain

controller but haven’t yet been converted to Win2K’s native mode). To use the snap-in, first register the associated

adsiedit.dll file at the command line:

regsrv32 adsiedit.dll

The ADSI Edit snap-in will be available from the MMC’s Console/Add/Remove snap-in menu. Once you’ve added

the snap-in, you can use the ADSI Edit console to make changes to AD objects and attributes.

For a more advanced discussion of the AD schema and its underlying constructs, I recommend The Definitive

Guide to Win2K Administration by Sean Daily and Darren Mar-Elia. Chapter 1 of this book describes advanced

AD schema design and management issues. You can find a link to this free eBook by Realtimepublishers.com

at http://www.realtimepublishers.com.

Page 17: The Definitive Guide to Active Directory Troubleshooting

LDAP specifies that every AD object be represented by a unique name. These names are formed by combininginformation about domain components, OUs, and the name of the target object, known as a common name.

Table 1.1 lists each of these LDAP name components and their descriptions.

Chapter 1 www.netpro.com7

Active Directory supports LDAP versions 2 and 3.

Table 1.1. LDAP name components.

Attribute Type

Domain-Component

Organizational-Unit-Name

Description

An individual element of the DNS domain name of the object’sdomain; e.g. com, org, edu, realtimepublishers, microsoft, etc.

Common-Name

Organization-Name

For example, the LDAP name for the user object for a person named "David Templeton" in the realtimepublish-ers.com domain’s Marketing OU would be as follows:

CN=David Templeton,OU=Marketing,DC=realtimepublishers,DC=com

The above form of an object’s name as it appears in the directory is referred to the object’s distinguished name(DN). Alternately, an object can also be referred to using its relative distinguished name (RDN). The RDN is theportion of the DN that refers to the target object within its container. In the above example, the RDN of the userobject would simply be ‘David Templeton.

DN Abbreviation

DC

OU An organizational unit container object within an AD domain.

CN Any object other than domain components and organizationalunits, such as printers, computers, users, etc.

O The name of a single organization, such as a company.Although part of the X.500 and LDAP standards, Organizationis generally not used in directories such as Active Directory thatuse Domain Components to organize the tree structure.

Locality-Name L The name of a physical locale, such as a region or a city.Although part of the X.500 and LDAP standards, Locality isgenerally not used in directories such as Active Directory thatuse Domain Components to organize the tree structure.

Country-Name C The name of a country. Although part of the X.500 andLDAP standards, Country is generally not used in directoriessuch as Active Directory that use Domain Components toorganize the tree structure.

Page 18: The Definitive Guide to Active Directory Troubleshooting

Domains, Trees, and ForestsA significant advantage of AD is that it allows for a flexible, hierarchical design. To facilitate this design, the ADstructure employs several different logical components. The first of these components is the domain. A domainserves as the core unit in AD’s logical structure, and is defined as a collection of computers that share a commondirectory database. In fact, this definition is basically identical to that of NT domains. Like NT domains, Win2Kdomains have unique names. However, unlike the NetBIOS-based domain names used in NT, Win2K domains usea DNS naming structure (e.g. realtimepublishers.com or mydomain.org).

Domains also have several other important characteristics. First, they act as a boundary for network security: eachdomain has its own separate and unique security policy, which defines things such as password expiration and simi-lar security options. Domains also act as an administrative boundary, since administrative privileges granted tosecurity principals within a domain do not automatically transfer to other domains within AD. Finally, domains actas a unit of replication within Active Directory: since all servers acting as domain controllers in a Win2K domainreplicate directory changes to one another, they contain a complete set of the directory information related to theirdomain.

Chapter 1 www.netpro.com8

Win2K domain names don’t have to be Internet-registered domain names ending in Internet-legal top-level

domains, such as .com, .org, .net, etc. For example, it is possible to name domains with endings such as

.priv, .msft, or some other ending of your choosing. This of course assumes that the domain’s DNS servers

aren’t participating in the Internet DNS namespace hierarchy (which is by far the most common scenario, due

to security considerations with exposing internal DNS servers to the Internet). If you do elect to use standard

Internet top-level domains in your Win2K domain names, you should register these names on the Internet

even if they don’t participate in the Internet DNS namespace. The reason for this is that most organizations

are connected to the Internet, and using unregistered internal domain names that may potentially be

registered on the Internet could cause name conflicts.

Win2K’s AD design also integrates the concepts of forests and trees. A tree is a hierarchical arrangement of Win2Kdomains within AD that forms a contiguous namespace. For example, assume a domain named xcedia.com exists inyour AD structure. The two subdivisions of xcedia.com are Europe and us, which are each represented by separatedomains. Within AD, the names of these domains would be us.xcedia.com and europe.xcedia.com. These domainswould form a domain tree since they share a contiguous namespace. This arrangement demonstrates the hierarchicalstructure of AD and its namespace: all of these domains are part of one contiguous related namespace in thedirectory; that is to say, they form a single domain tree. The name of the tree is the root level of the tree, in thiscase, xcedia.com.

Figure 1.2 shows the single-domain tree described in this example.

Page 19: The Definitive Guide to Active Directory Troubleshooting

A forest is a collection of one or more trees. A forest can be as simple as a single Win2K domain, or more complexsuch as a collection of multi-tiered domain trees.

Let's take our single-tree example scenario from earlier a step further. Assume that within this AD, the parentorganization Xcedia also has a subsidiary company with a Win2K/DNS domain name of Realtimepublishers.com.Although the parent company wants to have both organizations defined within the same AD forest, it wants theirdomain and DNS names to be unique. To facilitate this, you would define the domains used by the two organizationswithin separate trees in the same AD forest. Figure 1.3 illustrates this scenario. All domains within a forest (eventhose in different trees) share a schema, configuration, and global catalog (we’ll discuss the global catalog in a latersection). In addition, all domains within a forest automatically trust one another due to the transitive, hierarchicalKerberos trusts that are automatically established between all domains in a Win2K forest.

Chapter 1 www.netpro.com9

Figure 1.2: An Active Directory forest with a single tree.

The Kerberos Version 5 authentication protocol is a distributed security protocol based on Internet standards,

and is the default security mechanism used for domain authentication within or across Win2K domains.

Kerberos replaces Windows NT LAN Manager (NTLM) authentication used in Windows NT Server 4.0, as the

primary security protocol for access to resources within or across Win2K Server domains. Win2K domain

controllers still support NTLM to provide backward compatibility with Windows NT 4.0 machines.

Page 20: The Definitive Guide to Active Directory Troubleshooting

Chapter 1 www.netpro.com10

Figure 1.3: Example of a multi-tree Active Directory forest.

In the case of a forest with multiple trees, the name of the forest is the name of the first domain created

within the forest (i.e. the root domain of the first tree created in the forest).

Although cohabitation of different organizations within the same AD forest is appropriate in some circum-

stances, in others it is not. For example, unique security or schema needs may require two companies to

use entirely different AD forests. In these situations, Kerberos trusts aren’t established between the two

forests but you may create explicit trusts between individual domains in different forests.

There are several resources you might find helpful when planning your organization’s AD structure and

namespace. One is The Definitive Guide to Active Directory Design and Planning, another free eBook from

Realtimepublishers.com (a link to which can be found at http://www.realtimepublishers.com). There are

also several Microsoft white papers that contain valuable information on AD design and architectural

concepts, including "Active Directory Architecture" and "Domain Upgrades and Active Directory." These

and others technical documents related to AD can be found on Microsoft’s Web site at

http://www.microsoft.com/windows2000/server.

Page 21: The Definitive Guide to Active Directory Troubleshooting

Organizational UnitsAn organizational unit (OU) is a special container object that is used to organize other objects – such as computers,users, and printers – within a domain. OUs can contain all of these object types, and even other OUs (this typeof configuration is referred to as nested OUs). OUs are a particularly important element of Active Directory forseveral reasons. First, they provide the ability to define a logical hierarchy within the directory without creatingadditional domains. OUs allow domain administrators to subdivide their domains into discrete sections anddelegate administrative duties to others. More importantly, this delegation can be accomplished without necessarilygiving the delegated individuals administrative rights to the rest of the domain. As such, OUs facilitate theorganization of resources within a domain.

Chapter 1 www.netpro.com11

There are a number of models used for the design of OU hierarchies within domains, but the two most

common are those dividing the domain organizationally (e.g. by business unit) or geographically.

An example of OUs within a domain is shown in Figure 1.4.

Figure 1.4: Organizational units (OUs) within a domain.

The Global CatalogBecause Active Directory is the central component of a Win2K network, network clients and servers frequentlyquery it. In order to increase the availability of Active Directory data on the network as well as the efficiency ofdirectory object queries from clients, Win2K introduces a service known as the global catalog. The global catalog isa separate database from the Active Directory itself, and contains a partial, read-only replica of all the directoryobjects in the entire AD forest.

Only Win2K servers acting as domain controllers may be configured as global catalog servers. By default, the firstdomain controller in a Win2K forest is automatically configured to be a global catalog server (this can be moved laterto a different domain controller if desired; however, every Win2K forest must contain at least one global catalog). LikeActive Directory itself, the global catalog uses replication in order to ensure updates between the various globalcatalog servers within a Win2K domain or forest. In addition to being a repository of commonly queried ADobject attributes, the global catalog plays two primary roles on a Win2K network:

Page 22: The Definitive Guide to Active Directory Troubleshooting

Network logon authenticationIn native-mode Win2K domains (networks where all domain controllers have been upgraded to Win2K and the native mode election has been manually made by the administrator), the global catalog facilitates network logons for Active Directory-enabled clients. It does so by providing universal group membership information to the account sending the logon request to a domain controller. This applies not only to regular users but also to every type of object that must authenticate to the Active Directory (including computers, etc.). In multi-domain networks, at least one domain controller acting as a global catalog must be available in order for users to log on. Another situation that requires a global catalog server occurs when a user attempts to log on with a user principal name (UPN) other than the default. If a global catalog server is not available in these circumstances, users will only be able to login to the local computer (the one exception is members of the domain administrators group, who do not require a global catalog server in order to log on to the network).

Directory searches and queriesWith Active Directory, read requests such as directory searches and queries by far tend to outweigh write-oriented requests such as directory updates (e.g. by an administrator or during replication). The majority of Active Directory-related network traffic on a Win2K network is comprised of requests from users, administrators, and applications about objects in the directory. As a result, the global catalog is essential to a Win2K infrastructure since it allows clients to quickly perform searches across all domains within a forest.

Chapter 1 www.netpro.com12

Note: Although mixed-mode Win2K domains do not require the global catalog for the network logon authentication

process, global catalogs are still important in facilitating directory queries and searches on these networks and

should therefore be made available at each site within the network.

Physical Structure of ADThus far, our discussion of AD has focused on the logical components of the directory’s architecture; that is, thecomponents used to structure and organize network resources within the directory. However, an AD-based networkalso incorporates a physical structure, which is used to configure and manage network traffic.

Domain ControllersThe concept of a domain controller has been around since the introduction of Windows NT. As is the case withNT, a Win2K domain controller is a server that houses a replica of the directory (in the case of Win2K, the direc-tory being AD rather than the NT SAM database). Domain controllers are also responsible for replicating changesto the directory to other domain controllers in the same domain. Additionally, domain controllers are responsiblefor user logons and other directory authentication, as well as directory searches.

Fortunately, Win2K does away with NT’s restriction that converting a domain controller to a member server or

vice-versa requires reinstallation of the server OS. In Win2K, servers may be promoted or demoted to domain

controller status dynamically (and without reinstallation) using the Dcpromo.exe domain controller promotion

wizard.

Page 23: The Definitive Guide to Active Directory Troubleshooting

At least one domain controller must be present in a domain, and for fault tolerance reasons it’s a good idea to havemore than one domain controller at any larger site (e.g. a main office or large branch office). To prevent user logontraffic from crossing over WAN links, you should consider putting at least one domain controller in even the small-est of your branch offices and similar remote sites.

Directory ReplicationAs we’ve discussed, domain controllers are responsible for propagating directory updates they receive (e.g. a newuser object or password change) to other domain controllers. This process is known as directory replication, andcan be responsible for a significant amount of WAN traffic on many networks.

The Win2K Active Directory is replicated in a multi-master fashion between all domain controllers within adomain to ensure greater availability of the directory and network as a whole. The term multi-master indicates thatmultiple read/write copies of the database exist simultaneously on each Win2K domain controller computer. Thus,each Win2K domain controller is effectively a peer of the other controllers, and any domain controller can writedirectory updates and propagate those updates to other domain controllers. This is in notable contrast to NT 4.0’ssingle-master PDC/BDC replication topology wherein a single domain controller, the PDC, houses a read/writecopy of the database.

Chapter 1 www.netpro.com13

AD’s replication design means that different domain controllers within the domain may hold different data

at any given time – but usually only for short periods of time. As a result, individual domain controllers may

be temporarily out of date at any given time and unable to authenticate a login request. Active Directory’s

replication process has the characteristic of bringing all domain controllers up to date with each other; this

characteristic is called "convergence".

The Operations MastersAlthough multi-master replication is a central feature of Active Directory and Win2K networks, the potential forcollisions and conflict between multiple servers makes this functionality inappropriate for some network operationsand roles. Win2K accommodates these special cases by electing specific domain controllers to serve as operationsmasters (also referred to as flexible single master operations or FSMOs) for each of these network roles. There arefive different types of operations masters in Win2K: two that are forest-specific and three that are domain-specific.Win2K automatically elects the operation master servers during the creation of each Active Directory forest anddomain.

Page 24: The Definitive Guide to Active Directory Troubleshooting

When you use the Active Directory Installation Wizard to create the first domain in a new forest, all five of the]single master operations roles are automatically assigned to the first domain controller in that domain. In a smallActive Directory forest with only one domain and one domain controller, that domain controller continues to ownall the operations master roles. In a larger network, whether with one or multiple domains, you can re-assign theseroles to one or more of the other domain controllers.

The two forest-specific operations master roles are listed below:

Schema masterThe domain controller that serves the schema master role is responsible for all updates and modifications to the forest-wide Active Directory schema. The schema defines every type of object and object attribute that can be stored within the directory. Modifications to a forest’s schema can only be done by members of the Schema Administrators group, and can be done only on the domain controller that holds the schema master role.

Domain naming masterThe domain controller elected to the domain naming master role is responsible for making changes to the forest-wide domain name space of the Active Directory. This domain controller is the only one that can add or remove a domain from the directory, or add/remove references to domains in external directories.

The three domain-specific operations master roles are as follows:

PDC emulatorIf a Win2K domain contains non-AD-enabled clients or is a mixed-mode domain containing Windows NT backup domain controllers (BDCs), the PDC emulator acts as a Windows NT primary domain controller (PDC) for these systems. In addition to replicating the Windows NT-compatible portion of directory updates to all BDCs, the PDC emulator is also responsible for time synchronization on the network (which is important for Win2K’s Kerberos security mechanism), and processing account lockouts and client password changes.

RID masterThe RID (relative ID) master allocates sequences of RIDs to each domain controller in its domain. Whenever a Win2K domain controller creates an object such as a user, group, or computer, that object must be assigned a unique security identifier (SID). A SID consists of a domain security ID (this is identical for allSIDs within a domain), and a RID. When a domain controller has exhausted its internal pool of RIDs, it requests another pool from the RID Master domain controller.

Infrastructure masterWhen an object in one domain is referenced by an object in another domain, it represents the reference by the Globally Unique IDentifier (GUID), the Security IDentifier (SID – for objects that reference security principals), and the Distinguished Name (DN) of the object being referenced. The infrastructure master is the domain controller responsible for updating an object's SID and distinguished name in a cross-domain object reference. The infrastructure master is also responsible for updating all inter-domain references any time an object referenced by another object moves (for example, whenever the members of groups are renamed or changed, the infrastructure master updates the group-to-user references). The infrastructure masterdistributes updates using multi-master replication. Note: except where there is only one domain controller ina domain, never assign the infrastructure master role to the domain controller that is also acting as a global catalog server. If you use a global catalog server, the infrastructure master will not function properly. Specifically, the effect will be that cross-domain object references in the domain will not be updated. In a

Chapter 1 www.netpro.com14

Page 25: The Definitive Guide to Active Directory Troubleshooting

situation where all domain controllers in a domain are also acting as global catalog servers, the infrastructure master role is unnecessary since all domain controllers will have current data.

Because the operations masters play such critically important roles on a Win2K network, it’s essential for propernetwork operation that all the servers hosting these roles are continually available.

SitesThe final, and perhaps most important component of AD’s physical structure is a site. Sites allow administrators todefine the physical topology of a Win2K network, something that wasn’t possible under Windows NT. Sites can bethought of as areas of fast connectivity (e.g. individual office LANs), but are defined within AD as a collection ofone or more IP subnets. When you look at the structure of the IP protocol, this begins to make sense – differentphysical locations on a network are typically going to be connected by a router, which in turn necessitates the useof different IP subnets on each network. It’s also possible to group multiple, non-contiguous IP subnets together toform a single site.

So, why are sites important? The primary reason is that the definition of sites makes it possible for AD to gainsome understanding of the underlying physical network topology, and tune replication frequency and bandwidthusage accordingly (under NT, this could only be done via manual adjustments to the replication service). This"intelligence" conferred by the knowledge of the network layout has numerous other benefits. For example, itallows AD-enabled computers hosting users who are logging into the network to automatically locate their closestdomain controller and use that controller to authenticate. This helps prevent a situation commonly seen under NT,wherein logon authentication requests often traverse low-bandwidth WAN connections to remote domain con-trollers in situations where the local domain controller is temporarily busy and the client computer has erroneouslycached the faraway controller as the default controller to use for logons. In a similar fashion, sites also give othercomponents within a Win2K network new intelligence. For example, a client computer connecting to a serverrunning the Distributed File System (Dfs) feature in Win2K can use sites to locate the closest DFS replica server.

It’s important to remember that sites are part of the physical structure of AD and are in no way related to the logicalconstructs we’ve already discussed, such as domains and OUs. It’s possible for a single domain to span multiplesites, or conversely for a single site to encompass multiple domains. The proper definition of sites is an essentialaspect of AD and Win2K network design planning.

Chapter 1 www.netpro.com15

For sites that house multiple domains (e.g. an organization that divides business units into domains rather

than OUs, thus hosting multiple business unit domains on a single site), it’s important to remember to place

at least one, and possibly two, domain controllers for each domain that users will authenticate to from that

site. This outlines the biggest disadvantage of the business unit domain model: the potential for requiring

many domain controllers at each and every site.

Page 26: The Definitive Guide to Active Directory Troubleshooting

AD’s Backbone: DNSThe TCP/IP network protocol plays a far larger role in Win2K than with previous versions of Windows NT.Although other legacy protocols such as IPX and NetBEUI continue to be supported, most of the internalmechanics of Win2K networks and Active Directory are based on TCP/IP.

In Win2K, as with all TCP/IP-based networks, the ability to resolve names to IP addresses is an essential service. Abounded area within which a given name can be resolved is referred to as a namespace. In Windows NT-basednetworks, NetBIOS is the primary namespace, and WINS the primary name-to-IP address resolution service. WithWin2K, Microsoft has abandoned the use of NetBIOS as the primary network namespace and replaced it withDNS, which is also used on the Internet.

Like Active Directory, DNS provides a hierarchical namespace. Both systems also make use of domains, althoughthey define them somewhat differently. Computer systems (called "hosts") in a DNS domain are identified by theirfully qualified domain name (FQDN), which is formed by appending the host’s name to the domain name withinwhich the host is located. Multi-part domain names (i.e. domains that are several levels deep in the hierarchy of theDNS namespace) are listed with most important domain division (e.g. .com, .org, .edu, etc.) at right and the leastimportant – the host name – at left. In this way, a host system’s FQDN indicates its position within the DNShierarchy. For example, the FQDN of a computer named ‘mercury’ located in the domain ‘realtimepublishers.com’would be mercury.realtimepublishers.com.

Although it is possible to incorporate a DNS namespace within a Windows NT network for name-to-IP addressresolution, the use of DNS is optional and mainly of interest to enterprises running in heterogeneous environmentsor Internet-based applications. However, DNS plays a far more critical role in the Win2K Active Directory. InWin2K, DNS replaces NetBIOS as the default name resolution service (however, it is still possible to continueusing a NetBIOS namespace and WINS servers on a Win2K network for legacy systems and applications). Inaddition, Win2K domains use a DNS-style naming structure (e.g. a Win2K domain might have a name such as‘santarosa.realtimepublishers.com’ or ‘mydomain.net’), which means the namespace of Active Directory domains isdirectly tied to that of the network’s DNS namespace.

Chapter 1 www.netpro.com16

This namespace duplication will normally be limited to the internal DNS namespace for companies using the

Microsoft-recommended configuration of separate DNS configurations for the internal LAN and the Internet.

Page 27: The Definitive Guide to Active Directory Troubleshooting

Finally, Win2K and Active Directory use DNS as the default locator service; that is, the service used to convertitems such as Active Directory domain, site, and service names to IP addresses.

It’s important to remember that although the DNS and Active Directory namespaces in a Win2K network areidentical in regards to domain names, the namespaces are otherwise unique and used for different purposes. DNSdatabases contain domains and the record contents (e.g. host address/A records, server resource/SRV records, mailexchanger/MX records, etc.) of the DNS zone files for those domains, whereas the Active Directory contains a widevariety of different objects including domains, organizational units, users, computer, and group policy objects.

Another notable connection between DNS and the Active Directory is that Win2K DNS servers can be configuredto store their DNS domain zone files directly within the Active Directory itself, rather than in external text files.Although DNS doesn’t rely on the Active Directory for its functionality, the converse is not true: Active Directoryrelies on the presence of DNS for its operation.

Win2K includes an implementation of Dynamic DNS (defined by RFC 2136) that allows AD-enabled clients tolocate important Win2K network resources, such as domain controllers, through special DNS resource recordscalled SRV records. The accuracy of these SRV records is therefore critical to the proper functioning of a Win2Knetwork (not to mention the availability of the systems and services they reference).

Introduction to AD and Win2K MonitoringAs you’ve already learned, Win2K introduces a number of new and important infrastructure components that didnot exist in Windows NT networks. As a result, ensuring the health and availability of your Win2K servers meansthat you will need to account for these additional components in your network monitoring routine. Monitoringwill provide early warning indicators that will help mitigate the risk of loss associated with network downtime.

The network monitoring procedures employed by most organizations tend to fall into one of the following categories:

Limited or no proactive monitoring procedures in place.Unfortunately, IT departments in some organizations are purely reactive when it comes to network infrastructure problems, and either do not regularly monitor critical network resources and components, or havelimited monitoring in place. Some may conduct regular reviews of server event logs or generate reports based on these logs, but since such information is delivered in an on-demand fashion, it is of diminished value when compared to real-time monitoring systems. These organizations will be at high risk of downtime and financial loss in a Win2K environment.

Existing monitoring procedures in place using home-grown or built-in tools.A second category is where the need for proactive network monitoring is recognized, but has been implemented by the organization using basic, low-cost tools (e.g. shareware/freeware utilities or those provided with the OS or the Resource Kit). This includes tools such as Event Viewer and Performance Monitor, Resource Kit utilities (e.g. NLTEST, BROWMON, NETDOM, DOMMON, DATALOG, REPADMIN, REPLMON, DFSCHECK, and similar utilities), and freeware/shareware utilities (e.g. utilities that test machine and service availability using PING, NT service status queries, and queries to well-defined ports such as DNS, HTTP, and FTP, etc.) Although all of these tools can be helpful in ensuring network health, many require high levels of attention from administrators and suffer from significant limitations when it comes to scalability and identifying the various types of problems that may exist on the network.

Chapter 1 www.netpro.com17

Page 28: The Definitive Guide to Active Directory Troubleshooting

Existing monitoring procedures in place with full-featured network monitoring tools.The third category is organizations with network monitoring routines built on sophisticated, full-featured network monitoring software. In addition to many of the basic services provided by the tools that come withWindows NT/2000, the Resource Kits, and freeware/shareware utilities, these utilities typically include intelligent scripting to provide sophisticated testing as well as corrective actions in the event of failure. In addition, many network-monitoring tools include a knowledge base that helps administrators understand why a problem is happening and offer suggestions as to how to resolve it. For organizations running large or multi-site Win2K networks, this type of tool is highly recommended.

For administrators of Windows NT (or other operating systems) networks that have existing monitoring tools andprocedures, the migration to Win2K will mainly involve an upgrade of existing tools and staff knowledge about thevulnerabilities of the new environment. However, if your organization has employed a more reactive stance (i.e. fixit only when it breaks) with regards to resolving network problems, you’ll quickly find that this methodology canbe especially troublesome in a Win2K environment.

Although it is true that Win2K provides a far greater level of reliability and performance than its predecessors, italso involves a higher number of "moving parts" and dependencies that need to be accounted for. Although legacyWindows NT networks have their own set of dependencies and vulnerabilities, they are far fewer in number due toNT’s simpler (and less capable) network architecture. Let’s quickly review the primary monitoring considerations ina Windows NT environment are as follows:

PDC availability and performanceDue to the single-master nature of NT domains, there is a high dependence (and thus, a high availability requirement) on the Primary Domain Controller (PDC) of each Windows NT domain. Although Backup Domain Controllers exist to create fault-tolerance and load-balancing for client logon authentication, a Windows NT domain without a PDC essentially grinds to a halt until the PDC is brought back online or replaced via the manual promotion of a BDC to PDC status by a network administrator. In addition, network logon traffic loads on domain controllers should be monitored to assess domain controller performance and the ability to respond to client network logon authentication requests within an acceptable period of time.

Domain trust relationshipsOn multi-domain NT networks, there typically exists a complex array of trust relationships between domains in order to accommodate network access requirements for the business. NT trust relationships (formed between domain controllers) are notoriously fragile and prone to failure, and thus require continualmonitoring and testing in order to assure the availability of network resources to users.

Name serversAnother aspect of NT networks requiring continual monitoring is the availability of network name servers. For the majority of Windows NT-based networks (including those with Windows 95/98/ME/2000 clients), NetBIOS is the predominant namespace and Windows Internet Name Service (WINS) the predominant name-to-IP address resolution service. WINS databases and replication are also notoriously fragile elements of NT networks, and must be regularly monitored to ensure their functionality. Even for networks using DNS as the primary name resolution service, the availability of the DNS name servers is equally important as it is with WINS.

Chapter 1 www.netpro.com18

Page 29: The Definitive Guide to Active Directory Troubleshooting

Network browser serviceWindows NT, Windows Me, Windows 98, Windows 95, and other members of the Windows product familyrely on a network browsing service to build lists of available network resources (e.g. servers, shared directories, and shared printers). The architecture of this service, which calls for each eligible network node to participate in frequent elections to determine a browse master and backup servers for each network segment is another infamously unreliable aspect of Microsoft networks and which requires frequent attention and maintenance.

Other critical services and applicationsIn addition to name resolution services such as WINS and DNS, NT environments may house othermission-critical services required for proper operation of the network or the business in question. Forexample, critical applications such as backup, antivirus, mail, FTP, web, and database servers should be polled using intelligent service-level queries to verify that they are functioning properly and at acceptablelevels of performance.

Basic network and system metricsAll networks, Windows NT or otherwise, should be monitored to protect against problems stemming from resource allocation problems on individual servers or the network itself. For example, any good network monitoring regimen will include the monitoring of CPU, memory, and disk space resource usage on allcritical servers, as well as network connectivity and bandwidth usage.

AD and Win2K Monitoring ConsiderationsA functioning Win2K network is a complex mesh of relationships and dependencies involving a variety of differentsystems and services, including Active Directory, DNS, the global catalog, and operations master servers. Runningan effective Win2K network means having a handle of every aspect of your network environment at all times.

It’s no surprise that the primary monitoring consideration in Win2K is Active Directory and its related services andcomponents. This includes responsiveness to DNS and LDAP queries, AD inter-site and intra-site replication, anda special Win2K service called the Knowledge Consistency Checker (KCC). In addition, the health and availabilityof services such as DNS, the global catalog, and DFS are also important.

Chapter 1 www.netpro.com19

The Knowledge Consistency Checker (KCC) is a special Win2K service that automatically generates Active

Directory’s replication topology, and ensures that all domain controllers on the network participate in replication.

However, knowing what metrics to monitor is only a first step. By far, the most important and complex aspect ofmonitoring network health and performance isn’t related to determining what to monitor, but rather how to digestthe raw data collected from the array of metrics and make useful determinations from that data. For example,although it would be possible to collect data on several dozen metrics (e.g. via Performance Monitor) related toActive Directory replication, simply having this information at hand doesn’t tell you how to interpret the data, orwhat you should consider acceptable tolerance ranges for each metric. A useful monitoring system not only collectsraw data, but also understands the inter-relation of that data and how to use the information to identify problemson the network. This kind of artificial intelligence represents the true value of network monitoring software.

Page 30: The Definitive Guide to Active Directory Troubleshooting

In order to ensure the health and availability of the Active Directory as well as other critical Win2K network services,organizations will need to regularly monitor a number of different services and components, which are listed inTable 1.2.

Chapter 1 www.netpro.com20

Category

Domain Controllers/ActiveDirectory

Potential Problems

Low CPU or memory resources on domain controllers

Low disk space on volumes housing the SYSVOL folder, the Active Directorydatabase (NTDS.DIT) file, and/or the Active Directory transactional log files

Slow or broken connections between domain controllers (within a site or across sites)

Slow or failed client network logon authentication requests

Slow or failed LDAP query responses

Slow or failed Key Distribution Center (KDC) requests

Slow or failed Active Directory synchronization requests

NetLogon (LSASS) service not functioning properly

Directory Service Agent (DSA) service not functioning properly

Knowledge Consistency Checker (KCC) not functioning properly

Excessive number of SMB connections

Insufficient RID allocation pool size on local server

Problems with Transitive or External trusts to Win2K or downlevel NT domains

Low Active Directory Cache Hit Rate for Name Resolution Queries (e.g. due toinefficient Active Directory Design)

Replication Failed replication (e.g. due to domain controller or network connectivity problems)

Slow replication

Replication topology invalid/incomplete (lacks transitive closure/consistency)

Replication using excessive network bandwidth

Too many properties being dropped during replication

Update Sequence Number (USN) update failures

Other miscellaneous replication-related failure events

Global Catalog Slow or failed global catalog query responses

Global catalog replication failures

Page 31: The Definitive Guide to Active Directory Troubleshooting

Change Monitoring and AuditingIn addition to monitoring and troubleshooting problems within the Win2K network infrastructure, anotherdistinct advantage of monitoring software is the ability to monitor and audit changes made to the Active Directorydatabase. In many organizations, there may be dozens or even hundreds of administrators making daily changes tothe Active Directory. In order to manage the potential chaos this situation presents, it’s essential that a system be inplace to identify all recent changes made to objects within the directory, and to be able to ascertain who did what –and when. Examples of the kinds of changes that you might want to track include changes to the Active Directoryschema, OUs, contacts, computers, printers, and directory recovery actions taken by administrators (e.g. a siteadministration restoring Active Directory on a local domain controller).

Problem Resolution, Automation, and AlertingMonitoring and troubleshooting critical network infrastructure components is an important starting point, but itis by no means the only proactive measure that you can take to increase the availability of your network. Goodnetwork monitoring software provides a wide assortment of alerting options, such as console alerts, network pop-upmessages, event log entries, e-mail alerts, pager notifications, and SNMP traps.

In addition to providing problem identification and alerting features, many third-party products also provide auto-matic problem resolution features. For example, it is possible to configure many products to take specific correctiveactions when a problem is detected, such as restarting a particular service when it is found to be unresponsive.

Chapter 1 www.netpro.com21

DNS Missing or incorrect SRV records for domain controllers

Slow or failed DNS query responses

DNS server zone file update failures

Operation Masters (FSMOs) Inaccessibility of one or more operation master (FSMO) servers (PDC emulator,

RID allocation, Infrastructure, Domain Naming, Schema)

Forest or domain-centric operation master roles not consistent across domaincontrollers within domain/forest

Slow or failed role master responses

Miscellaneous Problems Low-level network connectivity problems

TCP/IP routing problems

DHCP IP address allocation pool shortages

WINS server query or replication failures (for legacy NetBIOS systems andapplications)

Naming context lost + found items exist

Application or service failures or performance problems

Table 1.2: Common Problems in Active Directory-based Win2K Networks.

Page 32: The Definitive Guide to Active Directory Troubleshooting

Many tools use scripting and/or the ability to call external utilities to accomplish these tasks. The most comprehensiveutilities base their decisions on rule sets derived from an internal database and/or intelligent escalation routines thatemulate what an administrator might do. For example, you might configure a system such that on the first failureof a given service, that service is restarted; the computer is restarted in the event the service restart fails; a differentmachine is promoted to replace that system in the event the computer restart attempt fails, and so on.

Other ConsiderationsThere are several considerations you should keep in mind when creating a Win2K network monitoring andtroubleshooting solution. One is the overall architecture of the application(s) being used in the solution. It’simportant to understand how the product collects its data and what impact this collection will have on yournetwork and servers. For example: Does the product employ local agents to gather metrics or does it use remotequeries? Do throttling features exist to control network bandwidth and system resource usage? Is there amachine/site/domain hierarchy that allows data to be passed to the central collection database in an efficientmanner? Does the product provide web-based management? All of these questions are important since they canhave a significant impact on your network environment and your overall satisfaction with the product.

Another differentiating feature about network monitoring software packages is whether or not they provide a supportknowledgebase of common problems and solutions. This kind of knowledge is invaluable from both a technicaland financial standpoint, since it serves to reduce the learning curve of the supporting IT staff, as well as theamount of time and money administrators must expend researching and resolving problems. Some utilities augmentthis capability by allowing administrators to add their own experience to the knowledgebase or a problem trackingand resolution database, thereby leveraging internal IT staff expertise and creating a comprehensive problemresolution system.

A final feature provided by some applications, and one that may be of interest to IT shops engaged in Service LevelAgreements (SLAs), is the ability to generate alerts and reports that address exceptions to, or compliance with SLAobligations.

SummaryAlthough Win2K and Active Directory represent a quantum leap forward in the NT product line, they also introducenew levels of network infrastructure complexity that must be properly managed in order to maintain an efficient andhighly available network. Real-time, proactive monitoring and management of the Active Directory and other criticalservices is an essential part of managing Win2K-based networks. In this chapter, we discussed the most importantfeatures and components of Win2K and Active Directory, their roles within the enterprise, differences betweenmanaging Windows NT 4.0-based networks and Win2K Active Directory-based networks, and some of the basicmetrics and statistics that Win2K network administrators need to watch to help them ensure high availability ontheir networks.

In the remaining chapters of this book, we’ll drill down and explore each of the vital areas of Active Directory andWin2K networks in detail, providing the information, tools, and techniques you’ll need to employ to maintain ahealthy and highly available Win2K network.

Chapter 1 www.netpro.com22

eBook Copyright NoticeThis site contains materials created, developed, or commissioned by Realtimepublishers.com, Inc. and is protected byinternational copyright and trademark laws. No material (including but not limited to the text, images, audio, and/orvideo) may be copied, reproduced, republished, uploaded, posted, transmitted, or distributed in any way, except that onecopy may be downloaded for your personal, non-commercial use on a single computer. In connection with such use, youmay not modify or obscure any copyright or other proprietary notice. If you have any questions about these terms, or ifyou would like information about licensing materials from Realtimepublishers.com, please contact us via e-mail [email protected]

Page 33: The Definitive Guide to Active Directory Troubleshooting

The Definitive Guide to Active Directory Troubleshooting

Chapter 2

Page 34: The Definitive Guide to Active Directory Troubleshooting

Designing an Effective Active DirectoryThe main function of Active Directory (AD) is to allow the network resources for Windows 2000 (Win2K) to beidentified and accessed. AD accomplishes this goal by providing a single namespace where users and applicationscan go to register and gain access to the information they need. AD can be set up and designed in many differentways to meet the needs of users and administrators. It’s your job as an administrator to properly set up and designyour AD for maximum efficiency.

The best way to troubleshoot AD problems is to avoid AD problems in the first place. To do that, you need to startwith a good design. The design of AD not only includes the layout of the forests, trees, domains, and OrganizationalUnits (OUs) but also the site and site links that represent the physical network.

In this chapter, I’ll give you a solid understanding of how to design an AD for your environment and network.Because the information in AD can be distributed across the network, there may be unique aspects of your designand implementation that only apply to your site. However, my goal is to give you enough information to ensurethat the design serves the needs of your users and administrators.

Active Directory’s Logical and Physical StructuresAs I mentioned in Chapter 1, AD has internal structures that can be categorized as logical and physical. Thesestructures are the building blocks you’ll use to design and build your AD service. Your challenge is to understandeach building block and use it to build an efficient AD. The concepts behind these structures are sometimes noteasy, but understanding them and using them correctly are the keys to a good design.

Logical StructuresA list of logical structures used in AD is shown in Table 2.1.

Chapter 2 www.netpro.com24

Namespace AD is a namespace because it resolves an object’s name to the object itself.

Naming context Represents a contiguous subtree of AD.

Organizational Unit A container object that allows you to organize your objects and resources.

Domain A partition in AD that provides a place to group users, groups, computers,

printers, servers, and other resources together.

Tree A grouping of domains.

Forest A collection of one or more trees.

Trust relationship A logical connection between two domains that forms one administrative unit.

Global catalog A central source for AD queries for users and other objects.

Logical Structure Description

Table 2.1: The logical structures of AD, which are used to design and build the object hierarchy.

Page 35: The Definitive Guide to Active Directory Troubleshooting

Two important logical structures that you need understand to design an AD are the namespace and naming context.Although these two concepts seem to mean the same thing, they’re actually different. To help you understand howthey differ, I’ll give you a quick overview of each. These structures are also discussed throughout the chapter.

NamespaceAnother term for a directory is namespace. A namespace refers to a logical space in which you can uniquely resolvea given name to a specific object in the directory. AD is a namespace because it resolves a name to the object nameand the set of domain servers that stores the object itself. Domain Name System (DNS) is a namespace because ittranslates easy-to-remember names (such as www.company.com) into an IP number address (for example,124.177.212.34).

Naming ContextThe naming context represents a contiguous subtree of AD in which a given name is resolved to an object. If youlook at the internal layout of AD, you see a structure that looks similar to a tree with branches. If you expand thetree, you see the containers, the objects that reside in them, and the attributes associated with the objects.

In AD, a single domain controller always holds at least three naming contexts.

DomainContains the object and attribute information for the domain of which the domain controller is a member

ConfigurationContains the rules for creating the objects that define the logical and physical structure of the AD forest

SchemaContains the rules for creating new objects and attributes.

Chapter 2 www.netpro.com25

AD depends on DNS and the DNS-type namespace that names and represents the domains in the forest. It’s

important to design your domain tree in a DNS-friendly way and to provide clients with reliable DNS services.

Although AD uses DNS to create its structure, DNS and AD are totally separate namespaces.

Page 36: The Definitive Guide to Active Directory Troubleshooting

Physical StructuresIn addition to the logical structures in AD, several physical structures help you implement the logical structures onyour network. These physical structures are listed in Table 2.2.

Designing Active DirectoryYour primary objective in designing AD is to build a system that reflects the network resources in your company.You need to arrange the forest and trees to reflect the location and placement of your network resources. You needto design the domains and OUs to implement an administrative and security structure for both users andadministrators. When designing the layout of AD, you also need to design the users’ groups and security policiesas well as the administrative methods you used.

From the list of logical and physical structures that you have to work with, four structures are critical to the designof AD: forests and trees, domains, OUs, and sites. Designing and implementing each of these four structures buildson the previous one. Implementing these structures properly is crucial to a successful design. Design your ADstructure in the following order:

1. Design the forest and trees

2. Design the domains for each tree

3. Design the OUs for each domain

4. Design the sites for the forest and domains.

In the next four sections, I’ll describe how to design each of these main structures.

Designing the Forest and TreesA forest is a collection of one or more trees. A forest can also be a set of domain trees that doesn’t form a commonnaming context. The trees in a forest share the same directory schema and configuration but don’t need to share thesame namespace. For example, a single directory can contain two companies or organizations. Figure 2.1 illustrateshow two companies named company1.com and company2.com form a single forest.

Chapter 2 www.netpro.com26

Object and attributes An object is defined by the set of attributes or characteristics assigned to it.

Objects include users, printers, servers, groups, computers, and security policies.

Domain controller A network server that hosts AD in a domain.

Directory server role A server that takes the role of Flexible Single Master Operation (FSMO).

Directory server roles are single-master servers that perform special roles for

AD, such as managing domains, managing schemas, and supporting down-level

clients (Windows NT).

Site A location on the physical network that contains AD servers. A site is defined as

one or more well-connected Transmission Control Protocol/Internet Protocol

(TCP/IP) subnets.

Global catalog server Stores the global catalog information for AD.

Physical Structure Description

Table 2.2: The physical structures of AD, which are used to implement the logical directory structures on the network

Page 37: The Definitive Guide to Active Directory Troubleshooting

Figure 2.1: Two companies or organizations named company1.com and company2.com can form a forest in AD.

The forest serves two main purposes. First, it simplifies workstation interaction with AD because it provides a globalcatalog where the client can perform all searches. Second, the forest simplifies administering and managing multipletrees and domains. A forest has the following key characteristics or components:

Global SchemaThe directory schema for the forest is a global schema, meaning that it’s exactly the same for each domain controllerin the forest. The schema exists as a naming context and is replicated to every domain controller. The schemadefines the object classes and the attributes of object classes.

Global Configuration ContainerThe configuration container exists as a naming context that is replicated to every domain controller in the forest.Thus, it’s exactly the same across the domain controllers in the forest. The configuration container contains theinformation that defines the structure of the forest. This information includes the domains, trust relationships,sites, site links, and the schema. By replicating the configuration container on every domain controller, each domaincontroller can reliably determine the structure of the forest, allowing it to replicate to the other domain controllers.

Complete TrustAD automatically creates bi-directional transitive trust relationships among all domains in a forest. This allows thesecurity principals, such as users and groups of users, to authenticate from any computer in the forest. However,this is only true if the users’ access rights have been set up correctly.

Global CatalogThe global catalog contains a copy of every object from every domain in the forest. However, it only stores a selectset of attributes from the objects. By default, the global catalog isn’t placed on every domain controller in the forest;instead, you determine which domain controllers should hold a copy.

Chapter 2 www.netpro.com27

Page 38: The Definitive Guide to Active Directory Troubleshooting

When designing the forest, you need to consider both the users and the administrators. For example, consider anorganization that has just acquired another company. If the two forests are merged into a single forest, all the userscan view the entire AD. However, the forests might not be merged because the two autonomous administrativegroups might not agree on how to manage the forests. The winner of this dispute depends on your priority: Doyour users have a higher priority than your administrators?

If the administrators win, the users inherit two forests and no longer have a single, consistent view of AD. Eachadministrative group manages its own forest independently.

The answer also depends on which type of organization your company is. If it isn’t important for the users to havea consistent view of AD, it might be appropriate to have multiple forests with separate administrators. For example,consider an application service provider (ASP) company, which hosts AD services on behalf of other companies. Theusers from those companies have no reason to view the host company’s information. In addition, each administrativegroup wants its independence.

Determining the Number of ForestsWhen determining the number of forests for your company, consider the requirements of the organization itself. Insmaller, centrally managed organizations, you typically need one forest. However, if the company is large and hasmultiple locations in one or multiple countries, you probably need multiple forests. To properly determine thenumber of forests for your company, you need to understand the maintenance and overhead of having one forestcompared to multiple forests.

An environment with a single forest is simple to create and maintain. All users view one AD using the global catalog.Maintaining a single forest is easy because you only need to apply configuration changes once to affect all the domainsin the forest. For example, when you add a domain to the forest, all the trust relationships are set up automatically.In addition, the new domain receives any additional changes made to the forest.

When deciding on the number of forests you need, remember that a forest has shared elements: the schema,configuration container, and global catalog. Thus, all the administrators need to agree on their content andmanagement. Managing these elements becomes more complicated when you add a forest because it incurs amanagement cost. The following is a brief list of the many management issues surrounding multiple forests:

• Each additional forest must contain at least one domain, domain controller, and someone to manage it.

• Each additional forest creates a schema. Maintaining consistency among schemas is difficult and creates overhead.

• Each additional forest creates a configuration container. Maintaining consistency among configuration containers when the network configuration changes is difficult and creates overhead.

• If you want user access among forests, you must create and maintain explicit one-way trusts for every relationship you establish.

• Users wanting access to resources outside their forest need to make explicit queries; this is difficult for theordinary user.

Chapter 2 www.netpro.com28

Page 39: The Definitive Guide to Active Directory Troubleshooting

• Any synchronization of components among multiple forests has to be done manually or using ametadirectory service or other synchronization solution.

• Users cannot easily access the network resources contained in other forests.

Setting Up and Managing Multiple ForestsHaving to set up and manage two forests might be necessary if two organizations in a company don’t trust oneanother or cannot agree on administrative policies. For example, let’s say a company has two locations in differentcountries—New York and London. Each location has its own administrative group, which needs to manage itsnetwork resources according to its own policies. In this case, two different forests can be used to separate theadministrative requirements.

In other situations, your company might have multiple forests, but you want to have a central administrativegroup. To set up central management of multiple forests, you need to add administrators to the Enterprise andSchema Administration groups of each forest. Because there is only one Enterprise and Schema Administrationgroup per forest, you must agree on a central group of administrators who can be members of these groups.

As mentioned previously, it’s difficult to manage user access between two or more forests. The simplest method isto create explicit one-way trusts among the domains that must trust one another. The one-way trust only allowsaccess among the domains in the direction that it’s set up. This approach of connecting the forest is shown inFigure 2.2.

Chapter 2 www.netpro.com29

One situation in which you may consider managing multiple forests occurs when two organizations merge or

participate in a joint venture. This puts the administration of your network into the hands of two autonomous

groups. For this reason, multiple trees are typically more costly to manage. To reduce this cost, organizations

such as partnerships and conglomerates need to form a central group that can drive the administrative

process. On the other hand, in short-lived organizations like joint ventures, it might not be realistic to expect

administrators from each organization to collaborate on forest administration.

There is another good example of a situation where multiple forests may be required. Many enterprise

organizations elect to maintain separate, parallel AD forests for testing purposes. Other organizations maintain

multiple forests because they have a disjointed organizational structure with no common infrastructure among

business units. Although you’ll certainly want to keep your network and AD design as simple as possible, your

forest structure should follow the organizational, administrative, and geographical structure of your organization.

Page 40: The Definitive Guide to Active Directory Troubleshooting

Chapter 2 www.netpro.com30

Figure 2.2: Two forests can allow user access between its domains by establishing explicit one-way trusts.Only the domains connected by the trusts can allow access between the forests.

Figure 2.3: The namespace of the domain tree shows the hierarchical structure of the tree.

Page 41: The Definitive Guide to Active Directory Troubleshooting

Explicit one-way trusts aren’t transitive. The one-way trusts in Win2K are the same as the one-way trusts that existedin Windows NT. Creating one-way trusts among multiple forests or trees can be complicated, so it’s important tokeep it simple by limiting the domains that trust one another.

Determining the Number of TreesA tree is simply a grouping of domains. The domains that form a tree are arranged hierarchical and share a commonand contiguous namespace. Trees can be viewed one of two ways. The first view is the trust relationships amongdomains. The second view is a view of the namespace of the domain trees as shown in Figure 2.3.

In a single forest in which all domains trust one another, the tree relationship is defined by the namespace that isnecessary to support the domain structure. For example, the root domain called company.com can have twosubdomains (or child domains) named seattle.company.com and chicago.company.com. The relationship betweenthe root domain and the two child domains is what forms the tree, or namespace.

In the previous section, I emphasized that multiple forests in an organization are generally not recommended.However, this isn’t to say that there aren’t situations in which multiple trees are appropriate or even recommended.For one, multiple trees allow you to have multiple namespaces that coexist in a single directory. Multiple trees giveyou additional levels of separation of the namespaces, something that domains don’t provide. Although multipletrees work well in most situations, I recommend that you start by creating one tree until the circumstances arisethat call for more.

You may be wondering if there are any extraordinary benefits to having multiple trees. For example, do multipletrees reduce the replication or synchronization that occurs among domain servers? The answer is no because theschema and configuration container are replicated to all domain controllers in the forest. In addition, the domainpartitions are only replicated among the domain controllers that are in the domain. Having multiple trees doesn’treduce replication.

Likewise, you may be wondering whether multiple trees cause any problems. For example, does having multipletrees require you to establish and maintain explicit one-way trust relationships? Again, the answer is no because thetransitive trust relationships are automatically set up among all domains in the forest. This includes all domainsthat are in separate trees but in the same forest.

Designing the DomainsThe next task in planning AD is creating and designing the domains. A domain gives you a place to group users,groups, computers, printers, servers, and other resources that belong together. In addition, a domain is a securityboundary for these objects, and it defines the set of information that is replicated among domain controllers. Thedomain in AD works like the domain that exists in Windows NT.

A domain can also be called a partition, which is a physical piece of AD that contains the object information.Figure 2.4 shows a domain structure with its contents of network resources.

Chapter 2 www.netpro.com31

Page 42: The Definitive Guide to Active Directory Troubleshooting

The purpose of domains is to logically partition the AD database. Most large implementations of AD need todivide the database into smaller pieces. Domains enable you to partition AD into smaller, more manageable unitsthat you can distribute across the network servers.

Domains are the basic building blocks of AD and Windows 2000. Domains can be connected to form the treesand forests, and what connects them are the trust relationships, which are automatically established in AD. Thesetrusts allow the users of one domain to access the information contained in the other domains. When multipledomains are connected by trust relationships and share a common schema, you have a domain tree. Every ADinstallation consists of at least one domain tree.

It’s your role as an administrator to decide the structure of domains and which objects, attributes, groups, andcomputers are created. The design of a domain includes DNS naming, security policies, administrative rights, andhow replication is handled. When you design domains, follow these steps:

• Determine the number of domains

• Choose a forest root domain

• Assign a DNS name to each domain

• Partition the forest

• Place the domain controllers for fault tolerance and high network availability

• Determine the explicit trust relationships that need to be established, if any.

Chapter 2 www.netpro.com32

Figure 2.4: The domain structure is a piece of AD. It contains the users, groups, computers, printers, servers, and other resources.

Page 43: The Definitive Guide to Active Directory Troubleshooting

Determining the Number of DomainsI recommend that you start with one domain in your environment. This is true even if there are two or more physicallocations. This design keeps the layout of your domain simple and easy to maintain. You can then add otherdomains as needed. The mistake some people make is to create a bunch of domains and not know what to do withthem. One domain will be adequate for most companies, especially smaller ones.

Although one domain will work in most circumstances, other circumstances necessitate having more than onedomain for an entire organization. Some of these are described below; you must decide which of these fit yourneeds.

Administrative RightsIf your organization has multiple administrative groups that want guaranteed autonomy, you may need to create additional domains and give each group its individual rights. For example, if two companies merge together, one group may need to operate and maintain autonomous activities.

International SettingIf your organization is international, you may need to create additional domains to support anotherlanguages. (Administrators, users, and others may need to access AD in their first language, and the schemacontains language-specific attribute display names.)

Replication TrafficBecause the AD database can be distributed, you may want to create additional domains to hold thedistributed partitions. The need to create additional domains typically arises when you have a single domain trying to replicate across wide area network (WAN) links. If the replication is too slow, you can alleviate the problem by splitting the domain into two. You can then place the two domains on each side ofthe WAN so that they can be closest to the users.

Account Security SettingsAccount security settings apply to the entire domain. Account security settings include password length andexpiration period, account lockup and intruder detection, and Kerberos ticket policy. These settings cannot be changed in the domain for individual OUs or groups. If you need to have unique settings, you may need to create another domain.

Preserve Existing Windows NT DomainIf your company already has an existing Windows NT domain, you may want to keep it instead of consolidating it into an AD domain. This requirement could produce more domains than planned.

Determining the number of domains for your organization is an individual effort. No one can tell you definitivelyhow many domains to have and how to split them without knowing more about your company’s organization andnetwork. However, using these simple guidelines, you can establish parameters that enable you to effectively designdomains and determine the appropriate number for your company.

Chapter 2 www.netpro.com33

Page 44: The Definitive Guide to Active Directory Troubleshooting

Choosing a Forest Root DomainThe first domain that you create becomes the forest root domain (or root domain), which is the top of the forest.The forest root domain is extremely important because it determines the beginning of the namespace and establishesthe forest. Because the AD forest is established with the first domain, you need to make sure that the name of theroot domain matches the top level in the namespace. For example, root domains are domains with names such ascompany.com and enterprise.com. These domain names are the roots of the DNS structures and the root of AD.Any subsequent domains you create or add to the tree form the tree hierarchy.

The first domain you create in an AD forest contains two forest-wide groups that are important to administeringthe forest, the Enterprise Administrators group and the Schema Administrators group. Containing these twogroups makes the root domain special. You cannot move or re-create these groups in another domain. Likewise,you cannot move, rename, or reinstall the root domain. In addition to these groups, the root domain contains theconfiguration container, or naming context, which also includes the schema naming context.

After you install the root domain, I recommend that you back it up often and do everything you can to protect it.For example, if all the servers holding a copy of the root domain are lost in a catastrophic event and none of themcan be restored, the root domain is permanently lost. This is because the permissions in the EnterpriseAdministrator and Schema Administrator groups are also lost. There is no method for reinstalling or recovering theroot domain and its groups in the forest other than completely backing it up and restoring it.

Using a Dedicated Root DomainAs I described above, the first domain you create in AD becomes the forest root domain. For smaller implementationsof AD, you may only need to create the root domain, nothing more.

For a larger implementation with multiple locations around the world, however, you’ll probably want to use adedicated root domain.

A dedicated root domain is a root domain that is kept small, with only a few user account objects. Keeping theroot domain small allows you to replicate it to other locations at low cost (that is, with little impact on networkusage and bandwidth). Figure 2.5 illustrates how you can replicate a dedicated root domain to the other locationson your network.

Chapter 2 www.netpro.com34

For more information on determining the number of domains, see "Determining the Number of Domains"

earlier in this chapter.

Page 45: The Definitive Guide to Active Directory Troubleshooting

A dedicated root domain focuses on the overall operations, administration, and management of AD. There are atleast two advantages to using a dedicated root domain in a larger implementation of AD.

• By keeping the user and printer objects out of the root domain, you enhance security by restricting access to only a few administrators

• By keeping the root domain small, you can replicate it to other domain controllers on the network. This approach helps increase the availability of the network.

Because domain administrators can access and change the contents of the Enterprise Administrators and SchemaAdministrators groups, having a dedicated root domain limits normal access. Membership in these built-in groupsshould only be given to the enterprise administrators, and they should only access the domain when doing officialmaintenance. In addition, membership in the Domain Administrators group of the root domain should be grantedonly to the enterprise administrators. Taking these steps allows you to avoid any accidental changes to the rootdomain. You should also create a regular user account for each of your administrators so that they don’t carryadministrative privileges while doing regular work.

As I mentioned earlier in "Choosing a Forest Root Domain," always replicate the root domain to multiple serversin an effort to provide fault tolerance for this domain. Because a dedicated root domain is small (no user or printerobjects), it can be replicated across the network more quickly and easily. In addition to replicating the root domainacross the local area network (LAN), you can replicate the root domain across the WAN to reduce the trust-traversaltraffic among trees.

Chapter 2 www.netpro.com35

Figure 2.5: A dedicated root domain is small enough to efficiently replicate copies tothe other locations on your network

Page 46: The Definitive Guide to Active Directory Troubleshooting

Assigning a DNS Name to Each DomainAfter you’ve determined the number of domains and installed the root domain, you need to determine the DNSnames for each domain. DNS is a globally recognized, industry-standard system for naming computers and networkservices that are organized in a hierarchy. AD clients make queries to DNS in an attempt to locate and log on todomains and domain controllers.

Network users are better at remembering name-based addresses, such as www.company.com, than they are atremembering number-based addresses, such as 124.177.212.34. DNS translates an easy-to-remember name address(www.company.com) into a number address (124.177.212.34).

As I’ve mentioned, the domain is identified by a DNS name. You use DNS to locate the physical domain controllerthat holds the objects and attributes in the domain. DNS names are hierarchical (like AD domains). In fact, theDNS name for a domain indicates the position of the domain in the hierarchy. For example, in the domain namecompany.com, the DNS name tells us that the domain has to be at the top of the forest and is the root domain.Another example is:

marketing.chicago.company.com

From this domain name, we know that the domain is the Marketing Department’s domain in the Chicago locationof the company. The domain is two levels from the root domain, or top of the tree. The Chicago domain is a childdomain of the root domain, or company, and the Marketing domain is a child domain under Chicago.When you create DNS names for the domains in AD, I recommend that you follow these guidelines:

• Use an Internet-registered name for the top-level domain

• Use Internet standard characters

• Use locations to name child domains

• Never use the same name twice.

Using an Internet-Registered Name for the Top-Level DomainWhen you name your top-level domain, I recommend that you only use a DNS name that has been registered onthe Internet and is thus globally unique. For example, realtimepublishers.com is a top-level domain name andshould be registered with the Internet Corporation for Assigned Names and Numbers (ICANN). You don’t need toregister the names of underlying domains because they fall under the control and jurisdiction of the top-level domainowner/registrant. For example, the domain name research.realtimepublishers.com doesn’t need to be registered.

Using Internet Standard CharactersWhen you assign DNS names, you’re restricted to using only Internet standard characters to ensure compatibilitywith AD and the Internet. The basic standards for naming are as follows:

• Domain names contain only letters, numbers, and hyphens (-)

• Domain names cannot begin or end with a hyphen

• The domain names of .com, .net, and .org cannot exceed 67 characters

Chapter 2 www.netpro.com36

Page 47: The Definitive Guide to Active Directory Troubleshooting

• Relative domain names (that is, the components between the dots in a fully qualified domain name) cannot exceed 22 characters (this doesn’t include any extensions)

• Domain names aren’t case sensitive

• Domain names cannot include spaces.

Using Locations to Name Child DomainsWhen you determine the name of the child domains, I recommend that you use the geographical locations of yourcompany. This applies to the first layer of child domains that you create under the root domain. The key to designingand naming this layer is the WAN layout: Try to match the domains to the locations in your company’s WAN. Forexample, the first layer of domains that you create under the root could have names based on physical locations.Figure 2.6 portrays the first layer of domain names, representing the physical, or WAN, locations on your network.I make this recommendation because other naming schemes based on business structures and organizations areprone to constant change. In fact, I guarantee that they’ll change and change many times. The AD domain hierarchy

Chapter 2 www.netpro.com37

isn’t nearly as fluid or adaptable as the business itself. Once you create and name domains, you cannot move orrename them easily. In addition, you cannot move or rename the root domain.

Using locations to name child domains is more flexible because physical locations on a network seldom change.The organization at the specific site may change, but not the physical location itself. This design allows the tree tobe more flexible to everyday changes. However, if the physical location is changed or removed, the resources aremoved. This includes the physical resources, such as domain controllers, printers, and other equipment supportingthe site.

Figure 2.6: The first layer of domains directly under the root domain is named after the physical, orWAN, locations on the network.

Page 48: The Definitive Guide to Active Directory Troubleshooting

If your company is smaller and contained in one physical location, you could name domains after the company ororganization. These domains then hold all the objects and attributes for your company. This is an easy and efficientdesign. However, if your company has multiple physical locations, with network resources spread across them,you’ll want to create a second layer of domains (under the root domain) and give the domains location names. Theorganizational structures of business units, divisions, and departments will then be placed under each of theselocation domains.

Never Using the Same Name TwiceI recommend that you never use the same DNS name twice, even if the names are used on different networks. Thissimple guideline will help eliminate any confusion down the road. For example, let’s say you decide to use thedomain name engineering.company.com. Don’t use this name for any other domain, even if the domain is on adifferent network. A client may connect to both networks and query engineering.company.com. Depending on thelayout of the network, the client may locate the wrong domain in the wrong forest.

Dividing the ForestIn larger organizations, the implementation of AD can become quite large. As it grows, I recommend that youbreak it into smaller pieces, known as partitions. By definition, a partition is a domain or portion of the directorytree that extends from the beginning of a branch (or naming context) to the bottom of the tree. The domainphysically stores the containers, objects, and attributes in that branch. Several rules control creating partitions inAD and how they operate.

• The topmost partition is the root domain

• Partitions don’t overlap (one object cannot be held in two partitions)

• The partitions contain all the information for the naming context.

In AD, the basic unit of partitioning is the domain. So when you create your first partition, you’re actually creatinga child domain under the root domain. The domains in AD act as partitions in the database. This means that eachdomain represents a partition in the overall AD database. Partitioning this database increases its scalability. As youpartition AD, you break it into smaller, more manageable pieces that can be distributed across the domaincontrollers, or network servers. Figure 2.7 illustrates how you can divide the AD database into smaller pieces thatcan be distributed to the domain controllers.

Chapter 2 www.netpro.com38

Page 49: The Definitive Guide to Active Directory Troubleshooting

Breaking AD into smaller pieces and distributing them among multiple servers places a smaller load and lessoverhead on any one server. This approach also allows you to control the amount and path of traffic generated toreplicate changes among servers. Once you create a partition, replication occurs among servers that hold copies.

In AD, you can create many partitions at multiple levels in the forest. In addition, copies of the domain can bedistributed to many different servers on the network. Although AD is distributed using partitions, any user canaccess the information completely transparently. Users can access the entire AD database regardless of which serverholds which data. Of course, users must have been granted the proper permissions.

Although a single domain controller may not contain the entire AD database, users can still receive whateverinformation they request. AD queries the global catalog on behalf of a user to identify the requested object, thenresolves the name to a server (domain controller) address using DNS. Again, this process is entirely transparent tothe user.

Placing the Domain Controllers for Fault ToleranceAfter you’ve partitioned AD, you need to decide how to distribute each new domain or partition across the networkservers. The domain controllers are the servers that store the domains and their distributed copies. One domaincontroller holds only one copy of an AD partition or domain unless it’s a global catalog. The domain controllerstores the objects for the domain or partition to which it belongs.

The availability of domain information is strictly determined by the availability of the domain controllers. It’sobvious that the domain controllers must be available so that users can log on and access AD information. For thispurpose, never have only one domain controller for any domain. I recommend that you have at least two domaincontrollers for each domain to provide redundancy and fault tolerance for every domain in your organization.

Chapter 2 www.netpro.com39

Figure 2.7: You can partition the AD database into smaller pieces, then distribute them among net-work servers or domain controllers.

Page 50: The Definitive Guide to Active Directory Troubleshooting

Determining Trust Relationships Trust relationships are logical connections that combine two or more domains into one administrative unit. Trustrelationships allow permissions to be associated and passed from one domain to another. Without some sort oftrust among domains, users cannot communicate or share resources. In this section, I’ll describe the advantages ofusing bi-directional trusts, one-way trusts, and cross-link trusts.

Using Bi-directional Transitive TrustsIn AD, trust relationships are automatically established between every domain and its parent domain in the tree orforest. This greatly reduces the overhead of managing trust relationships. The types of trusts that are created arenew in Win2K and are called bi-directional transitive trusts. The best way to understand the concept of transitivetrusts is to use an example. Figure 2.8 shows bi-directional trusts being established among all the domains and theirchild domains.

Chapter 2 www.netpro.com40

Figure 2.8: Each domain has a bi-directional transitive trust relationship between itself and each of itschild domains.

One of the advantages of these new trusts is that they’re automatically established among all domains; this allowseach domain to trust all the other domains in the forest. Another advantage is that these bi-directional trusts,which areautomatically established using Win2K’s Kerberos security mechanism, are much easier to set up and administerthan Windows NT–style trusts. Having bi-directional trusts also reduces the total number of trust relationshipsneeded in a tree or forest. For example, if you tried to accomplish the same thing in NT, you’d have to create two-ways trusts between one domain and every other domain. This would increase the total number of trusts exponen-tially with the number of domains.

If you have experience with Windows NT domains, you may know something of trust relationships. However, thetrusts in AD differ from NT trusts because they’re transitive. To help you understand what this means, I’ll providean example. Win2K transitive trusts work much like a transitive equation in mathematics. A basic mathematicaltransitive equation reads as follows:

A=B, B=C, therefore A=C

Page 51: The Definitive Guide to Active Directory Troubleshooting

When applying this transitive concept to trust relationships, you get an understanding of how transitive trusts workamong domains. For example, if Domain A trusts Domain B, and Domain B trusts Domain C, Domain A trustsDomain C. Figure 2.9 illustrates this idea. Transitive trust relationships have been set up between Domain A andDomain B and between Domain B and Domain C. Thus, Domain A trusts Domain C implicitly.

Chapter 2 www.netpro.com41

Figure 2.9: A domain tree viewed in terms of its transitive trust relationships. Because transitivetrust relationships have been set up between Domain A and Domain B and between Domain B

and Domain C, Domain A trusts Domain C implicitly.

In Windows NT, trusts were non-transitive, so they didn’t allow this implicit trust to exist. For one domain to trustanother domain, an explicit trust relationship had to be created between them.

When domains are created in an AD forest, bi-directional trust relationships are automatically established. Becausethe trust is transitive and bi-directional, no additional trust relationships are required. The result is that every domainin the forest trusts every other domain. Transitive trusts greatly reduce your overhead and the need to manuallyconfigure the trusts. Because trusts are automatically set up, users have access to all resources in the forest as long asthey have the proper permissions.

Transitive trusts are a feature of the Kerberos authentication protocol. The protocol is used by AD and providesdistributed authentication and authorization. The parent-child relationship among domains is only a naming andtrust relationship. This means that the trust honors the authentication of the trusted domain. However, having alladministrative rights in a parent domain doesn’t automatically make you an administrator of a child domain.Policies set in a parent don’t automatically apply to child domains because the trust is in place.

Page 52: The Definitive Guide to Active Directory Troubleshooting

Using One-Way TrustsOne-way trusts aren’t transitive and are used among domains that aren’t part of the same forest. If you’re familiarwith the one-way trusts in Windows NT, the one-way trusts that exist in Win2K are just the same. However,they’re only used in a handful of situations.

First, one-way trusts are often used when new trust relationships must be established among domains of differentforests. You can use them among domains to isolate permissions. For example, you can use one-way trusts to allowaccess among forests and among the domains of the same tree. Figure 2.10 shows how you can create a one-waytrust between two domains in two different forests. Setting up a one-way trust allows users to access networkresources in the direction of the trust. The actual user rights depend on the access control lists (ACLs) governingthe domains.

Chapter 2 www.netpro.com42

Figure 2.10: A one-way trust is established between a domain in Forest 1 and a domain in Forest2. The trust allows access to network resources in each domain.

The second use of one-way trusts is to create a relationship from an AD domain to backward-compatible domains,such as Windows NT. Because Windows NT domains cannot naturally participate in AD transitive trusts, youmust establish a one-way trust to them. You have to manage one-way trusts manually, so try to limit the numberyou use.

In both these situations, you can create two one-way trusts among the domains. However, two one-way trusts don’tequal a bi-directional transitive trust in AD.

Using Cross-Link TrustsCross-link trusts are used to increase performance among domains. Cross-link trust relationships help increase thespeed at which users authenticate among domains. However, cross-link trusts are only needed between twodomains that are both far from the root domain. To completely understand the need for the cross-link trusts, youfirst need to understand how user authentication works in AD.

Page 53: The Definitive Guide to Active Directory Troubleshooting

When a user needs to authenticate to a resource that doesn’t reside on its own domain, the client first has todetermine where the resource is and locate it. If the resource isn’t in the local domain, the domain controller willpass back a referral list of other domain controllers that might have the resource. The workstation then contacts theappropriate servers in the referral list to find the resource. This process continues until the requested resource isfound. This process is often referred to as chasing referrals and can take time, especially on large or complex ADnetworks.

Walking up and down the domain tree branches lengthens the time it takes to query each domain controller andrespond to the user. To speed this process up, you can establish a cross-link, or shortcut, trust relationship betweentwo domains. If you decide to use a cross-link trust, I recommend that you place it between the two domains thatare farthest from the root domain.

For example, suppose you have a domain tree that has domains 1, 2, 3, 4, and 5 in one branch and domains 1, A,B, C, and D in another branch. Domains 5 and D are located farthest from the root domain. Let’s say that a userin Domain 5 needs to access a resource in Domain D. To accomplish this request, the authentication process musttraverse up the first branch and down the second branch while talking to each domain controller. Continuousauthentications like this create a significant amount of network traffic. To alleviate this problem, you can establish across-link between Domain 5 and Domain D. Figure 2.11 illustrates the layout of the domain tree with the cross-linkestablished between the two domains.

Chapter 2 www.netpro.com43

Figure 2.11: The domain tree has two branches, Domains 1, 2, 3, 4, and 5 are one branch, and domains 1, A, B, C, and Dare the second branch. The cross-link trust is established between domains 5 and D.

The cross-link that has been established serves as an authentication bridge between the two domains. The result is abetter authentication performance between the domains.

Page 54: The Definitive Guide to Active Directory Troubleshooting

Designing Organizational Units for Each DomainAn OU is a container object that allows you to organize your objects and tie a Group Policy Object (GPO) to it.Using the OU, you can group similar objects into logical structures in a domain. OUs can also be nested to build ahierarchy in a domain. This hierarchy of containers is typically named after divisions, departments, and groups inyour company. When you’re designing and creating the hierarchical structure in each domain, it’s important tounderstand the following characteristics of OUs:

OUs can be nestedAn OU can contain other OUs, enabling you to build a hierarchy inside each domain.

OUs can help delegate administrationYou can delegate administrative tasks to subordinate administrators by creating subordinate OUs. Using nested OUs, you can fine-tune the level of control you need.

OUs aren’t security principalsYou cannot make an OU a member of a group. You cannot grant users permissions to resources because they reside in a particular OU. Because OUs are used to delegate administration, they can specify who manages the resources in the OUs, but they don’t indicate the resources a user can access.

OUs can be associated with a GPOA GPO enables you to define configurations for users and computers in OUs. For example, you can create a desktop policy that every user in the OU will use.

OUs don’t need to be viewed by usersIt isn’t necessary for you to design OUs with user navigation in mind. Although users can view the OU structure, it isn’t an efficient method for finding resources. The preferred method for users to find resourcesis by querying the global catalog.

Now that you understand a few of the basic characteristics for OUs, you need to consider the following guidelinesfor designing an efficient and effective OU structure:

• Create OUs to delegate administration

• Create OUs to reflect your company’s organization

• Create OUs for Group Policy

• Create OUs to restrict access.

Creating OUs to Delegate AdministrationI’ve mentioned that OUs can be used to create administrative areas in a domain. Using OUs, you can delegateadministrative tasks to subordinate administrators. For example, suppose the Engineering Department wants toadminister its own objects and resources in the Chicago domain. You can accomplish this by creating the following OU:

engineering.chicago.company.com

After you’ve created the new OU and placed all the objects and resources in it, you can grant explicit permissionsto the administrators of the Engineering Department so that they can control their own objects. Figure 2.12illustrates how you can create the Engineering OU in the Chicago domain.

Chapter 2 www.netpro.com44

Page 55: The Definitive Guide to Active Directory Troubleshooting

Another nice feature that I mentioned earlier (see "Designing Organizational Units for Each Domain") is that OUscan be nested. This enables you to build a hierarchy in each domain. For example, let’s say the Testing Group inthe Engineering Department wants full administrative control over all its resources, such as users, printers, andcomputers. To accommodate this request, you simply create a new OU directly under the Engineering OU in theChicago domain. The hierarchical structure now looks like the following:

testing.engineering.chicago.company.com

After you’ve created the new OU and placed the resources, you can give full privileges to the Testing group’sadministrator. If an OU is nested, it inherits the properties of the parent OU by default. For example, if theEngineering OU has certain security or Group Policy objects set, they’re passed down to the Testing OU. TheTesting OU is considered nested under the Engineering OU.

Be careful to limit the number of OU layers you create. Creating too many layers can increase the administrativeoverhead. Limiting the number of OU layers also increases user logon performance. When a user logs on to AD,the security policies take effect. To find all these policies, the workstation must search all layers of the OU structure.Having fewer OU layers allows the client to complete this search more quickly.

Creating OUs to Reflect Your Company’s OrganizationIf you don’t create OUs in your domain, all users, printers, servers, computers, and other resources are displayed ina single list. This type of layout makes it difficult to search for resources. This problem increases as the number ofobjects in the domain grows.

Chapter 2 www.netpro.com45

Figure 2.12: You can create the Engineering OU in the Chicago domain, then assign permissions o the Engineering Department administrators to manage all the objects.

Page 56: The Definitive Guide to Active Directory Troubleshooting

One of the many benefits of creating OUs in a domain is the organization of this flat layout. OUs allow you tocreate an organization that reflects your company’s divisions, departments, and groups. In fact, you can use yourcompany’s organizational chart or a similar document to help you. Figure 2.13 illustrates how you can create OUsbased on an organizational chart.

Chapter 2 www.netpro.com46

Figure 2.13:OUs have been created in a domain based on an organizational chart.

Creating OUs for Group PolicyGroup Policy enables you to define Win2K desktop configurations for users and computers. Desktop configurationshave settings that govern and control the user’s experience at the workstation. GPOs can also be associated with anOU. This association allows all users and computers in the OU and any nested OUs to receive the settings definedin Group Policy. These settings configure a number of items, including installed software, Registry settings, andlogon scripts—just to name a few.

For more information on Group Policy, I recommend that you check out another free eBook from

Realtimepublishers: The Definitive Guide to Windows 2000 Group Policy by Darren Mar-Elia, a link to

which can be found at http://www.realtimepublishers.com.

The ability to set Group Policy on OUs allows you to control a large set of users and computers from a centralpoint. If you have a special need for certain users and computers, you can create an OU and establish GroupPolicy. For example, if the Accounting Department needs specific settings on its desktops, you can create anOU=Accounting and establish the specific policy. Group Policy will then apply to every user and computer in thenew OU.

Page 57: The Definitive Guide to Active Directory Troubleshooting

As I mentioned above, GPOs can be associated with OUs as well as the domain and site objects in AD. BecauseGPOs can be associated with each of these objects, you can create implementations using GPOs to generate variouscombinations. If you aren’t careful, these combinations can become very complicated and cause you headaches.

Creating OUs to Restrict AccessThe quickest and easiest way to restrict total access to network resources is to create a new OU and place thenetwork resources in it. You can then restrict access to the OU, thereby removing access to the network resources.In addition, the objects representing the network resources are no longer visible.

Users who don’t have the right to read an object, can normally still see it in AD. This may be a problem if you havehighly secure network resources that you don’t want anyone else to see. You can restrict and hide the resources bycreating a new OU in the domain and limiting access to only the few who need it.

Designing the Sites for the Forest Sites are locations on the physical network that contain AD servers. A site is stored in AD as an object and isdefined as one or more well-connected TCP/IP subnets. By "well-connected," I mean that the network connectivityamong the subnets is highly reliable and supports a data-transfer rate of at least 10 megabits per second.

Designing sites and site links in AD takes advantage of the physical network layout. (For an explanation of sitelinks, see "Creating Sites and Site Links Based on Network Topology" below.) The basic assumption is that serversand workstations with the same subnet address are connected to the same network segment and have LAN speeds.Defining a site as a set of subnets allows administrators to easily configure AD access and replication topology totake advantage of the physical network. Sites also help you locate network servers so that they’re physically close tothe users who depend on them.

It’s your role as an administrator to design the site objects and site links for your tree or forest that assure the bestnetwork performance. It’s also your job to determine what speed assures this performance and reduces serverdowntime as a result of network outages. Establish site objects and site links based on network and subnet speed.While many subnets can belong to a single site, a single subnet can’t span multiple physical sites. To help youestablish a design for the sites in your forest, you need to consider the following guidelines:

• Create sites and site links based on network topology• Use sites to determine the placement of domain controllers• Use sites to determine the placement of global catalog servers.

Creating Sites and Site Links Based on Network TopologyWhen you create sites and site links for your tree or forest, use the physical layout of your network, or topology.Before you can properly create sites and site links, you need a solid understanding of what they are.

Chapter 2 www.netpro.com47

Page 58: The Definitive Guide to Active Directory Troubleshooting

About SitesSites are groups of computers (or subnets) that share high-speed-bandwidth connections on one or more TCP/IPsubnets. Subnets are groups of local segments on the network that are physically located in the same place.Multiple site objects create a site topology. Figure 2.14 portrays a site with TCP/IP subnets that exist between theservers and workstations. A LAN always connects a site.

Chapter 2 www.netpro.com48

Figure 2.14: A site is one or more TCP/IP subnets or LAN networks that exist between the servers and workstations.

One domain can span more than one site, and one site can contain multiple domains. However, for design

purposes, it’s important to remember that sites define how replication occurs among domain controllers and

which domain controller a user’s workstation contacts for initial authentication. Normally, the workstation first

tries to contact domain controllers in its site.

About Site LinksSite links are objects that represent the WAN links on your network. They also represent any low-bandwidthconnections between two locations. Site links connect two or more sites together. They help you determine thereplication schedule and latency, and they help you determine where to place network servers. The rule is to createa site link when a connection is slower than LAN-speed connection. Defining site links allows administrators toconfigure AD and replication to take advantage of the network.

Page 59: The Definitive Guide to Active Directory Troubleshooting

The site link object has four settings.

CostHelps the replication process determine the path of the communication among domain controllers

Replication ScheduleDetermines what time of day the replication process can execute

Replication IntervalHelps the replication process determine how often to poll the domain controllers on the other side of the link

TransportHelps the replication process determine which transport protocol to use during communications.

Site and site link objects are stored in a special container called the configuration container. The configurationcontainer is stored and replicated to every AD domain controller, providing each server with complete details ofthe physical network topology. A change to any of the information in the site or site link objects causes replicationto every domain controller in the forest.

Creating the Site TopologySites and site links create the site topology, as shown in Figure 2.15. The site topology helps the replication processdetermine the path, cost, and protocol among domain controllers.

Chapter 2 www.netpro.com49

Figure 2.15: The site topology is created from the site objects and the site links. The site topology helps the replication processdetermine the path, cost, and protocol among domain controllers.

When you create the site topology, it’s useful to have a complete set of physical LAN and WAN maps. If yourcompany has campus networks at one or more locations, you’ll need to have the physical maps of those locations.These maps should include all the physical connections, media or frame types, protocols, and speed of connections.

Page 60: The Definitive Guide to Active Directory Troubleshooting

When defining the sites, begin by creating a site for every LAN or set of LANs that are connected by high-speedbandwidth connections. If there are multiple physical locations, create a site for each location that has a LANsubnet. For each site that you create, keep track of the IP subnets and addresses that comprise the site. You’ll needthis information when you add the site information to AD.

Chapter 2 www.netpro.com50

Site names are registered in DNS by the domain locator, so they must be legal DNS names. You must also

use Internet standard characters—letters, numbers, and hyphens. (For more information, see "Using

Internet Standard Characters" earlier in this chapter.)

After you’ve created the sites, you need to connect them with site links to truly reflect the physical connectivity ofyour network. To do this, you need to first assign each site link a name. Site links are transitive, just like trustrelationships in Win2K. This means that if Site A is connected to Site B and Site B is connected to Site C, it’sassumed that Site A can communicate with Site C.

The process of generating this site topology is automatic, and it’s handled by a special Win2K service called theKnowledge Consistency Checker (KCC). If you don’t like the topology that the KCC generates for you, you cancreate it manually.

The purpose of creating the site topology is to ensure rapid data communications among AD servers. The sitetopology is used primarily when setting up replication of AD. However, the placement of the domain controllersand partitions govern when and how replication takes place.

Using Sites to Determine the Placement of Domain ControllersAfter you’ve properly created site and site link objects, you can use them to help you decide how to properlydistribute AD partitions across the network servers. Network servers are the domain controllers that store ADdomains and their copies. One domain controller holds only one copy of an AD partition or domain. The serverworks to authenticate users and provide responses to queries about the objects and attributes.

Your responsibility is to determine where to place the domain controllers on the network to best suit the needs ofthe users. I recommend that they be located on or near the users’ subnet or site. When a workstation connects tothe network, it typically receives a TCP/IP address from DHCP. This TCP/IP address identifies the subnet or siteto which the workstation is attached. If the workstation has a statically assigned IP address, it’ll also have staticallyconfigured subnet information.

In either case, when users log on to the network, their workstations can reach the closest domain controller site byknowing the assigned address and subnet information. Because computers in the same site are physically close toeach other in, communication among them is reliable and fast. Workstations can easily determine the local site atlogon because they already know what TCP/IP subnet they’re on, and subnets translate directly to AD sites.

If no domain controller is available in the local site, user traffic will cross the WAN links and sites to find otherservers. To place the domain controller for best overall connectivity, select the site where the largest numbers ofusers are located. All the users in that site will authenticate to the local domain controller. This approach guaranteesthat the users will retrieve their object information from the global catalog partition. The location of the server isimportant because users are required to access a global catalog server when they log on.

Page 61: The Definitive Guide to Active Directory Troubleshooting

Using Sites to Determine the Placement of DNS ServersI’ve already mentioned that DNS and AD are inseparably connected. AD uses DNS to locate the domain controllers. The DNS service enables users’ workstations to find the IP addresses of the domain controllers. TheDNS server is the authoritative source for the locator records of the domains and domain controllers on the network.To find a particular domain controller, the workstation queries DNS for the appropriate service (SRV) and address(A) resource records. These records from DNS provide the names and IP addresses of the domain controller.

The availability of DNS directly affects the availability of AD and its servers. As mentioned, users rely on DNS as aservice. To guarantee DNS as a service, I recommend that you place or have available at least one DNS server forevery site on your network. This allows all users to access the DNS service locally. You don’t want users to have toquery DNS servers that are offsite to locate the domain controllers that are on their own subnet.

Chapter 2 www.netpro.com51

The AD domain controllers query DNS to find each other during replication. A new domain controller

participates in replication by registering its locator records with DNS. Likewise, each domain controller must

be able to look up these records. This is the case even if the domain controllers are on the same subnet.

If you depend on an outside DNS service, you may need to adjust the number of DNS servers and physical placement,if possible. You’ll also need to verify that the outside DNS service supports the required SRV resource record. If itdoesn’t, you may need to install and configure your own implementation of Microsoft’s DNS to support AD.

If you don’t want to depend on an existing DNS service or a DNS service that is offsite, you may want to installthe Microsoft DNS service that is integrated into AD. The Microsoft DNS service stores the locator records for thedomain and domain controllers in AD. You can then have one or more domain controllers provide the DNS service.Again, I recommend that you place at least one DNS server for each site object in your environment. Using theMicrosoft DNS service is an optional configuration, and storing the locator records in AD may have a negativeimpact on replication traffic on large networks.

SummaryMy first recommendation for troubleshooting AD was to make sure that its components are designed andimplemented correctly. In addition, the efficiency of AD depends on the design and implementation of key structures— forests, trees, domains, and OUs. I also recommended that the sites and site links be properly established tosupport the distribution and replication of the system. I also discussed the placement of other supporting servers,such as domain controllers, global catalog servers, and DNS servers. The design and implementation of thesestructures is strictly your responsibility as network administrators. Before you can effectively troubleshoot AD, makesure you feel confident about your design.

eBook Copyright NoticeThis site contains materials created, developed, or commissioned by Realtimepublishers.com, Inc. and is protected byinternational copyright and trademark laws. No material (including but not limited to the text, images, audio, and/orvideo) may be copied, reproduced, republished, uploaded, posted, transmitted, or distributed in any way, except that onecopy may be downloaded for your personal, non-commercial use on a single computer. In connection with such use, youmay not modify or obscure any copyright or other proprietary notice. If you have any questions about these terms, or ifyou would like information about licensing materials from Realtimepublishers.com, please contact us via e-mail [email protected]

Page 62: The Definitive Guide to Active Directory Troubleshooting

The Definitive Guide to Active Directory Troubleshooting

Chapter 3

Page 63: The Definitive Guide to Active Directory Troubleshooting

Monitoring and Tuning the Windows 2000 System and NetworkA Windows 2000 (Win2K) network is a system of devices that work together toward a common goal of providingcommunication among users, servers, and applications. The most important of these devices are Win2K domaincontrollers. This book is primarily interested in the components or software that are required for Win2K ActiveDirectory (AD) services. In this chapter, I’ll focus on how you can monitor Win2K domain controllers and theirsubsystems to help you reduce downtime and improve AD performance.

Monitoring Windows 2000 Domain Controllers As previously mentioned, domain controllers are the single most important type of device on an AD-based Win2Knetwork. These devices share the responsibility of storing the directory information, and they interact with eachother to replicate the directory information and keep it up to date. In addition, domain controllers are responsiblefor authenticating user logons and servicing other requests for access to directory and network resources. Becausedomain controllers are crucial to the performance and operation of the directory, it’s critical that you continuallymonitor these servers. A poorly performing or misbehaving domain controller can easily cause network downtimeand loss of directory functionality. For example, when the directory slows down significantly or is unavailable, userscan’t log on, there is no Address Book for Exchange, and users can’t print or access Web-based applications.

When you consider how you’ll monitor your domain controllers, first remember that no one domain controllercontains all of the directory information. In any well-built Win2K network, each domain partition in the directorytypically has two or more domain controllers hosting the domain to provide fault tolerance for directory services.With this kind of redundancy in place, you might initially be fooled into thinking that monitoring each domaincontroller for performance and downtime isn’t all that important.

However, each domain controller plays a role in supporting your users. For example, if two domain controllers inthe same directory partition are placed in separated sites or subnets, users in each site will use the domain controllernearest them. However, if one of the domain controllers goes down, users in that location must traverse the widearea network (WAN) to log on and access the directory. This is usually undesirable, especially if there are too manyusers and/or if the WAN link is slow.

Another example of why you need to monitor domain controllers is that some domain controllers on a Win2Knetwork (no matter how many domain controllers it may have) are unique. For example, some Win2K domaincontrollers perform special duties called Flexible Single-Master Operation (FSMO) roles. Although the replicationof AD is multimaster, the FSMO roles held by these domain controllers are single-master (much like a WindowsNT 4.0 primary domain controller, or PDC). This essentially means that these domain controllers don’t haveadditional copies or replicas to provide fault tolerance if the domain controller hosting a particular role is down.

As I discussed in Chapter 1, these FSMO domain controllers perform special roles for AD, such as managing thedomain, managing the schema, and supporting down-level clients. If any of these critical domain controllers godown, the directory loses functionality and can no longer update or extend the schema, or add or remove a domainfrom the directory.

Chapter 3 www.netpro.com53

Page 64: The Definitive Guide to Active Directory Troubleshooting

Failing to monitor domain controllers can adversely affect a network’s performance and availability. For example, ifan entire department is unable to access the domain controller or directory, users lose time, and the company losesmoney. To help you ensure that your domain controllers are available, you can, and should, monitor and analyzeWin2K in the five following areas:

• Overall system • Memory and cache • Processor and thread • Disk • Network.

I’ll discuss each of these areas, and the reason for their importance, in the following sections. I’ll discuss monitoringAD itself in Chapter 4.

Monitoring the Overall System Monitoring a Win2K domain controller means watching the operation of both the server’s operating system (OS)and its hardware subsystems. When you monitor domain controllers, I recommend that you begin by establishing aperformance and reliability baseline for each—that is, a nominal and acceptable level of operation under real-worldconditions on your network. Establishing a baseline allows you to track the operation of the domain controller overtime. If a potential problem or bottleneck occurs, you can recognize it immediately it because you can comparethat behavior to the baseline established for that domain controller.

Monitoring domain controllers means watching for problems or bottlenecks in the OS and its subsystems. A simpleexample of a bottleneck occurs when a domain controller’s processor is running at 100 percent usage because oneapplication has tied up the central processing unit (CPU). Almost every NT/Win2K administrator has seen thisoccur at some point or another.

Win2K provides several utilities that can assist you in monitoring your domain controllers and their subsystems.These tools provide features that will help you search for bottlenecks and other problems. They’re described below.

Task ManagerGives you a quick view of which applications and processes are running on the domain controllers. This utility allows you to view a summary of the overall CPU and memory usage for each of these processes andthreads.

Performance consoleAllows you to view the current activity on the domain controller and select the performance information that you want collected and logged. You can customize Win2K’s performance-counter features and architecture to allow applications to add their own metrics in the form of objects and counters, which you can then monitor using the Performance console. By default, the Performance console has two applications, System Monitor and Performance Logs and Alerts.

System Monitor enables you to monitor nearly every aspect of a domain controller’s performance and establish a baseline for the performance of your domain controllers. Using System Monitor, you can see the performance counters graphically logged and set alerts against them. The alerts will appear in Event Viewer.

Chapter 3 www.netpro.com54

Page 65: The Definitive Guide to Active Directory Troubleshooting

Performance Logs and Alerts enable you to collect information for those times when you can’t detect a problem in real time. Performance Logs and Alerts allows you to collect domain controller performance data for as long as you want—days, weeks, or even months.

Event ViewerAllows you to view the event logs that gather information about a domain controller and its subsystems. There are three types of logs: the application log, the system log, and the security log. Although the event logs start automatically when you start the domain controller, you must start Event Viewer manually.

Chapter 3 www.netpro.com55

When monitoring domain controllers using the Performance console’s logging feature, make sure you don’t

actually create a problem by filling the computer’s disk with large log files.

First, be sure to only include those statistics in the logging process that you absolutely need. Keep the

sampling period to the minimum required to evaluate domain controller performance and usage. To select

an appropriate interval for your computer, establish a baseline of performance and usage. Also, take into

account the amount of free disk space on your domain controller when you begin the logging process.

Finally, make sure that you have some application in place (such as the Performance console) that continu-

ally monitors the domain controller to ensure that it has plenty of free disk space.

In addition to monitoring the local domain controller, you can use the Performance console to monitor

domain controllers remotely and store the log files on a shared network drive. This enables you to monitor

all the domain controllers in a directory from one console or utility.

At the heart of the Performance console and Task Manager are the performance counters that are built into theWin2K OS. I’ll introduce each of these monitoring utilities briefly in the upcoming sections and demonstrate howthey can help you monitor specific subsystems. Keep in mind that this chapter isn’t intended to be an in-depthstudy of all the capabilities of these utilities. Instead, my intention is to provide a general introduction to them andshow you how you can use them to assist you in monitoring your domain controllers.

Using Task ManagerThe easiest and quickest way to view how each application or system process is using the CPU and memory is byusing Win2K’s Task Manager. This utility allows you to see which processes or threads are running on a Win2Kdomain controller at any given moment; it also shows a summary of overall CPU and memory usage. To launchTask Manager, either right-click the taskbar (typically at the bottom of the screen) and choose Task Manager orpress Ctrl+Alt+Del, then click Task List from the menu. Figure 3.1 shows an example of Task Manager.

Page 66: The Definitive Guide to Active Directory Troubleshooting

Task Manager supplies three pages of information: Applications, Processes, and Performance. Each of these pageswill help you understand more about a domain controller’s processes and memory. I’ll discuss each of these screensin greater detail later in this chapter.

Using the Performance ConsoleIn Win2K, one of the main utilities for monitoring a domain controller is the Performance console. The Performanceconsole allows you to view the current activity on the domain controller and select the performance informationthat you want collected and logged.

Chapter 3 www.netpro.com56

Figure 3.1: Win2K’s Task Manager allows you to view and manage the applicationsand processes running on a domain controller and manage their performance.

In Windows NT, the Performance console was known as Performance Monitor, and like most NT administration

utilities, it was a standalone utility rather than an MMC snap-in.

The Performance console helps you accurately pinpoint many of the performance problems or bottlenecks in yoursystem. It monitors your Win2K domain controller by capturing the selected performance counters that relate tothe system hardware and software. The performance counters are programmed by the developer of the related sys-tem. The hardware-related counters typically monitor the number of times a device has been accessed. For example,the physical disk counters indicate the number of physical disk reads or writes and how fast they were completed.Software counters monitor activity related to application software running on the domain controller. To launch thePerformance console, choose Start>Programs>Administrative Tools>Performance.

Page 67: The Definitive Guide to Active Directory Troubleshooting

The first application that starts in the Performance console is System Monitor. Using System Monitor, you canview the current activity on the domain controller and select information to be collected and logged for analysis.You can also measure the performance of your own domain controller as well as that of other domain controllerson your network. System Monitor is shown in Figure 3.2.

Chapter 3 www.netpro.com57

Figure 3.2: The Performance console includes both System Monitor and Performance Logs and Alerts.

When it starts up, System Monitor isn’t monitoring any counters or performance indicators for the system. Youdetermine which counters System Monitor tracks and displays. To add a counter, click the Plus (+) tool on thetoolbar or right-click anywhere in the System Monitor display area and choose Add Counters from the shortcutmenu. Using either approach, the Add Counters dialog box appears, as shown in Figure 3.3, where you can choosewhich counters to monitor.

Page 68: The Definitive Guide to Active Directory Troubleshooting

Once you choose the counter that you want to view, System Monitor tracks its performance in real time. When youfirst start using System Monitor, the number of counters that are available seems overwhelming because there arecounters for almost every aspect of the computer. However, in the spirit of the age-old 80/20 rule, you’ll probablyfind that you tend to use about 20 percent of the available counters 80 percent of the time (or more), using theother counters only when you need specific monitoring or troubleshooting information.

Chapter 3 www.netpro.com58

Figure 3.3: In System Monitor, you can choose which counters you want totrack and monitor on the display.

If you don’t understand the meaning of a particular Performance console counter, highlight it and click Explain.

The informational dialog box that appears provides a description of the selected counter (and, in some cases,

what the various values or ranges might indicate).

In later sections of this chapter, I’ll discuss how you can use System Monitor to monitor memory, view processes,and monitor network components on a domain controller as well as monitor the disk subsystem.

Event ViewerAs with its NT predecessor, Win2K uses an event logging system to track the activity of each computer and itssubsystems. The events that are logged by the system are predetermined and tracked by the OS. In addition,Win2K provides Event Viewer, which allows you to view the events that have been logged.

Events Tracked in Event LogsWin2K domain controllers occasionally encounter serious error conditions and halt operation. This situation iscalled a stop error (also informally known to some users as the "blue screen of death," or BSOD). The error messageis displayed on a solid blue background on a domain controller’s console. A number of stop errors are worthwhilemonitoring because they affect the reliability of a computer. Fortunately, stop errors are recorded in a domaincontroller’s Event Log when the computer restarts. To view the stop errors in the Event Log, you need to launchEvent Viewer.

Page 69: The Definitive Guide to Active Directory Troubleshooting

In addition to stop errors, if a Win2K domain controller restarts, these events are also recorded in the system logsection of the Event Log. The reasons for a restart could include OS crashes, OS upgrades, and hardware maintenance.

Another type of event that a domain controller tracks in the Event Log is application crashes. Win2K uses the Dr.Watson utility (Drwtsn32.exe) to record problems and failures in applications running on the domain controller.Failures are recorded in the application log section of the Event Log. Again, you can use Event Viewer and theinformation in the Event Log to analyze problems with an application.

Types of Event LogsWhen you use Event Viewer, the Event Log is separated into three logs, as follows:

Application LogContains events logged by applications or programs such as Exchange or Microsoft Internet Information Server (IIS) that are running on the computer. The developer of an application decides which events to record.

System LogContains events logged by the subsystems and components of the domain controller. For example, if a disk driver has problems or fails, it records the events in the system log. You can use this log to determine the general availability and uptime of the domain controller.

Security LogRecords security events, such as when a user successfully logs on or attempts to log on. This log also records events that relate to file access. For example, an event is recorded when a file is created, opened, or deleted. By default, the security log can only be seen by system administrators.

Starting Event ViewerThe event logs start automatically when you start the domain controller; you must start Event Viewer manually. Tostart or display Event Viewer, choose Start>Run, type eventvwr, then click OK, or chooseStart>Programs>Administrative Tools>Event Viewer. Event Viewer starts and displays the following screen (seeFigure 3.4).

Chapter 3 www.netpro.com59

Page 70: The Definitive Guide to Active Directory Troubleshooting

Types of Events Logged by Event ViewerEvent Viewer logs several types of events, each of which has a different severity to help you analyze a problem. Thetypes of events are as follows:

ErrorSignifies that a severe problem has occurred. This means that data was lost or functionality was lost. For example, if a service fails to load during startup or stops abruptly, an error is logged.

WarningLess significant than an error and indicates that a problem could occur in the future. For example, a warning is logged if disk space becomes too low.

InformationDescribes important situations that need noting. This event is typically used to notify when an operation is successful—for example, a disk driver loaded successfully and without errors.

Success auditLogs successful access to a secured system resource such as a file or directory object. A success audit event is a successful security-access attempt. For example, if a user attempts to log on to the system and is successful, a success audit event is logged.

Chapter 3 www.netpro.com60

Figure 3.4: The startup screen or display for Event Viewer.

Only a user with administrative privileges can view the security log. Regular users can only view the application

and system logs.

Page 71: The Definitive Guide to Active Directory Troubleshooting

Failure auditIs the opposite of the success audit event. For example, if a user attempts to log on to the system or access a secured resource and fails, a failure audit is logged.

Sorting and Filtering EventsUsing Event Viewer, you can sort events on the screen so that you can easily review and analyze the information.To sort events on the screen, choose View>Newest First (the default) or Oldest First.

In addition to selecting the sort order for events, you can filter them. Filtering events allows you to select and viewonly the events that you want to analyze. To set a filter for events in Event Viewer, choose View>Filter Events.Figure 3.5 illustrates the dialog box that appears to help you specify the filter characteristics.

Chapter 3 www.netpro.com61

Figure 3.5: Events can be filtered in Event Viewer to restrict the list of events that are displayed.

Exporting EventsIn addition to sorting and filtering events in Event Viewer, you can export events in a variety of formats to use withapplications such as Microsoft Excel. To export events, choose Action>Export List. When the Save As dialog boxappears (shown in Figure 3.6), you can type a file name with the .xls extension, or choose a file type, such as Text(Comma Delimited) (*.csv).

Page 72: The Definitive Guide to Active Directory Troubleshooting

Monitoring Memory and Cache One of the most common performance problems with Win2K domain controllers (and all servers, for that matter)is excessive paging, which is caused by insufficient random access memory (RAM). In this situation, one of thegreatest performance gains you can achieve is adding more physical RAM to a system. Therefore, I recommendthat the first subsystem you monitor be the domain controller’s memory and cache. Problems caused by lack ofmemory can often appear to be problems in other parts of the system. For instance, a lack of memory can causeinsufficient file system cache, which can lead to and be seen as a performance problem in the disk subsystem.

Before I give you the details of monitoring your domain controller’s memory, I’ll first briefly introduce the memorymodel for Win2K. Memory in Win2K provides a page-based virtual memory management scheme (called VirtualMemory Manager, or VMM) that allows applications to address 4 gigabytes (GB) of memory. Memory in Win2Kis able to do exactly that by implementing virtual addresses. Each application is able to reference a physical chunkof memory, at a specific virtual address, throughout its life. VMM takes care of whether the memory should bemoved to a new location or swapped to disk completely independently of the application.

Because everything in the system is realized using pages of physical memory, it’s easy to see that pages of memorybecome scarce rather quickly. VMM uses the hard disk to store unneeded pages of memory in one or more files calledpaging files. Paging files represent pages of data that aren’t currently being used but may be needed spontaneouslyat any time. By swapping pages to and from paging files, VMM is able to make pages of memory available toapplications on demand and provide much more virtual memory than the available physical memory.

One of the first monitoring or troubleshooting tasks you’ll carry out is to verify that your domain controller hasenough physical memory. Table 3.1 shows the minimum memory requirements for a Win2K domain controller.

Chapter 3 www.netpro.com62

Figure 3.6: The events in Event Viewer can be exported for use with various applications.

This

Minimum Installation

Server running a basic set of services

Server running an expanded set of services

Requires this amount of memory

64 MB

128 MB

512 MB

Table 3.1: Minimum memory requirements for a Win2K domain controller.

Page 73: The Definitive Guide to Active Directory Troubleshooting

These recommendations are minimum physical memory requirements. Your physical memory requirements foractual production servers will typically be much higher. Because your Win2K domain controllers will at least berunning AD, I recommend that you always start with at least 512 megabytes (MB) RAM. If you want to load otherapplications that come with their own memory requirements, you’ll need to add memory to support them. If thereisn’t enough memory on your domain controller, it will start running slower as it pages information to and from itshard drive. When physical memory becomes full and an application needs access to information not currently inmemory, VMM moves some pages from physical memory to a storage area on the hard drive called a paging file.

As the domain controller pages information to and from the paging file, the application must wait. The wait occursbecause the hard drive is significantly slower than physical RAM. This paging also slows down other system activitiessuch as CPU and disk operations. As I mentioned earlier, problems caused by lack of memory often appear to beproblems in other parts of the system. To maximize the performance and availability of your domain controller servers,it’s important for you to understand and try to reduce or eliminate wherever possible the performance overheadassociated with paging operations.

Fortunately, there are a couple of utilities that you can use to track memory usage. Two of the most common areutilities I’ve already introduced: Task Manager and the Performance console.

Using Task Manager to View Memory on a Domain ControllerYou can use Task Manager to view memory usage on a domain controller. To do so, click the Performance tab.Figure 3.7 shows an example of the Performance page in Task Manager running on Win2K.

Chapter 3 www.netpro.com63

Page 74: The Definitive Guide to Active Directory Troubleshooting

The Performance page in Task Manager contains eight informational panes. The first two are CPU Usage andCPU Usage History. These two panes and the Totals pane all deal with usage on the CPU, or processor. Theremaining panes can be used to analyze the memory usage for the domain controller and include the following:

MEM UsageA bar graph that shows the amount of virtual memory your domain controller is currently using. This pane is one of the most useful because it can indicate when VMM is paging memory too often and thrashing. Thrashing occurs when the OS spends more time managing virtual memory than it does executing application code. If this situation arises, you need to increase the amount of memory on the system to improve performance.

Memory Usage HistoryA line graph that tracks the size of virtual memory over time. The history for this pane is only displayed in the line graph and not recorded anywhere. You can use this information to help determine if there is a problem with virtual memory over a longer period of time.

Physical MemoryTells you the total amount of RAM in kilobytes (K) that has been installed on your domain controller. This pane also shows the amount of memory that is available for processes and the amount of memory used for system cache. The amount of available memory will never go to zero because the OS will swap data to the hard drive as the memory fills up. The system cache is the amount of memory used for file cache on the domain controller.

Chapter 3 www.netpro.com64

Figure 3.7: The Performance page of Win2K’s Task Manager allows you to view adomain controller’s memory usage.

Page 75: The Definitive Guide to Active Directory Troubleshooting

Commit ChargeShows three numbers, which all deal with virtual memory on the domain controller: Total, Limit, and Peak. The numbers are shown in kilobytes K. Total shows the current amount of virtual memory in use. (You’ll notice that this number matches the number shown in MEM Usage.) Limit is the maximum possible size of virtual memory. (This is also referred to as the paging limit.) Peak is the highest amount of memory that has been used since the domain controller was started.

Kernel MemoryShows you the total amount of paged and non-paged memory, in kilobytes, used by the kernel of the OS. The kernel provides core OS services such as memory management and task scheduling.

I mentioned that you can easily and quickly check the memory usage on your domain controller using Task Manager. Task Manager allows you to see the amount of virtual memory in use.

Using the Performance Console to Monitor Memory on a Domain ControllerIn addition to using Task Manager, you can use the Performance console to determine whether the current amountof memory on a domain controller is sufficient. The System Monitor application in the Performance console allowsyou to graphically display memory counters over time. I also recommend that you display the memory cache counters.

Available Memory CountersTo determine if there is a bottleneck in memory, you need to check three memory counters:

• Available Bytes (under the Memory object in System Monitor)

• Available Kbytes (kilobytes or KB)

• Available Mbytes (megabytes or MB).

You can use any of these three counters to understand your domain controller’s memory commitment. I recommendthat you reserve at least 20 percent of available memory for peak use.

To view one or all of the available memory counters, either click the Plus (+) tool on the toolbar or right-click any-where in the display area and choose Add Counters from the shortcut menu. Once the Add Counters dialog boxappears, choose Performance Object>Memory, then choose one of the available memory counters. Figure 3.8 showsthe Available Bytes counter of the memory Performance Object.

Chapter 3 www.netpro.com65

Page 76: The Definitive Guide to Active Directory Troubleshooting

The Available Bytes counter shows the amount of physical memory available to processes running on the domaincontroller. This counter displays the last observed value only; it isn’t an average. It’s calculated by summing spaceon three memory lists.

FreeMemory that is ready or available for use.

ZeroedPages of memory filled with zeros to prevent later processes from seeing data used by a previous process.

StandbyMemory removed from the working set of a process and en route to disk, but still available to be recalled.

If Available Bytes is constantly decreasing over a period of time and no new applications are loaded, it indicatesthat the amount of working memory is growing, or it could signal a memory leak in one or more of the runningapplications. A memory leak is a situation where applications or processes consume memory but don’t release itproperly. To determine the culprit, monitor each application or process individually to see if the amount of memoryit uses constantly increases. Whichever application or process constantly increases memory without decreasing it isprobably the culprit.

Page-Fault CountersWhen a process or thread requests data on a page in memory that is no longer there, a domain controller issues apage fault. Here, the page has typically been moved out of memory to provide memory for other processes. If therequested page is in another part of memory, the page fault is a soft page fault. However, if the page has to beretrieved from disk, a hard page fault has occurred. Most domain controllers can handle large numbers of soft pagefaults, but hard page faults can cause significant delays.

Chapter 3 www.netpro.com66

Figure 3.8: Using the Available Bytes memory counter to monitor or track how much memory is left forusers or applications.

Page 77: The Definitive Guide to Active Directory Troubleshooting

Page-fault counters help you determine the impact of virtual memory and page faults on a domain controller.These counters can be important performance indicators because they measure how VMM handles memory.

Page Faults/secIndicates the number of page faults without making a distinction between soft page faults and hard page faults.

Page Reads/secIndicates the number of times the disk was read to resolve hard page faults. This counter indicates the impact of hard page faults.

Pages Input/secIndicates the number of pages read from disk to resolve hard page faults. This counter also indicates the impact of hard page faults.

Figure 3.9 illustrates how you can use System Monitor to track page-fault counters.

Chapter 3 www.netpro.com67

Figure 3.9: The Page Faults/sec, Page Reads/sec, and Pages Input/sec counters determine the impact ofvirtual memory and paging.

If the numbers recorded by these counters are low, your domain controller is responding quickly to memory requests.However, if the numbers are high and remain consistently high, it’s time to add more RAM to the domain controller.

Page 78: The Definitive Guide to Active Directory Troubleshooting

Paging File UsageAnother important set of counters helps you determine the size of virtual memory. These counters are related topaging file usage. Before I discuss how you can effectively use these counters, it’s important that you better understandthe paging file and its function. The paging file is the space on a domain controller that enables the OS to swapout memory to the hard drive. As the domain controller loads more applications than it can run in actual memory,it pages some memory to the hard drive to create room for the new applications.

You can see how much the paging file is being used by watching two counters under the Paging File object.

% UsageIndicates the current usage value that was last recorded.

% Usage PeakIndicates the high-water mark for the paging file.

If a domain controller was perfect, the OS would have enough memory for every application that was loaded andwould never page memory out. Both the % Usage counter and the % Usage Peak counter would be at zero. Theopposite is that the domain controller is paging memory as fast as possible, and the usage counters are high. Anexample of a bad situation is one in which your domain controller has 128MB of memory, the % Usage Peakcounter is at 80 percent, and the % Usage counter is above 70 percent. In this situation, it’s fairly certain that yourdomain controller will be performing poorly.

By default, Win2K automatically creates a paging file on the system drive during installation. Win2K bases the sizeof the paging file on the amount of physical memory present on the domain controller (in most cases, it’s between768MB and 1536MB). In addition to this paging file, I recommend that you create a paging file on each logicaldrive in the domain controller. In fact, I recommend that you stripe the paging file across multiple physical harddrives, if possible. Striping the paging file improves performance of both the file and virtual memory becausesimultaneous disk access can occur on multiple drives simultaneously.

Chapter 3 www.netpro.com68

The recommendation for using disk striping on the paging file works best with Small Computer System Interface

(SCSI) drives rather than those based on Integrated Device Electronics (IDE) interfaces. This is because SCSI

handles multiple device contention more efficiently than IDE and tends to use less CPU power in the process.

Also, I don’t recommend that you spread the paging file across multiple logical drive volumes (partitions) located

on the same physical drive. This won’t generally aid paging file performance—and it may actually hinder it.

To change or set the virtual memory setting on your domain controller, right-click My Computer, then chooseProperties from the shortcut menu. In the System Properties dialog box, click the Advanced tab, then clickPerformance Options. Notice that the Performance Options dialog box allows you to see the current setting forVirtual Memory. Next, click Change to display more information and to change the paging file settings.

Changing the paging file size or location is unfortunately one of those rare setting changes in Win2K that

requires you to restart the domain controller before the change takes effect. So, if you decide to change any

settings for the paging file, do it during a scheduled maintenance time when it’s safe to take the domain

controller down and doing so won’t affect your users.

Page 79: The Definitive Guide to Active Directory Troubleshooting

System CacheIn addition to tracking the amount of memory and virtual memory in the domain controller, you need to keep aneye on the computer’s system cache settings. The system cache is an area in memory dedicated to files and appli-cations that have been accessed on the domain controller. The system cache is also used to speed both file systemand network input/output (I/O). For example, when a user program requests a page of a file or application, thedomain controller first looks to see if it’s in memory (system cache). That’s because a page in cache responds morequickly to user requests. If the requested information isn’t in cache, the OS fulfills the user request by reading thefile page from disk.

If the system cache isn’t large enough, bottlenecks will occur on your domain controller. The Cache object inSystem Monitor and its counters help you understand caching in Win2K. In addition, several counters under theMemory object help you determine the amount of file cache. Two of the counters that best illustrate how the filecache is responding to requests are described below.

Copy Read Hits %A counter under the Cache object that tracks the percentage of cache-copy read requests that are satisfied by the cache. The requests don’t require a disk read to give the application access to the page. A copy read is a file-read operation that is satisfied by a memory copy from a page in the cache to the application’s buffer.

Cache Faults/secA counter under the Memory object that tracks the number of faults that occur when a page sought in the system cache isn’t found. The page must be retrieved from elsewhere in memory (a soft fault) or from the hard drive (a hard fault).

Chapter 3 www.netpro.com69

When you’re considering using the Copy Read Hits % counter to assess file-cache performance, you might

also consider tracking the Copy Reads/sec counter, which measures the total number of Copy Read operations

per second. By assessing these numbers together, you’ll have a better sense of the significance of the data

provided by the Copy Reads Hits % counter. For example, if it were to spike momentarily without a corresponding

jump (or perhaps even a decrease) in the number for overall Copy Reads/sec, the data might not mean much.

Ideally, you can identify a cache bottleneck when there is a steady decrease in the Copy Read Hits % counter

with a relatively flat Copy Reads/sec figure. A steady increase in both counters, or an increase in Copy Read

Hits % and a relatively flat Copy Reads/sec, indicates good file cache performance.

Thus, the Copy Read Hits % counter records the percentage of successful file-system cache hits, and the CacheFaults/sec counter tracks the number of file-system cache misses. Figure 3.10 shows these counters in SystemMonitor. Remember that one of the counters is a percentage and the other is a raw number, so they won’t exactlymirror each other.

Page 80: The Definitive Guide to Active Directory Troubleshooting

Generally speaking, I recommend that a domain controller have at least an 80 percent cache hit rate over time. Ifthese two counters show that your domain controller has a low percentage of cache hits and a high number ofcache faults (misses), you may want to increase the total amount of RAM. Increasing the RAM allows the domaincontroller to allocate more memory for system cache and should increase the cache hit rate.

Monitoring Processors and Threads I find that many people mistakenly believe that monitoring a domain controller is focused primarily on the domaincontroller’s physical processor(s), or CPUs. However, the truth is that the processor doesn’t do anything unless thereare processes and threads to run. In Win2K, a process is made up of one or more threads that run on the CPU. Abottleneck at the processor typically means that either one or more processes are consuming most of the processortime or there are too many threads contending for the CPU.

A process is an executable program that follows a sequence of steps. Each process requires a cycle from the domaincontroller’s processor as it runs. A thread is the component of a process that is being executed at any time. Thus, aprocess must contain at least one thread before it can perform an operation. A single process executing more thanone thread is referred to as being multithreaded. Win2K is a multithreaded OS that is capable of running multipleprocessor threads simultaneously—even, when they’re present, across multiple CPUs.

Chapter 3 www.netpro.com70

Figure 3:10: The Copy Read Hits % and the Cache Faults/sec counters show how the domaincontroller’s cache is responding.

Page 81: The Definitive Guide to Active Directory Troubleshooting

When an application is developed, the developer determines the number of threads each process will use. In asingle-threaded process, only one thread is executed at one time. In a multithreaded process, more than one threadcan be executed concurrently. Being multithreaded allows a process to accomplish many tasks at the same time andavoid unnecessary delay caused by thread wait time. To change threads, the OS uses a process called contextswitching, which interrupts one thread, saves its information, then loads and runs another thread.

In addition to the multithreaded and multitasking approach to handling processes and threads, Win2K allows pri-orities to be assigned to each process and thread. The kernel of the Win2K OS controls access to the processorusing priority levels.

Using Process Viewer to Monitor Processes and ThreadsEvery Win2K domain controller comes with the Process Viewer utility (Pviewer.exe). This utility is part of theWin2K Support Tools, which is located in the Support folder on the Win2K CD. Process Viewer is a great tool forlooking at the various processes and associated threads currently running on your domain controller. To launchProcess Viewer on your computer, choose Start>Programs>Accessories>Command Prompt, then type pviewer.Figure 3.11 shows an example of Process Viewer.

Chapter 3 www.netpro.com71

Figure 3.11: The Process Viewer utility allows you to view the processes and threadsrunning on your domain controller.

Using this utility, you can view the name of each process, the amount of time it’s been running, the memory allocatedto it, and its priority. You can also view each thread that makes up a selected process. For each thread, you can seehow long it’s been running, its priority, context switches, and starting memory address.

Page 82: The Definitive Guide to Active Directory Troubleshooting

In addition to the information you see on the main screen, you can display the memory details for a process. Figure3.12 illustrates the Memory Details dialog box that is shown when you select a process, then click Memory Details.

Chapter 3 www.netpro.com72

Figure 3.12: Memory details for each process are displayed by clicking Memory Details inProcess Viewer’s main window.

When using Process Viewer, you can stop or kill a process that is running on a domain controller by selecting

it and clicking Kill Process. However, be sure you understand the function and impact of killing a process

before doing so—it may be vital to your domain controller’s functionality. Worse yet, by killing a process, you

can irrecoverably lose or corrupt the data.

Using Task Manager to View Processes on a Domain ControllerEarlier in this chapter (see "Using Task Manager to View Memory on a Domain Controller"), I described using TaskManager as the quickest and easiest method of monitoring the performance of the CPU. You can also use TaskManager to see which processes or threads are running on the Win2K domain controller and to view a summary ofoverall processor or CPU usage. To launch Task Manager, either right-click the taskbar and choose Task Manageror press Ctrl+Alt+Del, then select Task List from the menu. Then click the Processes tab. A list is displayed of theprocesses currently running on the domain controller. Figure 3.13 shows an example of the processes running onWin2K and displayed in Task Manager.

Page 83: The Definitive Guide to Active Directory Troubleshooting

This view provides a list of the processes that are running—their names, their process identifiers (PIDs), the per-centage of CPU processing they’re taking up, the amount of CPU time they’re using, and the amount of memorythey’re using. Notice the System Idle Process, which always seems to be toward the top of the process list. This is aspecial process that runs when the domain controller isn’t doing anything else. You can use the System Idle Processto tell how the CPU isn’t loaded because it’s the exact opposite of the CPU Usage value on the Performance tab.For example, if the CPU Usage value is 5, the System Idle Process value will be 95. A high value for the SystemIdle Process means that the domain controller isn’t heavily loaded, at least at the moment you checked.

Working with the List of ProcessesYou can sort this list of processes according to one of the column labels mentioned above. For example, if you wantto view the processes in order of the amount of memory used, simply click that column’s label. The display or listwill change accordingly.

Task Manager also allows you to customize the columns, thereby receiving additional information about theprocesses and being able to assess as many as 23 parameters. To customize the columns, on the Processes page,choose View>Select Columns. As shown in Figure 3.14, notice that many additional columns of information canbe displayed. These additional columns will help you monitor and tune each process more completely. For moreinformation about each of the additional columns, refer to the Help menu in Task Manager.

Chapter 3 www.netpro.com73

Figure 3.13: Win2K’s Task Manager allows you to view and manage processes that arecurrently running on the system.

Page 84: The Definitive Guide to Active Directory Troubleshooting

In addition, you can see which of these processes belong to an application. To do so, click the Applications tab,right-click one of the applications on the Applications page, then click Go To Process. This will take you to theassociated application’s process on the Process tab. This feature helps you associate applications with their processes.

Chapter 3 www.netpro.com74

Figure 3.14: The Select Columns dialog box in Task Manager allows you to monitor additionalimportant statistics about the processes that are running on your domain controller.

As you may know, highlighting a process in Task Manager, then clicking End Process, stops that process from

running. This is a useful feature because it allows you to stop processes that don’t provide any other means

of being stopped. However, I recommend that you only use this method as a last resort because the process

stops immediately and doesn’t have a chance to clean up its resources. Using this method to stop processes

may leave domain controller resources unusable until you restart. It may also cause data to be lost or corrupted.

Viewing Information about ProcessesIf you view the list of processes from either Process Viewer or Task Manager and don’t know what a process is, youcan use the Computer Management utility to view the processes and their associated file paths or locations.Computer Management also lets you view the version, size, and date of the application or module for each process.

To view the Computer Management utility, choose Start>Programs>Administrative Tools>Computer Management.Once the utility has loaded, select System Tools, System Information, Software Environment, Running Tasks.Figure 3.15 displays processes and their associated information in the Computer Management utility.

Page 85: The Definitive Guide to Active Directory Troubleshooting

Using the Performance Console to View Processes on a Domain ControllerYou can use the System Monitor application in the Performance console to view the values of the performancecounters for the processor, processes, and threads. This utility allows you to graphically display these counters overtime. In the next few sections, I’ll discuss some of the counters and how you can use them.

% Processor Time CounterThe first counter that you should check when monitoring the domain controller is % Processor Time. This countergauges the activity of the computer’s CPU. It shows the percentage of time that all processors in the domain controllerare busy executing code other than the System Idle Process. Acceptable processor activity ranges between 1 percentand 85 percent, although the actual amount depends on the type of applications loaded on your domain controller.

The % Processor Time counter is a primary indicator of processor activity. This counter is calculated by measuringthe time that the processor spends executing the idle process, then subtracting that value from 100 percent. It canbe viewed as the percentage of useful work that the processor executes. To view the % Processor Time counter, youuse System Monitor. Figure 3.16 shows the % Processor Time counter in System Monitor.

Chapter 3 www.netpro.com75

Figure 3.15: The Computer Management utility allows you to view the processes that are running on your domaincontroller as well as the path and file name information associated with each process.

Page 86: The Definitive Guide to Active Directory Troubleshooting

If the % Processor Time counter is consistently high, there may be a bottleneck on the CPU. I recommend thatthis counter consistently stay below 85 percent. If it pushes above that, you need to find the process that is using ahigh percentage of the processor. If there is no obvious CPU "hog," you may want to consider adding anotherprocessor to the domain controller or reducing that domain controller’s workload. Reducing the workload mightinvolve stopping services, moving databases, removing directory services, and so on.

Interrupts/sec CounterThe Interrupts/sec counter measures the rate of service requests from the domain controller’s I/O devices. Thiscounter is the average number of hardware interrupts that the processor is receiving and servicing each second. Ifthis value increases without an associated increase in system response, there could be hardware problems on one ofthe I/O devices. For example, a network interface card installed in the domain controller could go bad and causean excessive amount of hardware interrupts. To fix the problem, you need to replace the offending network card’sdriver or the physical card.

The Interrupts/sec count doesn’t include deferred procedure calls; they’re counted separately. Instead, this countertracks the activity of hardware devices that generate interrupts, such as the system clock, mouse, keyboard, diskdrivers, network interface cards, and other peripheral devices. (For example, the system clock interrupts the CPUevery 10 milliseconds.) When an interrupt occurs, it suspends the normal thread execution until the CPU hasserviced the interrupt.

During normal operation of the domain controller, there will be hundreds or thousands of interrupts per second.System Monitor displays the counter as a percentage of the real number. This means that if the domain controllerhas 560 interrupts in one second, the value is shown as 5.6 on the graph. Figure 3.17 displays the Interrupts/seccounter using System Monitor.

Chapter 3 www.netpro.com76

Figure 3.16: The % Processor Time counter give you the ability to view the amount of time that the proces-sor is doing real work.

Page 87: The Definitive Guide to Active Directory Troubleshooting

Unfortunately, it’s difficult to suggest a definite threshold for this counter because this number depends on theparticular processor type in use and the exact role and use of the domain controller. I therefore recommend thatyou establish your own baseline for this counter and use it as a comparison over time. This will help you knowwhen a hardware problem occurs. For example, a network interface card installed in the domain controller couldgo bad and cause an excessive number of hardware interrupts. By having an established baseline, you can quicklyidentify that there is a problem.

Processor Queue Length CounterThe Processor Queue Length counter under the System object displays the number of processes or threads waitingto be executed in the run queue that is shared among all processors on the server. This counter displays the lastobserved value only; it’s not an average. If there are too many threads waiting for the CPU and the CPU cannotkeep up, the system is processor-bound and starts to slow down. Figure 3.18 illustrates how System Monitor showsProcessor Queue Length.

Chapter 3 www.netpro.com77

Figure 3.17: The Interrupts/sec counter allows you to view the impact the hardware I/O devices have onthe performance of the domain controller.

In System Monitor, you can make changes to the graph display. To do this, right-click anywhere on the graph, then

choose Properties from the shortcut menu. The System Monitor Properties dialog box appears, containing several

tabs to change the display and effect of the graph and data. For example, if you want to change the graph scale,

click the Graph tab and change the Vertical Scale parameters. To confirm the change, click Apply, then OK.

Page 88: The Definitive Guide to Active Directory Troubleshooting

I recommend that your domain controller not have a sustained Processor Queue Length of greater than twothreads. If the number of threads goes above two, performance slows down, as does responsiveness to the users. Thedomain controller shown in the figure could be in trouble, especially if this type of activity is sustained. There areseveral ways to alleviate the domain controller slowing down. You can replace the CPU with a faster processor, addmore processors, and reduce the workload. In some situations, the Processor Queue Length counter will increase ifthe system is paging heavily, so adding memory or RAM could be needed. To determine if you need more RAM,monitor the paging counters.

Monitoring the Disk Because the hard drives in the domain controller have moving parts, they’re always the slowest subsystem in thecomputer. In fact, the hard drive subsystem will typically be over 100,000 times slower than the memory subsystem.As a result, the architects of Win2K designed the file-system caching service. Its sole responsibility is to move dataoff the hard drives and into the faster memory subsystem. This minimizes the performance penalty of retrievingdata from the domain controller.

By nature, a hard drive is massive and cheap. The disk subsystem can contain hundreds of GB storing millions orbillions of files. In turn, memory is relative small and expensive. Therefore, the architects of Win2K designed thevirtual memory system to store pieces of memory on the hard drive, thereby allowing more room for users andapplications. However, as I discussed earlier (see "Paging File Usage"), you pay a performance price for paging.

Chapter 3 www.netpro.com78

Figure 3.18: The Processor Queue Length counter indicates how congested the processor is.

Page 89: The Definitive Guide to Active Directory Troubleshooting

Using the Performance Console to Monitor the Disk SubsystemBecause system performance depends so heavily on the disk subsystem, it’s important that you understand how tomonitor it. To properly monitor the disk subsystem, you need to monitor disk usage and response time, whichincludes the number of actual reads and writes plus the speed with which the disk accomplishes each request. Theprimary utility you use to monitor these attributes is System Monitor in the Performance console. Using SystemMonitor, you can view key counters that apply to physical device usage and the logical volumes on the drives.

% Disk Time and % Idle Time CountersThe % Disk Time counter under the PhysicalDisk object allows you to view how busy the domain controller’s harddrive is. This counter is a percentage of elapsed time that the hard drive is busy servicing read and write requests.The % Idle Time counter under the PhysicalDisk object reports the percentage of time the hard drive is sitting idle.

Using these counters, you can monitor the physical activity of the hard drives in each computer. Figure 3.19illustrates the % Disk Time and % Idle Time counters in System Monitor.

Chapter 3 www.netpro.com79

Figure 3.19: The % Disk Time counter allows you to view how busy a physical disk drive is, and the % IdleTime counter tracks the percentage of time a drive is idle.

The figure shows that as you might expect, % Disk Time and % Idle Time basically mirror each other. I recommendthat if the value for % Disk Time is consistently above 70 percent, you consider reorganizing the domain controllerto reduce the load. However, if the domain controller is a database server, the threshold can go as high as 90 percent.The threshold value depends on the type of server that has been implemented and what has caused the disk I/O.For example, if VMM is paging heavily, it can drive up the % Disk Time counter. The simplest solution here is toadd memory.

Page 90: The Definitive Guide to Active Directory Troubleshooting

Disk Reads/sec and Disk Writes/sec CountersIn addition to the percentage of time the disk is busy, you can also see what the disk is doing. You can monitor thisusing two counters under the PhysicalDisk object in System Monitor.

Disk Reads/sec counterTracks the rate of read operations on the disk.

Disk Writes/sec counterTracks the rate of write operations on the disk.

Normally, a domain controller will perform twice as many (if not more) read operations than write operations; itcan also service a read request at least twice as fast. This is because the write request has to write the data, then verifythat it was written. You can see the Disk Reads/sec and Disk Writes/sec counters in System Monitor, as shown inFigure 3.20.

Chapter 3 www.netpro.com80

Figure 3.20: The Disk Reads/sec and the Disk Writes/sec counters show how the domain controller ishandling the disk requests that come to it.

Using these counters, watch for spikes in the number of disk reads when your domain controller is busy. If youhave the appropriate amount of memory on your domain controller, most read requests will be serviced from thesystem cache instead of hitting the disk drive and causing disk reads. You want at least an 80 percent cache hit rate;this means that only 20 percent of read requests are forced to the disk. This is valid unless you have an applicationthat reads a lot of varying data at the same time—for example, a database server is by nature disk-intensive andreads varying data. Obtaining a high number of cache hits with a database server may not be possible.

Page 91: The Definitive Guide to Active Directory Troubleshooting

Current Disk Queue Length CounterThe Current Disk Queue Length counter represents the number of requests outstanding on the disk at any onetime. The disk has a queue, or list, that can hold the read and write requests in order until they can be serviced bythe physical device. This counter shows the number of requests in service at the time the sample is taken. Mostdisk devices installed on your domain controller are single-spindle disk drives. However, disk devices with multiplespindles, such as some Redundant Array of Independent Disks (RAID) disk systems, can have multiple reads andwrites active at one time. Thus, a multiple-spindle disk drive can handle twice the rate of requests of a normal device.

Figure 3.21 displays System Monitor tracking the length of the disk queue.

Chapter 3 www.netpro.com81

Figure 3.21: The Current Disk Queue Length counter represents the number of outstanding read and writerequests. Using this counter, you can monitor the performance of the queue for the disk drives.

If the disk drive is under a sustained load, this counter will likely be consistently high. In this case, the read andwrite requests will experience delays proportional to the length of this queue, divided by the number of spindles onthe disks. For decent performance, I recommend that the value of the counter average less than 2.

Because gathering disk counters can cause a modest increase in disk-access time, Win2K doesn’t

automatically activate all the disk counters when it starts up. By default, the physical disk counters are on,

and the logical disk counters are off. The physical disk counters monitor the disk driver and how it relates to

the physical device. The logical disk counters monitor the information for the partitions and volumes that have

been established on the physical disk drives.

Page 92: The Definitive Guide to Active Directory Troubleshooting

% Free Space CounterAn example of a logical disk counter is the % Free Space counter. This counter is the percentage of the free spaceavailable on the logical disk or volume. Free space is calculated as a ratio of the total usable space provided on thevolume of the logical disk drive. This counter is obviously an important one to monitor because it allows you toview the amount of disk space that is left for user and application requests.

This counter allows you to monitor the performance of the disk drives as they start to fill up. This task is importantbecause as a disk drive starts to run out of space, each write request becomes tougher to perform and slows downoverall disk performance. The reason for this is that as the drive fills up, each write takes longer to search for space.The longer it takes the disk to write the data, the less it does, so performance slows. Thus, as the drive fills up, itworks harder to service requests; this is often called thrashing. I recommend that you always leave at least 10 percentof the disk free to minimize thrashing.

Monitoring the Network Each Win2K domain controller depends on the network to move information to its users and to other servers.However, if the network becomes too crowded and traffic exceeds capacity, performance for all users and domaincontrollers will suffer. You need to monitor the network components for each domain controller on your networkto help eliminate bottlenecks. Monitoring the network typically consists of observing usage on network componentsand measuring the amount of traffic on the network.

Using Network Monitor to Watch Network TrafficNetwork Monitor (Netmon.exe) allows you to analyze in-depth, low-level network traffic and enables you to detectand analyze problems on your local networks and WAN connections. Network Monitor captures and displays thenetwork packets that the Win2K domain controller receives from users and other servers to provide real-time trafficmonitoring and analysis. You can also display the traffic in a post-capture mode to help you analyze it after the fact.

In real-time mode, Network Monitor allows you to monitor and test network traffic for a specific set of conditions.If the conditions are detected, it displays the events and prompts you for the appropriate action. In post-captureanalysis, network traffic is saved in a proprietary capture file and can be parsed by protocol to pick out specificnetwork frame types.

Network Monitor does the following:

• Captures network data in real-time or delayed mode• Provides filtering capabilities when capturing network packets • Uses parsers for detailed post-capture analysis.

Chapter 3 www.netpro.com82

To start the domain controller with the logical disk counters on, you use the DISKPERF utility. At the command

prompt, type DISKPERF –YV. This sets the domain controller to gather counters for both the logical disk

devices and the physical devices the next time the system is started. For more information about using the

DISKPERF utility, type DISKPERF /? at the command prompt.

Page 93: The Definitive Guide to Active Directory Troubleshooting

Using the Performance Console to Monitor Network Components on a Domain ControllerYou can use System Monitor in the Performance console to monitor the domain controller’s network performance.Specific performance counters allow you to watch the computer’s network throughput and network interfaces.

Domain Controller Network ThroughputThe easiest way to measure domain controller throughput and the bandwidth of each network component is todetermine the rate at which the computer sends and receives network data. Several performance counters under theServer object in System Monitor can help you measure the data transmitted through your domain controller’s networkcomponents. These counters represent all the network traffic sent to and received from the domain controller andall the network interface cards installed on it.

Bytes Total/secThe number of bytes the domain controller has sent and received from the network each second. This value provides an overall indication of how busy the domain controller is, servicing network requests. It can help you determine whether any network components are creating bottlenecks for network traffic.

Bytes Transmitted/secThe number of bytes that the domain controller has sent on the network. It indicates the network traffic that has been sent.

Bytes Received/secThe number of bytes that the domain controller has received from the network. It indicates the network traffic that has been received.

The advantage of the last two counters is that they break out the values for traffic sent and received.

I recommend that once you’ve monitored these counters, you compare the results to your domain controller’s totalnetwork throughput. To do this, I suggest that you establish a baseline of data rates and averages. Establishing abaseline allows you to know what to expect from the domain controller. If a potential problem or bottleneck innetwork throughput occurs, you can recognize it immediately because you can compare it against the baselineyou’ve established.

You can also make some estimates as to where a bottleneck exists if you know the network and bus speeds of thedomain controller. If the data rate through the card is approaching the network limit, segmenting and adding acard may help. If the aggregate data rate is approaching the bus speed, it may be time to split the load for thedomain controller and add another one or go to clustering.

Network Interface ThroughputIf you want to break down the amount of traffic to each individual network adapter or interface card, you’ll wantto use the Network Interface object in System Monitor. The counters that display the amount of traffic processedby each network interface card are Bytes Total/sec, Bytes Sent/sec, and Bytes Received/sec. The counters in the

Chapter 3 www.netpro.com83

Network Monitor is a complex tool that allows you to monitor all kinds of network traffic and troubleshoot a

variety of network problems. Thus, a detailed explanation of it is beyond the scope of this chapter and book.

Page 94: The Definitive Guide to Active Directory Troubleshooting

previous section have similar names, but they display the amount of traffic for the entire domain controller, regardlessof the actual number of interface cards installed. Using the counters assigned to each network adapter allows you todrill down and see how each performs individually.

Bytes Total/secThe number of bytes the network interface card has sent and received from the network each second. This value measures the rate at which bytes are both sent and received on the network interface card; this includes all frame and media types. This value also provides an overall indication of how busy the network adapter is.

Bytes Sent/secThe rate at which bytes are sent on the network interface. This value breaks down the amount of traffic being sent.

Bytes Received/secThe rate at which bytes are received. This value breaks down the amount of traffic being received.

Figure 3.22 illustrates how you can use the Bytes Total/sec, Bytes Sent/sec, and Bytes Received/sec counters inSystem Monitor to monitor the domain controller’s network adapter.

Chapter 3 www.netpro.com84

Figure 3.22: The Bytes Total/sec, Bytes Sent/sec, and Bytes Received/sec counters allow you to monitorthe domain controller’s network adapter.

Page 95: The Definitive Guide to Active Directory Troubleshooting

Summary As a network administrator, a critical part of your job is making sure that each and every domain controller hostingAD is functioning properly. To accomplish this task, you need to properly monitor each of these Win2K domaincontrollers; this in turn means watching over the critical OS components and hardware subsystems. To help youmonitor a domain controller and its subsystems, Win2K provides several utilities, and this chapter discussed themost important ones: Task Manager, the Performance console, and Event Viewer. Using these utilities, you canwatch server resources and subsystems in real time while they work to support the requests by users, applications,and other servers.

Chapter 3 www.netpro.com85

eBook Copyright NoticeThis site contains materials created, developed, or commissioned by Realtimepublishers.com, Inc. and is protected byinternational copyright and trademark laws. No material (including but not limited to the text, images, audio, and/or video)may be copied, reproduced, republished, uploaded, posted, transmitted, or distributed in any way, except that one copymay be downloaded for your personal, non-commercial use on a single computer. In connection with such use, you maynot modify or obscure any copyright or other proprietary notice. If you have any questions about these terms, or if youwould like information about licensing materials from Realtimepublishers.com, please contact us via e-mail at [email protected]

Page 96: The Definitive Guide to Active Directory Troubleshooting

The Definitive Guide to Active Directory Troubleshooting

Chapter 4

Page 97: The Definitive Guide to Active Directory Troubleshooting

Monitoring Active DirectoryTroubleshooting Active Directory (AD) and AD-based networks requires that you become familiar with AD constructsas well as monitoring tools and techniques. Monitoring AD allows you to determine whether problems are occurring inany part of the directory. However, it’s sometimes difficult to accurately determine the cause of a problem because AD isdistributed across domain controllers and interacts with a number of external services and protocols, such as:

• Domain Name System (DNS)—for name resolution

• Lightweight Directory Access Protocol (LDAP)—for directory lookups

• Transmission Control Protocol/Internet Protocol (TCP/IP)—for transport.

AD also has a complex infrastructure containing many different components. To ensure the health of the directoryas a system, you must monitor all of these components. You also need to understand AD’s internal processes, suchas replication.

In this chapter, I’ll describe which infrastructure components you need to continually monitor to ensure ADavailability as well as some of the built-in and third-party utilities that are available to help you do so. It’s always agood idea to have a sound understanding of one’s tools before using them, so I’ll start by introducing the tools inour monitoring tool set.

Using the Monitoring ToolsYou can use several tools to monitor the individual areas of AD and AD as a service or system. These tools includebuilt-in Win2K utilities and Support Tools/Resource Kit utilities as well as those available from third-partyindependent software vendors (ISVs). This chapter will give you an overview of all of these utilities and describehow they can help you monitor the directory. It won’t give a comprehensive list of all utilities built into Windows2000 (Win2K) or available on the market, and it won’t provide extensive documentation on any of these tools.Rather, it’ll focus on tools that represent, in my opinion, the best of the built-in and third-party tools.

Third-Party ToolsIn this section, I’ll discuss NetPro’s DirectoryAnalyzer and NetIQ’s AppManager.

DirectoryAnalyzer from NetProDirectoryAnalyzer from NetPro was one of the first AD monitoring tools on the market, and it performs real-timemonitoring and alerting on all aspects of the AD infrastructure. Instead of monitoring individual domain controllers,it monitors the directory as a whole. It does this by monitoring all domain controllers and directory processes atonce as a background process. If a problem occurs at any level in the directory, DirectoryAnalyzer alerts, or notifies,your users. If the problem is critical, its integrated knowledge base contains descriptions and troubleshooting methodsthat will help you solve it.

DirectoryAnalyzer monitors the individual structures and components of AD—replication, domains, sites, GlobalCatalogs (GCs), operations master roles, and DNS (inasmuch as it relates to AD). Each of these components is

Chapter 4 www.netpro.com87

Page 98: The Definitive Guide to Active Directory Troubleshooting

vital to the operation of AD. DirectoryAnalyzer can monitor and alert on specific conditions and problems in eachof the individual structures. The alerts are then recorded at the DirectoryAnalyzer client or console for viewing.

Alerts have two levels of severity—warning and critical. Warning alerts indicate that a predetermined threshold hasbeen met in one of the directory structures. Warning alerts help you identify when and where problems may occur.Critical alerts indicate that a predetermined error condition has been met. Critical alerts are problems that needyour immediate attention; if you ignore them, AD could lose functionality or the directory altogether.

By clicking Current Alerts under View Status in the sidebar, you can display all of the alerts with their associatedtype, time, and description. Figure 4.1 shows the Current Alerts screen in DirectoryAnalyzer. The alerts have beenrecorded for the AD domain controllers, directory structures, and directory processes.

Chapter 4 www.netpro.com88

Figure 4.1: DirectoryAnalyzer allows you to monitor the entire directory for problems.

You can also send alerts to enterprise management systems using Simple Network Management Protocol (SNMP).This allows you to integrate DirectoryAnalyzer alerts with management consoles such as HP OpenView and Tivoli.Alerts can also be recorded in the Event Log of the Win2K system and viewed using the Event Viewer utility. (See"Event Viewer" later in this chapter.)

DirectoryAnalyzer logs all alert activity to a history database. You can export the database and analyze alert activityover time using a variety of formats, such as Microsoft Excel, Hypertext Markup Language (HTML), DynamicHTML (DHTML), and Rich Text Format (RTF). You can also identify trends in the data, finding cycles or periodsof high and low alert activity.

Page 99: The Definitive Guide to Active Directory Troubleshooting

AppManager from NetIQAppManager from NetIQ Corporation is a suite of management products that manages and monitors theperformance and availability of Win2K. One of these management products allows you to monitor the performanceof AD. For example, AppManager verifies that replication is occurring and up-to-date for the directory by monitoringthe highest Update Sequence Number (USN) value for each domain controller. The USN is discussed in moredetail later in this chapter (see "Monitoring Replication"). In addition, inbound and outbound replication statisticsare tracked, as are failed synchronization requests for the directory.

AppManager also allows you to monitor the number of directory authentications per second and monitor the cachehit rate of name resolution. Using this tool, you can monitor and track errors and events for trust relationships.You can also log errors and events to enterprise management systems using SNMP. This means that SNMP trapsare generated and routed to a configured network manager.

In addition, you can use or run a set of prepackaged management reports that allow you to further analyze currenterrors and events. You can also set up this utility to send e-mail and pager alerts when an event is detected.

Built-In ToolsIn this section, I’ll discuss System Monitor, Event Viewer, and REPADMIN.

System MonitorFor the domain controller in AD, one of the main monitoring utilities is System Monitor. This utility allows youto watch the internal performance counters that relate to the directory on the domain controller. The directoryperformance counters are software counters that the developers of AD have programmed into the system.

Using System Monitor, you can monitor current directory activity for the domain controller. Once you’ve installedAD on a server, several performance counters—for replication activity, DNS, address book, LDAP, authentication,and the database itself—measure the performance of the directory on that computer.

I discussed how to launch and use System Monitor in Chapter 3, so I won’t repeat that information here. Instead,I’ll focus on how to use some of the more important performance counters that are available for AD. Remember,System Monitor tracks all of its counters in real time. For this reason, I recommend that you always establish abaseline or normal operation that you can compare the real-time values against. When adding AD counters toSystem Monitor, if you don’t understand the meaning of any counter, highlight it, then click Explain. The ExplainText dialog box appears and provides a description of the counter.

You can also graph the performance counters and set alerts against them. The alerts will appear in the Event Viewer.

Chapter 4 www.netpro.com89

Page 100: The Definitive Guide to Active Directory Troubleshooting

Event ViewerTo view and analyze the events that have been generated by a Win2K domain controller, you can use the EventViewer. This utility allows you to monitor the event logs generated by Win2K. By default, there are three eventlogs: the application log, the system log, and the security log. (These three logs were described in the "EventViewer" section of Chapter 3.) In addition, after you install AD, three more logs are created.

Directory service logContains the events that are generated by AD on the domain controller. You can use this log to monitor activity or investigate any directory problems. By default, the directory records all critical error events.

DNS server logContains the events generated by the DNS service installed on your domain controller. For example, when the DNS service starts or stops, it writes a corresponding event message to this log. More critical DNS events are also logged—for example, if the service starts but cannot locate initializing data, such as zones or other startup information stored in the domain controller’s Registry or AD. The DNS log exists only if the DNS service is running on the server. The DNS service typically runs on only a few domain controllers in the forest.

File Replication service logContains events generated by file replication on the domain controller. The File Replication service (FRS) isa replication engine used to replicate files among different computers simultaneously. AD uses this service to replicate Group Policy files among domain controllers.

Depending on how you configure your AD installation, you may have one or all of these logs on your domaincontroller. Figure 4.2 shows the Event Viewer startup screen on a domain controller after you’ve installed AD with DNS.

Chapter 4 www.netpro.com90

Page 101: The Definitive Guide to Active Directory Troubleshooting

Replication Diagnostics (REPADMIN)The Replication Diagnostics tool is simply referred to as REPADMIN. It’s a command-line utility that allows youto monitor and diagnose the replication process and topology in AD. It also provides a number of switches that youcan use to monitor specific areas of replication. For example, you can force replication among domain controllersand view the status.

During normal replication, the Knowledge Consistency Checker (KCC) manages and builds the replication topologyfor each naming context on the domain controller. The replication topology is the set of domain controllers thatshares replication responsibility for the domain. REPADMIN allows you to view the replication topology as seenby the domain controller. If needed, you can use REPADMIN to manually create the replication topology, althoughthis isn’t usually beneficial or necessary because it’s generated automatically by the KCC.

You can also view the domain controller’s replication partners, both inbound and outbound, and some of the internalstructures used during replication, such as the metadata and up-to-dateness vectors.

You can install the REPADMIN.EXE utility from the Support\Tools folder on the Microsoft Windows 2000 CD.Running the SETUP program launches the Win2K Support Tools Setup wizard, which installs this tool along withmany other useful support tools to the Program Files\Support Tools folder. Figure 4.3 shows the interface forREPADMIN.

Chapter 4 www.netpro.com91

Figure 4.2: The Event Viewer startup screen lists additional event logs that have been created for AD.

Page 102: The Definitive Guide to Active Directory Troubleshooting

Monitoring the AD InfrastructureAn important aspect of any AD deployment is always monitoring the environment and infrastructure. Theinfrastructure of AD is the set of processes and data structures that the directory service uses to function properly.By constantly monitoring the infrastructure, you can detect issues that arise in the environment and correct thembefore they affect your users. For example, users will be affected if there is an intermittent failure of a bridgeheadserver or if a Flexible Single Master Operation (FSMO, pronounced "fizmo") role-holding server goes down.

The first task in troubleshooting AD is to constantly monitor critical areas of the directory deployment. Irecommend that you continuously monitor at least the following directory structures and components:

Chapter 4 www.netpro.com92

Figure 4.3: The REPADMIN utility allows you to view the replication process and topology.

Page 103: The Definitive Guide to Active Directory Troubleshooting

Domain controllersThese are critical to the proper operation of AD. If one domain controller isn’t functioning properly, the directory and some users will lose performance and possibly functionality. If the domain controller that is having problems has also been assigned additional vital roles (such as being a DNS or GC server), the directory may become unavailable to all users. Thus, it’s critical to monitor and track the performance of alldomain controllers on the network at all times.

Domain partitionStores AD objects and attributes that represent users, computers, printers, and applications. The domain partition is also used to accomplish a number of management roles, which include administration and replication. You must monitor the performance and availability of the domain partition so that the services it supports are constantly available.

GC partitionStored on domain controllers throughout the network. Only a few domain controllers store a copy of the GC partition, and they need to be monitored for the GC. GCs are specialized domain controllers whose availability is necessary for clients to be able to log on to the network. The GC streamlines directory searches because it contains all of the objects in the forest but only a few of their key attributes.

Operations mastersThese (or FSMO role holders) are single-master domain controllers that perform special roles for AD. It’s important that you monitor and track the performance of each operations master so that the service itperforms is maintained. If any operations master stops functioning, its functionality is lost in the directory.

Replication process and topologyAre critical to the operation of AD. If changes have been made to a directory object on one domain controller,the replication process needs to propagate the changes to all of the other domain controllers that have replicas of that object. If replication isn’t functioning, different portions of the directory get out of sync. This confusesusers, and they lose access to directory resources.

For example, if an administrator has changed a Group Policy but the change hasn’t been synchronized to all copies, users using the older copies may access the wrong information. In addition, once the synchronizationamong directory replicas is lost, it’s very difficult and time-consuming to get back. Thus, it’s critical to constantly monitor the replication process and topology for problems.

Monitoring the Domain ControllersBecause AD can be distributed across many domain controllers, you need to constantly monitor individual domaincontrollers. If one domain controller isn’t functioning properly, the directory and your users will lose performanceand possibly functionality. If multiple domain controllers aren’t functioning properly, the network can becomeunusable. For this reason, I recommend that you always check or monitor that the domain controller’s hardwareand subsystems are operating correctly. (For details on how to monitor the hardware components of the domaincontroller, refer to Chapter 3.) After you’re confident that the hardware is performing well, you need to monitorthe AD services running on the domain controllers for errors and other problems.

Chapter 4 www.netpro.com93

Page 104: The Definitive Guide to Active Directory Troubleshooting

Using DirectoryAnalyzerMany third-party tools (such as those I discussed earlier) provide you with an easy way to monitor all of thedomain controllers in your forest from one management console. For example, in DirectoryAnalyzer, click BrowseDirectory By Naming Context; the directory hierarchy is displayed. If you expand the naming contexts, you see allof the associated domain controllers. To see the alerts for just one domain controller, select a domain controllerobject, then click Current Alerts. The alerts that are displayed have exceeded a warning or critical threshold andshow the severity, subject, associated type, time, and description. Figure 4.4 shows an example of usingDirectoryAnalyzer to view all alerts for each domain controller.

Chapter 4 www.netpro.com94

DirectoryAnalyzer is an extremely useful utility because it monitors all of the domain controllers in the AD forestas a background process and allows you to periodically view the results. It also monitors the most critical directorystructures and processes—for example, the configuration and activity for the domain partitions, GC partitions,operations master roles, sites, DNS, the replication process, and the replication topology.

Figure 4.4: DirectoryAnalyzer allows you to monitor all the domain controllers in your forest for problems and see thealerts that have been recorded for each domain controller.

To see the alerts and other information for each domain controller, you can also use the Browse Directory

By Site option. It allows you to browse the directory layout according to sites and their associated domain

controllers. In addition, it permits you to view the status of each site and the site links.

Page 105: The Definitive Guide to Active Directory Troubleshooting

In addition to viewing the alerts from the domain controllers, you can click any alert and see a more detaileddescription of the problem. If you don’t understand the alert, you can double-click it; the Alert Details dialog boxwill appear and provide more description, as shown in Figure 4.5.

Chapter 4 www.netpro.com95

Figure 4.5: DirectoryAnalyzer provides more information about an alert in the Alert Details dialog box.

Once you’ve been notified of the alert and viewed more information about it in the Alert Details dialog box, youcan use the integrated knowledge base to help resolve the problem. The knowledge base provides you with adetailed explanation of the problem, helps you identify possible causes, then helps you remedy or repair the problem.To access the knowledge base, click More Info in the Alert Details dialog box or choose Help>Contents in the console.Figure 4.6 shows an example of the information available in the knowledge base.

Page 106: The Definitive Guide to Active Directory Troubleshooting

As you know by now, domain controllers are the workhorses of AD. They manage and store the domain informationand accept special functions and roles. For example, a domain controller can store a domain partition, store a GCpartition, and be assigned as a FSMO role owner. Domain controllers, in turn, allow the directory to manage userinteraction and authentication and oversee replication to the other domain controllers in the forest.

In addition to displaying alerts for each domain controller, DirectoryAnalyzer displays detailed configurations. Forexample, when you choose Browse Directory By Naming Context, you see several icons for each domain controller.An icon that includes a globe indicates that the domain controller stores a GC partition. When an icon displayssmall triangles, it indicates that the domain controller is also providing the DNS service. An icon that displays botha globe and small triangles indicates that the domain controller has both a GC and a DNS.

Chapter 4 www.netpro.com96

Figure 4.6: DirectoryAnalyzer’s in-depth knowledge base helps you find solutions to problems in AD.

Page 107: The Definitive Guide to Active Directory Troubleshooting

If you select a domain controller, then click the DC Information tab, you can view detailed information about howthe domain controller is operating and handling the directory load. Figure 4.7 shows the DC Information pane inDirectoryAnalyzer.

Chapter 4 www.netpro.com97

Figure 4.7: You can view detailed information about a domain controller using the DC Information pane in DirectoryAnalyzer.

DirectoryAnalyzer provides a high-level summary of how each domain and its associated domain controllers arefunctioning. Click Browse Directory By Naming Context to see a high-level status of all the domain controllers in adomain. To view the status for a particular domain, select it, then click the DC Summary tab. Figure 4.8 illustratesthe DC Summary pane, which uses green, yellow, and red icons to indicate the status of each domain controller ina domain.

Page 108: The Definitive Guide to Active Directory Troubleshooting

You can also quickly view where the domain controller resides, if it’s a GC, and who manages the computer. If anyof the domain controllers aren’t showing a green (clear) status icon, there is a problem that you need to investigateand fix.

Using NTDS Performance CountersNTDS (NT Directory Service) performance counters are internal domain controller counters used by multipleaspects of AD. Once AD has been installed on the domain controller, these directory counters are added to thesystem. These counters allow you to monitor and track the domain controller’s replication activity, LDAP traffic,and authentication traffic. Table 4.1 describes the more useful NTDS performance counters and how to use themto track AD activity on a domain controller.

Chapter 4 www.netpro.com98

Figure 4.8: The DC Summary pane in DirectoryAnalyzer provides a high-level status of all domain controllers in a domain.

DRA InboundBytes Total/sec

Indicates the total amount of inbound replication traffic overtime. If a small number of bytes are being sent, either thenetwork or the server is slow. Other issues that might limit thenumber of bytes being sent include few changes being madeto the naming contexts hosted by the domain controller,replication topology problems, and connectivity failures.Of course, you need to check this value against a baselineof activity.

Tracks the totalnumber of bytesper second receivedon the server duringreplication with otherdomain controllers.

This Counter Does This How to Use It

Page 109: The Definitive Guide to Active Directory Troubleshooting

Chapter 4 www.netpro.com99

DRA InboundObject UpdatesRemaining in Packets

Indicates that the server is receiving changes but is taking along time to apply them to the AD database. The value of thiscounter should be as low as possible. A high value indicatesthat the network is slow during replication or the domaincontroller is receiving updates faster than it can apply them.Other issues that can affect speed of update are high domaincontroller load, insufficient hardware (memory, disk, or CPU),the disk becoming full or fragmented, other applicationsusing too many resources, and so on.

Tracks the numberof object updatesreceived in the ADreplication updatepacket but not appliedto the local domaincontroller.

This Counter Does This How to Use It

DRA OutboundBytes Total/sec

Indicates the total amount of outbound replication trafficover time. If this value remains low, it can indicate a slowserver or network or few updates on this domain controller.In the latter case, it can mean that clients are connecting toother domain controllers because this one is slow or thatthere are topology problems. For best results, test the currentvalue against an established baseline value.

Tracks the numberof bytes that are sentfrom the serverduring replicationto other domaincontrollers.

DRA PendingReplicationSynchronizations

Indicates the backlog of directory synchronizations for theselected server. This value should be as low as possible. Ahigh value could indicate a slow server or a problem withthe server’s hardware.

Tracks the number ofpending requests fromreplication partners forthis domain controllerto synchronize withthem. Synchronizationsare queued, ready forprocessing by thedomain controller.

DS Threads in Use Indicates how the directory service on the server is respondingto client requests. When a client requests information, ADspawns a thread to handle the request. If the number ofthreads remains constant, Win2K clients may experiencea slow response from the domain controller.

Tracks the currentnumber of threadsthat are being usedby the directoryservice running on thedomain controller.

KerberosAuthentications/sec

Indicates how the domain controller is responding to clientrequests for authentications. If this counter doesn’t showactivity over time, clients could be having a problem contactingthe domain controller.

Tracks the currentnumber of authentica-tions per second forthe domain controller.

LDAP Bind Time This counter tracks only the last successful bind for anLDAP client. The value of this counter should be as low aspossible to indicate that the domain controller was quick toauthenticate the LDAP client. If the value is high, thedomain controller was slow to authenticate LDAP. A highvalue can indicate a server problem, the domain controller istoo busy, insufficient hardware (memory or CPU), or otherapplications using too many resources.

Tracks the amount oftime (in milliseconds)required to process thelast LDAP bindrequest from the client.A bind is described asauthenticating theLDAP client.

Page 110: The Definitive Guide to Active Directory Troubleshooting

NTDS counters enable you to monitor the performance of AD for the selected domain controller. You can viewthese counters under the NTDS object in System Monitor (see Figure 4.9). By default, System Monitor is startedwhen you choose Start>Administrative Tools>Performance Console.

Chapter 4 www.netpro.com100

LDAP ClientSessions

If your domain controller has LDAP clients trying to connect,the value of this counter should show activity over time. If thevalue remains constant, the server or client may have problems,the domain controller may be too busy running other applica-tions, or there is insufficient hardware (memory or CPU).

Tracks the currentnumber of LDAPsessions on the selecteddomain controller.

This Counter Does This How to Use It

LDAP Searches/sec Indicates how many LDAP search requests the domaincontroller is servicing per second. You typically view differentsearch rates depending on the domain controller’s hardware,the number of clients connected to the domain controller,and what sorts of things the clients are doing.

Tracks the numberof LDAP searchoperations that wereperformed on theselected domaincontroller per second.LDAP clients connect-ing to the serverperform the LDAPsearch operations.

LDAP SuccessfulBinds/sec

Indicates how the domain controller responds to authenticationsfrom the clients. This value allows you to view the number ofsuccessful binds per second for LDAP clients. Again, if this valueremains constant over time, there can be a network, client, orserver problem. For example, there is a bad network component,the client is too busy, or the server is too busy.

Tracks the number ofLDAP binds persecond that occursuccessfully.

NTLMAuthentications

Allows you to see whether there are authentications fromWindows 98 and NT clients for this domain controller. Ifyou’re supporting Windows 98 and NT and the value remainsconstant over time, there is a network problem. For example,the network could have a bad or poorly configured component,or the client could be too busy.

Tracks the totalnumber of WindowsNT LAN Manager(NTLM) authentica-tions per secondserviced by thedomain controller.

Table 4.1: A few of the NTDS performance counters that allow you to track how a domain controller is responding to replicationtraffic, LDAP traffic, and authentication traffic.

Page 111: The Definitive Guide to Active Directory Troubleshooting

Monitoring the Domain PartitionsDomain partitions in AD are often referred to as naming contexts, and they provide a security and replicationboundary. Each domain partition exists in the NTDS.DIT database on the domain controllers that participate inthe domain. The domain partition stores all the users, printers, servers, computers, and application data. Becauseusers depend on the domain to access other network resources, it’s important that you constantly monitor the stateof the domain partition.

Using DirectoryAnalyzerDirectoryAnalyzer allows you to monitor the alerts for each domain in AD and the associated domain controllers.These alerts monitor the domain controllers, replicas, group policies, trust relationships, DNS, and other activityfor a domain. If you see any critical alerts, you need to investigate and fix the problems.

To view the alerts for a domain, click Browse Directory By Naming Context. Select a domain, then click theCurrent Alerts tab. The display shows the current alerts for that domain (see Figure 4.10).

Chapter 4 www.netpro.com101

Figure 4.9: NTDS performance counters allow you to monitor and track load and performance of the ADimplementation on each domain controller.

Page 112: The Definitive Guide to Active Directory Troubleshooting

In addition to displaying alerts for each domain, DirectoryAnalyzer allows you to view configuration information.Using the Naming Context Information tab, you can view the current number of alerts that are active for thefollowing areas: Naming Context (or Domain), Replica, DNS Server, and DC Server.

The Naming Context Information tab also displays the number of domain controllers for the domain and whetherthe domain supports mixed mode. When a domain supports mixed mode, it allows replication and communicationwith down-level domain controllers and clients to occur. In addition, you can see which domain controllers in thedomain are performing the operations master roles and an operations master consistency check. And finally, youcan view all the trust relationships that exist for the domain. Figure 4.11 shows the Naming Context Informationpane in DirectoryAnalyzer.

Chapter 4 www.netpro.com102

Figure 4.10: DirectoryAnalyzer allows you to monitor each domain partition for problems.

Page 113: The Definitive Guide to Active Directory Troubleshooting

To further monitor the domain, DirectoryAnalyzer provides a high-level summary of each domain controller. ClickBrowse Directory By Naming Context, then click the DC Summary tab. (The DC Summary pane is shown inFigure 4.8 earlier in this chapter.)

Using Domain Database Performance CountersIn AD, the database for the domain has been implemented as an indexed sequential access method (ISAM) record ortable manager. This table manager is often referred to as the Extensible Storage Engine (ESE) and is implemented byESENT.DLL on the server. By default, the associated database file is stored on the Win2K server as<drive>\WINNT\NTDS\NTDS.DIT.

Chapter 4 www.netpro.com103

Figure 4.11: The Naming Context Information pane in DirectoryAnalyzer allows you to see detailed information for a domain.

If necessary, you can relocate the NTDS.DIT database on a domain controller using the NTDSUTIL utility.

Using this database engine, AD provides a set of database performance counters that allow you to monitor thedomain in depth. These counters provide information about the performance of the database cache, database files,and database tables, and they help you monitor and determine the health of the database for the domain controller.By default, database performance counters aren’t installed on the domain controllers. (For instructions on installingthem, see "Installing the Counters" below.)

You can view and monitor database counters using the System Monitor utility. Table 4.2 gives you a generaldescription of the more useful database performance counters and how to use them to track the activity of thelow-level database for each domain.

Page 114: The Definitive Guide to Active Directory Troubleshooting

Chapter 4 www.netpro.com104

Cache % Hits Indicates how database requests are performing. Thevalue for this counter should be at least 90%. If it’s lowerthan 90%, the database requests are slow for the domaincontroller, and you should consider adding physicalmemory to create a larger cache.

Tracks the percentageof database page requestsin memory that weresuccessful. A cache hit isa request that is servicedfrom memory withoutcausing a file-readoperation.

This Counter

Cache PageFaults/sec

Indicates how the database cache is performing. I recommendthat the computer have enough memory to always cachethe entire database. This means that the value of this countershould be as low as possible. If the value is high, you needto add more physical memory to the domain controller.

Tracks the number ofrequests (per second)that cannot be servicedbecause no pages areavailable in cache. Ifthere are no pages, thedatabase cache managerallocates new pages forthe database cache.

File OperationsPending

Indicates how the OS handles the read/write requests tothe AD database. I recommend that the value for thiscounter be as low as possible. If the value is high, youneed to add more memory or processing power to thedomain controller. This condition can also occur if thedisk subsystem is bottlenecked.

Tracks the number ofpending requests issuedby the database cachemanager to the databasefile. The value is thenumber of read and writerequests that are waitingto be serviced by the OS.

File Operations/sec Indicates how many file operations have occurred for theAD database. I recommend that this value be appropriatefor the purpose of the domain controller. If you thinkthat the number of read and write operations is too high,you need to add memory or processing power to thecomputer. However, adding memory for the file systemcache on the computer reduces file operations.

Tracks the number ofrequests (per second)issued by the databasecache manager to thedatabase file. The value isthe read and writerequests per second thatare serviced by the OS.

Does This How to Use It

Table Open CacheHits/sec

Indicates how the AD database is performing. The valuefor this counter should be as high as possible for goodperformance. If the value is low, you may need to addmore memory.

Tracks the number ofdatabase tables openedper second. The databasetables are opened by thecached schemainformation.

Table 4.2: Some of the more useful database performance counters, which allow you to monitor the database for the domainpartition that stores all of the AD objects and attributes.

Page 115: The Definitive Guide to Active Directory Troubleshooting

Installing the CountersBy default, database performance counters aren’t installed on the domain controller. To install them, you must use thedynamic-link library (DLL) file called ESENTPRF.DLL. The instructions for installing the counters are as follows:

1. Copy the %System%\System32\ESENTPRF.DLL file to a different directory. For example, you cancreate a directory named C:\Perfmon, then copy the file to it.

2. Run the REGEDT32.EXE or REGEDIT.EXE Registry Editor and create the following Registry subkeys if they don’t already exist:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ESENTHKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ESENT\Performance

3. Under the Performance subkey that you added in Step 2, add and initialize the data of the following Registry values:Open: REG_SZ: OpenPerformanceDataCollect: REG_SZ: CollectPerformanceDataClose: REG_SZ: ClosePerformanceDataLibrary: REG_SZ: C:\Performance\esentprf.dll

4. Change directory to the %SystemRoot%\System32 folder (for example, C:\Winnt\System32).

5. Load the counter information into the Registry by executing the following statement:LODCTR.EXE ESENTPERF.INI

Once you’ve installed the database performance counters, you can use them to track and monitor the database onthe domain controller. As mentioned earlier in "Using NTDS Performance Counters," you can view and track eachcounter using the System Monitor utility in the Performance Console.

Monitoring the Global CatalogAs I’ve discussed in previous chapters, special servers on Win2K networks store a Global Catalog (GC) partition,which is replicated in AD. The domain controllers that contain the GC partition are referred to as GC servers.Because only the first domain controller installed in a forest is made a GC, you need to determine and specifywhich subsequent domain controllers will act as GC servers. In addition, you need to constantly monitor the GCpartition to ensure that it remains healthy.

The GC has been designed to support two crucial functions in an AD forest: user logons and forest-wide queries orsearches. It does this by storing all of the objects in the forest and the key attributes for each. It doesn’t store all theattributes for each object; instead, it stores only the attributes it needs to perform queries and support the logonprocess. One of these attributes is the distinguished name of the object.

Chapter 4 www.netpro.com105

Page 116: The Definitive Guide to Active Directory Troubleshooting

Once users query and retrieve the distinguished name from the GC, they can issue a search on their local domaincontroller, and LDAP will chase the referrals to the domain controller that stores the real object information. Inaddition, universal group membership is stored in the GC. Because universal groups can deny access to resources, auser’s membership in this group must be discovered during logon to build the logon access token. The requestsmade to the GC are automatic and not seen by the user.

You can use DirectoryAnalyzer to monitor the GC partition and how it’s performing. It monitors and tracks thefollowing conditions:

Domain Controller: Global Catalog Load Too HighIndicates that the domain controller that stores the GC partition has too much traffic. This is LDAP trafficcoming from workstations and servers.

Domain Controller: Global Catalog Response Too SlowIndicates that the domain controller that stores the GC partition isn’t responding in time to queries and other traffic.

Replica: GC Replication Latency Too HighIndicates that replication is taking too long to synchronize the GC stored on the domain controller. Ifreplication latency (the time it takes to replicate changes to all GCs in the forest) is too high, an alert is generated.

Site: Too Few Global Catalogs in SiteIndicates that there aren’t enough GC servers in the site.

Figure 4.12 shows how DirectoryAnalyzer monitors and tracks alerts for the GC.

Chapter 4 www.netpro.com106

Figure 4.12: DirectoryAnalyzer allows you to monitor the GC partition that exists on various domain controllers throughout the forest.

Page 117: The Definitive Guide to Active Directory Troubleshooting

Monitoring Operations MastersTo prevent conflicting updates in Win2K, AD provides a single-master server to update certain operations. In asingle-master model, only one server is allowed to provide updates for the forest or domain. When a domaincontroller takes on the responsibility of the single-master operation, it’s taking on a role. Thus, this method ofupdates is called single-master operation roles. When only one domain controller can take on the role at onetime, it’s referred to as a Flexible Single Master Operation (FSMO) role.

There are currently five types of operations masters in AD. The directory automatically elects the operations masterservers during the creation of each AD forest and domain. (For more detail on these FSMOs, see Chapter 1.)

Two operations masters manage forest-wide operations, so have forest-specific FSMO roles.

Schema masterResponsible for schema extensions and modifications in the forest

Domain naming masterAdds and removes domains in the forest.

Three operations masters manage domain operations and so have domain-specific FSMO roles.

Infrastructure masterUpdates group-to-user references in a domain

RID masterAssigns unique security IDs in a domain

PDC emulatorProvides primary domain controller support for down-level clients in a domain.

Chapter 4 www.netpro.com107

The three domain-specific FSMO roles exist in every domain. Thus, an AD forest with a total of 3 domains

would have 11 FSMO roles in all: 9 domain-specific roles and 2 forest-wide roles.

Because there is only one of each of the forest-specific FSMO roles, it’s extremely important that you constantlymonitor and track the activity and health of the operations masters. If any of them fail, the directory loses functionalityuntil the computer is restarted or another appropriate domain controller is assigned the role.

To monitor operations masters, you can use DirectoryAnalyzer. It monitors, checks the status of, and alerts on severaltypes of conditions and situations relating to operations masters, such as which domain controllers are holding theoperations masters. Click Browse Directory By Naming Context, and click the Naming Context Information tab.Under Operations Master Status, you see which domain controller is holding which FSMO. Figure 4.13 shows thestatus of operations masters in the Naming Context Information pane.

Page 118: The Definitive Guide to Active Directory Troubleshooting

You can also use the Naming Context Information pane (shown in Figure 4.13 above) to check the consistency ofthe operations masters across all of the domain controllers on the network. DirectoryAnalyzer monitors what eachdomain controller reports for the FSMO assignments. If not all of the domain controllers report the same valuesfor all of the operations masters, the word No appears beside Operations Master Consistent.

To investigate the problem, click Details. The Operations Master Consistency dialog box appears, indicating thatoperations master information is inconsistent. It displays the names of the domain controllers and which domaincontroller holds each operations master. In Figure 4.14 below, the domain controller COMP-DC-04 has inconsistentinformation about the true owner of the PDC operations master because it shows domain controller COMP-DC-01as the owner when it should be COMP-DC-03. Thus, the owner of the PDC operations master is inconsistent.

Chapter 4 www.netpro.com108

Figure 4.13: DirectoryAnalyzer displays which domain controllers are holding which operations masters for the naming context.

Page 119: The Definitive Guide to Active Directory Troubleshooting

In addition to showing the status and consistency checks, DirectoryAnalyzer monitors and displays alerts for eachoperations master. The alerts that are monitored and tracked provide information about the availability of the FSMOs.To monitor the availability of the operations masters, you can click Current Alerts in the sidebar on the main screen.To display the alerts for a domain or each domain controller, click Browse Directory By Naming Context.

The alerts indicate that the domain controller that holds the operations master isn’t responding. This could mean thatthe domain controller and AD are down and not responding. It could also mean that the domain controller no longerhas network connectivity, and this could indicate DNS or Internet Protocol (IP) addressing problems. Finally, thisalert could simply mean that the domain controller or the directory that is installed is overloaded and responding tooslowly. Figure 4.15 shows how DirectoryAnalyzer monitors and tracks alerts for each operations master.

Chapter 4 www.netpro.com109

Figure 4.14: DirectoryAnalyzer allows you to monitor and check consistency for each operations master.

Page 120: The Definitive Guide to Active Directory Troubleshooting

Monitoring Replication AD is a distributed directory made up of one or more naming contexts, or partitions. Partitions are used to distributethe directory data on the domain controllers across the network. The process that keeps partition information upto date is called replication. Monitoring replication is critical to the proper operation of the directory. Before Idiscuss how to monitor replication, however, I need to describe what it is and how it works.

In AD, replication is a background process that propagates directory data among domain controllers. For example,if an update is made to one domain controller, the replication process is used to notify all of the other domaincontrollers that hold copies of that data. In addition, the directory uses multimaster replication; this means thatthere is no single source (or master) that holds all of the directory information. Using multimaster replication,changes to the directory can occur at any domain controller; the domain controller then notifies the other servers.

Because AD is partitioned, not every domain controller needs to communicate or replicate with each other. Instead,that system uses a set of connections that determines which domain controllers need to replicate to ensure that theappropriate domain controllers receive the updates. This approach reduces network traffic and replication latency(the time to replicate a change to all replicas). The set of connections used by the replication process is thereplication topology.

Chapter 4 www.netpro.com110

Figure 4.15: DirectoryAnalyzer monitors and tracks the availability of each operations master.

Page 121: The Definitive Guide to Active Directory Troubleshooting

Using Directory Partition ReplicasA directory partition replica can be a full replica or a partial replica. A full replica contains all of the objects andattributes of a partition and is read- and write-accessible. A partial replica contains a subset of the objects andattributes and is read-only. Partial replicas are stored only on a GC server. Each domain controller stores at leastthree full directory partitions, or naming contexts, which include the schema partition, configuration partition, anddomain partition.

Schema PartitionThe schema partition contains the set of rules that defines the objects and attributes in AD. This set of rules is usedduring creation and modification of the objects and attributes in the directory. The schema also defines how theobjects and attributes can be manipulated and used in the directory.

The schema partition is global; this means that every domain controller in the forest has a copy, and these copiesneed to be kept consistent. To provide this consistency, the replication process in the directory passes updatedschema information among the domain controllers to the copies of the schema. For example, if an update is madeto the schema on one domain controller, replication propagates the information to the other domain controllers, orcopies of the schema.

Configuration PartitionThe configuration partition contains the objects that define the logical and physical structure of the AD forest.These objects include sites, site links, trust relationships, and domains. Like the schema partition, the configurationpartition exists on every domain controller in the forest and must be exactly the same on each one.

Because the configuration partition exists on every domain controller, each computer has some knowledge of thephysical and logical configuration of the directory. This knowledge allows each domain controller to efficientlysupport replication. In addition, if a change or update is made to a domain controller and its configuration partition,replication is started, which propagates the change to the other domain controllers in the forest.

Domain PartitionThe domain partition contains the objects and attributes of the domain itself. This information includes users,groups, printers, servers, organizational units (OUs), and other network resources. The domain partition is copied,or replicated, to all of the domain controllers in the domain. If one domain controller receives an update, it needsto be able to pass the update to other domain controllers holding copies of the domain.

A read-only subset of the domain partition is replicated to GC servers in other domains so that other users canaccess its resources. This allows the GC to know what other objects are available in the forest.

Using Directory UpdatesAD updates are changes made to an object or attribute stored on a domain controller. When an update occurs, thedomain controller that receives it uses replication to notify other domain controllers holding replicas of the samepartition. The domain controller that receives the update (called the originating domain controller) notifies itsreplication partners of the change first, then the partners requesting the appropriate changes.

A write request from a directory client is called an originating write. When an update that originates on onedomain controller is replicated to another domain controller, the update is called a replicated write. Using thisapproach, AD can distinguish update information during replication.

Chapter 4 www.netpro.com111

Page 122: The Definitive Guide to Active Directory Troubleshooting

AD replication doesn’t use date or time stamps to determine what changes need to be propagated among domaincontrollers; instead, it uses Update Sequence Numbers (USNs). A USN is a 64-bit counter that is associated witheach object. It increments each time a change is initiated, then it’s associated with the change. To view the USN ofan object, use the following command at a command prompt:

REPADMIN /showmeta <object DN>

In addition to maintaining USNs, AD maintains an up-to-dateness vector, which helps the domain controllersinvolved in replication track updates. The up-to-dateness vector is a table containing one entry per naming context,which are the high-watermark USNs for each replication partner. During replication, the requesting domaincontroller sends the up-to-dateness vector with its replication request so that the originating domain controllersends only those updates that the requesting domain controller doesn’t already have.

The up-to-dateness vector also helps with the problems of multiple replication paths among domain controllers.AD allows multiple replication paths to exist so that domain controllers can use more than one path to send andreceive replication traffic. When multiple replication paths exist, you might expect redundant traffic and endlesslooping during replication, but the directory allows domain controllers to detect when replication data has alreadybeen replicated. This method is called propagation dampening.

AD prevents these potential problems by using the up-to-dateness vector and the high-watermark vector. Theup-to-dateness vector contains server–USN pairs and represents the latest originating update. The high-watermarkvector holds the USNs for attributes that have been added or modified in the directory and that are stored in thereplication metadata for that attribute. Using both vectors, propagation dampening can occur and unnecessarydirectory updates avoided.

As I’ve mentioned, the values in the up-to-dateness vector can determine which updates need to be sent to thedestination domain controller. For example, if the destination domain controller already has an up-to-date valuefor an object or attribute, the source domain controller doesn’t have to send the update for it. To view the contentsof the up-to-dateness vector for any domain controller, type the following command at a command prompt:

REPADMIN /showvector <NC name>

To help resolve conflicts during replication, AD attaches a unique stamp to each replicated value. Each stamp isreplicated along with its corresponding value. To ensure that all conflicts can be resolved during replication, thestamp is compared with the current value on the destination domain controller. If the stamp of the value that wasreplicated is larger than the stamp of the current value, the current value (including the stamp) is replaced. If thestamp is smaller, the current value is left alone.

Chapter 4 www.netpro.com112

Page 123: The Definitive Guide to Active Directory Troubleshooting

Using the Replication TopologyAs I mentioned earlier, the replication topology is the set of connections used by the domain controllers in a forestto synchronize the directory partition replicas. The replication topology is created automatically on the basis ofinformation in AD by the Knowledge Consistency Checker (KCC), a built-in process that runs on all domaincontrollers. By default, the KCC runs at 15-minute intervals and designates the replication routes among domaincontrollers on the basis of the most favorable connections available at that time.

The KCC automatically generates replication connections among domain controllers in the same site. This localreplication topology is called an intra-site topology. If you have multiple wide area network (WAN) locations, youcan configure site links among the sites, then the KCC can automatically create the respective replication connectionobjects. The replication topology that is created among remote locations is called an inter-site topology. The sets ofdomain controllers that replicate directly with each other are called replication partners. Each time the KCC runs,these replication partners are automatically added, removed, or modified.

Chapter 4 www.netpro.com113

Although you can disable the KCC and create connection objects by hand, I strongly recommend that you

use the KCC to automatically generate the replication topology. The reason is that the KCC simplifies a

complex task and has a flexible architecture, which reacts to changes you make and any failures that

occur. However, if your organization has more than 100 sites, you may need to manually create the

replication topology; above this number, the KCC doesn’t scale well.

The KCC uses the following components to manage the replication topology:

ConnectionsThe KCC creates connection objects in AD that enable the domain controllers to replicate with each other. A connection is defined as a one-way inbound route from one domain controller to another. The KCC manages the connection objects and reuses them where it can, deletes unused connections, and creates new connections if none exist.

ServersEach domain controller in AD is represented by a server object. The server has a child object called NTDS Setting. This setting stores the inbound connection objects for the server from the source domain controller.Connection objects are created in two ways—automatically by the KCC or manually by an administrator.

SitesThe KCC uses sites to define the replication topology. Sites define the sets of domain controllers that are well connected in terms of speed and cost. When changes occur, the domain controllers in a site replicate with each other to keep AD synchronized. If the domain controllers are local (intra-site topology), replicationstarts as needed with no concern for speed or cost—within five minutes of an update occurring. If the two domain controllers are separated by a low-speed network connection (inter-site topology), replication is scheduled as needed. Inter-site replication occurs only on a fixed schedule, regardless of when updates occur.

SubnetsSubnets assist the KCC to identify groups of computers and domain controllers that are physically close or on the same network.

Page 124: The Definitive Guide to Active Directory Troubleshooting

Site linksSite links must be established among sites so that replication among sites can occur. Unless a site link is placed, the KCC cannot automatically create the connections among sites, and replication cannot take place. Each site link contains the schedule that determines when replication can occur among the sites that it connects.

Bridgehead serversThe KCC automatically designates a single server for each naming context, called the bridgehead server, to communicate across site links. You can also manually designate bridgehead servers when you establish each site link. Bridgehead servers perform site-to-site replication; in turn, they replicate to the other domaincontrollers in each site. Using this method, you can ensure that inter-site replication occurs only among designated bridgehead servers. This means that bridgehead servers are the only servers that replicate across site links, and the rest of the domain controllers are updated within the local sites.

Using DirectoryAnalyzerDirectoryAnalyzer allows you to monitor replication among domain controllers and report any errors or problems.It allows you to track the following problems and issues:

Replication CycleThe time during which the requesting domain controller receives updates from one of its replicationneighbors. You can view the successful replication cycle as well as any errors that occurred during that time.

Replication LatencyThe elapsed time between an object or attribute being updated and the change being replicated to all the domain controllers that hold copies. If replication latency is too high, DirectoryAnalyzer issues an alert.

Replication TopologyThe paths among domain controllers used for replication. If the replication topology evaluates that the topology is transitively closed (meaning that it doesn’t matter on which domain controller an update occurs), the topology will provide for that update to be replicated to all other domain controllers.

Replication FailuresOccur when a domain controller involved in replication doesn’t respond. Each time there are consecutive failures from the same domain controller, an alert is issued. Many things can cause failures—for example, a domain controller may be too busy updating its own directory information from a bulk load.

Replication PartnersSets of domain controllers that replicate directly with each other. DirectoryAnalyzer monitors domaincontrollers and pings them to make sure that each is still alive and working. If a replication partner doesn’t respond, an alert is issued.

Replication ConflictOccurs when two objects or attributes are created or modified at exactly the same time on two domaincontrollers on the network. AD resolves this conflict automatically, and DirectoryAnalyzer issues an alert so that you’ll know that one of the updates was ignored by replication.

Chapter 4 www.netpro.com114

Page 125: The Definitive Guide to Active Directory Troubleshooting

DirectoryAnalyzer is a unique utility because it allows you to browse AD for information on, for example, thereplication cycle and replication partners. Figure 4.16 shows the Replication Information pane, which displays thelast successful replication cycle for each domain controller, replication partners, and any errors that occurred duringreplication.

Chapter 4 www.netpro.com115

Figure 4.16: DirectoryAnalyzer allows you to view the replication cycle and replication partners for each domain controller.

Using DirectoryAnalyzer, you can monitor and track the replication process for errors. If a problem occurs, the utilitywill issue an alert to indicate what type of problem has occurred. You can double-click the alert to see more detailedinformation, then use the knowledge base to find troubleshooting methods to help you solve the problem. TheCurrent Alerts screen displays the more recent alerts that have been logged for replication (see Figure 4.17 below).

Page 126: The Definitive Guide to Active Directory Troubleshooting

You can also view the replication-related alerts that have been stored in the Alert History file in DirectoryAnalyzer.To display these alerts, on the Current Alerts screen, choose Reports>Alert History. On the Report page, select oneof the report options to specify what alerts you want to include. Then select Preview to display the report on thescreen. You can print the report or export it to a file. Figure 4.18 illustrates an Alert History report.

Chapter 4 www.netpro.com116

Figure 4.17: The Current Alerts screen in DirectoryAnalyzer allows you to view the most recent alerts for the replication process.

Page 127: The Definitive Guide to Active Directory Troubleshooting

SummaryBefore you can accurately troubleshoot AD, you must be able to effectively monitor it for problems. This meansthat you must be able to monitor the directory that has been distributed across domain controllers on the network.You can do this by using the monitoring tools described in this chapter. These tools allow you to watch the directorycomponents individually and as they interact with each other. For example, you can monitor the domain controllers,the domain partition, the GC partition, the operations masters, and the replication process and topology. Monitoringthese components ensures the health of the directory as a system.

Chapter 4 www.netpro.com117

Figure 4.18: Using DirectoryAnalyzer, you can produce a report of replication-related alerts.

eBook Copyright NoticeThis site contains materials created, developed, or commissioned by Realtimepublishers.com, Inc. and is protected byinternational copyright and trademark laws. No material (including but not limited to the text, images, audio, and/orvideo) may be copied, reproduced, republished, uploaded, posted, transmitted, or distributed in any way, except that onecopy may be downloaded for your personal, non-commercial use on a single computer. In connection with such use, youmay not modify or obscure any copyright or other proprietary notice. If you have any questions about these terms, or ifyou would like information about licensing materials from Realtimepublishers.com, please contact us via e-mail [email protected]

Page 128: The Definitive Guide to Active Directory Troubleshooting

The Definitive Guide to Active Directory Troubleshooting

Chapter 5

Page 129: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com118

Troubleshooting Active DirectoryTroubleshooting Active Directory (AD) means analyzing and identifying problems that occur in your AD networkand subsequently repairing them. Troubleshooting a production AD environment can often be difficult because it’sdynamic and complex by nature, but there are techniques and tools available to make the job easier. In this chapter,you’ll learn how to apply these techniques and tools and develop an AD network-troubleshooting methodology.

The troubleshooting process primarily involves isolating and identifying a problem. Few problems are difficult tosolve once you know exactly what is going wrong and where. Troubleshooting, in general, is more an art born outof experience than an exact science. Your approach to solving a problem can depend largely on the specifics of yourdirectory, system, and network. This chapter outlines some common techniques and approaches that you can use tohelp troubleshoot and maintain your implementation of AD.

Following a Specific Troubleshooting MethodologyWhen you troubleshoot AD, I recommend that you follow a specific methodology of diagnosing and troubleshootingproblems in the system. This methodology is a set of steps you can follow to identify situations, diagnose problems,and repair AD components. The first step of this methodology is a set of questions that you can use to identifyparticular situations or problems.

• Is network communication working?• Does the name resolution work?• Are the domain controllers responding?• Are the operations masters working?• Is the replication topology working?

When a problem doesn’t exhibit the characteristics of a typical failure, and when monitoring tools fail to provideenough information to isolate the problem, the next step is to try to eliminate pieces of the system until you endup with a small, predictable failure. As I mentioned earlier, use the process of elimination to rule out as manytechnologies and dependencies as possible. Even if the problem seems overly complex at first, you can simplify itby eliminating all of the possibilities—one by one.

Troubleshooting Network ConnectivityYou can troubleshoot network connectivity in a number of ways. For example, you can:

• Test that the hardware you’re using has network connectivity• Test that Internet Protocol (IP) addresses are correct using the IPCONFIG utility• Test that Transmission Control Protocol/Internet Protocol (TCP/IP) connections are working using the

PING utility• Perform other troubleshooting tests using DirectoryAnalyzer.

Testing for Network ConnectivityThe first step toward identifying and diagnosing AD problems is to verify that each domain controller and userworkstation has network connectivity. At a minimum, you need to check that your domain controller’s hardwareis functioning correctly, including the computer’s local area network (LAN) adapters, drivers, cables, and networkhub. For example, if you look in the Network and Dial-up Connections screen under Control Panel and the LocalArea Connection icon is marked with a red X, the network cable isn’t connected.

Page 130: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com119

Figure 5.1 shows that the domain controller has a local area connection problem. Because the domain controller’scable isn’t connected to the network, there is a simple solution to the problem: reconnect the cable.

Testing the IP AddressesAnother method of checking network connectivity on the LAN is to make sure that the IP addresses are correct. Toperform an IP check, use the IP Configuration utility (IPCONFIG). IPCONFIG allows you to view and modify thedomain controller’s IP configuration details on the command line. It also checks that the default gateway is on thesame subnet as the local computer’s IP address. For Domain Name System (DNS) dynamic updates, you can useIPCONFIG to register the computer’s entries in the DNS service.

To view a computer’s TCP/IP configuration, type the following command in a Command Prompt window on thedomain controller or workstation:

IPCONFIG /ALL

The default display shows only the IP address, subnet mask, and default gateway for each adapter bound toTCP/IP. Figure 5.2 shows an unsuccessful TCP/IP configuration and network connection.

Figure 5.1: A red X on the Local Area Connection icon indicates that the network cable is disconnected from your domain controller.

Page 131: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com120

Listing 5.1 shows a well-connected LAN. Notice that the IP addresses are displayed with appropriate values.

Figure 5.2: An unsuccessful TCP/IP configuration and network connection, shown using IPCONFIG.

C:> ipconfig /allWindows 2000 IP Configuration Host Name .......................... : cx266988-SPrimary DNS Suffix ................ : company.comNode Type .......................... : Hybrid IP Routing Enabled .................. : NoWINS Proxy Enabled .................. : NoSearch List ........................ : company.com

Ethernet adapter Local Area Connection:Connection-specific DNS Suffix ..... : company.comDescription ........................ : Netelligent 10/100TX PCI EmbeddedUTP Coax ControllerPhysical Address ................... : 00-80-5F-A9-C0-74IP Address ......................... : 10.0.0.10Subnet Mask ........................ : 255.255.0.0Default Gateway .................... : 10.0.0.1

If you want to save the results of running IPCONFIG for further analysis, you can capture the results in a text file.At the command line, enter the following command:

IPCONFIG /ALL > <local_drive>:\<text_file.txt>

Listing 5.1: A well-connected LAN, shown using IPCONFIG.

There are many advanced features and switches available with IPCONFIG. To view the available switches,enter:

IPCONFIG /?

Page 132: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com121

If everything looks normal when you run IPCONFIG, go on to test the TCP/IP connection.

Testing the TCP/IP ConnectionYou can test the TCP/IP connection among connected servers and workstations using the PING utility. PING allowsyou to determine whether the LAN adapter and TCP/IP are working and whether you have network connectivity tothe default gateway or the Dynamic Host Configuration Protocol (DHCP) server. In this case, you can use thePING command to test TCP/IP connectivity among the domain controllers that support AD or on a workstationthat uses AD. You can start the PING command on one domain controller to test the connectivity to another.

When a domain controller fails to connect to the targeted computer, the PING utility returns a "Request timedout" or "Destination host unreachable" message. This message is repeated at least four times as PING retries theconnection. In addition, the utility shows statistics gathered during the test.

You can use the PING utility to perform a series of steps to troubleshoot connectivity problems among domaincontrollers. The first test is called the loop-back address test, which verifies whether TCP/IP is working on the localcomputer. To perform this test on the local computer, type the following command in the Command Prompt window.(Instead of using 127.0.0.1, you can use the keyword localhost.)

PING 127.0.0.1

If the PING command fails on the loop-back address test, check the TCP/IP configuration settings and restart thelocal domain controller.

After you verify that TCP/IP is configured properly and the PING loop-back address test succeeds, you need to testthe local TCP/IP address of the local domain controller. To do this, type the following command:

PING <local_TCP/IP_address>

If the PING test for the local address fails, restart the domain controller and check the routing tables using theROUTE PRINT command at a command prompt on the computer. The ROUTE PRINT command displays thecurrent IP address assigned to the local computer plus all of the active and persistent network routes. This commandallows you to view and troubleshoot the network configurations that exist at the time that the command is executed.After you’ve verified that the local address is working properly, use the PING command to check the communication

WNTIPCFG.EXE in the Windows 2000 Resource Kit

If you’re wondering why Windows 2000 (Win2K) doesn’t contain a graphical TCP/IP configuration utility similarto the WINIPCFG.EXE file provided with Windows 95/98/Me, you’re not alone. A graphical user interface(GUI)-based TCP/IP configuration utility (WNTIPCFG.EXE) is included in the Windows 2000 Professional andWindows 2000 Server resource kits, but by default, the ResKit Setup utility doesn’t install it. To install it,you need to manually extract the WNTIPCFG.EXE file from the NETMGMT.CAB file, included in the ResourceKit’s main installation folder, onto your hard disk. (A similar situation existed with NT 4.0. The Windows NTServer Resource Kit included WNTIPCFG.EXE but didn’t install it by default.) What’s more, the Windows2000 Resource Kit Supplement 1 doesn’t contain the WNTIPCFG.EXE file, so to obtain it, you need theoriginal Resource Kit.

Page 133: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com122

to the other domain controllers in the same location or subnet. For example, you can check connectivity to theother domain controllers on the same subnet as follows:

PING <domain_controller1_address>PING <domain_controller2_address>

PING <domain_controller3_address>

In the PING statements, the domain controller address is represented as the domain name (that is, COMPANY.COM)or the IP address of the domain controller (that is, 10.0.0.10). If communication among the domain controllers onthe local subnet fails, you need to check that each computer is operational and that the network hubs and switchesare working properly. If the domain controllers are separated by a wide area network (WAN) connection, you needto ping the default gateways that route the TCP/IP traffic among WAN locations.

Start by pinging the IP address of your default gateway. If the PING command fails for the gateway, you need toverify that the address for the default gateway is correct and that the gateway (router) is operational.Next, ping the IP address of the remote domain controllers on the remote subnet as follows:

PING <remote_domain_contoller1_address>PING <remote_domain_contoller2_address>

PING <remote_domain_contoller3_address>

In the PING statements, the remote domain controller address is represented as the domain name (that is,REMOTE.COMPANY.COM) or the IP address of the domain controller (that is, 20.0.0.20). If the PINGcommand fails, verify the address of each remote domain controller and check whether each remote domaincontroller is operational. In addition, check the availability of all of the gateways or routers between yourdomain controller and the remote one.

In addition to pinging the domain controllers, you need to ping the IP address of the DNS server. If this commandfails, verify that the DNS server is operational and the address is correct.

Performing Other Troubleshooting Tests Using DirectoryAnalyzerYou can perform several other tests of network connectivity in AD using DirectoryAnalyzer from NetProComputing. You can test the network connection, view the current status and name, query IP addresses, andperform server lookups. You can use DirectoryAnalyzer to perform the following troubleshooting tests:

• Domain Controller Connectivity Test• Domain Connectivity Test• Site Connectivity Test.

Each performs a different type of connectivity test among the domain controllers in specific AD domains and sites.

Domain Controller Connectivity TestThe Domain Controller Connectivity test allows you to test the connectivity between a selected domain controller inthe forest and one or more target domain controllers. This test is useful for testing communications among anydomain controllers in the forest. To perform this test, choose Troubleshoot>DC Connectivity. The Test DomainController Connectivity dialog box appears, where you can select the domain controllers involved in the test.

Page 134: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com123

First, select the source domain controller from the Source list. Next, select the destination domain controller(s) thatthe source will communicate with during the test by clicking the check box to the left of each domain controller inthe Destination list. Then click Start Test. Figure 5.3 shows the results of running the Domain ControllerConnectivity test.

After the test is completed, the results are displayed at the bottom of the dialog box.

DestinationShows the name of each destination domain controller you selected.

TestShows the type of test that was performed. The type of test varies according to the services that have been assigned to the domain controller.

TimeShows the amount of time (in milliseconds) it took to perform each test. If a test is performed in less than 10 milliseconds, it’s displayed as < 10 ms; otherwise, the actual time is displayed.

ResultShows whether a test was successful. If the test failed, this column displays a brief description of why.

Figure 5.3: Running the Domain Controller Connectivity test to troubleshoot the communication path among domaincontrollers in the forest.

Page 135: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com124

Domain Connectivity TestThe Domain Connectivity test allows you to test the connectivity of a domain controller in a selected domain againstdomain controllers in the destination domain(s). To perform this test, choose Troubleshoot>Domain Connectivity.The Test Domain Connectivity dialog box appears, where you can select the domains involved in the test.

First, select the source domain/domain controller from the Source list. Next, select the destination domain(s) thatthe source domain/domain controller will communicate with during the test by clicking the check box to the leftof each domain in the Destination list. Then click Start Test. Figure 5.4 shows the results of running the DomainConnectivity test.

After the test is completed, the results are displayed at the bottom of the dialog box.

DestinationShows the name of each destination domain/domain controller you selected.

TestShows the type of test that was performed. The type of test varies according to the services that have been assigned to the domain controller.

TimeShows the amount of time (in milliseconds) it took to perform each test. If a test is performed in less than 10 milliseconds, it’s displayed as < 10 ms; otherwise, the actual time is displayed.

ResultShows whether a test was successful. If the test failed, this column displays a brief description of why.

Figure 5.4: Running the Domain Connectivity test to troubleshoot the communication between the domain controller inthe source domain and the domain controllers in the destination domain.

Page 136: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com125

Site Connectivity TestThe Site Connectivity test allows you to test the connectivity of a domain controller in a selected site against domaincontrollers in the destination site. To perform this test, choose Troubleshoot>Site Connectivity. The Test SiteConnectivity dialog box appears, where you can select the site and domain controllers involved in the test.

First, select the source site/domain controller from the Source list. Next, select the destination site(s) that the sourcedomain/domain controller will communicate with during the test by clicking the check box to the left of each namein the Destination list. Then click Start Test. Figure 5.5 shows the results of running the Site Connectivity test.

After the test is completed, the results are displayed at the bottom of the dialog box.

DestinationShows the name of each destination site/domain controller you selected.

TestShows the type of test that was performed. The type of test varies according to the services that have been assigned to the domain controller.

TimeShows the amount of time (in milliseconds) it took to perform each test. If a test is performed in less than 10 milliseconds, it’s displayed as < 10 ms; otherwise, the actual time is displayed.

ResultShows whether a test was successful. If the test failed, this column displays a brief description of why.

Figure 5.5: Running the Site Connectivity test to troubleshoot the communication between a site/domain controller andthe domain controllers in the destination site.

Page 137: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com126

Troubleshooting Name ResolutionDNS is the de facto name-resolution system used to locate computers and domain controllers in AD. For example,a workstation or member server finds a domain controller by querying DNS. If you have problems connecting toAD and you’ve successfully tested network connectivity, a name-resolution problem may exist. For example, if youcannot find domain controllers or network resources when you perform queries, it might mean that DNS domainnames aren’t being resolved to IP addresses.

Understanding Name ResolutionThe first step in identifying and diagnosing AD name-resolution problems is to review how the Win2K-basedcomputer registers names and locates domain controllers. For example, whenever you start a Win2K domaincontroller, it registers two types of names:

• A DNS domain name with the DNS service• If the computer has Network Basic Input/Output System (NetBIOS) enabled, a NETBIOS name with

Windows Internet Name Service (WINS) or with another transport-specific service.

The DNS resource records (RRs) registered by the domain controllers in AD include multiple service (SRV)records, address (A) records, and CNAME (canonical name) records, all of which identify the domain controllers’location in a domain and forest. When the domain controller is started, the Netlogon service registers these records.It also sends DNS dynamic-update queries for the SRV records, A records, and CNAME records every hour toensure that the DNS server always has the proper records.

When you use AD-integrated zones, the DNS server stores all of the records in the zone in AD. To run AD-integratedzones, the DNS service must be running on the domain controller. It’s possible that a record is updated in AD buthasn’t replicated to all DNS servers loading the zone. This might cause consistency problems. By default, all DNSservers that load zones from AD poll the directory at set intervals (every five minutes, but you can change this) toupdate the directory’s representation of the zones.

Checking That DNS Records Are RegisteredIf DNS records aren’t registered on the DNS server, no other domain controller or workstation can locate thedomain controller. There are a few ways that you can check for this.

Page 138: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com127

Using Event ViewerIf DNS records aren’t registered in DNS—for example, if the DNS client has problems dynamically updating DNSrecords—errors are recorded in the System Log in Event Viewer. Figure 5.6 shows how the System Log tracks DNSerrors in Event Viewer.

Figure 5.6: Using Event Viewer to track DNS errors that occur on the selected domain controller.

If the domain controller is a DNS server, an additional log tracks all of the DNS basic events and errors for theDNS service on the server. For example, the DNS Server log monitors and tracks the starts and stops for theDNS server. It also logs critical events, such as when the server starts but cannot locate initializing data—for example,zones or boot information stored in the Win2K Registry or (in some cases) AD. Figure 5.7 shows how you canaccess the DNS Server log in Event Viewer.

Page 139: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com128

Figure 5.7: Using the DNS Server log in Event Viewer to track the errors for all DNS events that occur on a domain controllerthat supports a DNS server.

Using PINGAnother simple method for checking whether DNS records have been registered is to determine whether you canlook up the names and addresses of network resources using the PING utility. For example, you can check thenames using PING as follows:

PING COMPANY.COM

If this command works, the DNS server can be contacted using this basic network test.

Using NSLOOKUPNext, you need to verify that the DNS server is able to listen to and respond to basic client requests. You can dothis using NSLOOKUP, a standard command-line utility provided in most DNS-service implementations, includingWin2K. NSLOOKUP allows you to perform query testing of DNS servers and provides detailed responses as itsoutput. This information is useful when you troubleshoot name-resolution problems, verify that RRs are added orupdated correctly in a zone, and debug other server-related problems.

To test whether the DNS server can respond to DNS clients, use NSLOOKUP as follows:

NSLOOKUP

Once the NSLOOKUP utility loads, you can perform a test at its command prompt to check whether the hostname appears in DNS. Listing 5.2 shows entering a host name and the output you can receive:

Page 140: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com129

company.com

Server: ns1.company.comAddress: 250.45.87.13

Name: company.comAddress: 250.65.123.65

The output of this command means that DNS contains the A record and the server is responding with an answer:250.65.123.65. Next, verify whether this address is the actual IP address for your computer.

You can also use NSLOOKUP to perform DNS queries, examine the contents of zone files on the local and remoteDNS servers, and start and stop the DNS servers. If the record for the requested server isn’t found in DNS, youreceive the following message:

The computer or DNS domain name does not exist

Checking the Consistency and Properties of the DNS ServerYou can check the consistency and view the properties of DNS servers, zones, and RRs using another command-lineutility called DNSCMD. Windows 2000 Server provides DNSCMD as a command-line interface for managingDNS servers. You can use this tool to script batch files, help automate the management and updating of existingDNS server configurations, and set up and configure new DNS servers on your network. DNSCMD also allowsyou to manually modify DNS server properties, create zones and RRs, and force replication between a DNSserver’s physical memory and the DNS database and data files.

You can use DNSCMD for most tasks that you can perform from the DNS console, such as:

• Creating, deleting, and viewing zones and records• Resetting server and zone properties• Performing routine administrative operations, such as updating, reloading, and refreshing the zone• Writing the zone back to a file or to AD• Pausing and resuming the zone• Clearing the cache• Stopping and starting the DNS service• Viewing statistics.

You can install DNSCMD by copying it from the \Support\Tools folder located on the Windows 2000 CD-ROM.For help in using the command, enter the following at a command prompt:

DNSCMD /?

When the DNS Server Doesn’t Resolve Names CorrectlyWin2K includes a caching DNS-resolver service, which is enabled by default. For troubleshooting purposes, thisservice can be viewed, stopped, and started like any other Win2K service. The caching resolver reduces DNSnetwork traffic and speeds name resolution by providing a local cache for DNS queries.

Listing 5.2: A sample command and output received using NSLOOKUP.

Page 141: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com130

How the Caching DNS-Resolver Service WorksWhen a name is submitted to DNS, if the resolver is caching names, it first checks the cache. If the name is in thecache, the data is returned to the user. If the name isn’t in the cache, the resolver queries the other DNS servers thatare listed in the TCP/IP properties for each adapter. It does this in the following order:

1. The resolver sends the query to the first server on the preferred adapter’s list of DNS servers and waits one second for a response.

2. If the resolver doesn’t receive a response from the first server within one second, it sends the query to thefirst DNS servers on all adapters that are still under consideration and waits two seconds for a response.

3. If the resolver doesn’t receive a response from any server within two seconds, it sends the query to all DNS servers on all adapters that are still under consideration and waits another two seconds for a response.

4. If the resolver still doesn’t receive a response from any server, it sends the query to all DNS servers on alladapters that are still under consideration and waits four seconds for a response.

5. If it still doesn’t receive a response from any server, the resolver sends the query to all DNS servers on alladapters that are still under consideration and waits eight seconds for a response.

6. If the resolver receives a positive response, it stops querying for the name, adds the response to the cache,and returns the response to the client. If it doesn’t receive a response from any server by the end of the eight seconds, it responds with a time-out. Also, if it doesn’t receive a response from any server on aspecified adapter, it responds for the next 30 seconds to all queries destined for servers on that adapter with a time-out and doesn’t query those servers.

The resolver also keeps track of which servers answer queries more quickly, and it might move servers up or downon the search list based on how quickly they respond.

Using Other TechniquesA typical problem occurs when a DNS server doesn’t resolve names correctly and provides incorrect data forqueries. For example, if an administrator changed the IP address on a domain controller but DNS wasn’t properlyupdated, DNS would supply the incorrect IP address to clients.

When working with Windows 2000 Server and DNS entry changes, you may notice that the DNS server has staleRRs because they haven’t been updated recently. This means that if there have been previous lookups or name-resolutionactivity, the DNS server doesn’t see the changes to the RRs. (The server caches DNS information from previouslookups so that subsequent lookups are fast.) The typical method of fixing this problem is to restart the server.

You can also fix this problem using the IPCONFIG command. Entering the following command allows you toview the current list of DNS entries that the server has cached:

IPCONFIG /displayDNS

Entering the following command allows you to refresh all DHCP leases and re-register DNS names. (Wait fiveminutes for the DNS entries in the cache to be reset and updated with the RRs in the server’s database.)

IPCONFIG /registerDNS

Page 142: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com131

You can also use the IPCONFIG command to dump all of the DNS cache entries.

IPCONFIG /flushDNS

It’s worth noting that the DNS server should eventually refresh the cache because each entry has a Time-To-Live(TTL) associated with it. TTL indicates a length of time used by other DNS servers to determine how long tocache information for a record before discarding it. For example, most RRs created by the DNS Server serviceinherit the minimum (default) TTL of 1 hour from the start-of-authority (SOA) RR; this prevents overly longcaching by other DNS servers. TTL is automatically decremented and eventually expires and disappears or isflushed from the cache.

For an individual RR, you can specify a record-specific TTL that overrides the minimum (default) TTL inheritedfrom the SOA RR. You can also use TTL values of zero (0) for RRs that contain volatile data not to be cached forlater use after the current DNS query is completed.

Another problem that may occur is that the DNS server doesn’t resolve names for computers or services outsideyour immediate network. For example, the DNS server may not resolve names for computers located on an externalnetwork or the Internet. If a DNS server fails to resolve a name for which it’s not authoritative, the cause is usuallya failed recursive query. Recursion is used in most DNS configurations to resolve names that aren’t located in theconfigured DNS domain.

For recursion to work correctly, all DNS servers used in the path of the recursive query must be able to respond toand forward correct data. If the DNS server fails a recursive query, you need to review the server’s configuration.By default, all Win2K DNS servers have recursion enabled. You can disable recursion using the DNS console to modifyadvanced server options. In addition, recursion might be disabled if the DNS server is configured to use forwarders.

Troubleshooting the Domain ControllersYou have a number of options for troubleshooting domain controllers. Before I discuss them, though, it’s importantto review the AD database and its associated files.

Understanding the Active Directory Database and Its Associated FilesAD is stored on each domain controller in a local database. The database exists as a domain database and, marriedwith the directory services, performs authentication services to users and applications. The domain controllersreplicate their data with each other to ensure that copies of the domain database on other domain controllers arecurrent and accurate.

The AD database is implemented on an indexed sequential access method (ISAM) table manager that has beenreferred to as "Jet." The table manager is called the Extensible Storage Engine (ESE). The ESE database is managedon each domain controller by the ESE.DLL file. The database is a discrete transaction system that uses log files toensure integrity; it uses support rollback to ensure that the transactions are committed to the database.The following files are associated with AD:

NTDS.DITThe main database file; it grows as the database fills with objects and attributes. On the other hand, the log files have a fixed size of 10 megabytes (MB). Any changes made to the database are also made to the currentlog file and to the DIT file in the cache. Eventually the cache is flushed. If a computer failure occurs before the cache is flushed, ESE uses the log file to complete the update to the DIT file.

Page 143: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com132

By default, the AD database is stored in <DRIVE>\WINNT\NTDS\NTDS.DIT. The log files for the directory database are stored in the same directory by default. Their purpose is to track the changes in the directory database, and they can grow to be quite large. I recommend that you give all the room you can to the log files. For example, you can place the log files on different disk drives than the database file to reducedisk contention on a single drive.

EDB.LOG and EDBXXXXX.LOGEDB.LOG is the current log file for AD. When a change is made to the database, it’s written to this file. When EDB.LOG becomes full of database transactions, it’s renamed to EDBXXXXX.LOG, where XXXXXstarts at 00001 and continues to increment using hexadecimal notation. AD uses circular logging, which constantly deletes old log files. If you view the directory files at any time, you’ll notice the EDB.LOG file and at least one or more EDBXXXXX.LOG files.

EDB.CHKStores the database checkpoint, which identifies the point where the database engine needs to replay the logs. This file is typically used during recovery and initialization.

RES1.LOG and RES2.LOGPlaceholders designed to reserve the last 20MB of disk space on the disk drive. Saving disk space gives the log files sufficient room to shut down gracefully if other disk space is consumed.

To manage the database, Win2K provides a garbage-collection process designed to free space in the AD database.This process runs on every domain controller in the enterprise with a default lifetime interval of 12 hours. Thegarbage-collection process first removes "tombstones" from the database. Tombstones are remains of objects thathave been deleted. (When an object is deleted, it’s not actually removed from the AD database. Instead, it’s markedfor deletion at a later date. This information is then replicated to other domain controllers. When the time expiresfor the object, the object is deleted.) Next, the garbage collection-process deletes any unnecessary log files. Finally, itlaunches a defragmentation thread to claim additional free space.

Above the directory database is a database layer that provides an object view of the database information by applyingthe schema to the database records. The database layer isolates the upper logical layers of the directory from theunderlying database system. All access to the database occurs through this layer instead of allowing direct access tothe database files. The database layer is responsible for creating, retrieving, and deleting the individual databaserecords or objects and associated attributes and values.

In addition to the database layer, AD provides a directory service agent (DSA), an internal process in Win2K thatmanages the interaction with the database layer for the directory. AD provides access using the following protocols:

• Lightweight Directory Access Protocol (LDAP) clients connect to the DSA using the LDAP protocol• Messaging Application Programming Interface (MAPI) clients connect to the directory through the DSA

using the MAPI remote procedure call (RPC) interface• Windows clients that use Windows NT 4.0 or earlier connect to the DSA using the Security Account

Manager (SAM) interface• AD domain controllers connect to each other during replication using the DSA and a proprietary RPC

implementation.

Page 144: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com133

Comparing Directory InformationWhen you want to compare directory information on domain controllers or directory partitions, you can use theDSASTAT utility. DSASTAT detects and examines the differences among a user-defined scope of objects on twodifferent domain controllers. It retrieves capacity statistics such as megabytes per server, objects per server, andmegabytes per object class. For example, you can use DSASTAT to compare all users in the SALES OrganizationalUnit (OU) in the COMPANY.COM domain with those in another directory partition by specifying the following:

DSASTAT -S:Company1;Company2 -B:OU=SALES,DC=COMPANY,DC=COM -GCATTRS:ALL -SORT:TRUE -T:FALSE -P:16 -FILTER:

"(&(OBJECTCLASS=USER)(!OBJECTCLASS=COMPUTER))"

In this example, you can determine whether both domain controllers agree on the contents of theOU=SALES,DC=COMPANY,DC=COM subtree. DSASTAT detects objects in one domain and not the other(for example, if a creation or deletion hasn’t replicated) as well as differences in the values of objects that exist inboth. This example specifies a base search path at a subtree of the domain. In this case, the OU name is SALES.The filter specifies that the comparison is concerned only with user objects, not computer objects. Because computerobjects are derived from user objects in the class hierarchy, a search filter specifying OBJECTCLASS = USERreturns both user and computer objects.

DSASTAT also allows you to specify the target domain controllers and additional operational parameters using thecommand line or an initialization file. DSASTAT determines whether domain controllers in a domain have aconsistent and accurate image of their own domain.

DSASTAT also compares the attributes of replicated objects. You can use it to compare two directory trees acrossreplicas in the same domain or, in the case of a Global Catalog (GC), across different domains. You can also useit to monitor replication status at a much higher level than monitoring detailed transactions. In the case of GCs,DSASTAT checks whether the GC server has an image that is consistent with the domain controllers in otherdomains. DSASTAT complements the other replication-monitoring tools, REPADMIN and REPLMON, byensuring that domain controllers are up to date with one another.

Analyzing the State of the Domain ControllersThe first step in troubleshooting and repairing problems with AD on the domain controller is to verify that thedirectory portion is running without errors. The Domain Controller Diagnostic (DCDIAG) utility allows you toanalyze the current state of the domain controllers in a domain or forest. It automatically performs the analysis andreports any problems with a domain controller. DCDIAG requires a separate installation of the Support Tools fromthe Windows 2000 Server or Advanced Server CD-ROM; by default, it’s installed in \program files\support tools.

DCDIAG is intended to perform a fully automatic analysis with little user intervention. This means that youusually don’t need to provide too many parameters to it on the command line. DCDIAG doesn’t work whenrun against a Win2K workstation or server, and it’s limited to working only with domain controllers.

Page 145: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com134

DCDIAG consists of a set of tests that you can use to verify and report on the functional components of AD on thecomputer. You can use this tool on a single domain controller, a group of domain controllers holding a domain partition,or across a site. When using DCDIAG, you can collect either a minimal amount of information (confirmation ofsuccessful tests) or data for every test you execute. Unless you’re diagnosing a specific problem on only one domaincontroller, I recommend that you collect only the severe errors for each one.

DCDIAG allows you to run the following tests to diagnose the status of a domain controller:

Connectivity TestVerifies that DNS names for the domain controller are registered. It also verifies that the domain controller can be reached using TCP/IP and the domain controller’s IP address. DCDIAG checks the connectivity to the domain controller using LDAP and checks that communications can occur using an RPC.

Replication TestChecks the replication consistency for each of the target domain controllers. For example, this test checks whether replication is disabled and whether replication is taking too long. If so, the utility reports these replication errors and generates errors when there are problems with incoming replica links.

Topology Integrity TestVerifies that all domain controllers holding a specific partition are connected by the replication topology.

Directory Partition Head Permissions TestChecks the security descriptors for proper permissions on the directory partition heads, such as the schema,domain, and configuration directory partitions.

Locator Functionality TestVerifies that the appropriate SRV RRs are published in DNS. This test also verifies that the domaincontroller can recognize and communicate with operations masters. For example, DCDIAG checks whetherthe locator can find a primary domain controller (PDC) and GC server.

Inter-site Health TestIdentifies and ensures the consistency of domain controllers among sites. To do this, DCDIAG performsseveral tests, one of which identifies the inter-site topology generator and identifies the bridgeheads for eachsite. This test determines whether a bridgehead server is functioning; if not, the utility identifies and locatesadditional backup bridgeheads. In addition, this test identifies when sites aren’t communicating with other sites on the network.

Trust Verification TestChecks explicit trust relationships—that is, trusts between two domain controllers in the forest. DCDIAG cannot check transitive trusts (Kerberos V5 trust relationships). To check transitive trusts, you can use the Windows 2000 Resource Kit’s NETDOM utility (not described in this chapter).

For more information on the Windows 2000 Resource Kit’s NETDOM utility, refer to the Resource Kitdocumentation or The Definitive Guide to Windows 2000 and Exchange 2000 Migration(Realtimepublishers), a link to which can be found at http://www.realtimepublishers.com.

Page 146: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com135

Diagnose Replication Latencies TestAnalyzes incoming replications and watches for delays or preemption of a higher-priority job. If thereplication process is delayed or preempted, latencies have occurred that slow down the process. This problem typically occurs because a higher-priority task hasn’t relinquished the computer’s processor or because a large number of replication requests or tasks is pending. New replication tasks are delayedbecause the domain controller is overloaded with replication requests.

Replication of Trust Objects TestChecks whether the computer account object has been replicated to all additional domain controllers inthe domain. It also checks whether the DSA object has been replicated to all replicas of the configurationdirectory partition.

File Replication Service (FRS) TestVerifies that FRS has started successfully on all domain controllers. If it hasn’t, this test delays theNETLOGON service from advertising that domain controller.

Critical Services Check TestVerifies that these key services are running: FRS, Inter-site Messaging Service, Kerberos Key Distribution Center Service, Server Service, Workstation Service, Remote Procedure Call Locator Service, Windows Time Service, Distributed Link Tracking Client Service, Distributed Link Tracking Server Service, and NETLOGON service. You can also use DCDIAG with the /repairmachineaccount command-line switch, which re-creates the domain controller’s machine account if it’s been accidentally deleted.

Using NTDSUTILThe Directory Services Management utility (NTDSUTIL.EXE) is a command-line utility included in Win2K thatyou can use to troubleshoot and repair AD. Although Microsoft designed NTDSUTIL to be used interactively viaa command-prompt session (launched simply by typing NTDSUTIL at any command prompt), you can also runit using scripting and automation. NTDSUTIL allows you to troubleshoot and maintain various internal componentsof AD. For example, you can manage the directory store or database and clean up orphaned data objects that wereimproperly removed.

You can also maintain the directory service database, prepare for new domain creations, manage the control of theFlexible Single Master Operations (FSMOs), purge metadata left behind by abandoned domain controllers (thoseremoved from the forest without being uninstalled), and clean up objects and attributes of decommissioned ordemoted servers. At each NTDSUTIL menu, you can type help for more information about the available options.(See Figure 5.8.)

Page 147: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com136

Figure 5.8: Viewing a list of available commands in the utility and a brief description of each.

Locating the Directory Database FilesBefore you use the NTDSUTIL utility to carry out troubleshooting and integrity checking on the AD database,you can use its Info command to determine the location and size of the directory database files. The Info command:

• Reports the free space for all disks installed on the domain controller• Reads the Registry keys and associated location of the AD database files• Reports the size of each of the database files, log files, and other associated files.

Before you perform this check, you have to either run NTDSUTIL after having booted the domain controller viathe special ‘Directory Service Restore Mode’ mode Safe Boot option or set the environment variableSAFEBOOT_OPTION to a value of DSREPAIR under a normal boot of Windows 2000 (e.g. via the commandSET SAFEBOOT_OPTION=DSREPAIR).To execute the Info command:

1. Choose Start>Programs>Accessories>Command Prompt.

2. In the Command Prompt window, type NTDSUTIL, then press Enter.

3. At the ntdsutil: prompt, enter the word files. The utility responds by displaying a ‘file maintenance’ prompt.The following commands have been entered and displayed to this point:

C:\>SET SAFEBOOT_OPTION=DSREPAIRC:\>NTDSUTILntdsutil: files

file maintenance:

4. At the File Maintenance prompt, enter the word info to display the location and sizes of AD database files, log files, and other associated files.

Page 148: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com137

Figure 5.9: Using the Info command in NTDSUTIL to display the location and size of AD database files.

Figure 5.9 shows the output of this command on a domain controller.

Checking for Low-Level Database Corruption

One of the first items you need to check when troubleshooting a domain controller in AD is that the underlyingdatabase is functioning properly. To do this, you can use NTDSUTIL’s Integrity option to detect any low-leveldatabase corruption of the directory files. The Integrity option checks that the headers for the database itself arecorrect and that all of the internal database tables are functioning and consistent with each other.

Before you perform a low-level database-integrity check, you need to start the domain controller in DirectoryService Restore mode. To do this:

1. Restart the domain controller. When you’re prompted, press F8 to display the Windows 2000Advanced Option menu.

2. Select Directory Service Restore Mode and press Enter.

3. Log on using the Administrator account and password that you assigned during the DCPROMO process.

Using NTDSUTIL, you can relocate or move AD database files from one location to another on the disk ormove the database files from one disk drive to another in the same domain controller. You can also movejust the log files from one disk to another to free up space for the data files. (See "Moving the ActiveDirectory Database or Log Files" later in this chapter.)

Page 149: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com138

To run the NTDSUTIL Integrity option:

1. Choose Start>Programs>Accessories>Command Prompt.

2. In the Command Prompt window, type NTDSUTIL, then press Enter.

3. At the ntdsutil: prompt, enter the word files. The utility responds by showing you the File Maintenance category.

The commands to this point appear in the Command Prompt window as follows:

I:>NTDSUTILntdsutil: files

file maintenance:

4. At the File Maintenance prompt, enter the word integrity to start the low-level database check on the domain controller. (The Integrity command reads every byte of the directory data file and displays the percentage of completion as a graph. Depending on the size of your database and the type of hardware you’re using for the domain controller, this process can take a considerable amount of time.)

Figure 5.10 shows the results of examining the low-level database structures in AD.

Figure 5.10: Using the Integrity option in NTDSUTIL to examine the AD database on a domain controller.

Page 150: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com139

Checking for Inconsistencies in the Database ContentsIn addition to using NTDSUTIL to verify that the AD database is functioning properly, you can use it to help youcheck the consistency of the contents of the AD database. The option in NTDSUTIL that performs a contentscheck is the Semantic Checker. The Semantic Checker option differs from the Integrity option in that it addressesthe contents (objects and attributes) of the directory database, not just its low-level structures.

When you run the Semantic Checker, it performs the following checks:

Reference Count CheckCounts the number of references in the database tables and matches the results with the values that are stored in the data file. This operation also ensures that each object has a globally unique identifier (GUID) and distinguished name (DN). For a previously deleted object, this operation ensures that the object has a deleted time and date but doesn’t have a GUID or DN.

Deleted Object CheckEnsures that the object has a time and date as well as a special relative distinguished name (RDN), given when the object was originally deleted.

Ancestor CheckEnsures that the DN tag is equal to the ancestor list of the parent. This could also be stated as a check that the distinguished name of the object minus its RDN is equal to its parent’s DN.

Security Descriptor CheckEnsures that there is a valid descriptor and that the discretionary access control list (DACL) isn’t empty.

Replication CheckVerifies that there is an up-to-dateness vector in the directory partition and checks to see that every object has metadata.

Like the Integrity option described above, you can run the Semantic Checker option only when the domaincontroller is in Directory Service Restore mode. To run in this mode:

1. Restart the domain controller. When you’re prompted, press F8 to display the Windows 2000Advanced Option menu.

2. Select Directory Service Restore Mode and press Enter.

3. Log on using the administrator account and password that you assigned during the DCPROMO process.

To troubleshoot and repair the AD database, you can use the Integrity option only while the domain controlleris in Directory Service Restore mode.

Page 151: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com140

To run the Semantic Checker option:

1. Choose Start>Programs>Accessories>Command Prompt.

2. In the Command Prompt window, type NTDSUTIL, then press Enter.

3. At the ntdsutil: prompt, type semantic database analysis, then press Enter.

4. Type verbose on. This displays the Semantic Checker.

5. To start the Semantic Checker without having it repair any errors, type go. To start it and have it repair any errors that it encounters in the database, enter go fixup.The commands to this point appear in the Command Prompt window as follows:

I:>NTDSUTILntdsutil: semantic database analysissemantic checker: verbose onVerbose mode enabled.semantic checker: go

Figure 5.11 shows the results of using the NTDSUTIL Semantic Checker.

Figure 5.11: Using the NTDSUTIL Semantic Checker option to check the consistency of the contents of the directory database.

Cleaning Up the MetadataThe NTDSUTIL program allows you to clean up the metadata that is left behind after a domain controller isdemoted. The utility that you use to demote a domain controller is the DCPROMO utility (DCPROMO.EXE).This utility is used to promote a server to a domain controller and demote a domain controller to a member server.As part of the demotion process, DCPROMO removes the configuration data for the domain controller from AD.This data takes the form of an NTDS Settings object, which exists as a child to the server object in the ActiveDirectory Sites and Services Manager and is located in AD as the following object:

CN=NTDS Settings,CN=<server_name>,CN=Servers,CN=<site_name>,CN=Sites,

CN=Configuration,DC=<domain>...

Page 152: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com141

The attributes of the NTDS Settings object contain values about the domain controller’s replication partners,naming contexts, whether the domain controller is a GC server, and the default query policy. The NTDS Settingsobject is also a container that may have child objects that represent the replication partners. This data is requiredfor the domain controller to synchronize quickly but is retired upon demotion. If the NTDS Settings object isn’tproperly removed when the domain controller is demoted, you can use the NTDSUTIL utility to manually removethe NTDS Settings object.

To clean up the metadata:

1. Choose Start>Programs>Accessories>Command Prompt.

2. At the command prompt, type NTDSUTIL, then press Enter.

3. At the ntdsutil prompt, type metadata cleanup, then press Enter. Based on the options returned to the screen, you can use additional configuration parameters to ensure that the removal occurs correctly.

4. Before you clean up the metadata, you must select the server on which you want to make the changes. To connect to a target server, type connections, then press Enter.

5. If the user who is currently logged on to the computer running NTDSUTIL doesn’t have administrativepermissions on the target server, alternate credentials need to be supplied before making the connection.To supply alternate credentials, type the following command, then press Enter:

set creds <domain_name user_name password>

6. Type connect to server <server_name>, then press Enter. You should receive confirmation that theconnection has been successfully established. If an error occurs, verify that the domain controller you specified is available and that the credentials you supplied have administrative permissions on the server.

7. When a connection has been established and you’ve provided the right credentials, type quit, then press Enter, to exit the Connections menu in NTDSUTIL.

8. When the Metadata Cleanup menu is displayed, type select operation target and press Enter.

9. Type list domains, then press Enter. A list of domains in the forest is displayed, each with an associated number. To select the appropriate domain, type select domain <number>and press Enter (where <number> is the number associated with the domain of which the domain controller you’re removing is a member). The domain you select determines whether the server being removed is the last domain controller of that domain.

10. Type list sites, then press Enter. A list of sites, each with an associated number, is displayed.

Before you manually remove the NTDS Settings object for any server, check that replication has occurredafter the domain controller has been demoted. Using the NTDSUTIL utility improperly can result in partialor complete loss of AD functionality. (For a description of how to check whether replication has occurred,see Chapter 4.)

Page 153: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com142

11. Type select site <number> and press Enter (where <number> is the number associated with the site of which the server you’re removing is a member). You should receive a confirmation, listing the site and domain you chose.

12. Type list servers in site and press Enter. A list of servers in the site, each with an associated number, is displayed.

13. Type select server <number> and press Enter (where <number> is the number associated with the serveryou want to remove). You receive a confirmation, listing the selected server, its DNS host name, and the location of the server’s computer account that you want to remove.

14. After you’ve selected the proper domain and server, type quit to exit the current NTDSUTIL submenu. When the Metadata Cleanup menu is displayed, type remove selected server and press Enter. You should receive confirmation that the server was removed successfully. If the NTDS Settings object has already been removed, you may receive the following error message:

Error 8419 (0x20E3)The DSA object couldn’t be found

15. Type quit at each menu to quit the NTDSUTIL utility. You should receive confirmation that theconnection disconnected successfully.

Moving the Active Directory Database or Log FilesThere are several common problems that occur with AD that all stem from the same source: low disk space. Theseproblems may surface as any of a number of errors messages in the Win2K Event Log. The most common of theseerrors is described below, along with their associated symptoms and solutions.

• The following error message may occur when you start AD on a domain controller:

Lsass.exe - System Error

Directory Services could not start because of the following error: There is not enough space on the disk.Error Status: 0xc000007f. Please click OK to shutdown this system and reboot into Directory Service Restore Mode, check the event logs for more detailed information.

When this error occurs, the following events are recorded in the Event Log for the directory service on the domain controller and can be viewed using Event Viewer:

Event ID: 1393Attempts to update the Directory Service database are failing with error 112.Since Windows will be unable to log on users while this condition persists,the Netlogon service is being paused. Check to make sure that adequate free disk space is available on the drives where the directory database and log files reside.

Event ID: 428NTDS (272) The database engine is rejecting update operations due to low free disk space on the log disk.

Page 154: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com143

• The following warning message is recorded in the System Log of the domain controller and can be viewedusing Event Viewer:

Event ID 2013:The D: disk is nearing Capacity. You may need to delete some files.

If the disk drive runs out of disk space, AD won’t start up. Win2K attempts to avoid this situation, but it can occurif you ignore warnings about low disk space in the System Log or if you run large scripts against AD for massdirectory imports.To resolve the problem of having no disk space, you can either make space available on the same disk drive ormove AD to a separate drive. The first method requires you to simply reduce the number of files or folders on thesame disk drive as the directory database.

If you want to move the AD database to another drive on the domain controller, you can use the NTDSUTIL utilityto move either the database file or the database log files. This method is ideal when you cannot move data toanother drive to free up space. If all drives are at capacity, you may need to install an additional hard disk in thedomain controller.

Before you move the directory database file or log files, you need to start the domain controller in DirectoryService Restore mode. To do this:

1. Restart the domain controller. When you’re prompted, press F8 to display the Windows 2000 AdvancedOption menu.

2. Select Directory Service Restore Mode and press Enter.

3. Log on using the administrator account and password that you assigned during the DCPROMO process.

To move the directory database file or log files:

1. Locate the drive containing the directory and log files. The directory database (NTDS.DIT) and log filesare located in the NTDS folder on the root drive by default. (However, the administrator may have changed their locations during the DCPROMO process.)

2. Choose Start>Programs>Accessories>Command Prompt.

3. In the Command Prompt window, type NTDSUTIL, then press Enter.

4. At the ntdsutil: prompt, enter the word files. The utility displays the File Maintenance category. The commands to this point should appear as follows:

I:>NTDSUTILntdsutil: files

file maintenance:

5. At the File Maintenance prompt, enter the word info to display the location of the AD database files, logfiles, and other associated files. Note the location of the database and log files.

Page 155: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com144

6. To move the database files to a target disk drive, type the following command at the ntdsutil: prompt:

MOVE DB TO %s (where %s is the target folder on another drive)

7. To move the log files to a target disk drive, type the following command at the ntdsutil: prompt. (The target directory where you move the database file or log files is specified by the %s parameter. The Move command moves the files and updates the Registry keys on the domain controller so that AD restarts using the new location.)

MOVE LOGS TO %s (where %s is the target folder on another drive)

8. To quit NTDSUTIL, type quit twice to return to the Win2K command prompt, then restart the domain controller normally.

Repairing the Active Directory DatabaseYou can use the NTDSUTIL Repair feature to repair the AD database file. However, you should use it only as alast resort for recovering the database—if a valid backup is available, always use it first to restore the data. The reasonis that repairing the directory database doesn’t always work correctly. For example, if a database file is corrupt, usingthe NTDSUTIL Repair feature may not restore all objects and attributes. In fact, in some cases, there is a risk thatusing the Repair feature will cause further data to be lost.

To repair the AD database file:

1. Choose Start>Programs>Accessories>Command Prompt.

2. In the Command Prompt window, type NTDSUTIL, then press Enter.

3. At the ntdsutil: prompt, enter the word files. The utility displays the File Maintenance category.

4. At the File Maintenance prompt, enter the word repair.The commands to this point should appear as follows:

I:>NTDSUTILntdsutil: files

file maintenance: repair

5. As soon as the repair operation has completed, run the NTDSUTIL Semantic Checker on the database. (For instructions, see "Cleaning Up the Metadata" earlier in this chapter.)

I highly recommend that you completely back up AD on the domain controller before you execute the Movecommand. In addition, I recommend that you back up AD after you move the directory database file andlog files; restoring the directory database will then retain the new file location.

Page 156: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com145

Figure 5.12 shows the results of using the NTDSUTIL Repair option.

Figure 5.12: Using NTDSUTIL as a last resort to repair the directory database files.

Page 157: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com146

Troubleshooting Secure Channels and Trust RelationshipsWhen a Win2K system joins a domain, a computer account is created. Whenever the system starts after that, ituses the password for that account to create a secure channel with the domain controller for its domain. Requestssent on the secure channel are authenticated, and sensitive information (such as passwords) is encrypted, but thechannel isn’t integrity-checked, and not all information is encrypted.

There are many reasons why a domain controller cannot communicate on a secure channel. For example, the useror domain controller may not have the appropriate access permissions or trust relationships. You can test the statusof secure channels and trust-relationship links using the Windows 2000 Resource Kit’s NLTEST command-line utility.

ADcheck: A Free AD and Windows 2000 Network Diagnostic Tool

Although Win2K and the Windows 2000 Resource Kit provide some basic tools for performing troubleshootingtasks, they aren’t especially easy to use. NetIQ Corporation provides an excellent—and free—utility forperforming a host of AD diagnostic and troubleshooting tasks. ADcheck provides five essential categoriesof Win2K diagnostics:

1. Test Domain ControllerChecks the availability of the domain controller, validates DNS records (for example, SRV RRs), and binds to the domain controller to verify AD status

2. List Domain ControllersLists each domain controller along with its name, availability, Active Directory Service Interfaces (ADSI) scripting location, and site location

3. List Operations MastersLists FSMO role holders, compares them to an internal best practices list, and recommends changes when necessary

4. Test ReplicationChecks domain replication topology and displays diagnostic information about replication partners

5. Show Domain Controller StatusProvides summaries of the status of domain controllers, including replication errors and partners,AD site analysis, and charts that show recommended changes to the placement of domaincontrollers.

ADcheck is also capable of generating some very detailed reports, each of which shows potential causesfor problems as well as the problems themselves. For some reports, it also compares the current configurationto an internal best practices guideline and may recommend changes. Given that it’s completely free, thistool is something that no Win2K network administrator should be without. You can download ADcheckfrom http://www.netiq.com/adcheck/download.asp.

Page 158: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com147

To validate access to resources in a trusting domain, the trusting domain controller establishes a secure channelwith a domain controller in the trusted domain. Pass-through authentication then occurs over this secure channel.However, in WAN environments, the trusted domain’s domain controllers may be dispersed over a wide variety offast and slow links. If a fast link is unavailable when the trusting domain controller wants to establish a securechannel, the secure channel may be established with a domain controller over a slow link. Even when the fast linkis reestablished, pass-through authentication may occur over the slow link to the trusted domain’s domain controller.

The mechanism for establishing a secure channel is very similar to the normal user-logon process. That is, the trustingdomain controllers send out logon requests to all known domain controllers in the trusted domain. The trustingdomain controllers then set up a secure channel with the first trusted domain controller that responds to this request.

Normally, this method is preferred because the first domain controller to respond to a logon request is typically thecontroller that is located across the fastest communication link. However, if that link is down or the "fast" domaincontroller is unavailable, a domain controller over a slower link may respond first, and all pass-through authenticationsoccur over the slow link.

There is a built-in mechanism in Windows 2000 that tracks how long authentication takes over the existing securechannel. If pass-through authentication takes longer than 45 seconds, that fact is noted. If two such authenticationsexceed that limit, a rediscovery process begins, the current secure channel is broken, and the trusting domain’s PDConce again sends out logon requests to all known trusted domain controllers. However, because this mechanismtracks only those communications that take longer than 45 seconds, users may see a 40-second delay every timethey attempt to use a resource without a secure-channel reset taking place.

You can run the NLTEST utility on the trusting domain controller to break and re-initialize a secure channel (forexample, when the secure-channel password was last changed) and obtain information about an existing trustrelationship. You can also use NLTEST to restart the discovery process for a new trusted domain controller. Thesyntax of NLTEST is:

NLTEST /sc_query:<account_domain>

Where <account_domain> is the name of the trusted domain. This command returns the name of the trusteddomain controller with which the trusting domain controller has a secure channel. If that domain controller isunacceptable, use the following syntax:

NLTEST /sc_reset:<account_domain>

Troubleshooting the Operations MastersThe operations masters in AD perform single-master operations for the forest and domains and are officially calledFlexible Single Master Operations, or FSMOs. A number of operations in the directory have single-master opera-tions—operations such as updating the schema, creating new domains in a forest, issuing new blocks of relativeIDs (RIDs), and supporting domains and clients that are running Windows NT 4.0 and earlier.

Page 159: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com148

The forest has two operations masters that manage certain forest-wide single-operation activities, and each domainhas three operations masters that manage certain domain-wide activities. For example, a forest with two domainswould have eight operations masters: two for the forest and three domain-specific operations master roles in eachdomain. The five FSMO roles are:

• Schema master—Forest-wide and one per forest• Domain Naming master—Forest-wide and one per forest• Relative ID master—Domain-specific and one for each domain• Primary Domain Controller emulator—Domain-specific and one for each domain• Infrastructure master—Domain-specific and one for each domain.

Because the operations masters are assigned to specific domain controllers in the forest and domains and are criticalto the operation of AD, your first step to troubleshoot each operations master is to use the domain-controllertroubleshooting techniques described in "Troubleshooting the Domain Controllers" earlier in this chapter. Onceyou’re assured that the domain controller itself is operating properly, you can turn your attention to the operations masters.

When Operations Masters FailIf a domain controller holding a FSMO (operations role) master fails, major network problems are almost guaranteed toensue. A list of the various operations master roles, their functions, and the effects of losing them are listed below:

Schema MasterIf the domain controller holding the forest-wide schema master role fails, you or your directory administrators won’tbe able to modify or extend the AD schema. Schema modifications typically occur when you install directory-enabledapplications such as management utilities that rely on the directory for information. These applications try to modifyor extend the current schema with new object classes, objects, and attributes. If the applications being installed cannotcommunicate with the domain controller that has been designated as the schema master, installation will fail.

The schema master solely controls the management of the directory schema and propagates updates to the schemato the other domain controllers as modifications occur. Because only directory administrators are allowed to makechanges, the schema operations master isn’t visible to directory users and doesn’t affect them.

Domain Naming MasterIf the domain naming master role holder in a forest fails, you lose the functionality of adding and removingdomains in the forest. When a new domain is created or deleted from the forest structure, the domain controllerthat has been designated as the domain naming master is contacted and verifies that the change operation can becompleted.

The domain naming master is the only domain controller that controls the creation and deletion of domains, andit propagates the changes to the other domain controllers as necessary. Because only directory administrators areallowed to make structural domain changes to the forest, the domain naming operations master isn’t visible todirectory users and doesn’t affect them.

Page 160: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com149

Relative ID (RID) MasterIf the domain controller that stores the relative ID (RID) master role fails or stops communicating, domain controllersin the same domain cannot obtain the RIDs they need. (RIDs are unique security IDs.) Domain controllers useRIDs when they create users, groups, computers, printers, and other objects in the domain; each object is assigneda RID. The RID master role allocates blocks of RIDs to other domain controllers in its domain. As I mentioned atthe beginning of this section, there is only one RID master role per domain.

If a domain controller has remaining (unassigned) RIDs in its allocated block, the RID master role doesn’tneed to be available when new object accounts are created.

Infrastructure Master

If the domain controller that stores the infrastructure master role fails, a portion of AD won’t function properly. Theinfrastructure master role controls and manages the updates to all cross-domain references, such as group referencesand security identifier (SID) entries in access control lists (ACLs). For example, when you add, delete, or rename auser who is a member of a group, the infrastructure master controls the reference updates. There is always only oneinfrastructure master role in each domain in a forest.

Because only one domain controller is assigned to perform this role, it’s important that it doesn’t fail. However, ifit does, it’s not visible to network users. In fact, it’s visible to only directory administrators when they’ve recentlymoved or renamed a large number of object accounts. In addition, having one domain controller assigned to thisrole can be a big security problem.

If you force a transfer of the infrastructure master role from its original domain controller to another domaincontroller in the same domain, you can transfer the role back to the original domain controller after you’vereturned it to production.

It is strongly recommended that you not put the infrastructure master role on any domain controller that isalso acting as a global catalog server. For more information about FSMO placement rules and best practices,see Microsoft Product Support Services article Q223346, at http://support.microsoft.com.

PDC EmulatorIf the PDC fails or no longer communicates, users who depend on its service are affected. These are down-levelusers from Window 95, Windows 98, and Windows NT 4.0 (or earlier). The PDC is responsible for changes tothe SAM database, changing passwords, account lockout for down-level workstations, and communications withthe domain controllers.

If you force a transfer of the PDC emulator role from its original domain controller to another domaincontroller in the same domain, you can transfer the role back to the original domain controller afteryou’ve returned it to production.

Page 161: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com150

Determining the Operations Master Role Holders LocationsAn important step in the process of troubleshooting problems with operations master role holders is identifyingwhich domains controllers hold the various forest- and domain-wide roles. There are actually several methods ofdetermining the location of FSMO role holder in Win2K.

Using the DSA and Schema MMC Snap-InsTo determine the RID, PDC, and Infrastructure FSMO Holders of a selected domain using Win2K’s built-in tools,follow these steps:

1. Click Start, Run, type dsa.msc, and press Enter or click OK.

2. Right-click the selected Domain Object in the top left pane, and then click Operations Masters.

3. Click the PDC tab to view the server holding the PDC master role.

4. Click the Infrastructure tab to view the server holding the Infrastructure master role.

5. Click the RID Pool tab to view the server holding the RID master role.

Determining the forest Schema Master role holder is a bit trickier, and involves the following:

1. Click Start, click Run, type mmc, and then click OK.

2. On the Console menu, click Add/Remove Snap-in, click Add, double-click Active Directory Schema, click Close, and then click OK.

3. Right-click Active Directory Schema in the top left pane, and then click Operations Masters to view the server holding the schema master role.

For the Active Directory Schema snap-in to be listed as an available choice in Step 2, you’ll have to havealready registered the Schmmgmt.dll file. If it doesn’t appear as an option, follow these steps to registerit: click Start, Run, type regsvr32 schmmgmt.dll in the Open box, and click OK. A message will be displayedconfirming that the registration was successful.

Determining the forest’s Domain Naming Master role holder requires the following steps:

1. Click Start, Run, type mmc, and then click OK.

2. On the Console menu, click Add/Remove Snap-in, click Add, double-click Active Directory Domains and Trusts, click Close, and then click OK.

3. In the left pane, click Active Directory Domains and Trusts.

4. Right-click Active Directory Domains and Trust, and click Operations Master to view the server holdingthe domain naming master role in the Forest.

Although the above methods certainly work, they aren’t necessarily the easiest. The following sections describesome additional methods for determining FSMO role holders on your network.

Page 162: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com151

Using NTDSUTILNTDSUTIL is a tool included with Windows 2000 Server, Windows 2000 Advanced Server, and Windows 2000Datacenter Server. This tool is can be used to verify change certain aspects of the Active Directory. The following isthe steps needed to to view the Flexiible Single Master Operation (FSMO) roles on a given Domain Controller.Ntdsutil.exe is the only tool that shows you all the FSMO role owners. You can view the PDC emulator, RIDmaster, and infrastructure master role owners in Active Directory Users and Computers. You can view the schemamaster role owner in the Active Directory Schema snap-in. You can view the domain naming master role owner inActive Directory Domains and Trusts.

1. Click Start, click Run, type cmd in the Open box, and then press Enter.

2. Type ntdsutil, and then press Enter.

3. Type domain management, and then press Enter.

4. Type connections, and then press Enter.

5. Type connect to server <server_name>, where <server_name> is the name of the Win2K domaincontroller you want to view, and then press Enter.

6. Type quit, and then press Enter.

7. Type select operation target, and then press Enter.

8. Type list roles for connected server, and then press Enter.

Using the Windows 2000 Resource Kit’s Dumpfsmos.cmdThe Win2K Resource contains a batch file named Dumpfsmos.cmd that you can use to quickly list FSMO roleowners for your current domain and forest. The .cmd file uses Ntdsutil.exe to enumerate the role owners.Dumpfsmos.cmd takes a single argument, the name of the domain controller to which it should connect whenquerying for FSMO locations. The usage of the command is:

Dumpfsmos <server_name>

Using DCDIAGAnother method involves use of the DCDIAG command. On a Windows 2000 Domain Controller, run thefollowing command: dcdiag /test:knowsofroleholders /v . Note that the use the /v switch here is required. Thisoperation lists the owners of all FSMO roles in the enterprise known by that domain controller.

Using AD Replication MonitorA final method to accomplish this is to use the Active Directory Replication Monitor (Replmon.exe) utility. Before

you can use AD Replication Monitor, you’ll need to install it. The AD Replication Monitor utility is part of theWindows 2000 Support Tools, which are located on the Windows 2000 Server CD-ROM in the \Support\Toolsfolder. Run the Setup.exe file in this folder to install the tools. Once installed, you can start the AD ReplicationMonitor utility by choosing Start\Programs\Support Tools\Tools, and selecting Active Directory ReplicationMonitor. Once the utility is running, follow these steps to determine the operations master role holders for theforest and domain:

Page 163: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com152

1. Right-click Monitored Servers, and then add one or more servers using the wizard.

2. Right-click the servers, and then click Properties.

3. Click the FSMO Roles tab.

4. The domain controllers that hold the operations master roles are displayed under the "Owner" column.

5. To test the connectivity to each of the operations master role holders, click Query to the right of each role.

Using Third-Party UtilitiesCertain third-party utilities, such as NetPro’s DirectoryAnalyzer (http://www.netpro.com) and NetIQ’s ADCheckutility (http://www.netiq.com/ADcheck/Download.asp), provide features to determine the domain controllersacting as FSMO role holders servers.

An example of viewing the schema master using DirectoryAnalyzer is shown in Figure 5.13.

Figure 5.13: Using a third-party utility to determine which domain controller in your forest holds a particular FSMO role.

Seizing an Operations Master RoleIf a domain controller holding one or more operations master roles is down during a critical time, will be unavailablefor a long time, or is permanently out of service, you’ll need to take steps to force the transfer of the role(s) toanother domain controller. You can accomplish feat by using NTDSUTIL. Forcing this type of operations master

Page 164: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com153

role transfer is also referred to as seizing the role on a domain controller. However, before you decide whether toseize the role of a operations master, be aware that doing so is a major step that should be taken only if the affecteddomain controller will never be brought back online. Forcing a transfer in this case should be a permanent action.

Once you’ve determined where the current operations master role holders in a domain or forest are (using theinformation in the previous section), you can use the NTDSUTIL program to transfer the operations master rolefrom one domain controller in the forest or domain to another in the same forest or domain. To seize an operationsmaster role on a selected domain controller, follow these steps:

1. On the target domain controller (the domain controller that will be taking over the forest- or domain-wideoperation master role), choose Start>Run. In the Open dialog box, type NTDSUTIL, then click OK.

2. If you’re not running NTDSUTIL on the target domain controller, you need to select and connect to it. At the ntdsutil: prompt, type connections, then press Enter. Type connect to server <server_name>,where <server_name> is the name of the server you want to use, then press Enter. To supply additional credentials, type set creds <domain_name user_name_password> and press Enter. At the Server Connections prompt, type quit, then press Enter again.

3. At the ntdsutil: prompt, enter the word roles.

4. To seize the role on the currently connected domain controller, enter seize <role_type>, where <role_type> is one of the following: schema master, domain naming master, rid master, infrastructure master, or pdc. (For a list of roles that you can seize, enter ? at the FSMO Maintenance prompt or see the list of roles at the beginning of this section.)

5. After you seize the roles, type quit, then press Enter, to return to the previous menu in the NTDSUTIL interface. Repeat this step until you’ve exited the utility.

6. Reboot the domain controller that seized the operations master role to complete the role change operation.

If the current operations master role holder domain controller is online and accessible, or can berepaired and brought back online, it’s recommended that you transfer the role using NTDSUTIL’stransfer command rather than the seize command. For more information on seizing and transferringflexible single master operation roles, see Microsoft Product Support Services articles Q255504 andQ223787, at http://support.microsoft.com.

Checking for Inconsistencies among Domain-Wide Operations MastersAnother way to troubleshoot problems on operations masters is to check for inconsistencies among the domaincontrollers in a domain. If the domain controllers don’t report operations masters consistently, long-term problems,such as replication problems, can arise. There are a number of third-party utilities on the market capable of detectingdomain controller inconsistencies. One example of such a utility is NetPro’s DirectoryAnalyzer, which is capable ofinspecting exactly what each domain controller believes are the domain-wide master role assignments. If all domaincontrollers fail to report the same values for all the operations masters, there is a problem, and this will be reported.

Page 165: The Definitive Guide to Active Directory Troubleshooting

Figure 5.14 shows an example of using a third-party utility (NetPro’s DirectoryAnalyzer) to check for operationsmaster role holder inconsistencies. As the figure shows, the domain controller COMP-DC-04 lists COMP-DC-01as the owner of the PDC operations master, while domain controller COMP-DC-O3 is the actual owner. Thus,the owner of the PDC operations master is inconsistent across the domain controllers.

Chapter 5 www.netpro.com154

Figure 5.14: Checking for consistency of the operations masters on domain controllers.

Troubleshooting the Replication TopologyWhen you troubleshoot replication problems and errors, it’s important to know who the replication partners of aspecific domain controller are and the status of replication with each one.

Viewing the Replication Partners for a Domain ControllerYou can view the replication partners for a specific domain controller using two tools, DirectoryAnalyzer andREPADMIN.

Using DirectoryAnalyzerWhen you use DirectoryAnalyzer to see replication partners, you’re viewing the replication topology for the selecteddomain controller in a forest, and you can check replication consistency among replication partners. In addition,DirectoryAnalyzer constantly checks the replication topology to ensure that it’s transitively closed. If it isn’t,DirectoryAnalyzer generates an alert.

Figure 5.15 shows the Replication Information tab in the Browse Directory By Site view. This tab allows you toview the replication topology and the last successful replication cycle for each replication partner. It also shows thereplication partners and any errors that occurred during replication.

Page 166: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com155

Figure 5.15: Using DirectoryAnalyzer to view the replication partners for each domain controller.

Using REPADMINYou can also use the Replication Administration utility (REPADMIN) to monitor the current links to other replicationpartners for a specific domain controller, including the domain controllers that are replicating to and from theselected domain controller. Viewing these links shows you the replication topology as it exists for the current domaincontroller. By viewing the replication topology, you can check replication consistency among replication partners,monitor replication status, and display replication metadata.

To use REPADMIN to view the replication partners for a domain controller, enter the command:

REPADMIN /SHOWREPS

Forcing Domain Controllers to Contact Replication PartnersIf you detect errors when viewing the replication partners using either the DirectoryAnalyzer utility or the REPADMINtool, you can manually force the domain controller to contact its replication partners and authenticate with them.This is necessary to create the replication links. You can use the following command to force contact:

REPADMIN /KCC

Page 167: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com156

During normal operation, the Knowledge Consistency Checker (KCC) generates automatic replication topologyfor each directory partition on the domain controllers. You don’t need to manually manage the replicationtopology for normal operation.

In addition to tracking replicated changes, many third-party utilities also constantly evaluate replicationlatency across all domain controllers. If the latency exceeds the specified threshold, theutility will generate an administrative alert and/or generate a log entry reporting the condition.

Tracking Replicated ChangesAfter the replication links have been re-created, future replication processes should occur automatically at the normalscheduled time. You can check whether replication is occurring normally among replication partners by tracking aparticular replicated change. This allows you to ensure that the target domain controller is receiving the change. Toperform this check, enter the following for a specific object in AD:

REPADMIN /SHOWMETA CN=CJOHNSON,OU=ENGINEERING,DC=COMPANY,DC=COM

<domain_controller>

In this command, <domain_controller> is the host name of the target domain controller for which you’re trackingreplicated changes for CJOHNSON in the ENGINEERING OU in the COMPANY.COM domain. The outputfrom this command shows the Update Sequence Number (USN), originating DSA, date and time, version number,and replicated attribute.

Forcing Replication among Replication PartnersThere are several methods that you can use to initiate replication among direct replication partners in a commonname context. For each of the following methods, the source domain controller describes the domain controllerthat replicates changes to a replication partner. The destination domain controller receives the changes.

To force replication among replication partners, you can use REPADMIN to issue a command to synchronize thesource domain controller with the destination domain controller by using the object GUID of the source domaincontroller. To accomplish the task of forcing replication, you need to find the GUID of the source server. Enter thefollowing command to determine the GUID of the source domain controller:

REPADMIN /SHOWREPS <destination_server_name>

You can find the GUID for the source domain controller under the Inbound Neighbors section of the output.First, find the directory partition that needs synchronization and locate the source server with which the destination isto be synchronized. Then note the GUID value of the source domain controller. Once you know the GUID, youcan initiate or force replication by entering the following command:

REPADMIN /SYNC <directory_partition_DN> <destination_server_name>

<source_server_objectGUID>

Page 168: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com157

An example of running this command to initiate replication between DC1 and DC2 of the domain partition calledCOMPANY.COM is seen below. The replication is forced from the source domain controller, DC1, to the destinationdomain controller, DC2. To perform the replication, use the following command:

REPADMIN /SYNC DC=COMPANY,DC=COM DC1 d2e3ffdd-b98c-11d2-712c-

0000f87a546b

If the command is successful, the REPADMIN utility displays the following message:

REPLICASYNC() FROM SOURCE: d2e3badd-e07a-11d2-b573-0000f87a546b,

TO DEST: DC1 IS SUCCESSFUL.

Optionally, you can use the following switches at the command prompt:

• /FORCE: Overrides the normal replication schedule• /ASYNC: Starts the replication event without waiting for the normal replication to finish.

You’ll typically force replication only when you know that the destination domain controller has been down oroffline for a long time. It also makes sense to force replication to a destination domain controller if network con-nections haven’t been working for a while.

Viewing Low-Level AD Replication StatusYou can troubleshoot the replication topology another way by viewing the low-level status of AD replication. TheReplication Monitor (REPLMON) utility allows you to do this. Because this tool is graphically based, you can viewthe replication topology in graphical form and monitor the status and performance of replication among domaincontrollers.

REPLMON provides a view only from the domain controller perspective. Like REPADMIN, you can install itfrom the \Support\Tools folder on the Windows 2000 CD-ROM. REPLMON has two options that you’ll findhelpful when monitoring AD:

Generate Status ReportGenerates a status report for the domain controller. The report includes a list of directory partitions for the server, the status of the replication partners for each directory partition, and the status of any Group Policy objects. It also includes the status of the domain controllers that hold the operations master roles, a snapshot of performance counters, and the Registry configuration of the server.

Show Replication TopologiesDisplays a graphical view of the replication topology. This option can also display the properties of the domain controller and any intra-site or inter-site connections that exist for the domain controllers.

Checking for KCC Replication ErrorsAnother method of troubleshooting replication problems is to check the entries that might appear in the DirectoryService log in Event Viewer. Event Viewer lists errors that pertain to replication, such as Knowledge ConsistencyChecker (KCC) errors. For example, you might see the entry (ID 1311 from event source NTDS KCC) in theDirectory Service log; it means that the Directory Service consistency checker has determined that for changes topropagate across all sites, replication cannot be performed with one or more critical domain controllers.

Page 169: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com158

When the KCC generates this error message, it’s in a mode where it doesn’t remove any connections.Normally, the KCC cleans up old connections from previous configurations or redundant connections.Thus, you might find that there are extra connections during this time. The solution is to correct thetopology problem so that the spanning tree can form.

This error could also indicate that there isn’t enough physical connectivity to create a spanning tree connecting allof the sites. A spanning tree is a network algorithm that most network switches use to build tables of media-access-control-address and port-number associations. This behavior can occur if the KCC has determined that a site hasbeen orphaned from the replication topology.

One domain controller in a specific site owns the role of creating inbound replication connection objects amongbridgehead servers from other sites. This domain controller is known as the Inter-Site Topology Generator. Whileanalyzing the site link and site link bridge structure to determine the most cost-effective route to synchronize anaming context between two points, it might determine that a site doesn’t have membership in any site link andtherefore has no means of creating a replication object to a bridgehead server in that site.

The first site in AD (named Default-First-Site-Name) is created automatically. It’ a member of the default site link(DEFAULTIPSITELINK) that is also created automatically and used for RPC communication over TCP/IP amongsites. If you create two additional sites—for instance, Site1 and Site2—you need to define a site link that each siteis going to be a member of before these sites can be written to AD. However, you can also edit the properties of asite link and modify which sites reside in it. If you remove a site from all site links, the KCC displays the errormessage listed above to indicate that a correction needs to be made to the configuration.

Troubleshooting Using Change ManagementNo discussion of troubleshooting AD would be complete without mentioning one of the most important techniquesavailable to the network administrator: change management. In any hardware- or software-troubleshooting endeavor,one of the most important questions an administrator can ask is: What changed to cause this problem? As you’veprobably learned from your own experience, most problems don’t occur in a vacuum. Rather, they develop as aresult of some change that is made or that occurs in the system. Thus, the ability to know what was changed onthe network—and when—is an invaluable troubleshooting tool.

Obtaining this type of change-management data for AD has been difficult or impossible. Win2K doesn’t recordsuch changes automatically in its event logs, and logging AD infrastructure changes manually is cumbersome andprone to error. Even third-party AD-management tools have been challenged in this area: Although a number oftools are available to diagnose problems with and monitor real-time events related to AD infrastructure components,the ability to analyze the progression of changes to these components over time has remained elusive.

This situation changed recently when NetPro Computing released a new tool, DirectoryInsight. DirectoryInsight isunique in that it gives administrators a wide array of information about changes to an AD network that occur overtime. This information includes:

• Records of all AD infrastructure and configuration changes across the enterprise• A historical record of AD changes, including modifications to such key infrastructure elements as directory

structure, replication, security, and schema• An enterprise-level view of changes to the AD object population that occur over time• Object population counts for GPOs, groups, OUs, domains, domain controllers, servers and computers,

sites, and users at any time.

Page 170: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com159

Figure 5.16: NetPro’s DirectoryInsight AD change-management tool.

It’s a good idea to use change-management information proactively in addition to using it reactively. Forexample, you might use the object-population information to analyze and plan network capacity and topredict future trends and infrastructure needs. This information is invaluable for management reportsand Information Technology (IT) budget planning.

With this kind of information in hand, it becomes much easier to investigate the origin of a problem and resolveit as well as to keep yourself generally "in the loop" about important changes that are being made to your ADinfrastructure. You can view at a glance events such as additions or deletions to AD elements like domains, OUs,domain controllers, groups and Group Policy, and the AD schema, and you can view the changes to the populationof these elements over time (or at a particular time).

For example, when users at a particular site report suddenly slow network logins that began that morning, youmight analyze your change log and determine that the only domain controller serving the domain on that site wasremoved by a site administrator the previous evening. DirectoryInsight consolidates all of the collected data into asingle, centralized database and provides convenient access to it using a Web-based administration tool (an ActiveXcontrol running in an Internet Explorer browser window.)

A sample DirectoryInsight screen is shown in Figure 5.16.

Page 171: The Definitive Guide to Active Directory Troubleshooting

Chapter 5 www.netpro.com160

SummaryTroubleshooting AD means identifying and analyzing problems that occur and repairing them in the various systems.The troubleshooting process is mostly about isolating and identifying a problem. To troubleshoot AD, you firstcheck to see whether the domain controllers in the forest can communicate with each other. Next, you need toensure that AD has access to DNS and that DNS is working properly. After you verify that DNS is working, youneed to check that the individual domain controllers and operations masters are working properly and supportingthe directory functions. Last, you need to verify that replication is working and that no consistent errors are beinggenerated.

eBook Copyright NoticeThis site contains materials created, developed, or commissioned by Realtimepublishers.com, Inc. and is protected byinternational copyright and trademark laws. No material (including but not limited to the text, images, audio, and/orvideo) may be copied, reproduced, republished, uploaded, posted, transmitted, or distributed in any way, except that onecopy may be downloaded for your personal, non-commercial use on a single computer. In connection with such use, youmay not modify or obscure any copyright or other proprietary notice. If you have any questions about these terms, or ifyou would like information about licensing materials from Realtimepublishers.com, please contact us via e-mail [email protected]

Page 172: The Definitive Guide to Active Directory Troubleshooting

The Definitive Guide to Active Directory Troubleshooting

Chapter 6

Page 173: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com161

Chapter 6: Backing Up and Recovering Windows 2000 and Active Directory

As a systems administrator, you must protect your network against data loss, computer failures, and loss of directo-ry services. To accomplish this task, you must back up and restore each Windows 2000 (Win2K) server and theservices each computer provides. You also need to back up and recover critical system data such as the ActiveDirectory (AD) database, the Registry, and other service-specific configuration databases and configuration files.Microsoft refers to this set of critical data as System State data.

It’s an unfortunate truth that, regardless of what precautions you take or what quality of hardware you’ve pur-chased, the odds are that someday, somewhere, you’ll experience a system failure. If your organization depends onits Win2K systems, you need to know how to troubleshoot and repair them when they fail. Although Microsofthas greatly improved system reliability and recoverability in Win2K, things still can and do go wrong. Fortunately,Microsoft provides tools to help you maintain and recover your system.

In this chapter, I’ll cover these new recovery and troubleshooting tools, including the Safe mode startup options,Win2K Setup’s Emergency Repair Process, and the Recovery Console. I’ll also discuss some advanced recoverytricks and techniques that you can use to assist in maintaining a high level of OS and network availability. I’ll alsomention some third-party tools that complement Win2K’s built-in recovery features.

Building a Fault-Tolerant System that Includes a Backup and Restore Strategy

The first level of defense against system disaster is ensuring that you’ve implemented at least minimal levels of faulttolerance on your network servers. Fault tolerance is generally defined as the resilience of a particular computer,subsystem, or component to various kinds of failures.At the network level, implementing system fault tolerance might involve installing power-backup equipment (suchas an Uninterruptible Power Supply, or UPS, with line conditioning) on critical network devices and creating afault-tolerant routing topology using dynamic routing protocols. In the case of network servers, implementing sys-tem fault tolerance usually involves creating backup or redundant resources for critical subsystems. The array offault-tolerant features for any important network system should always include, at least, the following:

•Disk redundancy—Using one or more levels of Redundant Array of Independent Disks (RAID) technology, in which the disk subsystem can withstand the failure of any one drive. You can implement RAID either using software, such as Win2K’s built-in RAID features (either RAID 1/disk mirroring or RAID 5/disk striping with parity), or using a hardware RAID device or disk controller).•Power redundancy—In the form of a UPS or power-conditioned backup generator capable of keeping the system up and running if the power fails. Backup power devices should ideally be connected to the server(s)they protect using hardware and software (for example, serial or network cabling and either Win2K’s UPS software or the UPS vendor’s software). This software can inform the server, network administrator, and/or network users about various power events that affect system availability. The software can also automaticallyinitiate a proper system shutdown after a specified amount of time or when batteries are running low.•UPS monitoring and management—For example, Win2K includes a UPS monitoring and management utility written by American Power Conversion (a leading UPS vendor). In addition, most UPS devices also include their own customized UPS management software.

Page 174: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com162

•Clustering solutions—Group servers into single logical entities. Windows 2000 Advanced Server and Windows 2000 Datacenter Server include a clustering service as well as a Transmission Control Protocol/Internet Protocol (TCP/IP) load-balancing component called the Network Load Balancing service. In addition, there are many third-party Win2K server-clustering solutions available on the market.

•Regular system backups—Maintain offline data redundancy for all critical system data. For example, you should develop and thoroughly test for each of your production servers and domain controllers a backup and recovery plan that includes the operating system (OS) and directory services.

Here are some goals and objectives for developing an effective backup and recovery strategy. You should adaptand expand these strategies to your environment and situation.

•Develop backup and restore methods that ensure that you can quickly recover lost data.•Make sure that you have a backup for each volume that contains the data. In case of total failure, this

strategy allows you to restore the entire volume.•Create a backup of AD that includes the entire domain on the domain controller. The backup should

include user accounts and security information.•Use the log created during backup to determine which files were opened by other applications during

the backup. The names of any files that aren’t backed up appear in the log, so you can use the log to keep track of which files aren’t being backed up and why. In addition, arrange a time to perform a separate backup of these files during times when the application isn’t running.

•Keep at least three copies of the backup media and rotate them. In addition, keep at least one copy offsite in case of a catastrophic failure.

•Perform a trial restore periodically to verify that your files are being properly backed up. A trial restore can uncover problems that don’t show up using backup verifications.

•Secure both the storage device and the backup media to prevent an administrator for another server from restoring stolen data onto your server. Ideally, this should include keeping an offsite copy as well as on-site copies of backups.

By far the most important step you can take to prepare yourself for a critical disaster is to back up the systemregularly. Every network should have in place a backup process that regularly copies the system data to secondaryor offline storage devices.

The process should back up the user data files and applications that would need to be recovered if they were lostor damaged as a result of a disk failing or data becoming corrupted. Backups should also include critical systemdata such as the AD database, the Registry, and other service-specific configuration databases and configurationfiles.

The acronym RAID originally stood for Redundant Array of Inexpensive rather than Independent Disksbecause the idea was to use a greater number of cheaper disks rather than a Single Large Expensive Disk(SLED). However, the definition was changed because many modern RAID systems are built strictly forredundancy rather than cost savings, and they often use disks that aren’t necessarily inexpensive.Wherever possible, choose a hardware-based RAID solution over Win2K’s software-based RAID. Most hard-ware RAID solutions offer better performance (many RAID controllers include on-board cache randomaccess memory (RAM) and dedicated processors), advanced RAID levels that offer improved flexibility andperformance, better management features, and easier recovery if a primary disk in a mirror set (RAID-1volume) fails.

Page 175: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com163

When you develop your backup strategy, you need to know how long it takes to back up an active server or domaincontroller on the network. To assist with this strategy, remember not to back up the server during peak hours.Instead, schedule backups during off-peak hours so that the impact of the backup process is minimal.

Another important consideration in any backup routine is the timing of backups in relation to the availability ofthe data files being backed up. For example, many backup utilities can’t back up files that are being used (by usersor applications), so it’s important to schedule backups to occur during off-hours, when network usage is at its low-est and the maximum number of files are available to be backed up.

Backing up data during off-peak hours is also important from a performance standpoint. For example, backing upa computer’s data taxes the computer’s resources and makes it less accessible. Also, performing a backup from aremote computer over the system’s primary network connection can significantly reduce network performance.

Using the Windows 2000 Backup and Restore UtilityWin2K includes a graphical user interface (GUI)-based backup and restore utility called Backup. (The file is calledNtbackup.exe.) You can use this utility to back up an entire Win2K system, including AD and other critical data.

The Win2K backup utility has been integrated to copy the core Windows 2000 Server distributed services, whichinclude AD, File Replication service (FRS), and Certificate Services. Backup lets you back up and restore theseservices by checking the System State check box in the utility. Backup also supports Remote Storage, RemovableStorage, disk-to-disk operations, and other new Win2K services and features. You can back up data to a tape drive,a logical drive, a removable disk, or an entire library of disks or tapes organized into a media pool and controlled bya robotic changer.

The backup utility lets you perform the following tasks:

•Back up selected user files and folders located on the domain controller’s hard drive•Back up the domain controller’s System State, which are the files central to system operation (the Registry,

AD and SYSVOL, the COM+ Class Registration database, and boot files)•Restore backed-up files and folders to your server’s hard drive•Schedule regular backups•Create an Emergency Repair Disk (ERD), which helps you repair system files that become corrupted.

Backup and restore operations can only be performed by backup operators and local administrators. Members ofthe Backup Operators group can back up and restore data on the domain controller. (This group is one of thebuilt-in groups provided by Win2K.) Any domain user can be granted the user rights to back up and restore filesand directories. Members of the Local Administrators group can back up and restore data files, directories, and theSystem State of the domain controller.

To start the backup utility, choose Start>Programs>Accessories>System Tools>Backup. Figure 6.1 shows theWelcome page.

To reduce the risk of losing data, and to improve data security, store backup sets in a safe that is bothheatproof and fireproof. In addition, as part of your backup cycle, regularly rotate one or more full backupsets to an offsite location. Thus, if all of the equipment at the primary site is destroyed, you can stillrecover your data.

Page 176: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com164

The Welcome page provides three main options to assist you in backing up and restoring your data.

•Backup Wizard—Helps you back up your programs and files to help prevent data loss and damage caused by disk failures, power outages, virus infections, and other potentially damaging events.•Restore Wizard—Helps you restore your previously backed-up data in the event of a hardware failure, accidental erasure, or other data loss or damage.•Emergency Repair Disk—Helps you create an ERD that you can use to repair and restart Win2K if it’s damaged. This option doesn’t back up your files or programs, and it’s not a replacement for regularly backing up your system.

In addition to these options, the tabs on the Welcome page allow you to bypass the wizards and select backup andrestore options manually.

Using the Backup WizardTo perform a basic backup of your domain controller, click Backup Wizard and follow the on-screen instructions.

Specifying Default OptionsWhen you run the Backup Wizard and accept the default settings, all of your computer’s data files and System

Figure 6.1: The Welcome page of the Win2K backup utility.

You can also start the backup utility from a command-line interface or prompt. At the command prompt onthe local computer, simply type Ntbackup.

Page 177: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com165

State data are backed up as a set. The Backup Wizard lets you back up the data to a file or tape on the target com-puter. When you back up the data, you have to designate a file name and location. Backup files typically have the.bkf extension, but you can use any extension to override the default. You can save a backup file to a hard drive,floppy disk, or removable or non-removable media.

Figure 6.2 shows the What to Back Up dialog box, where you select what you want to back up.

In the next dialog box, you can select the type of backup media and the location and name of the destination back-up file. (By default, the file is called Backup.bkf.)

You can use the backup utility to back up and restore data on either file allocation table (FAT) or NT file system(NTFS) volumes on your system or domain controller. If you back up data from an NTFS 5 (Win2K NTFS) vol-ume, you should in most cases restore it to an NTFS 5 volume. If you restore the data to a FAT or Windows NT4.0 or earlier NTFS volume, you’ll lose certain file and folder features, and you’ll lose data as well—for example,file permissions, Encrypting File System (EFS) settings, disk quota information, mounted drive information, andremote storage information.

The Completing the Backup Wizard dialog box, shown in Figure 6.3, presents a summary of your current selec-tions and allows you to specify additional options, if needed. (These are discussed in "Specifying AdvancedOptions" below.) Clicking Finish starts the backup, and the Backup Progress dialog box tracks the progress of thebackup. (See Figure 6.4.)

Figure 6.2: The Backup Wizard prompts you to select what you want to back up on the localcomputer.

Page 178: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com166

Figure 6.3: The Completing the Backup Wizard dialog box displays a summary of your currentselections and allows you to choose additional backup options.

Figure 6.4: The Backup Progress dialog box tracks the progressof the backup.

Page 179: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com167

Specifying Advanced OptionsInstead of clicking Finish in the Completing the Backup Wizard dialog box, you can click Advanced and customizethe backup. The Type of Backup dialog box appears, where you can select the specific type of backup to perform.(See Figure 6.5.)

The backup utility allows you to perform the following types of backup:

•Normal—Copies all selected files and marks each one as having been backed up. (This clears the archive attribute.) Performing a normal backup of all files restores the state of your computer to the time of the backup using only the one normal backup. If you then create incremental or differential backups, you have to restore all the incremental backups or the latest differential after restoring the normal backup.A normal backup creates a backup of the files, folders, and drives selected with the entire System State of the server or domain controller while it’s online. (See "Maintaining System State Backups" later in this chapter.) If the server is a domain controller, AD is backed up as part of the System State. When you back up the System State for the local computer, you cannot choose to back up individual components of the System State data because of dependencies among them.•Copy—Allows copies of all selected files but doesn’t mark each one as having been backed up. (This means that the archive attribute is cleared.) If you want to back up files between normal and incremental backups, copying is useful because it doesn’t affect these other backup operations.•Incremental—Allows you to back up only those files that have been created or changed since the last normal or incremental backup. It marks files as having been backed up. If you use a combination of normal and incremental backups, restoring files and folders requires that you have the last normal backup set as well as all subsequent incremental backup sets.

Figure 6.5: The Type of Backup dialog box allows you to choose the mode, or type, of backup toperform and whether to back up the data files that have been migrated to Remote Storage.

Page 180: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com168

•Differential—Allows you to back up files created or changed since the last normal or incremental backup. It doesn’t mark files as having been backed up. (This means that the archive attribute isn’t cleared.) If you’re performing a combination of normal and differential backups, restoring files and folders requires that you have the last normal as well as the last differential backup.•Daily—Allows you to copy all selected files that have been modified on the day the daily backup is performed. The backup files aren’t marked as having been backed up.

You can also perform a combination of backups. For example, backing up your domain controller data using nor-mal and incremental backups requires the least amount of storage space and is the quickest backup method.However, recovering files can be time-consuming and difficult because the backup sets can be stored in several dif-ferent places, disks, and/or tapes. On the other hand, backing up your domain controller’s data using a combina-tion of normal and differential backups is more time-consuming, especially if your data changes frequently, but it’seasier to restore the data because the backup set is typically stored on only a few disks or tapes.

After you’ve selected the type of backup and whether to back up data files migrated to Remote Storage, you can usethe How to Back Up dialog box (shown in Figure 6.6) to specify whether you want the backup to be verified.Verification reads the backed-up data to ensure its integrity. Verification takes extra time but helps ensure that thebackup is successful.

You can also specify whether you want the backup to use hardware compression, if available. Hardware compres-sion allows the backup to try and reduce the size of the data files and System State being backed up. Backup filesthat have been backed up using hardware compression must be restored to hard drives that support compression.

Figure 6.6: The How to Back Up dialog box provides options to verify the backup and supporthardware compression, if available on the local server.

Page 181: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com169

Another dialog box (not shown) allows you to specify whether to append the backed-up data to an existing backupfile or to replace the existing backup file with the new backup data. The next dialog box, the Backup Label dialogbox, allows you to customize the backup and media labels that are stored with the backup file. (See Figure 6.7.) Tomake your life easier, the labels include the date and time of the backup.

In the next dialog box, you can decide when to perform the backup. You can choose Now to perform it immediate-ly or Later to schedule a time (preferably during off-peak hours) to have the backup run. (Scheduled backups arediscussed in "Backing Up Using Manual Selection" below. After you’ve selected all the advanced options, theCompleting the Backup dialog box appears again and summarizes your backup configuration. Click Finish to runthe backup. When it completes, a backup status report is displayed, as shown in Figure 6.8.

Figure 6.7: The Backup Label dialog box allows you to customize the backup and media labels.

Page 182: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com170

Figure 6.8: The backup status report shows the details of the finished backup.

Backing Up Using Manual SelectionAs I mentioned earlier, instead of using the Backup Wizard, you can manually select the options you want to beavailable during backup. To do this, on the Welcome page of the backup utility, click the Backup tab. In theBackup dialog box (shown in Figure 6.9), you can select the data files, folders, and drives to be backed up. You canalso select the storage media, file location, and name for the backup data.

Page 183: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com171

Figure 6.9: The Backup dialog box allows you to select the data files, folders, and drives that you want to be backedup as well as the backup file name, location, and media type.

Here you can perform the following tasks and set up the backup configuration:

•Select the data files, folders, and drives to be backed up—The Backup dialog box provides a tree view of the local files and folders that you can back up. You can use this tree view the same way you use WindowsExplorer to open devices and select files and folders.

•Select storage media and backup destination—Backups provide two options for selecting storage media: removable and non-removable storage devices and tape devices. Storage devices can be hard drives, floppy disks, and Zip and Jaz drives. The tape device option is only available when the local computer has a tape drive installed.

After you’ve made your selections, click Start Backup. The Backup Job Information dialog box appears (shown inFigure 6.10). Here you can customize the backup by specifying a name and a label for the backup and whether thedata should be appended to the backup media.

When you’ve selected these options, click Start Backup to start the backup. The backup utility displays the BackupProgress dialog box (see Figure 6.4), where you can track the backup procedure.

One handy new feature in the Win2K backup utility is its ability to back up to alternative backup mediasuch as hard disks, Jaz and Zip drives, optical drives, compact disc-recordable/rewritable (CD-R/RW)drives, and digital video disc-ROM (DVD-ROM) drives. As a rule, if it has a logical drive letter and Win2Kcan write files to it, Win2K Backup can use it as a destination backup device.

Page 184: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com172

If you want to schedule the backup to run at a later time (and unattended), click Schedule; the backup utility willwait and run the backup at the scheduled date and time you specify. To specify additional backup options, clickAdvanced to select the backup type, whether to verify the backup, and whether to back up remote media. You canspecify the type of backup (Normal, Copy, Incremental, Differential, or Daily), indicate whether you want a log fileto record the actions, exclude certain file types from the backup, and verify whether the backup was successful. (Forinformation on the types of backup, see "Specifying Additional Options" above.)

Maintaining System State BackupsWin2K allows you to back up and restore system components known as System State data or System State backups.The contents of a System State backup depend on the type of Win2K computer and include:

•The Registry•System boot files (including all system files)•AD service•COM+ Class Registration database•Certificate Services database•SYSVOL folder•Cluster service information

Here is a breakdown of System State backup components by OS:

•Windows 2000 Professional—A copy of the system Registry hive files (as with an ERD), the COM+ Class Registration database, the system boot files, including Ntldr, Ntdetect.com, and other system data, such as performance-counter configuration files and all files protected by Windows File Protection (WFP).•Windows 2000 Server—The Registry, COM+ Class Registration database, system boot files, and, if the server is a certificate server, Certificate Services information.•Win2K server that is also a domain controller—In addition to the components for Windows 2000 Server, a copy of the AD database (Ntds.dit), the log and checkpoint files, and the contents of the SYSVOL folder.•Win2K server that is also running Domain Name System (DNS)—In addition to the components for Windows 2000 Server, Directory Services (DS)-integrated as well as non-DS-integrated DNS zone information.•Windows 2000 Advanced Server acting as cluster members—In addition to the components for Windows 2000 Server, a copy of the quorum recovery log file.

Figure 6.10: The Backup Job Information dialog box allows you to customizethe backup job you’ve created.

Page 185: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com173

To back up System State data, run the Win2K backup utility and click the Backup tab. In the tree list at the left,click the System State check box. You have the option of selecting only System State data for a backup or includingit as part of a larger backup that includes local disk volumes. Finally, note that the files migrated to near-line stor-age by Win2K Remote Storage (RS) and those protected by WFP are only included during a backup if you selectBack Up Data That Is in Remote Storage and Automatically Back Up System Protected Files with the System State,respectively.

When you back up the System State, a copy of the Registry is placed on the local system partition in a folder under%Systemroot%\Repair\Regback (such as C:\Winnt\Repair\Regback). If the system Registry files are deleted ordamaged, you can use these backup copies of the Registry hive files to repair your system (for example, using theRecovery Console) without performing a full restore of the System State data using the Win2K backup utility.

To restore the System State on a domain controller, you must first start your computer in Directory ServicesRestore mode. This allows you to restore AD and the SYSVOL folder. When you back up or restore the SystemState data, you get all the System State data that is relevant to your server or domain controller. You cannot backup or restore individual components of the data because of dependencies among the components of the SystemState. However, you can restore the System State data to an alternate location; if you do, only the Registry files,SYSVOL folder files, cluster service information, and system boot files are restored.

Using the Restore WizardIf you want to restore the data files and System State data that you’ve backed up, you can choose Restore Wizard onthe backup utility’s Welcome page. If a hard disk fails, the Restore Wizard allows you to recover the entire system toan operational system or restore the system from the ground up.Before using the Restore Wizard, however, I recommend that you collect the information in Table 6.1; you’ll needit to perform a successful restore.

For This Do This

Disk configuration Using the Disk Management system utility, record the volumes and sizes of the disksin your system. If a disk should completely fail, you’ll use this information to re-cre-ate the disk configuration. All disk configurations must be restored before restoringthe System State or data to your computer. If you don’t, the restore process may failor the computer may not start after the restore has finished.

Server name Record the server name. You’ll use it to restore a computer of the same name andavoid changing many different client configuration settings.

Internet Protocol (IP)addresses

If the computer has a static IP address, record it so you can set it up manually if youneed to. However, in most cases, the IP address should be restored with the Registry.

Domain information Know what domain this computer belongs to and be prepared to set up a new com-puter account for it. Even if the computer name doesn’t change, you may need to re-establish a new account.

Local administrativepassword

You must know the local computer’s administrative password used when the backupwas created. If you don’t know it, you can’t log on to the computer once it’s restoredto establish a domain account for the computer. If you’re not part of the domain,you can’t use a domain account; this applies even if you’re domain administrator.Moreover, the local administrator’s password is also required to restore the SystemState on a domain controller.

Table 6.1: Information required to perform a successful restore.

Page 186: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com174

Figure 6.11: The Welcome page for the Restore Wizard.

When you click Next on the Welcome page, the What to Restore dialog box appears, where you can select the spe-cific data set that you previously backed up (shown in Figure 6.12). The Restore Wizard provides you with a treeview of the files and folders the data was in when it was backed up. You can navigate the tree to select the files andfolders in the same way that you use Windows Explorer. The label that you selected for the backup identifies thedata set.

Figure 6.11 shows the Welcome page for the Restore Wizard.

I don’t recommend using the restore procedure for copying a system from one computer to another. Thebackup of the System State is server-specific.

Page 187: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com175

When you click Next, the Completing the Restore Wizard dialog box appears. Like its equivalent in the BackupWizard, this dialog box shows your current selections and allows you to select advanced options, if needed. Youstart the restore by clicking Finish. As in the Backup Wizard, a progress dialog box appears, where you can trackthe restore.

Specifying Advanced OptionsInstead of running the restore, you can click Advanced on the Completing the Restore Wizard to specify advancedoptions. In this case, the Where to Restore screen appears, which lets you select one of three destinations for yourrestored files. (See Figure 6.13.)

Figure 6.12: The What to Restore dialog box provides a tree view of the data sets that havebeen created. You navigate the tree to select the files and folders that you want to restore.

Page 188: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com176

You can restore to:

•Original location—The folder(s) the data was in when it was backed up. This option is useful if you’re restoring files and folders that have been damaged or lost.•Alternate location—Retains the structure of the backed-up folders and files. This option is useful if you know you’ll need some old files, but you don’t want to overwrite or change any of the current files or folders on your disk.•Single folder—Doesn’t retain the structure of the backed-up folders and files. Only the backed-up files are placed here. This option is useful if you’re searching for a file and you don’t know its location.After you select the location for the restored files, the How to Restore dialog box appears, where you select how you want your files and folders restored. Again, you have three options.•Don’t replace the file on my computer—Prevents files from being overwritten on your hard disk. This is the safest method of restoring files.•Replace the file on the disk only if the file is older—Allows you to make changes to the files since you last backed up your data. This option ensures that you don’t lose the changes you’ve made to the files.•Always replace the file on my disk—Replaces all of the files on your hard drive with files in your backup data set. If you’ve made any changes to the files since you last backed up, your data will be replaced with the backed-up data.

Once you’ve made your selection, the Advanced Restore Options dialog box allows you to specify whether youwant to restore security, the Removable Storage database, and/or junction point data. (See Figure 6.14.)

Figure 6.13: The Where to Restore dialog box allows you to select the location of restoredfiles and folders.

Page 189: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com177

The last dialog box in the Restore Wizard provides a summary of the selections that you’ve made and lets you startthe restore process. Like the Backup Wizard, once you start the restore process, a progress dialog box appears,which provides details during the restore and completion information once it’s done.

Restoring Required ServicesIf you have to perform a restore, several server services require special attention to make them operational. Table6.2 lists the services that require additional effort. The sections below the table provide additional informationabout restoring each service.

Figure 6.14: The Advanced Restore Options dialog box allows you to restore security and/orspecial system files.

When you restore the System State, your recovery plan should take into account the fact that the age ofthe backup tape mustn’t exceed the AD tombstone lifetime (the length of time that deleted objects aremaintained in AD before the system permanently removes them). The default for this value is 60 days. If atape older than the tombstone is restored, the restore application programming interfaces (APIs) will rejectall of the data as being out of date. Microsoft discusses this issue in Product Support Services articleQ216993, "Backup of the Active Directory Has 60-Day Useful Life." This underscores the importance ofperforming regular backups of AD.

Page 190: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com178

This Service Requires This

Windows Internet Naming Service (WINS)On a TCP/IP network, the Windows Internet Naming Service (WINS) dynamically maps IP addresses to comput-er names (Net BIOS names). Because of this, WINS lets users access resources by name instead of requiring themto use IP addresses that are difficult to recognize and remember. WINS servers support clients running NT 4.0 andearlier versions of Microsoft OSs.

WINS The WINS database is restored to the state it was in at the time of the back-up. This overrides the current state of the server.

DHCP DHCP leases are restored to their state at the time of the backup. You mustperform several steps to reconcile the state of this database with the currentstate of the network.

Remote Storage During a restore operation, the Remote Storage database is recalled from tapemedia when you restart the service—but only if the tapes are available.

Certificate Services server After a restore operation, this server may have outstanding certificates that arenow unknown. You can revoke and re-issue these certificates or leave the oldcertificates orphaned.

Windows MediaServices server

After a restore operation, you may have to re-install this server because thedatabase containing setup information may be lost.

Internet InformationServices (IIS) server

If you perform a complete restore, no problems with IIS should arise. If youperform a partial restore, you must follow the backup/restore procedures spe-cific to the IIS service.

AD In a network with more than one domain controller, the default restoremethod (non-authoritative) is generally the preferred method for restoring afailed server. Use the authoritative restore process outlined later in this chapter(see "Authoritative Restores") only if you want to get the system back to thestate at the time the backup was made. (You’d want to do this, for example, ifyou erroneously deleted AD objects from the database and it would be diffi-cult to re-create them. For more information, see "Developing a Backup andRestore Strategy for Active Directory" later in this chapter.)

SYSVOL If the computer being restored is the only domain controller on the network,you must select a restore option in the Advanced Restore Options dialog boxin the backup utility. Otherwise, you need to use the default (non-authorita-tive) restore process.

Table 6.2: How to handle required server services when performing a restore.

Page 191: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com179

When a Win2K server receives a request from a client computer asking for a mapping from a friendly name to anIP address, WINS responds. When a restore is completed, the WINS database is restored, but it may be out of datebecause the information on the network is dynamic. The database updates itself over time and within a day or twoshould be consistent. During this time, some name requests may go unanswered or contain incorrect mappings. Ifthe WINS database is replicated among several WINS servers (the recommended procedure), you should initiatereplication, which synchronizes the database with the up-to-date server. If no other server is available, it’s best to letthe database synchronize on its own.

Dynamic Host Configuration Protocol (DHCP)

Dynamic Host Configuration Protocol (DHCP) is a networking protocol that offers dynamic configuration of IPaddresses for computers. DHCP ensures that address conflicts don’t occur and helps conserve the use of IP address-es by centrally managing address allocation.

The DHCP server allocates IP addresses and other network-configuration information to DHCP-aware networkclients. Using DHCP is the most common way to distribute IP addresses in a modern network. The restore processrestores the DHCP database. However, it’s restored to the date the backup was performed, and this can result induplicate IP addresses being issued. Having duplicate addresses causes those computers to cease all network opera-tions.

To avoid this, DHCP has a Safe mode of operation. In Safe mode, DHCP broadcasts on the network to verify thatthe IP address it’s about to issue isn’t already in use. After a restore, you need to reconcile the database and enterSafe mode for a period of one-half of the IP lease duration. Because Safe mode significantly reduces network andserver performance, and because being in Safe mode for this period of time is enough to ensure that DHCP func-tions properly, Microsoft recommends quitting Safe mode as soon as the one-half lease duration is met.

To reconcile the DHCP database, run the DHCP snap-in, then choose Action>Reconcile while the scope is high-lighted. In the scope properties dialog box, choose Conflict Detection under Advanced and set the number ofattempts to 1.

Remote Storage ServiceThe Remote Storage service (the Win2K version of Hierarchical Storage Management) frees up disk space by mov-ing data from the local hard disk to a remote storage device (such as a tape), from which it can be recalled whenev-er needed. Users still see and access the data without knowing that it’s been archived.

The Remote Storage service cannot recall its database from the Remote Storage tape during the restore operationunless the Remote Storage tape is in the correct drive—that is, the drive configured to be the Remote Storagedevice—or in the robotic library. If any issues with the service exist, the tape will restore by using the database copythat the service stores on the tape. This is an automatic process that requires no user intervention.

Certificate ServerCertificate Server is the Win2K service that issues X.509 security certificates for a particular Certificate Authority(CA). It provides customizable services for issuing and managing certificates for the enterprise.

After performing a restore operation, you don’t have to take any special steps for Certificate Server. However, cer-tificates may exist on the network that were issued before the restore operation. Although Certificate Server is nowunaware of these certificates, they’re valid and will continue to function.

Page 192: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com180

Internet Information Server (IIS)Internet Information Server (IIS) is a set of software services that supports Web-site creation, configuration, andmanagement, along with other Internet functions. If you perform a complete system restore, you don’t need to takeadditional steps to restore IIS. If you perform a partial restore of a file only, you may need to use the IIS MicrosoftManagement Console (MMC) snap-in to restore the IIS database. You’ll find instructions on doing this in the IISHelp pages.

Active DirectoryFor a detailed discussion of how to back up and restore AD as a service, see "Developing a Backup and RestoreStrategy for Active Directory" later in this chapter.

SYSVOL

SYSVOL is a replicated data set that contains the policies and scripts that are used by some Windows security sys-tems. SYSVOL uses Win2K’s FRS for distribution throughout the network. The three restore options for SYSVOLare identical to the options for FRS: non-authoritative (the default), authoritative, and primary.

Creating an Emergency Repair Disk (ERD)You can also use the backup utility to create an Emergency Repair Disk (ERD), which contains critical informationabout your Win2K system’s settings. An ERD can be a very helpful resource when you attempt to fix an unstartableserver because it contains information that can help Win2K Setup’s Repair mode repair problems with yourRegistry, system files (if they’re accidentally erased or become corrupt), your startup environment, and/or the parti-tion boot sector on your boot volume.

To create an ERD, make sure you have a blank 1.44-megabyte (MB) floppy disk. Then on the Welcome page ofthe backup utility, click Emergency Repair Disk to start the ERD Wizard. From there, you have a single option—whether you also want to update the hard disk–based copy of the Registry files (which are stored in the%Systemroot%\Repair folder). By default, this folder contains versions of the Registry hive files that were createdright after the Win2K setup process was completed. Therefore, if you want to preserve the original set of Registryfiles created during Setup but still update the files when the ERD is created, you need to first copy the files in thisfolder to another location.

An ERD contains three files:

•Autoexec.nt—A copy of the %Systemroot%\System32\Autoexec.nt file, which initializes the MS-DOS Virtual DOS Machine (VDM) environment used to run MS-DOS and 16-bit Windows applications.•Config.nt—A copy of the %Systemroot%\System32\Config.nt file, which initializes the MS-DOS VDM environment used to run MS-DOS and 16-bit Windows applications.•Setup.log—A log of files installed during Win2K Setup, along with checksum information that can be used during the Setup Repair process (an option of Win2K Setup) to verify files against their original source copies on the Win2K installation source (for example, the Windows 2000 CD-ROM). The Setup.log file has read-only, system, and hidden attribute flags set, so it isn’t normally visible unless you configure Windows Explorer to show all files.

You need to restore SYSVOL and AD together; however, to clarify the issues involved, this chapterexplains them separately. For a detailed discussion of how to back up and restore AD as a service, see"Developing a Backup and Restore Strategy for Active Directory" later in this chapter.

Page 193: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com181

Options to Use When Your Server or Domain Controller Won’t StartIn addition to the internal reliability enhancements (such as WFP, driver signing, and the driver verifier) that makeWin2K less prone to crashes than Windows NT, Microsoft has introduced three new system-recovery features thatmake repairing an unstartable Win2K system easier. These are described briefly below and in more detail in the fol-lowing sections. Third-party tools are another option.

•Safe mode booting—The first option to try is Safe mode and related startup options. This option is easy and understandable to use. A related option is Last Known Good Configuration, which may provide a solution in situations where a recently installed driver or service is causing problems.•Recovery Console—If Safe mode fails, the next option to try is the Recovery Console. Only advanced or administrative users should use this option. This method of starting a failed system uses the OS Setup CD-ROM or ERD floppy disks you created from the Setup CD-ROM.•Windows 2000 Setup Emergency Repair Process— If the Safe mode and Recovery Console options don’t work, try rerunning the Setup program from the Windows 2000 CD-ROM. It may (but isn’t guaranteed to) repair the system and make it startable.•Third-party recovery tools—There are a number of excellent third-party recovery tools on the market that you can use to assist you in troubleshooting and recovering unstartable Win2K systems.

Safe ModeSafe mode is a boot option that lets you disable certain OS features so that you can successfully start the system.For example, you can remove offending configurations, such as a newly installed driver, which might be causing aproblem.

Safe mode has a number of startup options that allow you to start using varying system configurations. For exam-ple, the regular Safe mode starts Win2K in a bare-bones environment (set of drivers and services). It doesn’t provideaccess to high-resolution video drivers, nor does it provide networking or other optional services.

To access most of these choices, you press F8 on the Windows 2000 Boot Loader menu when you start up. Thisselection displays a menu of the following alternative safe-start options:

•Safe Mode—Starts Win2K with the minimal set of drivers and services necessary. These basic files and drivers support the mouse, keyboard, monitor, mass storage, and other common services.•Safe Mode with Networking—Similar to Safe mode but adds drivers and services necessary to enable networking.•Safe Mode with Command Prompt— Similar to Safe mode but starts Win2K with a Command Prompt window instead of Windows Explorer.•Enable VGA Mode—Starts Win2K in VGA mode by using the Vga.sys driver instead of the regular video driver.

When you create an ERD, information about your current system settings is saved in the%Systemroot%\Repair folder. Don’t delete or change this folder, or you may not be able to repair problemswith your system. There is a bug in the ERD creation utility that prevents the hard disk–based copies ofthe Registry files and the ERD floppy disk from being created if the %Systemroot%\Repair folder is emptywhen you start creating an ERD. Therefore, if this folder is empty (that is, you moved or deleted the files),you may need to restore them temporarily from copies made on either that system or another system tocreate the ERD. Also, don’t try to create an ERD when your system is already having problems; the ERDcannot fix future problems.

Page 194: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com182

•Last Known Good Configuration—Starts Win2K using a previous version of the system Registry hive. The Last Known Good Configuration is the most recent session in which a user successfully started up without any service or driver-initialization failures and logged on to the computer.•Directory Service Restore Mode—Recovers the AD database. This option is extremely helpful when you recover AD and is valid only for Win2K domain controllers. If you plan to start a domain controller in Safemode and then use the backup utility with removable storage or AD, this is the only option that supports this. Note that when you start up a domain controller in Safe mode, AD runs, and you need to log in with domain credentials—for example, as the domain Administrator. When you start up in Directory Service Restore mode, AD isn’t started, and you log in with local machine credentials.•Debug Mode—Enables a startup mode in which the system sends debugging information across a serial cable to another computer running a debugger. (The mode uses COM2 as the debugging port.)•Enable Boot Logging—Creates an extended log file of success events and failure events to initialize system components as they load when Win2K starts. (This behavior is the default for all Safe mode boot options except for the Last Known Good Configuration and Directory Service Restore mode.) The log file is namedntbtlog.txt and resides in the %Systemroot% folder (for example, C:\Winnt).

Depending on the type of problem you’re experiencing with a system, one or more of these options might beappropriate in a given set of circumstances.

The Recovery ConsoleIf Safe mode doesn’t start your system, try using the Recovery Console. The Recovery Console is a special bootmode that provides a command-line interface. You can use it to start and stop services, read and write data on alocal drive (including drives formatted with NTFS), copy data to and from a floppy disk or CD-ROM, formatdrives, fix the boot sector or master boot record, and perform other administrative tasks—all outside the Win2Kenvironment.

The Recovery Console is particularly useful if you need to repair your system by copying a file from a floppy diskor CD-ROM to your hard drive, or if you need to reconfigure a service that is preventing your computer fromstarting properly. For example, you can use the Recovery Console to replace an overwritten or corrupted driver filewith a good copy from a floppy disk.

Using the Recovery ConsoleTo use the Recovery Console:

1. Insert the Windows 2000 Setup CD-ROM into the CD-ROM drive.2. When the text-based part of the Setup begins, follow the prompts. Choose the REPAIR option

by typing R.3. When you’re prompted, choose the Recovery Console by typing C.4. If you have a system that has more than one OS installed, choose the Win2K installation that you need

to access.5. When you’re prompted, type the administrator password.6. At the system prompt, type Recovery Console commands, then type either Help for a list of commands

or Help <command name> for help on a specific command.7. Quit the Recovery Console and restart the computer by typing Exit.

The Recovery Console is extremely powerful, so I recommend it only for advanced users and administrators.

Page 195: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com183

Adding the Recovery Console to the Startup OptionsIf you want to be able to start Win2K using the Recovery Console, you can add it as an option to the Boot Loadermenu on a functioning computer. To do this:

1. With Win2K running, insert the Windows 2000 CD-ROM into the CD-ROM drive.2. Choose File>Run.3. In the Open dialog box, type: D:\I386\Winnt32 /Cmdcons (where D: is the drive letter assigned to your

CD-ROM drive)4. Follow the on-screen instructions to complete the installation.

The commands for the Recovery Console allow you to accomplish simple operations such as change to a differentdirectory or view a directory, and more powerful operations such as fixing the boot sector on the hard drive. Youcan display help for the commands by simply typing Help at the Recovery Console command prompt.

Windows 2000 Setup Emergency Repair ProcessAnother system-recovery option is using the Win2K Setup emergency repair process (or Repair) option. This modeof Setup, like its Windows NT predecessor, allows you to perform some basic system-repair operations.

Assuming you have an ERD for the non-booting system (for more information on this, see "Creating anEmergency Repair Disk" earlier in this chapter), make sure you have it ready during this procedure. (Even if youdon’t have an ERD, it may still be possible to use Setup Repair mode to recover the system; see the sidebar below,"Using Windows 2000 Setup Repair without an ERD.")

The following steps provide a general overview of how to use the emergency repair process from the Setup CD-ROM:

1. Start your computer from the Windows 2000 Setup disks or CD-ROM—You can only start your computer from the CD-ROM if your computer hardware and basic input/output system (BIOS) have been set up to support this option.2. Choose the repair option during setup—After your computer starts up, the Setup program starts, and you’re asked whether you want to continue installing the Win2K OS. Press Enter to continue. The installation process starts and allows you to repair your system. During installation, you can choose whether to install a fresh version of Windows 2000 Server or repair an existing installation of Win2K. To repair a damaged or corrupt system, type R. You’re then asked whether you want to repair your system using the Recovery Console or the emergency repair process. Type R to use the emergency repair process.3. Choose the type of repair—You can choose either the Fast Repair option (type F) or the Manual Repair option (type M). The Manual Repair option requires that you make all the repair selections and determine whether you want to repair system files, the partition boot sector, or startup environment problems. I recommend that only advanced users and administrators use the Manual Repair option and that you use it only to repair one item at a time when you know the others are intact. For example, if you’re confident thatthe partition boot sector and startup environment are both intact, attempt to repair only the system boot files. (The Manual Repair option doesn’t let you try to repair problems with the Registry. To manually repair individual Registry settings and files or replace the entire Registry, use the Recovery Console. However, administrators should be the only ones who use it.)

For a more extensive discussion of the capabilities, commands, and use of the Recovery Console, checkout Chapter 12 of the free eBook The Definitive Guide to Windows 2000 Administration by Sean Daily andDarren Mar-Elia. You’ll find a link to it at www.realtimepublishers.com.

Page 196: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com184

The Fast Repair option uses a backup copy of the Registry that was created when Setup was first run or installed on your system. If you choose this option, you may lose settings or preferences you’ve created sincethe system was first installed unless you’ve created a System State backup. In this case, the system is restoredto that state.4. Start the repair process—If you have the required ERD (on a 1.44-MB floppy disk), which you created using the backup utility, and the original Windows 2000 Server CD-ROM, you can start the repair process.If you don’t have an ERD, the emergency repair process can attempt to locate your Win2K installation and start repairing as best it can, but it may not be able to fix everything. If you don’t have the ERD and the emergency repair process can’t fix your system, you can try to use the Recovery Console or try to re-install Win2K.5. Restart your computer—If the emergency repair process was successful, your computer should automatically restart, and the repair procedure is complete.

Developing a Backup and Restore Strategy for Active DirectoryAs the heart and soul of a Win2K network, AD is an essential component that must be available to applicationsand users at all times. Because it’s a replicated database, AD is vulnerable to the same kinds of problems that canplague any database. These problems include, but aren’t limited to:

•A corrupted or invalid database schema (which defines the structure of the database—what type of data it contains and how the data is arranged)

•Missing DNS records•Damaged or corrupted information•Accidental misconfiguration by an administrator.

In case one of these events occurs, it’s imperative that your disaster-preparation routine and disaster-recovery planinclude provisions for backing up and restoring AD.

Using Windows 2000 Setup Repair without an ERDIf you haven’t prepared an ERD, you can attempt to repair a downed server by rerunning the Setup programfrom the Windows 2000 Server CD-ROM. Assuming that it can locate the Win2K installation folder (improvingthe ability to locate the Win2K installation is one of the primary benefits of having an ERD), Setup might beable to repair the system using information saved at the time that you originally ran it to install the server. Thisinformation is saved in the %Systemroot%\Repair folder on the system partition. If this information is physicallyaccessible and intact, Setup can use it during the repair process. However, some system settings may be lostunder these circumstances; for more information about this, refer to the Win2K documentation.

Winternals Software Recovery ToolsIf you need more help than even Windows 2000’s Setup Repair mode and the Recovery Console canoffer, you may find a savior in the tools available from Winternals Software. Winternals makes a numberof top-notch tools (including ERD Commander 2000, Disk Commander, NTRecover, and NTFSDOS)designed to help you recover Windows NT and 2000 systems, including even those that are totallyunstartable because of corruption or misconfiguration. Winternals even makes one tool (Remote Recover)that makes it possible to perform over-the-network recoveries on computers that you aren’t physicallypresent with. You can find a listing and description of these tools at www.winternals.com.

Page 197: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com185

Backing Up Active DirectoryBecause the files that make up AD (including Ntds.dit, the primary database file) are continually in use on Win2Kdomain controllers, you can’t simply copy the AD database as you might a standard user data file. However, as dis-cussed earlier in this chapter (see "Maintaining System State Backups"), the Win2K backup utility provides a fea-ture for performing an online backup of AD. AD is backed up whenever you include System State data as part of abackup on a Win2K domain controller.

When you back up System State data on a domain controller, you’re backing up all AD data that exists on the serv-er (along with other system components such as the SYSVOL folder and the Registry). You cannot choose to backup or restore individual components of the System State data or of AD; it’s an all-or-nothing process. This isbecause of the dependencies among the components of the System State.

Although the Win2K backup utility can certainly do the job of backing up AD, most information technology (IT)shops prefer to use more robust third-party applications for their backup solutions. If your organization is using athird-party backup application, you need to make sure that you’ve purchased a Win2K-compatible version of theproduct. It must be capable of backing up AD or provide a separate agent component that can do so. WindowsNT 4.0–era versions of backup utilities are incapable of understanding AD’s format or backing it up, so for mostcompanies, migrating to Win2K necessarily means upgrading their backup software.

Restoring Active DirectoryAlthough the process of physically restoring the AD database on a Win2K domain controller from a backup isn’tlogistically complex, there are some important logical and architectural issues you need to take into considerationbefore performing any type of AD restore operation. On networks with more than one Win2K domain controller,remember that AD is automatically replicated and updated among domain controllers and thus doesn’t exist in onlya single location. This has implications for restoring AD. For example, you need to ask questions such as:

Third-Party Windows 2000 Backup UtilitiesThere are several excellent third-party Win2K backup utilities that are 100% compatible with backing upand restoring AD. The leaders in the Win2K backup arena include:- Backup Exec from Veritas (www.veritas.com)

- ARCserveIT from Computer Associates (www.cai.com)- UltraBAC for Windows NT/2000 from UltraBac Software (www.ultrabac.com)- ERDisk for Active Directory from Aelita Software (www.aelita.com)

You can back up AD in its entirety only using a full backup; you can’t back up AD incrementally (that is, bybacking up only object data that has changed since the last backup)..

As discussed in "Specifying Advanced Options" earlier in this chapter, the useful life of a backup of AD isusually only 60 days, so you may experience problems if you attempt to restore AD backup images olderthan this into a replicated Win2K network. The reason for the 60-day figure is that the useful life of a back-up is identical to the tombstone lifetime setting for the enterprise, and in Win2K, the default value for thetombstone lifetime entry is 60 days. However, you can change this value using the Directory Service (NTDS)config object. The value is the "tombstoneLifetime" attribute of the CN=Directory Service, CN=Windows NT,CN=Services, CN=Configuration, DC=COMPANY, DC=COM (for a domain called "company.com"). You can setthe value using the LDP or ADSI Edit utility..

Page 198: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com186

•Is just the local domain controller’s copy of AD corrupted or damaged, or are other replicas on other domain controllers also in the same state?•Is the data being restored the definitive copy that should be used to overwrite all other copies of AD objectdata? If so, are there changes or structural modifications that might be lost by restoring this copy of AD as a "master" copy?•Do you need to restore AD on a local domain controller only to regain functionality on that domain controller (for example, is the problem isolated to the local copy of AD on that computer), and should it then receive updates from other domain controllers using AD replication to bring its data store up to date?

The answers to these questions will help you determine which of the three AD restore modes to choose: non-authoritative, authoritative, or primary.

•Non-Authoritative Restore—A normal restore; most restore operations are run using this restore mode. You usually perform a non-authoritative restore when the problem is limited to the local Win2K domain controller and the AD replicas housed on other Win2K domain controllers are believed to contain valid replicas. During a non-authoritative restore, any data that you restore (including AD objects) retains its original update sequence number (USN). (AD uses the USN to detect and propagate any changes to the other domain controllers.) In turn, AD replication uses the USN to detect and propagate any changes to the other domain controllers in the same domain.•Authoritative Restore—Used when the other Win2K domain controllers contain invalid replicas or undesirable data. In this case, you manually designate the copy of the AD database that you restore. You designate only the local domain controller to be authoritative (that is, the "master" copy from which all other domain controllers seed their own AD replicas). An authoritative restore modifies the USN of each AD object that is being restored to the domain controller. This allows the USN of each object to be higher thanany of those that are currently on the domain controller. After all of the objects have been restored, they’re replicated to the other replicas.

You can use backup data from one domain controller to restore only to the same domain controller. Youcannot use the backup of one domain controller to restore another computer. However, if the domain con-troller’s system fails and never comes back, you can restore the backup data to another computer that willtake the place of the original domain controller. In addition, to completely back up your environment, youneed to have a backup of every domain controller on the network. Keep this in mind when developing yourbackup strategy. Also, because of the importance to the entire system of the domain controller that wascreated first in the root domain, you need to frequently back it up.

If you’re using Win2K’s backup utility (Ntbackup.exe) to perform a restore, be aware of the following addi-tional conditions, which must be met for the System State (including AD) to be successfully restored. If anyof these conditions isn’t met, the restore will fail.

- The server name must be identical to the backed-up server name.- The drive on which the %Systemroot% folder is located must be the same as when it was backed up.- The %Systemroot% folder must be the same folder as when it was backed up.- If SYSVOL or other AD databases are located on another volume, the databases must exist and be on the

same drive as when they were backed up.

Page 199: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com187

Non-authoritative RestoresIf the AD replicas on domain controllers other than the one you’re performing the restore on are intact and valid,you’ll probably want to perform a non-authoritative restore. A non-authoritative restore is a normal, or typical,restore of the AD objects to the domain controller that originally contained them.You restore AD by starting the Win2K domain controller in Directory Services Restore mode (discussed in "Safe

Mode" earlier in this chapter). When the system starts up, press F8 at the Windows 2000 Boot Loader menu, thenselect Directory Services Restore Mode from the alternate boot menu. Win2K starts in a safe mode, and you canfollow these steps to restore AD information on a domain controller:

1. Log on as a member of the Administrators or Backup Operators groups. I should also point out that the members are local computer users and groups, not domain users and groups. Also, users who are backup administrators (but not regular administrators) can’t run Ntdsutil to do things like authoritative restores. This requires a user with administrative privileges.2. Run the Win2K backup program. On the Welcome page, click Restore Wizard, then select the System State check box.If you restore the System State data and you don’t designate an alternate location for the restored data, the backup utility erases the System State data that is currently on your computer and replaces it with the System State data you’re restoring. If you restore the System State files to an alternate location, the AD database, the Certificate Services database, and COM+ Class Registration database aren’t restored.3. Once the restore is complete, restart the domain controller.

After the non-authoritative restore is complete, the restored data (which may be out of date) becomes synchro-nized. Once you restart the domain controller, it should begin participating in AD replication and receive directoryupdates from the other domain controllers.

You can use a non-authoritative restore if the domain controller fails or the entire AD database is corrupt becauseyou can simply restore the entire system data non-authoritatively. In more technical terms, this means that a non-authoritative restore retains the original USN.

A non-authoritative restore provides a start point (the time at which the data was backed up) for data replication.This minimizes the replication traffic on the network because only changed data is replicated rather than the entiredirectory. In the absence of this start point, all data would be replicated from other servers.

While it’s possible to back up AD either online (while the directory services are running) or offline (whenthe services are stopped), you can restore AD only when the directory services are offline.

Another option for restoring a Win2K domain controller is to simply re-install Win2K and reconfigure thesystem as a domain controller on its domain. You can give the domain controller a different name, or ifyou rebuild it with the same name, you need to first remove the old domain controller from the domain.After you’ve done this, the normal AD replication process repopulates the domain controller with currentdirectory information.

Page 200: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com188

Authoritative RestoresAn authoritative restore allows you to recover a domain controller, restore it to a specific time, and mark objects inAD as being authoritative with respect to their replication partners. For example, if an administrator inadvertentlydeletes an organizational unit (OU) containing a large number of users, you can perform an authoritative restore torecover the information and mark it as the source to force replication to the other domain controllers.

By definition, an authoritative restore replicates any changes made to the current data set to its outbound replica-tion partners. To be the authoritative source for the restore, the authoritative restore modifies the USN of the ADobjects that are being restored to the domain controller so that each object has a higher value than any of those thatare currently on the domain controller. This forces the restored objects to be replicated to the other replicas of thesame domain controller.

Such a restore is unusual, but it has the effect of rolling back all of the AD objects in the domain controller to thetime of the original backup. You can do this to restore erroneously deleted information of a replicated set of data.For example, if you inadvertently delete or modify objects stored in AD, you may want to authoritatively restorethem so that they can be replicated again to the other domain controllers. If you don’t authoritatively restore theseobjects, they’re never replicated to the other domain controllers in the same domain because they appear to be olderthan the objects currently on your domain controller.

To help you accomplish an authoritative restore, you can use the Ntdsutil utility to mark the target objects forauthoritative restore. This ensures that the data you want to be restored is replicated to the appropriate domaincontrollers after the restore occurs. Table 6.3 describes the commands in Ntdsutil available to perform an authorita-tive restore.

This Command Does This

Authoritative restore(main menu option)

Allows you to use the authoritative restore submenu options listed below. Youcan use this option only on a domain controller that is operating in DirectoryServices Restore mode.

Restore database (submenu option)

Marks the entire AD database (Ntds.dit) as authoritative.

Restore database verinc %d(submenu option)

Marks the entire AD database (Ntds.dit) as authoritative and increments theversion number by %d. Use this option only to authoritatively restore over aprevious sequential authoritative restore.

Restore subtree %s(submenu option)

Marks a subtree (all objects in the subtree) as being authoritative. The subtreeis defined by using the distinguished name (DN) of the OU object.

Restore subtree %s verinc %d(submenu option)

Marks a subtree (all objects in the subtree) as being authoritative and incre-ments the version number by %d. The subtree is defined by using the DN ofthe OU object. Use this option only to authoritatively restore from a backupthat contains the objects you want to restore over.

Table 6.3: The Ntdsutil authoritative restore commands.

Page 201: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com189

To perform an authoritative restore of AD on a specific domain controller:

1. Choose Start>Run or open a Command Prompt session.2. Type Ntdsutil, then press Enter.3. At the NTDSUTIL prompt, type Authoritative Restore, then press Enter. (This puts Ntdsutil into

Authoritative Restore mode.)4. At the Authoritative Restore prompt, to set the entire database as authoritative, type Restore Database.

ORTo set only a subtree of the database as authoritative (for example, an individual OU), use the Lightweight Directory Access Protocol (LDAP) DN that identifies the portion of AD being authoritatively restored. For example, to authoritatively restore the Engineering OU in theREALTIMEPUBLISHERS.COM domain, type the following:

RESTORE SUBTREE "OU=ENGINEERING,DC=REALTIMEPUBLISHERS,DC=COM"

5. When you’re prompted to confirm the authoritative restore, answer YES.6. Type Quit, then press Enter twice to return to the command prompt.7. Close the Command Prompt session.

Once AD is restored, be certain to answer No to the option to restart the server. This step is critical because therestore will otherwise be non-authoritative when the server restarts, and you’ll risk re-inheriting unwanted datafrom other AD replicas.

Always authoritatively restore SYSVOL whenever you authoritatively restore AD and vice-versa. Thisensures that SYSVOL and AD stay synchronized.

There are a few potentially negative consequences of authoritative restores that you should be aware of.One such effect relates to trust relationships and computer account passwords. Both of these are automat-ically negotiated at a specified interval (every seven days, by default, except for computer accounts thatcan be disabled by the administrator). During an authoritative restore, a previously used password for theobjects in AD that maintain trust relationships and computer accounts can be restored. In the case of trustrelationships, this could void communication with other domain controllers from other domains. With com-puter account passwords, this could void communications between the member workstation or server and adomain controller of its domain. In this case, you may have to manually reestablish trusts to resume propercommunications.

Restoring FRS DataI want to make a point about using Win2K’s backup utility to restore replicated FRS data sets as part of restoring adomain controller. If you want to mark the restored data as the primary data for all replicas, you need to set a spe-cial option that ensures that restored FRS data is replicated to your other servers. Specifically, when you restoreSystem State data, select the Advanced option to access the Advanced Restore Options dialog box, then selectWhen Restoring Replicated Data Sets, Mark the Restored Data As the Primary Data for All Replicas. If this domaincontroller is a member of FRS replica sets other than the SYSVOL replica set, those other replica sets will also berestored as authoritative. If you want to restore only the SYSVOL replica set, select the option in the backup utility;then, after the restore is complete, delete the other replica sets.

If you don’t select this option, the FRS data that you’re restoring may not be replicated to other servers because therestored data will appear to be older than the existing data. The other servers will overwrite the restored data, andthis will effectively prevent you from restoring the FRS data. Third-party backup applications also provide similaroptions for performing primary restores.

Page 202: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com190

Verifying RestoresOnce you’ve restored AD, you can verify that it was successful. You can do this using either advanced verification orbasic verification. You must perform basic verification; advanced verification is optional, but you must run it beforeyou run basic verification.

AdvancedAdvanced verification is optional; it isn’t usually required for normal recovery operations. You run it regardless ofwhether you performed an authoritative restore. If you do run it, you must do so before you run basic verification.Incorrectly using the Ntdsutil utility may corrupt the AD database, and you’ll have to restore the database frombackup again.

To perform advanced verification:

1. Make sure that you’re in Directory Services Restore mode.2. Choose Start>Run.3. In the Open dialog box, type Regedit, then click OK.4. In REGEDIT, select the Registry key

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NTDS and check that it contains a subkey called Restore In Progress. (This key is automatically generated by the Backup utility during the restore and indicates to AD that the database files have been restored and need to be checked and re-indexed the next time AD is started. When this check is completed, this key is automatically removed; don’t manually add or delete it.)

5. Close REGEDIT.6. To check for the recovered AD database files, start Ntdsutil by typing Ntdsutil at a command prompt or

in the Start>Run dialog box.7. At the NTDSUTIL prompt, type Files. At the File Maintenance prompt, type Info. (If the AD files have

been recovered successfully, you can see the difference. Don’t select any other options.)8. To exit File Maintenance, type Quit. To exit Ntdsutil, type Quit. To exit the DOS prompt, type Exit.9. Restart the server in Normal mode. Log on to the system normally and perform basic verification (see

below).

BasicBasic verification consists of rebooting and logging on normally, then confirming that the restored services are in astate consistent with a successful restore. It also includes verifying that FRS and Certificate Services were successful-ly restored.

To perform basic verification:

1. Restart the computer. (AD will automatically detect that it’s been recovered from a backup, perform an integrity check, and re-index its database. Both AD and FRS will be brought up to date from their replication partners using the standard replication protocols for each of those services.)

2. Confirm that distributed services successfully restored. (You should be able to browse AD and confirm that all of the user and group objects that were present in the directory before the backup were restored.)

3. Confirm that files that were members of an FRS replica set and certificates that were issued by the Certificate Service are present.

Page 203: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com191

The Active Directory Backup BugThere was a fairly serious bug in early versions of Win2K. This bug makes Win2K-based domain controllers unableto start in AD mode after you restore System State backups that were created before installing Windows 2000Service Pack 2. Restoring these backups to a Win2K domain controller leaves the directory services unable to start,and the system records a number of errors in the system Event Log. The problem affects all applications that usethe backup APIs to perform AD backups; this includes System State backups performed with Win2K’s built-inbackup utility as well as most third-party backup applications.

Affected backups are corrupted in such a way that when they’re restored, they prevent the domain controller fromstarting and cause it to display a "Directory Service cannot start" error message. Also, if you run Ntdsutil using theSemantic Database Analysis option to run the database semantic checker, you receive error message 550: "Databaseis inconsistent." With the backup utility, the problem happens even when the Verify option is turned on.You can avoid these problems by installing SP2, then making new backups of the System State. The fix is preventa-tive only; it doesn’t resolve errors that occur if you restore System State backups that contain incorrect header infor-mation.

How It OccursAs mentioned earlier in this chapter, you prepare for AD disaster recovery by making System State backups fromthe console of Win2K-based domain controllers at regular intervals. The elements of AD that are captured in aSystem State backup include the AD database (Ntds.dit), transaction logs (Edb*.log), and a patch file (Edb.pat).You restore by starting Win2K domain controllers in Directory Service Repair mode and restoring the System Stateusing the Win2K backup utility, Ntbackup.exe, or a third-party equivalent. After performing the restore operation,you can optionally use Ntdsutil.exe to mark specified domain name paths or subtrees as authoritative when thedomain controller next starts in AD mode.

The following conditions also contribute to this problem:

•An initial backup of AD is performed on a Win2K domain controller. During the backup, a number of objects change as a result of either local changes or replication. The changes to these objects generate additional transaction logs, which in turn advance the Joint Engine Technology (Jet) database checkpoint. TheJet checkpoint maintains a list of unflushed data in the database. Two copies of the checkpoint data are stored: One in the database header of the Ntds.dit file and a second in-memory copy, which is written to the backup media.

•A second backup is performed on another, relatively inactive domain controller. During the backup, the log files aren’t generated, and the Jet checkpoint isn’t advanced. This second backup completes before the log files are generated and the Jet checkpoint advances in the first backup.

•The second backup is then restored.

In order for the problem to manifest itself, the new transaction logs and Jet checkpoint advancement that occurduring the first backup must happen after the second backup completes. As a result, a relatively large first backup ismore likely to produce the problem because there is a commensurately larger window of time for the second back-up to complete (and for the Jet checkpoint to advance). Domain controllers in busy production environments areless likely to experience this problem during typical activity (creating, deleting, and modifying objects) becausethese activities in AD result in a steady advance of the Jet checkpoint.

Page 204: The Definitive Guide to Active Directory Troubleshooting

Chapter 6 www.netpro.com192

The problem is more likely to occur in large backups (or when the backup media doesn’t have a fast backup rate)because the backup process takes longer and there is more opportunity for the checkpoint file to advance. Theessential problem is that a second backup is made before the checkpoint file advances, then the second backup isrestored.

The result of all this, and the core of the problem, is that an outdated record of required transaction log files andcheckpoint data is written to the backup media, then later restored as the second backup. The header in therestored database references logs that aren’t actually required for the AD recovery and that aren’t all included in thebackup. This explains why log entries appear stating, "Log files are missing from System State." However, suchentries are misleading because it isn’t a case of the log files being missing; instead, the number of log files referencedin the restored database header is incorrect.

Working Around ItIf you use backups as a method of recovery for Win2K-based domain controllers, you may want to consider doingthe following to work around the bug:

•Inventory, then clearly label backups that were made before installing SP2. Place pre-SP2 backup media in locked storage. Don’t forget backup media that are stored on the local drives of computers in your organization.

•Consistent with good change-management practices, install SP2 on domain controllers in a lab environment that is representative of your production configuration. Make multiple backups, then initiate restore tests.

•Install SP2 on production domain controllers. Create new System State backups and clearly label them as post-SP1 backups.

•Destroy pre-SP2 backups.

Repairing a Domain Controller in Active DirectoryIf a Win2K domain controller fails, you have several options for repairing it. You might need to use more than oneor even all of them.

•Use the ERD to either prepare a set of disaster-recovery disks, or use a set of previously established disks torepair specific components of a domain controller so that it can successfully start up. Log on using an account that has Administrator or Backup Operator privileges.

•Re-install the Windows 2000 Server OS and run the Active Directory Installation Wizard. If a major hardware failure or malfunction requires the computer to be completely rebuilt, you may need to re-install the OS. After the computer is running, reconfigure the original network connections and DNS settings.

•If you need to remove a domain, run the Active Directory Installation Wizard to remove AD from all domain controllers that you’re removing. Then use the NETDOM utility to remove the domain itself (including cross-reference and trusted domain objects). To do this, type the following at the command prompt:

netdom trust /remove /force

•To clean up metadata that has been left behind by decommissioned or failed domain controllers, you can use Ntdsutil with the CLEANUP command. This operation removes the defunct domain controller’s identification and information from AD.

Page 205: The Definitive Guide to Active Directory Troubleshooting

SummaryThe guidelines for backing up and restoring Win2K servers and domain controllers include information forWin2K and AD. You can use the Win2K backup utility to back up and restore data and perform emergency recov-eries. To back up and restore a failed server to an operational state, you use the backup utility’s Backup and Restorewizards.Alternatively, you can back up and restore selected user files and folders on the domain controller’s harddrive.

You can back up and restore the domain controller’s System State—the files central to system operation; theyinclude the Registry, AD and SYSVOL, the COM+ Class Registration database, and boot files. You can also sched-ule regular backups and create an ERD that helps you repair system files should your system become corrupted.

Chapter 6 www.netpro.com193

eBook Copyright NoticeThis site contains materials created, developed, or commissioned by Realtimepublishers.com, Inc. and is protected byinternational copyright and trademark laws. No material (including but not limited to the text, images, audio, and/orvideo) may be copied, reproduced, republished, uploaded, posted, transmitted, or distributed in any way, except that onecopy may be downloaded for your personal, non-commercial use on a single computer. In connection with such use, youmay not modify or obscure any copyright or other proprietary notice. If you have any questions about these terms, or ifyou would like information about licensing materials from Realtimepublishers.com, please contact us via e-mail [email protected]

Page 206: The Definitive Guide to Active Directory Troubleshooting

Without Sean Daily, The Definitive Guide to Active Directory Troubleshooting wouldn't be a definitive guide at all.Sean has been running Active Directory since early beta, and he knows it inside and out. Check out his beefy biobelow! And, don't miss the backgrounder on the company he writes for. NetPro's ebook is the second comprehensiveguide published by Realtimepublishers.com that's free and available on the web - and it won't be the last!

Sean Daily is a world-renowned expert on Windows NT/2000 and a Senior Contributing Editor at Windows 2000Magazine. In addition to being the author of numerous books, including The Definitive Guide to Windows 2000Administration (Realtimepublishers.com) and Optimizing Windows NT (IDG/Hungry Minds), Sean speaks andconsults internationally on Windows NT/2000 and related technologies. Sean also serves as Series Editor forRealtimepublishers.com's Definitive Guide and Tips and Tricks Guide series of ebooks.

About the Tech EditorNetPro's own Chief Scientist, Gil Kirkpatrick, performed the technical edits for the ebook. Kirkpatrick is an expertin the design and development of large-scale distributed software for enterprise networks and has over 24 years ofindustry related experience. He is a recognized authority on commercial network directories, including Banyan'sStreetTalk, Novell's NDS eDirectory and Microsoft's Active Directory. In his previous position as NetPro's Directorof Engineering, Kirkpatrick was responsible for the development of NetPro's directory management products andserved as the architect and lead engineer for NetPro's flagship product, DirectoryAnalyzer. Kirkpatrick joinedNetPro in 1994 when NetPro acquired his company, High Technology Systems. Kirkpatrick is also the author ofActive Directory Programming, published by MacMillan USA.

The Definitive Guide to Active Directory Troubleshooting About the Author

About the Author