Multi-Tenant Scalability Guidance for

Exchange Server 2013

Customer Advisory Team Exchange Product GroupMicrosoft Corporation

August 2013

ContentsIntroduction.................................................................................................................................................3

Scalability Guidance Conclusions.................................................................................................................3

Tenant Scalability....................................................................................................................................3

Modern Public Folders.............................................................................................................................4

Limits on Exchange Transport Rules and Offline Address Books.............................................................4

General Recommendations for High-Scale Deployments........................................................................5

AD Optimizations.....................................................................................................................................5

Exchange Server 2013 Multi-Tenant Scalability Testing Methodology........................................................6

Goals........................................................................................................................................................6

Testing Script...........................................................................................................................................6

Scale Testing Lab Setup...........................................................................................................................7

Distribution of Virtual Server Images on Hyper-V Root Servers...........................................................7

Configuration of Virtual Server Images (RAM, CPU, Disk)....................................................................8

Test Results at Measurement Points.......................................................................................................9

Results at Scale Target.............................................................................................................................9

Validation with Simulated Client Load.....................................................................................................9

Exchange Logging Sizing Observations..................................................................................................10

AD Sizing Observations..........................................................................................................................10

Summary...................................................................................................................................................10

Legal Notice...............................................................................................................................................11

2

Introduction This document describes Microsoft recommendations for scaling a multi-tenant deployment of Exchange Server 2013 using the methods described in the Multi-Tenancy and Hosting Guidance for Exchange Server 2013 white paper. Microsoft Exchange Server 2010 Service Pack 2 (SP2) introduced Address Book Policies (ABP), a feature which allows segmentation of directory information. The ABP feature, along with other supported configuration, continues to provide the platform for deploying a custom multi-tenant environment using Exchange Server 2013. For the purposes of this document, a tenant is a hosted organization which has been configured with a set of administrative policy objects, such as an ABP, along with a set of users.

In the Multi-Tenancy and Hosting Guidance for Exchange Server 2013 white paper Microsoft describes the caveats and requirements to implement a multi-tenant solution using Exchange Server 2013. This document provides additional guidance on the scalability of specific areas of concern with Exchange 2013. This document does not attempt to cover all potential scalability limits in the product, and any high-scale deployment should proceed with necessary caution during the planning, testing, deployment, and scaling phases. The testing that was performed in Microsoft’s lab is described here to assist you in adding context to the white paper’s recommendations.

Scalability Guidance ConclusionsTenant ScalabilityScale testing performed in Microsoft’s lab has demonstrated that Exchange 2013 supports up to 50,000 tenants configured in a single Active Directory (AD) forest. Microsoft recommends not exceeding 50,000 tenants in a single forest. Customers who need to deploy capacity for greater than 50,000 tenants should design a multi-forest architecture with a provisioning system that is capable of supporting multiple Exchange capacity forests. Additionally, you may need to reduce this per-forest limit significantly if you plan to provision Exchange transport rules, or the Offline Address Book (OAB) for each of your tenants. See the section below titled Limits on Exchange Transport Rules and Offline Address Books for additional information on these limitations.

Each of the tenants were created and given the following objects (additional details of this configuration are available in the testing section later in this document):

A tenant organizational unit (OU) in AD.

Two universal security groups (USG), one for the tenant’s admin; one for all of the tenant’s users.

An accepted SMTP domain.

A global address list (GAL).

An “All Rooms” address list.

An “All Users” address list.

An “All Contacts” address list.

An “All Groups” address list.

An ABP.

Three mailbox-enabled users: one admin and two users.

Two mail-enabled contacts

Many multi-tenant messaging deployments contain a large number of tenants and a small number of users per tenant. This testing was designed to evaluate the scalability of an Exchange 2013 solution in

3

this type of configuration, specifically validating the overall number of tenants that could be successfully provisioned in a single Exchange organization without focusing on the overall number of users configured per-tenant or globally within the system. Administrators interested in deployments which contain a large number of users per tenant can also benefit from this guidance, however they should pay careful attention to the overall impact of the total number of users on the scalability of the system as that may become a more limiting factor than the total number of tenants.

Modern Public FoldersA modern public folder hierarchy was created for 10% of the tenants (5,000 tenants total). Testing to this level indicated we would have been able to go higher as there was no degredation observed in any of the aspects of this part of the test. Note that this testing focused primarily on the creation and segregation (access rights) of the folder hierarchy, and did not attempt to test large volumes of data posted within the public folders. If you choose to offer modern public folders to your tenants, you will need to perform additional testing to insure your service can scale to the limits and quotas you offer your tenants. Additional guidance can be found in the Exchange Server 2013 Public Folders TechNet topic.

In this scale lab, eight public folder mailboxes were created, one on each of the mailbox databases that existed in the scale lab. Throughout the scale testing the creation of new public folders, as well as the setting of access rights, the replication of the public folder hierarchy; and the posting of new messages occurred successfully and without any measurable degredation in the time it took to accomplish each of these tasks.

The public folder hierarchy was created with the following access rights:

RESELLER: The empty Reseller folder is owned by the hoster’s administrator. The default access right was set to FolderVisible*.

TENANT: Only the sole Tenant administrators were given the Owner role and that tenant’s users were given the FolderVisible access right. Default access rights were set to None to hide this folder, and its subfolders, from all other tenants.

SUB FOLDERS: These folders inherited access rights from the Tenant parent folder. Then additional rights were assigned to give control of the folder and its contents to the tenant’s users.

Testing confirmed that the Tenant-level folders were segregated and viewable only by the respective tenant, that e-mail to the child folder posted successfully, and that edits, deletes and the creation of additional subfolders behaved as expected based on the roles and access rights assigned to each folder.

*Note: If you offer Public Folders to your tenants, you should be on Cumulative Update 2 (CU2) for Exchange Server 2013. There is an issue on earlier versions where you are unable to set explicit access rights when creating a new public folder. The new folder will replicate immediately with only inherited rights, and any explicit rights you set will miss the first replication and will take effect only after the next replication, which may be hours. This was fixed in CU2 by allowing explicit permissions to be set at the same time you create a new public folder.

Limits on Exchange Transport Rules and Offline Address BooksBased on testing performed for the creation of this document, as well as sizing and scalability of internal hosting deployments, Microsoft recommends not exceeding 5,000 transport rules or 25,000 OAB’s per forest.

Transport Rule behavior: Functional issues were observed in testing where the efficiency of the transport service degraded as additional rules were created above 5,000. High CPU consumption by the EdgeTransport.exe process was observed; as well as low sustained rates of message processing; and the slow creation of new transport rules. Given this limitation, if customer requirements involve one or more transport rules per tenant, then the number of transport rules

4

would limit the number of tenants that could be hosted in a given Exchange organization (or AD forest).

Offline Address Book (OAB) behavior: A functional issue was observed above 25,000 OABs that would cause Outlook client downloads to fail. Given this limitation, if customer requirements involve one or more OAB’s per tenant, then the number of OABs will limit the number of tenants that could be hosted in a given Exchange organization (or AD forest).

General Recommendations for High-Scale DeploymentsMicrosoft provides generic performance and scalability guidance for Exchange 2013 in the Scalability section of the Multi-Tenancy and Hosting Guidance for Exchange Server 2013. The sizing process for high-scale hosted deployments requires careful attention to domain controller scalability because hosted deployments tend to be associated with extremely large directories. The test results quoted later in this document provide a sense of AD database growth as the number of test tenants increased.

Sizing should be performed with an understanding of user concurrency. Rather than sizing for the number of provisioned tenants and users, the deployment should be sized for the maximum concurrent number of users expected on the system. Historical user concurrency should be measured from an existing hosted Exchange deployment, or if transitioning from another messaging platform, equivalent measurements for the supported messaging protocols can be used as a starting point. Without a clear understanding of user concurrency to base sizing calculations on, the safest option is to size for 100 percent concurrency per the Microsoft recommendation for a typical enterprise deployment.

The deployment should be sized in such a way that additional capacity of a given physical resource can be quickly provisioned to handle unanticipated increases in user demand. Microsoft recommends deploying a significant amount of extra capacity during the initial deployment as a buffer to handle these unanticipated increases, given the dynamic nature of your customer base. As with any high-scale deployment, Microsoft recommends lab testing prior to production deployment to ensure that the specified hardware is capable of meeting the workload requirements. Given the extreme scale of hosted deployments, it may be difficult or impossible to perform lab validation of an entire forest simultaneously, so Microsoft recommends that the design include a “building block” or “scale unit” approach in which a subset of the hardware can be validated as a unit and multiple units can be put together to build the forest.

AD OptimizationsLarge-scale hosted Exchange deployments may see a significant volume of authentication requests on the Client Access (CAS) servers, especially during peak periods when many users are trying to logon concurrently. The authentication configuration should be evaluated to help ensure that the authentication process works properly in a hosting deployment:

1. When client protocols use NTLM or basic authentication, the Netlogon service on the server that the credentials were presented to will be responsible for validating the credentials via an AD domain controller. A limited number of threads are available for concurrently processing these authentication request. The number of threads is adjustable via the MaxConcurrentApi registry key, as described in the Microsoft Knowledge Base article, 975363 -- You are intermittently prompted for credentials or experience time-outs when you connect to Authenticated Services.

The default number of threads available is dependent on the server operating system used for your CAS and domain controller servers. For example, Windows Server 2008 R2 and prior versions default to two threads, and can be increased to a maximum ten threads. Windows Server 2012 uses a default of ten threads, and this default was used in all scale testing covered in this document. Windows Server 2008 R2 with SP1 and Windows Server 2012 can be increased to a maximum of 150 threads. The impact of tuning the MaxConcurrentApi key may be

5

monitored via the Netlogon performance counters described in the Knowledge Base article, 975363.

2. This applies to Windows Server 2008 only: Kerberos authentication may also be improved via application of the hotfix described in Knowledge Base article 2545833 -- Slow performance occurs when many user authentication requests are handled in Windows Server 2008 R2.

Exchange Server 2013 Multi-Tenant Scalability Testing MethodologyGoals Functionality tests were carried out at defined points along the way as the number of tenant organizations grew. The principal goal of this scale testing was to incrementally provision up to 50,000 tenant organizations, each with a set of users, a unique ABP assigned to its users, one accepted domain, one transport rule, one OAB, etc. Then we determined if there were any performance bottlenecks, scalability limitations, lessons-learned, or other impacts to client functionality at that scale. Having said this, it is important to note that tenants who are configured with multiple accepted domains* per tenant, or need to assign transport rules and/or OAB’s to every tenant; this will impact the overall sizing of the Exchange deployment, as discussed earlier in this document.

*Note: In this scale testing only one accepted domain was created per tenant, for a total of 50,000 accepted domains. However, 150,000 accepted domains is the maximum recommended by Microsoft, as discussed in the Multi-Tenancy and Hosting Guidance for Exchange Server 2013 white paper.

Testing ScriptThe test team deployed a virtualized hosted lab environment of four Exchange Server 2013 Enterprise servers. Tenant organizations were provisioned via a custom Exchange Management Shell script. Each tenant organization included the following:

A tenant OU in AD.

Two global security groups.

An accepted SMTP domain.

A GAL, filtered by tenant [based on CustomAttribute1].

An “All Rooms” address list, filtered both by tenant (as above) and a RecipientDisplayType of “ConferenceRoomMailbox”.

An “All Users” address list, filtered both by tenant (as above) and ObjectClass=user.

An “All Contacts” address list, filtered both by tenant (as above) and ObjectClass=contact.

An “All Groups” address list, filtered both by tenant (as above) and ObjectClass=group.

An OAB for the first 25,000 tenants, based on the tenant organization’s GAL.

A single shared OAB that contained a single contact (hoster’s Support Desk, for example) which was assigned to all tenants that did not have their own unique OAB.

An ABP, with the tenant’s GAL, address lists, and OAB (if it existed) linked to it.

6

Three mailbox-enabled users, with their CustomAttribute1 set to the tenant name.

Two mail-enabled contacts, with their CustomAttribute1 set to the tenant name.

Ten percent of tenants (5,000 total) were given a Public Folder hierarchy, with access rights such that these folders were viewable by the tenant’s users only.

Mailboxes were evenly distributed across all databases; of which there were two databases per server for a total of eight databases on four multi-role servers. All databases were members of a two-node database availability group (DAG).

After all the above had been provisioned, the tenant’s OAB was updated using the Update- OfflineAddressBook PowerShell cmdlet, and a new transport rule was created based on a “SubjectContainsWords” filter. For reasons described earlier in this document and shown below in Table 2, OAB creation was stopped at 30,000 and OAB’s were then reduced to 25,000 total for the remainder of testing. Also transport rule creation was stopped at 10,000 rules and were then reduced to 5,000 total for the remainder of testing.

Milestone breakpoints were defined at the following tenant organization counts: 1,000; 5,000; 10,000; 20,000; 25,000; 30,000; 40,000; and 50,000. At each point, the following information was gathered:

Average end-to-end time to fully provision a tenant organization Average time for each provisioning cmdlet to run.

At each break point, the following functionality tests were run:

Outlook Web App Test: Login to a new tenant user’s mailbox (for example, user01@tenant1.com) using Outlook Web App

o Open the Address Book feature in Outlook Web App. Verify that only those address books assigned by the tenant’s ABP are visible. Verify each of those address lists are populated as expected.

o Send an e-mail to another user in the same tenant (for example, user02@tenant1.com).

Outlook Test: Create an Outlook 2010 user profile for another user in the same tenant (for example, user02@tenant1.com)

o Verify that profile creation succeeds.o Open the mailbox in Outlook 2010.o Open the Address Book feature in Outlook.

Verify that only those address books assigned by the tenant’s ABP are visible. Verify each of those address lists are populated as expected.

o Verify that the Outlook client can download the tenant OAB.o Switch to ‘offline’ mode and verify that the OAB is populated correctly.o Verify that the e-mail sent from Outlook Web App has arrived.o Reply to the e-mail from the previous test and verify that the replies appear in both the

Outlook Web App and Outlook clients.

Scale Testing Lab Setup Distribution of Virtual Server Images on Hyper-V Root ServersThis section provides an overview of the number and types of servers in the scale testing lab, and shows how they were distributed to Windows Server 2012 Microsoft Hyper-V host servers.

7

Distribution of Virtual Server Images on Windows 2012 Hyper-V Host Servers

Configuration of Virtual Server Images (RAM, CPU, Disk) Table 1 shows the hardware configuration of both the Hyper-V host (physical) servers, as well as the RAM and CPU allotments to the Hyper-V virtual server guests. Each of the four Exchange 2013 servers (MBX-CASx) were deployed as a CAS and Mailbox combined multi-role server. Don’t use this table as a reference architecture, it is simply a definition of a hypothetical test topology. The disk, CPU and memory sizing represented here are not designed to match the needs of any specific production workload and indeed a real world environment with thousands of tenants and a corresponding number of users would quite likely require an increase in hardware resources, above those that are listed below.

Table 1 – Hardware configuration of Hyper-V root (physical) servers with guest image allotmentsHyper-V physical host Guest virtual server name Guest cores Guest RAMHyper-V Host1: 8 Cores 2.27GHz Intel Xeon, 32GB RAM, 13-drive RAID 6 Array

AD1 8 16 GB

Hyper-V Host2: 8 Cores 2.4GHz Intel Xeon, 32GB RAM, 13-drive RAID 6 Array

AD2 8 16 GB

Hyper-V Host3: 8 Cores 2.27GHz Intel Xeon, 48GB RAM, 13-drive RAID 6 Array

AD3Win 8 Client with Outlook 2013

62

16 GB4 GB

Hyper-V Host4: 8 Cores 2.4GHz Intel Xeon, 32GB RAM, 13-drive RAID 6 Array

MBX-CAS1 8 16 GB

Hyper-V Host5: 8 Cores 2.27GHz Intel Xeon, 32GB RAM, 13-drive RAID 6 Array

MBX-CAS2 8 16 GB

8

Hyper-V Host6: 8 Cores 2.27GHz Intel Xeon, 32GB RAM, 13-drive RAID 6 Array

MBX-CAS3 8 16 GB

Hyper-V Host7: 8 Cores 2.4GHz Intel Xeon, 32GB RAM, 13-drive RAID 6 Array

MBX-CAS4 8 16 GB

Test Results at Measurement PointsThis section presents the performance benchmarks at each breakpoint of the scale testing.

The performance benchmarks listed in Table 2 were captured during maximum provisioning load; in other words, tenant organizations (including mailbox-enabled users and mail-enabled contacts) were being created as quickly as possible.

Table 2 – Performance benchmarks at each phase of the scale testing

Total tenants provisioned

Total transport rules created

Total OABs created

Average time to provision tenant

Outlook Web App Test

Outlook Test

1,000 1,000 1,000 9 seconds Success Success5,000 5,000 5,000 30 seconds Success Success10,000 10,000* 10,000 74 seconds* Success Success20,000 5,000 20,000 64 seconds Success Success25,000 5,000 25,000 74 seconds Success Success30,000 5,000 30,000* 88 seconds Success Failed**40,000 5,000 25,000 109 seconds Success Success50,000 5,000 25,000 135 seconds Success Success

* After the test environment had scaled to 10,000 tenants, including 10,000 transport rules, the transport service continued to service requests, however the transport service’s ability to process at scale was diminished. It was determined that a limit of 5,000 transport rules should not be exceeded in a single Exchange Organization (or AD Forest). The script was updated to not create additional transport rules and rules already created above 5,000 were removed.

**After the environment had scaled past 25,000 tenants and OAB’s, OAB creation began to fail and Outlook could no longer reliably download the OAB. It was determined that a limit of 25,000 OAB’s should be the limit in a single Exchange organization (or AD Forest). All other Outlook tests were successful. The script was updated to not create additional OAB’s, and OAB’s already created above 25,000 were removed.

Results at Scale TargetAfter reaching 50,000 tenant organizations and 150,000 total mailboxes, the environment was stable and demonstrated acceptable performance. The Outlook Web App and Outlook tests succeeded with no issues. No other scalability or performance issues were encountered.

Validation with Simulated Client LoadAfter the target scale of 50,000 tenant organizations had been reached, load testing was performed against a single virtualized Client Access server to ensure all features remained functional with client load at target scale.

Load testing consisted of Outlook Web App load, which was generated using the Microsoft Exchange Load Generator (LoadGen) tool. The first test pass of the Outlook Web App load consisted of 500 concurrent Outlook Web App sessions for 8 hours, running the Outlook Web App 2013 Enterprise script included with the LoadGen tool. The second test pass of Outlook Web App load consisted of 2,000

9

concurrent Outlook Web App sessions for 8 hours, again running the Outlook Web App 2013 Enterprise script.

Performance was evaluated after the load testing was complete, and no abnormal issues were found in the system.

The load simulation wasn’t designed to simulate realistic client load at scale. It was simply designed to ensure that the system remained functional with a lite amount of client load after the system had been provisioned to the target level of tenant scale.

Exchange Logging Sizing ObservationsNew to Exchange 2013 is the default-enabled logging of various Exchange components, including the capture of seven days of performance logs, as well as logs that capture the installation and configuration of Exchange. These log files are kept in the %ExchangeInstallPath%\Logging folder. During scale testing we captured the growth of this folder, the results of which are shown below in Table 3.

AD Sizing ObservationsAs the number of tenant organizations grows, the number of objects stored in AD grows. In turn, this increase results in an increase in disk space consumed by the AD database file (NTDS.DIT), as shown in Table 3. To assist with capacity planning, the NTDS.DIT size was captured during this scalability testing process. The actual NTDS.DIT size of a given Exchange deployment may vary greatly depending on total number of objects and utilization of various attributes. For example, storage of certificates, deployment of third party tools and/or additional Microsoft products which use the AD database as a configuration store.

Table 3 – Exchange Logging and AD NTDS.DIT sizing observations

Number of Tenant Organizations

Size of NTDS.DIT Size of Exchange Logging Folder

1,000 tenants 184 MB 2.1 GB5,000 tenants 694 MB 3.38 GB

10,000 tenants 1.33 GB 6.19 GB20,000 tenants 2.60 GB 12.57 GB25,000 tenants 3.20 GB 13.23 GB30,000 tenants 3.83 GB 13.87 GB40,000 tenants 5.64 GB 15.14 GB50,000 tenants 6.32 GB 16.40 GB

The total number of objects stored in the AD database may be a limiting factor on the scalability of a given multi-tenant Exchange forest because the size of the database may impact the time required to on-board new domain controllers, backup domain controller databases, and may also impact the ability of AD to successfully replicate changes throughout the AD topology. For more information about AD scalability limits, see Active Directory Maximum Limits - Scalability. Exchange forests which will be designed to grow to extremely large user counts will require additional pre-deployment testing to ensure that the design is achievable given anticipated rate of AD object churn, requirements for time to on-board new domain controllers, requirements for time to recover from a domain controller hardware failure, and other specific deployment requirements.

SummaryBuilding a large-scale multi-tenant Exchange 2013 deployment is a very complex undertaking. Great care should be taken during the design, pre-production testing, and deployment phases to ensure that all scalability aspects of the system have been considered and are being closely monitored. No large scale deployments of this type are the same—every deployment has customizations that impact how the system will function as the number of tenants and users increases. Therefore, testing, monitoring,

10

capacity planning, and risk mitigation are all critical aspects of a successful multi-tenant Exchange deployment.

This document provides some high-level scalability guidance and deployment recommendations. We welcome your feedback and comments on this document and will incorporate them into future versions if required. If you have feedback, please send it to ExHostingFeedback@microsoft.com.

Legal NoticeThis document is provided “as-is”. Information and views expressed in this document, including URL and other Internet Web site references, may change without notice. You bear the risk of using it.

Some examples depicted herein are provided for illustration only and are fictitious. No real association or connection is intended or should be inferred.

This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes.

© 2013 Microsoft Corporation. All rights reserved.

Microsoft, MS-DOS, Windows, Windows Mobile, Windows Server, Active Directory, ActiveSync, Forefront, Exchange, Outlook, and SharePoint are trademarks of the Microsoft group of companies.

All other trademarks are property of their respective owners.

11