web344 microsoft.com: understanding iis 6.0 architecture strategies for high availability casey...
TRANSCRIPT
WEB344WEB344Microsoft.com: Understanding Microsoft.com: Understanding IIS 6.0 Architecture Strategies for IIS 6.0 Architecture Strategies for High Availability High Availability
Casey Jacobs – Group Mgr, OperationsCasey Jacobs – Group Mgr, OperationsWayne King – Group Mgr, Web OperationsWayne King – Group Mgr, Web OperationsJeff Stucky – Debug Mgr, OperationsJeff Stucky – Debug Mgr, Operations
Microsoft.com OperationsMicrosoft.com OperationsMicrosoft CorporationMicrosoft Corporation
AgendaAgenda
Microsoft.com IntroductionMicrosoft.com Introduction
High Availability ArchitectureHigh Availability ArchitectureGlobal Solutions & NetworkingGlobal Solutions & Networking
Web & Database Hosting ModelsWeb & Database Hosting Models
64-Bit Adoption – Case Study64-Bit Adoption – Case StudyOverview of AdoptionsOverview of Adoptions
Key Takeaways of MigrationKey Takeaways of Migration
Lessons LearnedLessons Learned
Microsoft.com IntroductionMicrosoft.com Introduction
Vision, Strategy & InitiativesVision, Strategy & Initiatives
Vision:Vision: Achieve the highest availability on the Achieve the highest availability on the Internet while showcasing MS Technologies Internet while showcasing MS Technologies
Strategy:Strategy: Early Adopt Microsoft solutions, Early Adopt Microsoft solutions, provide positive feedback loop with Product provide positive feedback loop with Product Teams and showcase our Operational abilitiesTeams and showcase our Operational abilities
Do ItDo It
Prove ItProve It Share ItShare It
Know ItKnow It
Internet Hosting – Quick FactsInternet Hosting – Quick Facts
3 Internet Data Centers 3 Internet Data Centers
2 Content Delivery Networks2 Content Delivery NetworksAkamai & Savvis vendor relationshipsAkamai & Savvis vendor relationships
Cache Downloads & Product UpdatesCache Downloads & Product Updates
1606 Data Center servers1606 Data Center servers
500+ servers in Lab (Dev 500+ servers in Lab (Dev PPE) PPE)
109 Web Sites, 1069 DBs, 1000’s Apps109 Web Sites, 1069 DBs, 1000’s Apps
80+ Gigabit/sec Bandwidth80+ Gigabit/sec Bandwidth
Enterprise Web Solutions at ScaleEnterprise Web Solutions at Scale
Microsoft.com Microsoft.com 13M UUsers/Day & 70M Page Views/Day13M UUsers/Day & 70M Page Views/Day10k Req/sec & 300K ConConnections10k Req/sec & 300K ConConnectionsIIS ConfigurationsIIS Configurations
350 Virtual Roots350 Virtual Roots190 IIS Web Applications190 IIS Web Applications12 Application Pools12 Application Pools
Windows Update Windows Update 150M UClient scans/day (Interactive & AU)150M UClient scans/day (Interactive & AU)12k ASP.NET Req/sec12k ASP.NET Req/sec4 Gb/sec Web site egress (Web pages only…4 Gb/sec Web site egress (Web pages only…doesn’t include the downloads!)doesn’t include the downloads!)
Web Site Availability BenchmarkingWeb Site Availability BenchmarkingMeasured by Keynote Systems, Inc.Measured by Keynote Systems, Inc.
Year To DateYear To DateRank Site Avail %
1 Microsoft.com 99.81
2 WindowsUpdate 99.80
3 AOL 99.77
4 Google 99.75
5 IBM 99.56
6 Yahoo 91.50
7 Oracle 87.67
8 SUN 80.88
Q:Q: Who is Keynote? Who is Keynote?
A:A: They provide monitor They provide monitor
agents worldwide which agents worldwide which
‘ ‘ping’ systemsping’ systems
Q:Q: How do we use the data? How do we use the data?
A:A: RealTime & Trending RealTime & Trending
* 3 Yrs at #1 for Microsoft.com* 3 Yrs at #1 for Microsoft.com
** Top 10 MSCOM web sites out ** Top 10 MSCOM web sites out
ranked Benchmarks in 2004ranked Benchmarks in 2004
High Availability ArchitectureHigh Availability Architecture
Data Center 1 Data Center 2
Cisco Guard Devices- Packet Filtering
- Anomaly Tracking
Routing Rule Sets - ACL’s (80 & 443 Only)
- Dedicated LANs for Unique Req’s
Internet
`
Clients
CDN Partnerships - Edge Cache: Akamai & Savvis - Cluster Load Balancing - FirstPoint (Akamai) - ITM (Savvis)
High Availability ArchitectureHigh Availability ArchitectureGlobal Solutions & NetworkingGlobal Solutions & Networking
High Availability Architecture High Availability Architecture Global Solutions & NetworkingGlobal Solutions & Networking
Global SolutionsGlobal SolutionsAkamai & Savvis – Content CachingAkamai & Savvis – Content Caching
Global Load Balancing – Web Cluster Level MgmtGlobal Load Balancing – Web Cluster Level Mgmt
Network InfrastructureNetwork InfrastructureCisco Guards = Anomaly Detection & DOS FiltersCisco Guards = Anomaly Detection & DOS Filters
Attack Rates > 70x Normal TrafficAttack Rates > 70x Normal Traffic
4k Req/Sec vs Attack 280k Req/Sec/Server4k Req/Sec vs Attack 280k Req/Sec/Server
Cisco Router ArchitectureCisco Router Architecture
HTTP/S Only – Exceptions Require ReviewHTTP/S Only – Exceptions Require Review
““Cookie Cutter” - Simple, Scalable & ManageableCookie Cutter” - Simple, Scalable & Manageable
Agility - Quickly Swap LANs between Web SitesAgility - Quickly Swap LANs between Web Sites
Data Center 1 Data Center 2
Online Online
Read OnlyNLB
Online Offline
Read/WriteNLB
Log Ship
(Synch’d Copies)
WebServicesNLB Cluster
WebServicesNLB Cluster
Log ShipSecondary
Log Ship Secondary
Log Shipping Synch’s (Contingency Tactics)
Online OnlineDuplicates
Cluster 1 Cluster 5 Cluster 6Cluster 3Cluster 2 Cluster 4
ASP.NET, ASP, HTML NLB Load Balancing 3 Servers Min/Cluster Synch’d Code & Content
High Availability ArchitectureHigh Availability ArchitectureWeb & Database HostingWeb & Database Hosting
High Availability Architecture High Availability Architecture Web & Database HostingWeb & Database Hosting
Network Load Balancing (NLB) ClustersNetwork Load Balancing (NLB) ClustersServer Cluster Sizes: 3 – 8 Servers/ClusterServer Cluster Sizes: 3 – 8 Servers/Cluster
Positives: Easy Mgmt & Free with Windows SKU’sPositives: Easy Mgmt & Free with Windows SKU’s
Challenges: Switch Overhead & Connection AffinityChallenges: Switch Overhead & Connection Affinity
Server ConfigurationsServer ConfigurationsBaseline O/S, IIS, ASP.NET Configurations IdenticalBaseline O/S, IIS, ASP.NET Configurations Identical
Application Code & Content Unique per SiteApplication Code & Content Unique per Site
Standard Hosting ModelsStandard Hosting ModelsAgility = Quickly reallocate from System to SystemAgility = Quickly reallocate from System to System
Efficiency = Less Staffing & Equipment RequiredEfficiency = Less Staffing & Equipment RequiredConsistent ConfigurationsConsistent Configurations
Repeatable Infrastructure ArchitectureRepeatable Infrastructure Architecture
64-bit Adoption – Case Study64-bit Adoption – Case Study
Early Adoption ObjectivesEarly Adoption Objectives1 Year Runtime prior to Win2k3 SP11 Year Runtime prior to Win2k3 SP1
Began May 21, 2004Began May 21, 2004
100% Adopted by April 2005100% Adopted by April 2005
Identify Memory Management ImprovementsIdentify Memory Management Improvements
System & App Migration Gotcha’sSystem & App Migration Gotcha’s
Historical ChallengesHistorical ChallengesWin2k/IIS5 – Memory limits hosting ASP & ASP.NET Win2k/IIS5 – Memory limits hosting ASP & ASP.NET with no memory based recycling for protectionwith no memory based recycling for protection
Win2k3/IIS6 – App Pools helped Isolation protection, Win2k3/IIS6 – App Pools helped Isolation protection, but memory recycling intervals continually shortenedbut memory recycling intervals continually shortened
Windows Server 2003 x64 Edition Windows Server 2003 x64 Edition Adoption OverviewAdoption Overview
64-bit Migration Strategy64-bit Migration Strategywww.Microsoft.com Hardwarewww.Microsoft.com Hardware
HardwareHardware1.1. Proof of concept testing with AMD supplied reference Proof of concept testing with AMD supplied reference
hardwarehardware
2.2. Purchased 64-bit capable HardwarePurchased 64-bit capable Hardware
Ability to host both x86 & 64-bit Windows O/SAbility to host both x86 & 64-bit Windows O/S
Analyze vendor components: Proc’s & MemoryAnalyze vendor components: Proc’s & Memory
3.3. Install x86 Windows (Windows 2003)Install x86 Windows (Windows 2003)
Incrementally test S/W stack on new H/W platformIncrementally test S/W stack on new H/W platform
4.4. Migrated to 64-bit WindowsMigrated to 64-bit Windows
Use same H/W as (2) and (3) were validatedUse same H/W as (2) and (3) were validated
Allowed for phased migration while testing App’s Allowed for phased migration while testing App’s on the new 64-bit H/W without impacton the new 64-bit H/W without impact
64-bit Migration Strategy64-bit Migration Strategywww.Microsoft.com Softwarewww.Microsoft.com Software
Software ApproachSoftware Approach1.1. Rely on x86 code not yet ported to 64-bitRely on x86 code not yet ported to 64-bit2.2. Expect Zero App Dev – x86 IIS on WoW64 Expect Zero App Dev – x86 IIS on WoW64
allows for Transparent Migrationallows for Transparent MigrationASP.NET 1.1 App’s, x86 ISAPI Extensions & FiltersASP.NET 1.1 App’s, x86 ISAPI Extensions & FiltersKB 895976: KB 895976: “How to run 32-bit Worker Processes on 64-“How to run 32-bit Worker Processes on 64-bit”bit”
3.3. Target 4GB VM per ProcessTarget 4GB VM per Process32-bit Processes under WoW64 get full 4 GB of VM32-bit Processes under WoW64 get full 4 GB of VMFull benefits of 64-bit Kernel if running 32-bit Apps Full benefits of 64-bit Kernel if running 32-bit Apps in WOW64in WOW64
* Helped solve VM crunch we were running into on x86* Helped solve VM crunch we were running into on x86** Don’t Sacrifice Kernel Memory Pools for extra Virtual** Don’t Sacrifice Kernel Memory Pools for extra Virtual Memory into UserModeMemory into UserMode
Hardware Platform Hardware Platform Comparative Study: x86 vs x64Comparative Study: x86 vs x64
Key Take Away'sKey Take Away'sHardware Costs – lower cost per server for x64Hardware Costs – lower cost per server for x64
Hardware capable of running both x86 & x64 O/SHardware capable of running both x86 & x64 O/S
Greater CPU and Memory capacities lend towards Greater CPU and Memory capacities lend towards further Consolidation of systems or ReAllocationsfurther Consolidation of systems or ReAllocations
x86x86 x64x64
Hardware ModelHardware Model HP DL580HP DL580 HP DL585HP DL585
ProcessorProcessor 4 x 2.0 GHz (Intel)4 x 2.0 GHz (Intel) 4 x 2.2 GHz (AMD)4 x 2.2 GHz (AMD)
RAMRAM 4 GB4 GB 16 GB16 GB
OS VersionOS VersionWindows Server Windows Server 2003 Enterprise 2003 Enterprise Edition SP1Edition SP1
Windows Server Windows Server 2003 Enterprise 2003 Enterprise x64 Editionx64 Edition
CPU Utilization per Platform CPU Utilization per Platform Comparative Study ResultsComparative Study Results
x86x86 x64x64
HTTP Req/SecHTTP Req/Sec CPU %CPU % HTTP Req/SecHTTP Req/Sec CPU %CPU %
222222 65%65% 216216 35%35%
Key TakeAway'sKey TakeAway'sHuge Gains due to 64-bit H/W & Windows PlatformsHuge Gains due to 64-bit H/W & Windows PlatformsSeamless migration provided with WoW64 Seamless migration provided with WoW64 Enabled Enabled www.Microsoft.comwww.Microsoft.com to leverage saved to leverage saved infrastructure to enable Data Center Redundancyinfrastructure to enable Data Center RedundancyApp Pool Recycles Eliminated – Enjoying the new 4GB App Pool Recycles Eliminated – Enjoying the new 4GB VM address space running under WoW64!!VM address space running under WoW64!!Enabled more App Pools driving further Isolation of Enabled more App Pools driving further Isolation of Code & Content in shared hosting modelsCode & Content in shared hosting models
Average Server Response TimesAverage Server Response TimesComparative Study ResultsComparative Study Results
x86x86 x64x64
RequestRequestTypeType Req/secReq/sec ResponseResponse
Time (ms)Time (ms) Req/secReq/sec ResponseResponseTime (ms)Time (ms)
ASPASP 7.857.85 244244 7.417.41 5353
ISAPIISAPI 110.85110.85 248248 125.43125.43 1818
StaticStatic 41.941.9 135135 31.0131.01 33
Static (cached)Static (cached) 47.1147.11 11 54.5154.51 11
Key Take Away'sKey Take Away'sSignificantly Improved Response Times/RequestsSignificantly Improved Response Times/Requests
Explanation – Combo due to relieved CPU burdens Explanation – Combo due to relieved CPU burdens and increased Physical and Virtual Memoryand increased Physical and Virtual Memory
Worst Performing Web PagesWorst Performing Web PagesComparative Study ResultsComparative Study Results
x86 (Seconds)x86 (Seconds) X64 (Seconds)X64 (Seconds) Performance GainPerformance Gain
79.3 79.3 5.1 5.1 15.5x15.5x
53.5 53.5 4.7 4.7 11.3x11.3x
49.4 49.4 2.82.8 17.7x17.7x
47.7 47.7 2.72.7 17.4x17.4x
44.8 44.8 2.62.6 17.4x17.4x
Key Take Away'sKey Take Away'sHUGE drop in Page delivery for worst App’sHUGE drop in Page delivery for worst App’s
Systems no longer required to queue or keep open Systems no longer required to queue or keep open connections for poor performing App’sconnections for poor performing App’s
* Server Performance Advisor (SPA) used to analyze data* Server Performance Advisor (SPA) used to analyze data
32-bit vs 64-bit Performance32-bit vs 64-bit Performance
64bit Adoption – www.Microsoft.com64bit Adoption – www.Microsoft.comTop Lessons LearnedTop Lessons Learned
WoW64 Redirection BehaviorsWoW64 Redirection BehaviorsFile Sys Redirection for 32-bit Apps File Sys Redirection for 32-bit Apps windows\windows\system32 vs. windows\syswow64system32 vs. windows\syswow64
Registry Redirection Registry Redirection HKLM\Software\Wow6432NodeHKLM\Software\Wow6432Node
Script Host Gotcha’s (cscript.exe/wscript.exe)Script Host Gotcha’s (cscript.exe/wscript.exe)Scripts depending on x86 com objects will require Scripts depending on x86 com objects will require x86 scripting hosts (located in \SysWow64)x86 scripting hosts (located in \SysWow64)Scripts using x86 scripting host need to be aware of Scripts using x86 scripting host need to be aware of WoW64 Redirection behaviors (see Above)WoW64 Redirection behaviors (see Above)
64bit Adoption – www.Microsoft.com64bit Adoption – www.Microsoft.comTop Lessons Learned (cont’d)Top Lessons Learned (cont’d)
Verification of 3Verification of 3rdrd Party App Compatibility Party App CompatibilityCommon Admin Tools: Regmon, Filemon, etc.Common Admin Tools: Regmon, Filemon, etc.
Required 64-bit flavor of device drivers or filter Required 64-bit flavor of device drivers or filter drivers that run in the kerneldrivers that run in the kernel
Antivirus SoftwareAntivirus Software
Backup SoftwareBackup Software
Imaging/Deployment SoftwareImaging/Deployment Software
www.Microsoft.com 64-bit Adoption www.Microsoft.com 64-bit Adoption Future DirectionFuture Direction
ASP.NET 2.0 Beta 2 – Early AdoptionASP.NET 2.0 Beta 2 – Early Adoptionhttp://www.Microsoft.com/Downloadshttp://www.Microsoft.com/DownloadsASP.NET introduces 64-bit flavorASP.NET introduces 64-bit flavorASP.NET 1.1 is the current dependency for ASP.NET 1.1 is the current dependency for having to run 32-bit IIS in WoW64.having to run 32-bit IIS in WoW64.Looking forward to start testing native 64bit Looking forward to start testing native 64bit configuration (running full 64-bit IIS)configuration (running full 64-bit IIS)
Getting started with SQL server deploymentsGetting started with SQL server deploymentsWorking on getting some SQL 2000 servers Working on getting some SQL 2000 servers online to leverage 4 GB Virtual Memory benefitsonline to leverage 4 GB Virtual Memory benefitsYukon Early Adoption: Full native 64-bit SQL Yukon Early Adoption: Full native 64-bit SQL setup with SQL Server 2005setup with SQL Server 2005
ResourcesResources
Online Webcasts: http://IISWebCastSeries.com
TechNet Online: http://Microsoft.com/Technet
ASP.NET Sites: http://MSDN.Microsoft.com
http://GotDotNet.com
http://ASPNET.com
Your FeedbackYour Feedbackis Important!is Important!Please Fill Out a Survey forPlease Fill Out a Survey forThis Session on CommNetThis Session on CommNet
© 2005 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.