Office 365 Network Optimization and Troubleshooting

Jeff Mealiffe

Principal Program Manager

Microsoft – Office 365 Engineering

Our high-performing network is one of the top 3 in the world with public peering in 23 countries with 1,500 ISPs.

Microsoft has invested $15 billion in infrastructure, building over 100 datacenters and we are constantly evaluating new locations

Our Datacenters support over 20 Million businesses and over 200 Online Services. Office 365 is sold in 131 markets, 43 languages, and 25 currencies.

Chicago

Cheyenne

Dublin

Amsterdam

San Antonio

Microsoft has datacenter capacity around the world…and we’re growing

Boydton Quincy

Des Moines

Brazil

35+ factors in site selection:

Proximity to customers

Energy, Fiber Infrastructure

Skilled workforce

Other Microsoft DC locations

Office 365 DC locations

1+ million servers

100+ datacenters

in 40+ countries

Hong Kong

Singapore

Japan Shanghai

Australia

There are many other Microsoft edge nodes

that are not yet compliant with Office 365.

The green circles

represent Microsoft

Edge nodes live for the

Office 365 Portal.

EDGE

Client RTT Server RTT

Content Time

TCP connect

SSL connect

App Latency

TTFB

TTLB

Reusing existing connections

EDGE

San Antonio DC

Without Edge, entire request over ISP’s network

With Edge, Microsoft’s network brought closer to the user

Edge reuses connections between to further improve performance

•

•

•

Brisbane Australia

Melbourne Australia

Perth Australia

Sydney Australia

Vienna Austria

Luxembourg Luxembourg

Sao Paulo Brazil

Montreal Canada

Toronto Canada

Prague Czechoslovakia

Paris France

Frankfurt Germany

Hong Kong Hong Kong

Dublin Ireland

Milan Italy

Turin Italy

Tokyo Japan

Seoul Korea

Kuala Lumpur Malaysia

Amsterdam Netherlands

Auckland New Zealand

Wellington New Zealand

Moscow Russia

Singapore Singapore

Stockholm Sweden

Zurich Switzerland

Taipei Taiwan

London UK

Ashburn USA

Atlanta USA

Boston USA

Chicago USA

Dallas USA

Denver USA

Honolulu USA

Las Vegas USA

• Site data is published at http://www.peeringdb.com/view.php?asn=8075

• Some cities have multiple peering points

• Peering locations may be on-net or off-net

• Peering may involve physical connection and/or routing advertisements

• Data as of July 2014 is subject to change

Los Angeles USA

Miami USA

New York USA

Palo Alto USA

San Jose USA

Seattle USA

Browser-based

Clients also cache

with HTML5

Browser-based

mobile clients

Native clients on

tablets, PCs,

& desktops

Native clients

on mobile devices

Outlook, Outlook Web Access, OneDrive for Business

Office 365 offers a wide variety of options across devices for customers to access the service

OWA uses HTML 5 Offline Application Caching if enabled in Offline Settings

Internet

peering and

routing

Customer

Internet

connection

Microsoft

network

Microsoft

edge nodes

Content

delivery

network

Server

workloads

Rich client

applications

Exchange Online http://aka.ms/tune

• Similar to on-prem

• Estimates rely upon on-prem baselines

• Cached mode reduces impact and provides for latency tolerance (<325 mSec)

• Perfmon, Netmon, top, etc. to baseline

• Definite advantages to Outlook 2013 SP1

• Exchange Bandwidth Calculator

SharePoint Online

• HTTPS views of webpages, uploads/downloads of content

• Document editing with Office Web Apps or Office

• Bursty, but latency tolerant

• Without baseline, no real way to estimate

• Netmon to baseline

• Will ramp up as more content is loaded into SharePoint, MySites,

OneDrive for Business

Lync Online

• IM is bursty, latency tolerant, and very small

• Voice uses RTAudio • 50kbps low, 80kbps high, autodetermination

• Video depends upon resolution • 280kbps low, 4000kbps high, dependent upon resolution

• Desktop sharing depends upon desktop resolution

• Peer-to-Peer versus Client-Server

• Lync Bandwidth Calculator

Use your pilot to predict new requirements

• Baseline bandwidth requirements before Office 365

• Measure bandwidth baseline for pilot users

• Extrapolate for total bandwidth required for your workload

• Assumes enough time to order upgrades if needed

• Start with at least 20% head room

• Evaluate options to conserve bandwidth

1.

2.

•

•

3.

4.

5.

6.

7.

8.

1.

2.

3.

4.

5.

6.

7.

8.

9.

Other configuration topics

IP Addresses not as quickly updated

PAC files, CONNECT, and are they helping or hurting

Some customer scenarios have seen improved performance Never a silver bullet and other measures should be investigated Required to be disabled for debugging or support

Agenda

NAT, Measuring Latency/Round Trip Time

TCP Window Scaling

GEO DNS issues

Proxy and Firewall port exhaustion

Packet Loss

Routing and Peering

TCP Idle time settings

Proxy Authentication

DNS performance

SACK and TCP MSS

Lync tests to Server

SharePoint customization performance

PSPing Demo

Internal RTT (ms) External RTT (ms) Total RTT to O365

54.88 346 400.88

Here we can see clearly, the poor RTT is outside the customer’s environment, on the ISP link to Office 365. If

this RTT is unexpected, the customer can engage their ISP to investigate.

54.88ms

Proxy

Office 365 Datacentre

0.346ms

Client

Putting it all together

TCP Window Scaling

TCP data packets

TCP ACK

TCP Window Scaling

enabled? Maximum TCP receive

buffer (Bytes)

No 65535 (65k)

Yes 1073725440 (1gb)

Impact of TCP Window Scaling

Round Trip Time (ms) Maximum Throughput

(Mbit/sec) without scaling

Maximum Throughput (Mbit/sec) with scaling

300 1.71 447.36

200 2.56 655.32

100 5.12 1310.64

50 10.24 2684.16

25 20.48 5368.32

10 51.20 13420.80

5 102.40 26841.60

1 512.00 134208.00

Presuming a 1000 Mbps link here is the maximum throughput we can get with TCP window scaling disabled and then with it enabled

TCP Window Scaling

7692 12:28:03 14/03/2014 12:28:03.8450000 0.0000000 100.8450000 10.127.0.199 contoso47-48ipv4b.sharepointonline.com.akadns.net TCP TCP: [Bad CheckSum]Flags=......S., SrcPort=43511, DstPort=HTTPS(443), PayloadLen=0, Seq=3807440828, Ack=0, Win=65535 ( Negotiating scale factor 0x0 ) = 65535

7740 12:28:04 14/03/2014 12:28:04.1440000 0.2990000 101.1440000 contoso47-48ipv4b.sharepointonline.com.akadns.net 10.127.0.199 TCP TCP:Flags=...A..S., SrcPort=HTTPS(443), DstPort=43511, PayloadLen=0, Seq=3293427307, Ack=3807440829, Win=4380 ( Negotiated scale factor 0x2 ) = 17520

7692 12:28:03 14/03/2014 12:28:03.8450000 0.0000000 100.8450000 10.127.0.199 contoso47-48ipv4b.sharepointonline.com.akadns.net TCP : [Bad CheckSum]Flags=......S., SrcPort=43511, DstPort=HTTPS(443), PayloadLen=0, Seq=3807440828, Ack=0, Win=65535 ( Negotiating scale factor 0x0 ) = 65535 {TCP:818, IPv4:122}

7740 12:28:04 14/03/2014 12:28:04.1440000 0.2990000 101.1440000 contoso47-48ipv4b.sharepointonline.com.akadns.net 10.127.0.199 TCP TCP:Flags=...A..S., SrcPort=HTTPS(443), DstPort=43511, PayloadLen=0, Seq=3293427307, Ack=3807440829, Win=4380 ( Negotiated scale factor 0x2 ) = 17520 {TCP:818, IPv4:122}

Microsoft network

Internet egress point

Customer network

Tracing route to OUTLOOK-APACNORTH.OFFICE365.COM [132.245.65.146]

over a maximum of 30 hops:

1 1 ms 1 ms 1 ms SkyRouter.Home [192.168.0.1]

3 11 ms 11 ms 11 ms ip-89-200-132-100.ov.easynet.net [89.200.132.100]

4 10 ms 10 ms 10 ms igbtmdistc7503.msft.net [195.66.236.140]

5 84 ms 84 ms 84 ms xe-0-3-2-0.ash-96cbe-1a.ntwk.msn.net [207.46.45.227]

6 96 ms 95 ms 95 ms ae2-0.atb-96cbe-1a.ntwk.msn.net [207.46.33.228]

9 140 ms 142 ms 140 ms 191.234.83.150

10 142 ms 138 ms 139 ms ae11-0.lax-96cbe-1b.ntwk.msn.net [207.46.47.11]

11 256 ms 256 ms 256 ms ae2-0.tya-96cbe-1a.ntwk.msn.net [207.46.46.149]

12 265 ms 265 ms 265 ms ae0-0.tya-96cbe-1b.ntwk.msn.net [204.152.140.181]

13 288 ms 290 ms 292 ms xe-7-0-1-0.sin-96cbe-1a.ntwk.msn.net [207.46.38.252]

14 290 ms 288 ms 287 ms xe-5-3-1-0.sin-96cbe-1b.ntwk.msn.net [207.46.41.39]

15 279 ms 279 ms 279 ms ae1-0.sg2-96cbe-1a.ntwk.msn.net [191.234.80.90]

18 280 ms 280 ms 279 ms 132.245.65.146

http://channel9.msdn.com/events/SharePoint-Conference/2014/SPC3993

http://technet.microsoft.com/en-us/library/office-365-system-requirements.aspx

http://trippams.online.lync.com

http://trippdb3.online.lync.com

http://tripphkn.online.lync.com

http://trippbl2.online.lync.com

http://trippsn2.online.lync.com

http://trippsg1.online.lync.com

What you can do