all clouds-are-made-of-metal-banv-april-2432014-for-distribution

Ultimately All Clouds Are

Made of MetalHarry@Quackenboss.com

Why the Title?

Dave Roberts Tweet

“But all clouds are

ultimately made of

metal.”

Dave Roberts

Senior Vice President, ServiceMesh

Tweet, August 2, 2012

Most Everybody Thinks SDN

Is the Solution

Mobile NetworkDatacenter

Opinion (I)

The more you understand the metal,

the more you will understand SDN, NFV

and Network Virtualization

Opinion (2)

WAN SDN ≠ LAN SDN

WAN Virtualization ≠ LAN Virtualization

Differences come from:

Physics

Opinion (3)

Mobile WAN ≠ Wired WAN

Physics & Econ Differences

WAN LAN

LATENCY ∞ 0

BANDWIDTH 0 ∞

Do the same SDN

solutions apply?

Campus Wi-Fi

Datacenter

Mobile

Networks

Broadband and

Long haul Networks

WAN Providers’ Challenge“A common refrain among the network executives we’ve talked to is that, if IT services can be provided in secondswith cloud computing technologies, it should not take days or weeks to deliver the network services that support them.”

Practical Implementation of SDN & NFV in the WAN

Heavy Reading

October 2013

London Mobile Infrastructure

600 Base Stations

See the Cell Tower?

Mobile Operators’ Finance Problem

3G Mobile Network Metal

Big device population

Complex

Heterogeneous

10 year depreciation

3G May Be the Future of IoT

Mobile SDN/NFV Perspective

Been “SDN” since before SDN

Salivating over IoT

NFV Mandatory Requirements

Existing OSS/BSS integration

Existing equipment interoperability

• Including legacy 3G

“Wired” WAN

Sprint Long Haul Network

Latency and Light Speed300,000 km/sec (in vacuum)

~200,000 km/sec (in fiber)

Slower because of refraction

Ask your neighborhood physicist

http://en.wikipedia.org/wiki/Optical_fiber

Cross-Country Latency

Source: Comcast

(4,100/200,000) =~.020 seconds

Packets go through routers, which buffer

adding delay

WAN Latency Is Speed-Independent

Source: Comcast.com 40G/sec network

BANDWIDTH LATENCY

40Gbps 20 milliseconds

400Gbps 20 milliseconds

1Mbps 20 milliseconds

Ping (Half Moon Bay to Palo Alto)

Half Moon Bay to New York

Half Moon Bay to Beijing

Half Moon Bay to New Delhi

Latency Vs. BandwidthThought experiment:

Transfer 1 Petabyte of data on 1TB SATA drives from SFO to New Delhi• Note: SATA III throughput: 6 Gbits/sec

• SATA drive: ~100 Mbytes/sec (~1.1Gbps)

Alternatives

A. Lease dedicated 40G circuit

B. Airplane 1000 drives (~2000 lbs)

Latency Vs. BandwidthA: 40G pipe

1-way latency: ~100ms (best case)

• (first bit in to first bit out)

~70 hours

• Hypothetical ideal throughput

Option B: Airplane

~18 hours (assuming non-stop)

1-way latency: ~18 hours

WAN Cable CostsResidential broadband installation

CAPEX: ~$1,000/home passed

OPEX: ~$5/month/homehttp://fastnetnews.com/fiber-news/175-d/4835-fiber-economics-quick-and-dirtynth

Long distance fiber installation

$30K/mile for conduit in open

trench during road construction

(< 1% of cost of road)

Transportation Industry LessonsUtilization is key to low cost

Synchronize

Schedule in advance

Take time to pack (doesn’t hurt overall time much)

Load big stuff first

Make sure there is a full buffer at on ramp

Make sure there isn’t a bottleneck unloading

Datacenter LANs

Samsung Backup

Datacenter Fire

April 20,2014

Mice + Elephants

Source: Facebook OIC 2013 presentation

East-West Traffic Example

Source: Facebook OIC 2013 presentation

Typical Cloud Data Center20 to 30 1U(1.75”) servers/rack

2x1U Top of Rack switches

Switches are 5% to 15% of rack

space, CAPEX and electricity

BUT: about 3 switch ports per server

(15% to 45% of server CAPEX)

Datacenter Topology

(HVQ Note: For each TOR Switch server port, there will be ~1 to 2

aggregation ports)

Source: Nathan Farrington Facebook OIC13 Presentation

TOR (Top of Rack) Switch48x10GE + 4x40GE

Vol. price (est.)

~$2500-$4000

Per port $50+

Source: Opencomute.org Intel draft

Example OCP

Bare Metal Switch

Cabling Cost > TOR CostCopper Twinax: ~$75/link

Intra-rack (server to TOR)

Fiber optics: ~$150 - $400

(2Xoptics + cable)

Rack to spine switch

Aggregation CablingMTP/MPO multifibre cables

12,24…up to72 fibers

Intel/Corning Silicon Photonics (future)

Up to 64 fibers - 25Gbps each

http://www.intel.com/content/www/us/en/research/intel-labs-silicon-photonics-mxc-connector.html

Data Center Metal Economics

Datacenter Metal Key PointsServer networking switches CAPEX is

20% to 30% of server cost

TOR switch hardware is pretty much same for everybody

Weakening perception that brand name SW is better is reducing margins

In-rack connectivity with copper cables is cheap, but still as much or more than the switch $/port

Limited opportunities to reduce hardware CAPEX

Bandwidth Vs CPU1 instruction needs 1 byte of

memory and generates 1 bit/sec of bandwidth

(1994) SPARC10: 60Mhz+100Mbps

(2000) 1Ghz CPU+1GbE

(2014) (3Ghz*15 cores) * 2 sockets

90Gbps (!)

Do datacenters have

enough NIC bandwidth?

Performance in Datacenter LANs

Latency and Bandwidth

Server Latency (One Way)

TCP/IP

GBE NIC

Server With GbE

~15µsTCP/IP

10GE NIC

Server With GbE

Note: numbers may vary based

on specific benchmarks

Switch Latency Is Less Than

Host Stack Latency

First bit in to first bit out

Measurement favors “cut-through” (versus “store and forward” switches

Typical 10GE (per-chip) .3µs to 1.5µs

Data Link Data Link

Switch Latency With Buffering

Switch buffers if output port busy

Dramatic increase in latency

60µs observed in benchmark tests

Dropped packets/retransmits: ∞ latency

Data Link Data Link

BUFFER

Source: HP Mellanox Low Latency Benchmark Report 2012

Round-trip user space to

user space

Who Cares About LAN Latency?Not if traversing the WAN

But high latency and dropped hurt

server throughput

Applications spin or get swapped

HFT cares!

HFT: Milliseconds = $$$ While You Were Blinking

High-frequency-trading activity is not constant; it occurs in microbursts. The line at the bottom of this graphic is the stock-market activity involving General Electric shares over 100 milliseconds (one-tenth of a second) at 12:44 p.m. on Dec. 19, 2013. The gray box magnifies a five-millisecond window, during which GE experienced heavy bid and offer activity and a total of 44 trades.

Graphic: CLEVERºFRANKE.

Data source: IEX.

44 trades in 5 milliseconds

http://www.nytimes.com/2014/04/06/magazine/flash-boys-michael-lewis.html?_r=0

Leveling the HFT Playing FieldTo allow customers to

pick and choose

different pricing tiers

based on different

latencies, HFT systems

add latency using fiber

38 miles of fiber cable =

~260 microseconds

http://www.nytimes.com/2014/04/06/magazine/flash-boys-michael-lewis.html?_r=0

Latency Key PointsFrom viewpoint of end-end across WAN, LAN

latency is near 0

10GE Host NIC and IP Stacks have improved dramatically

1GE is > 10X higher than 10GE

Few 1GE switches support cut through

Higher chance of busy ports

Switch latency in cut through is great

If switches have to buffer because of busy output port, latency goes way up

Latency spikes are short duration so hard to see, let alone measure

OpenFlow For Datacenter LANs

Note: The control plane

network switches are

not shown in this

diagram

OpenFlow Reaction Time in LAN

OpenFlow

Application

Transport Layer

Internetwork Layer

OpenFlow

Transport Layer

Internetwork Layer

1000µs

20-100µs

1000µs

OpenFlow Controller

App + IP Stack

OpenFlow Switch

Agent + IP Stack

OpenFlow MicroControllers?

http://www.theregister.co.uk/2012/12/11/pica8_sdn_bundles/

OpenFlow

MicroControllerOpenFlow

MicroController

Master

OpenFlow

Controller

Lessons From Transportation

Multi-pathingHOV lanes

Priority based on

policy

Inefficient utilization

What if nobody

changed lanes?

(802.11ag and ECMP

are static)

Dynamic Load Balancing Can Increase

Throughput

“It's tough to make predictions, especially about the future.”

― Yogi Berra

http://www.pcper.com/news/General-Tech/Atom-aint-dead-yet-New-ultra-low-power-Avoton-chips-servers

Lower Performance =

Higher Efficiency

New Rack DesignsSince the 1U server was invented by

Cobalt Networks (acquired by Sun)

the processors are > 100X faster

Network switching has

(mostly)lagged Moore’s Law

Will ratio of servers to network

switches decreases over time?

Time for new form factors?

TOR Switch Obselescence

Source: Intel IDF 14 Shenzhen presentation

Data Center Server Future

TOR Switch Evolves to Tray Switch

Conclusions

WAN Issues and OpportunitiesMobile: physical infrastructure

complexity has led to management

complexity

NFV promise is to reducing

physical devices, simplify mgt.

Long haul: utilization of capacity is

still opportunity, SDN offers ways to

simplify mgt. and improve utilization

Datacenter LANsLow cost of switching has limited

opportunities for independent SDN controller sales

Distributed SDN MicroController hierarchy should broaden appeal

Controller needs to be bundled with switch or switch software

Effective dynamic load balancing will improve throughput and latency

Important DifferencesWAN LAN

COST ∞ 0

BANDWIDTH 0 ∞LATENCY ∞ 0

Ultimately All Clouds Are

Made of MetalHarry@Quackenboss.com

all clouds-are-made-of-metal-banv-april-2432014-for-distribution

Technology

banv meetup-contrail

2018-19 - dpsludhiana.com book-class 2.pdf · and clouds...

chapter 5 clouds and stability chapter 5 clouds and...

clouds weight the of a word. clouds clouds clouds

cloud formation - how clouds form and different types of...

polar mesospheric clouds (pmcs) -also known as- noctilucent...

mobile phones, internet plans, voip & it services -...

clouds privadas

open contrail slides for banv meetup

clouds. what are clouds?

banv spring management 2012 refresher class

clouds, rain, clouds again

subject: types of clouds agata Łazarz. internet...

formation of clouds/types of clouds

introduction to clouds introduction to clouds€¢cirrus,...

storm clouds. cumulonimbus clouds cold rain process

types of clouds. different kinds of clouds lead to ... ·...

why do clouds turn gray? clouds - mrspruillscience€¦ ·...

clouds clouds clouds

clouds and climate: forced changes to clouds soee3410 ken...