aws re:invent 2016: another day, another billion packets (net401)

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Steve Mueller, Senior Product Manager, VPC, EC2 Networking

November 30, 2016

Another Day, Another Billion Packets

NET401

We Have the Cloud

Amazon

EBS

Amazon

RDS

Amazon

ElastiCache

Amazon

Redshift

Amazon EC2 Elastic Load

Balancing

Customers Have Datacenters

Whiteboard Engineering

Amazon

EBS

Amazon

RDS

Amazon

ElastiCache

Amazon

Redshift

Amazon

EC2

Elastic Load

Balancing

EC2 as It Was

10.44.12.4 10.44.12.5

10.44.92.1710.44.12.27

10.108.6.4

Why That Doesn’t Work

192.168.0.0/16

Routing Table

• 192.168.0.0/16: stay here

• 10.44.12.4/32: AWS

• 10.44.92.17/32: AWS

• 10.108.6.4/32: AWS

10.44.0.0/16

10.44.12.4 10.44.12.5

10.44.92.1710.44.12.27

10.108.6.4

Requirements

Customer selected IP addresses

Route aggregation for external connectivity

Conformance with existing network designs

172.31.0.0/18

192.168.0.0/16

Routing Table

• 192.168.0.0/16: stay here

• 172.31.0.0/18: AWS

172.31.1.0/24 172.31.2.0/24

172.31.1.7

172.31.1.8

172.31.1.9

172.31.2.12

172.31.2.51

Amazon Virtual Private Cloud (Amazon VPC)

This Is Just Virtual Networking!

Subnet ~= VLAN

VPC ~= VRF (virtual routing and forwarding)

But…

Scaling Challenges

VLAN ID space is constrained

• 12 bits => 4096 total VLANs

VRF support is constrained

• Large routers => 1-2 thousand VRFs

Fixed ratio of VLANs:VRFs

Router and Capacity Dimensions

Big Router

Data Plane

Control

Plane

Big Router

Data Plane

Control

Plane

An Example

Average router configuration line: 50 chars

Config per VPC: 10 lines

Subnets per VPC: 4

Config per subnet: 5 lines

Total VPCs: 2,000

Config size: 3 MB

But…

Doesn’t scale

• 12 bit VLAN ID = 4096 VLANs (not enough)

• BIG routers support 4000 VRFs ($200k+)

Large VLANs make NEs cry

Tied to vendor bugfix cycles (6 months +)

We want commodity, fungible network gear

• BIG virtual routers are built by few companies

• Interoperability of advanced features is marginal

Silos of Capacity

A

C

B

FE

D

G

A AA

A

B

C

B B

B B

C

D

F FF

D

D

B

G G

/4 /4

/40 /40

0

0

0

0

1324 132

C

G G

3 27

D DD

9910

F F F F F

1815 40

BB B B B

BB B B B

BB B B B

B B

Implementation Requirements

Scale to millions of environments the size of Amazon.com

Any server, anywhere in a region can host an instance

attached to any subnet in any VPC

Concepts

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

…

10.0.0.2

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

Server:

Physical host in an

Amazon data center

Instance:

Amazon EC2

instance owned by a

customer

VPC:

Amazon Virtual

Private Cloud

owned by a

customer

VPC ID:

Identifier for a VPC

such as vpc-

1a2b3c4d

Mapping Service:

Distributed lookup

service. Maps VPC

+ Instance IP to

server

Layer 2 (L2): Ethernet

10.0.0.2

10.0.0.3

L2 Src: MAC(10.0.0.2)

L2 Dst: ff:ff:ff:ff:ff:ff

ARP Who has

10.0.0.3?

The switch floods the

ARP request out all

ports

Ethernet Switch

L2 Src: MAC(10.0.0.3)

L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.3 is at

MAC(10.0.0.3)

The switch snoops the

ARP response and

learns the port for

MAC(10.0.0.3).

L2 Src: MAC(10.0.0.2)

L2 Dst: MAC(10.0.0.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.0.3

ICMP/TCP/UDP/…

Layer 2 (L2): VPC

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

L2 Src: MAC(10.0.0.2)


ARP Who has

10.0.0.3?

L2 Src: MAC(10.0.0.3)

L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.3 is at

MAC(10.0.0.3)

Src: 192.168.0.3

Dst: Mapping Service

Query:

Blue 10.0.0.3

Src: Mapping Service

Dst: 192.168.0.3

Reply:

Host: 192.168.1.4

MAC: MAC(10.0.0.3)

10.0.0.2

Server 192.168.0.3

Server 192.168.0.4

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

…

L2 Src: MAC(10.0.0.2)

L2 Dst: MAC(10.0.0.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.0.3

ICMP/TCP/UDP/…

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.1.4

Src: 192.168.1.4


Validate:

Blue 10.0.0.2 is at

192.168.0.3


Dst: 192.168.1.4

Mapping valid:

Blue10.0.0.2 is at

192.168.0.3

Layer 2 (L2): VPC

…

VPC Isolation

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

Src: 192.168.0.4


Query:

Grey 10.0.0.3

L2 Src: MAC(10.0.0.4)


ARP Who has

10.0.0.3?

VPC Isolation

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

Src: 192.168.0.4


Query:

Blue 10.0.0.3

L2 Src: MAC(10.0.0.4)


ARP Who has

10.0.0.3?

192.168.0.4 is not

hosting any instances

in VPC Blue.

Mapping Denied

Alarm Raised

VPC Isolation

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

…

L2 Src: MAC(10.0.0.4)

L2 Dst: MAC(10.0.0.3)

L3 Src: 10.0.0.4

L3 Dst: 10.0.0.3

ICMP/TCP/UDP/…

VPC: Blue

Src: 192.168.0.4

Dst: 192.168.1.4

Src: 192.168.1.4


Validate:

Blue 10.0.0.4 is at

192.168.0.4


Dst: 192.168.1.4

Mapping invalid!

192.168.1.4 does not

deliver the packet to

the instance.

Alarm Raised.

Layer 3 (L3): IP Routing

10.0.0.2

10.0.1.3

L2 Src: MAC(10.0.0.2)


ARP Who has

10.0.0.1?

Ethernet Switch

L2 Src: MAC(10.0.0.1)

L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.1 is at

MAC(10.0.0.1)

L2 Src: MAC(10.0.0.2)

L2 Dst: MAC(10.0.0.1)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

RouterEthernet Switch

L2 Src: MAC(10.0.1.1)

L2 Dst: MAC(10.0.1.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

Layer 3 (L3): VPC

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.1.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

L2 Src: MAC(10.0.0.2)


ARP Who has

10.0.0.1?

L2 Src: MAC(10.0.0.1)

L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.1 is at

MAC(10.0.0.1)

Src: 192.168.0.3


Query:

Blue 10.0.0.1


Dst: 192.168.0.3

Reply:

Host: Gateway

MAC: MAC(10.0.0.1)

10.0.0.2

Layer 3 (L3): VPC

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.1.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

Src: 192.168.0.3


Query:

Blue 10.0.1.3


Dst: 192.168.0.3

Reply:

Host: 192.168.1.4

MAC: MAC(10.0.1.3)

10.0.0.2

L2 Src: MAC(10.0.0.2)

L2 Dst: MAC(10.0.0.1)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.1.4

Src: 192.168.1.4


Validate:

Blue 10.0.0.2 is at

192.168.0.3


Dst: 192.168.1.4

Mapping valid:

Blue 10.0.0.2 is at

192.168.0.3

L2 Src: MAC(10.0.1.1)

L2 Dst: MAC(10.0.1.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

Caching

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

…

10.0.0.2

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

L2 Src: MAC(10.0.1.1)

L2 Dst: MAC(10.0.1.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

10.0.0.0/18

172.16.0.0/16

10.0.0.0/24 10.0.1.0/24

10.0.0.7

10.0.0.8

10.0.0.9

10.0.1.12

10.0.1.51

VPC: Blue

Src: 192.168.0.3

Dst: ???

L3 Src: 10.0.0.7

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

Getting Home – or Anywhere, Really

Edges

Server 192.168.0.3

Server 192.168.0.4

…

Edge 192.168.4.3

Edge 192.168.4.4

10.0.1.3

10.0.0.4

10.0.0.2

Mapping Service

10.0.0.2

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

Host 10.0.0.4 192.168.0.4

Host 10.0.1.4 192.168.0.4

…

172.16.0.0/16 Edge 192.168.4.3

…

Edges: VPN

Edge 192.168.4.3VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

IPSEC Stuff

Src: 54.68.100.245

Dst: 205.251.242.54

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

VPN

Edges: Direct Connect


Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

802.1Q VLAN Tag

Src: 54.68.100.245

Dst: 205.251.242.54

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

AWS

Direct Connect

Edges: Internet (IGW)


Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

L3 Src: 10.0.0.2

L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

Internet

54.148.157.46

Edges: Recap

VPNEdge 192.168.4.3

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

IPSEC Stuff

Src: 54.68.100.245

Dst: 205.251.242.54

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

AWS Direct ConnectEdge 192.168.4.3

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

802.1Q VLAN Tag

Src: 54.68.100.245

Dst: 205.251.242.54

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

InternetEdge 192.168.4.3

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

L3 Src: 54.148.157.46

L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

Image credit: Wikipedia

https://en.wikipedia.org/wiki/1918_Eighth_Avenue

A Brief Diversion

VPC Pricing

Cost per VPC: $0.00

Cost per subnet: $0.00

Upcharge per instance: $0.00

Nov 10, 2010

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7

172.31.1.8

172.31.2.12

172.31.2.51

VPC as a Platform

VPC as a Platform

VPN and Direct Connect

Security group egress filtering

Network ACLs

Routing tables

Elastic Network Interfaces (ENIs)

Multiple IPs

Amazon S3 Endpoints

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7 172.31.2.12

Server 192.168.0.3

Server 192.168.0.4

…

Edge 192.168.4.3

Edge 192.168.4.4

10.0.1.3

10.0.0.4

10.0.0.2

10.0.0.2

L3 Src: 10.0.0.2

L3 Dst: 54.231.33.89

TCP/HTTP/…

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.4

L3 Src: 10.0.0.2

L3 Dst: 54.231.33.89

TCP/HTTP/…

EdgesMapping Service

Host 10.0.0.4 192.168.0.4

Host 10.0.1.4 192.168.0.4

…

172.16.0.0/16 Edge 192.168.4.3

S3.us-east-1 Edge 192.168.4.4

…

A New Edge: S3


Src: 192.168.0.3

Dst: 192.168.4.4

L3 Src: 10.0.0.2

L3 Dst: 54.231.33.89

TCP/HTTP/…

VPC Endpoint 1a2b3c4d

Src: 54.68.100.245

Dst: 54.231.33.89

L3 Src: 10.0.0.2

L3 Dst: 54.231.33.89

TCP/HTTP/…

S3 endpoint

S3 Endpoints and Policy

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7 172.31.2.12

{

"Statement": [

{

"Sid": "Access-to-specific-bucket-only",

"Principal": "*",

"Action": [

"s3:GetObject",

"s3:PutObject"

],

"Effect": "Allow",

"Resource": ["arn:aws:s3:::my_secure_bucket",

"arn:aws:s3:::my_secure_bucket/*"]

}

]

}

{

"Statement": [

{

"Sid": "Access-to-specific-VPC-only",

"Principal": "*",

"Action": "s3:*",

"Effect": "Deny",

"Resource": ["arn:aws:s3:::my_secure_bucket",

"arn:aws:s3:::my_secure_bucket/*"],

"Condition": {

"StringNotEquals": {

"aws:sourceVpc": "vpc-111bbb22"

}

}

}

]

}

Simple Complex

Limited Flexible

EC2 VPC

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7

172.31.1.8

172.31.1.9

172.31.2.12

172.31.2.51

Default VPC

Simple Complex

Limited Flexible

EC2 - VPC

Related Sessions

NET201 - Creating Your Virtual Data Center: VPC Fundamentals and Connectivity Options

NET303 - NextGen Networking: New Capabilities for Amazon’s Virtual Private Cloud

NET304 - Moving Mountains: Netflix's Migration into VPC

NET402 - Deep Dive: AWS Direct Connect and VPNs

NET404 - Making Every Packet Countr

Remember to complete

your evaluations!

Thank you!

aws re:invent 2016: another day, another billion packets (net401)

Technology