another day, another billion packets

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Shahbaz Alam, AWS Professional Services

August 2016

Another Day, Another Billion

Packets

Deja Vu

We have the cloud

Amazon

EBS

Amazon

RDS

Amazon

ElastiCache

Amazon

Redshift

Amazon EC2 Elastic Load

Balancing

We have customers

Some customers have existing data centers

Customers want to make their datacenters

work with the cloud

???

Whiteboard engineering

Amazon

EBS

Amazon

RDS

Amazon

ElastiCache

Amazon

Redshift

Amazon

EC2

Elastic Load

Balancing

EC2 as it was

10.44.12.4 10.44.12.5

10.44.92.1710.44.12.27

10.108.6.4

Why that doesn’t work

192.168.0.0/16

Routing Table

• 192.168.0.0/16: stay here

• 10.44.12.4/32: AWS

• 10.44.92.17/32: AWS

• 10.108.6.4/32: AWS

10.44.0.0/16

10.44.12.4 10.44.12.5

10.44.92.1710.44.12.27

10.108.6.4

Design Requirements

• Customer selected IP addresses

• Route aggregation for external connectivity

• Conformance with existing network designs

172.31.0.0/18

192.168.0.0/16

Routing Table

• 192.168.0.0/16: stay here

• 172.31.0.0/18: AWS

172.31.1.0/24 172.31.2.0/24

172.31.1.7

172.31.1.8

172.31.1.9

172.31.2.12

172.31.2.51

Amazon Virtual Private Cloud (VPC)

This is just virtual networking!

Subnet ~= VLAN

VPC ~= VRF (virtual routing and forwarding)

But…

Scaling challenges

VLAN ID space is constrained

• 12 bits => 4096 total VLANs

VRF support is constrained

• Large routers => 1-2 thousand VRFs

Fixed ratio of VLANs:VRFs

Router and capacity dimensions

Big Router

Data Plane

Control

Plane

Big Router

Data Plane

Control

Plane

An example

Average router configuration line: 50 chars

Config per VPC: 10 lines

Subnets per VPC: 4

Config per subnet: 5 lines

Total VPCs: 2,000

Config size: 3 MB

But…

This doesn’t scale

• 12 bit VLAN ID = 4096 VLANs (not

enough)

• BIG routers support 4,000 VRFs

($200k+)

Large VLANs make Network Engineers cry

Tied to vendor bugfix cycles (6 months +)

BIG virtual routers are built by few

companies

Interoperability of advanced features is

marginal

$$$

Silos of capacity (illustrative)

A

C

B

FE

D

G

A AA

A

B

C

B B

B B

C

D

F FF

D

D

B

G G

/4 /4

/40 /40

0

0

0

0

1324 132

C

G G

3 27

D DD

9910

F F F F F

1815 40

BB B B B

BB B B B

BB B B B

B B

Functional requirements

• Scale to millions of environments the size of

Amazon.com

• Any server, anywhere in a region can host an instance

attached to any subnet in any VPC

Let’s review: L2 – Ethernet

10.0.0.2

10.0.0.3

L2 Src: MAC(10.0.0.2)

L2 Dst: ff:ff:ff:ff:ff:ff

ARP Who has

10.0.0.3?

The switch floods the

ARP request out all

ports

Ethernet Switch

L2 Src: MAC(10.0.0.3)

L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.3 is at

MAC(10.0.0.3)

The switch snoops the

ARP response and

learns the port for

MAC(10.0.0.3).

L2 Src: MAC(10.0.0.2)

L2 Dst: MAC(10.0.0.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.0.3

ICMP/TCP/UDP/…

Let’s review: L3 – IP routing

10.0.0.2

10.0.1.3

L2 Src: MAC(10.0.0.2)


ARP Who has

10.0.0.1?

Ethernet Switch

L2 Src: MAC(10.0.0.1)

L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.1 is at

MAC(10.0.0.1)

L2 Src: MAC(10.0.0.2)

L2 Dst: MAC(10.0.0.1)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

RouterEthernet Switch

L2 Src: MAC(10.0.1.1)

L2 Dst: MAC(10.0.1.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

VPC Concepts

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

…

10.0.0.2

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

Server:

Physical hypervisor

in an Amazon data

center

Instance:

Amazon EC2

instance owned by a

customer

VPC:

Amazon Virtual

Private Cloud

owned by a

customer

VPC ID:

Identifier for a VPC

such as vpc-

1a2b3c4d

Mapping Service:

Distributed lookup

service. Maps VPC

+ Instance IP to

server

L2 - VPC

Server 192.168.0.3

Server 192.168.0.4

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

L2 Src: MAC(10.0.0.2)


ARP Who has

10.0.0.3?

L2 Src: MAC(10.0.0.3)

L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.3 is at

MAC(10.0.0.3)

Src: 192.168.0.3

Dst: Mapping Service

Query:

Blue 10.0.0.3

Src: Mapping Service

Dst: 192.168.0.3

Reply:

Host: 192.168.1.4

MAC: MAC(10.0.0.3)

10.0.0.2

Server 192.168.0.3

Server 192.168.0.4

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

…

L2 Src: MAC(10.0.0.2)

L2 Dst: MAC(10.0.0.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.0.3

ICMP/TCP/UDP/…

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.1.4

Src: 192.168.1.4


Validate:

Blue 10.0.0.2 is at

192.168.0.3


Dst: 192.168.1.4

Mapping valid:

Blue 10.0.0.2 is at

192.168.0.3

L2 - VPC

…

VPC isolation

Server 192.168.0.3

Server 192.168.0.4

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

Src: 192.168.0.4


Query:

Grey 10.0.0.3

L2 Src: MAC(10.0.0.4)


ARP Who has

10.0.0.3?

VPC isolation

Server 192.168.0.3

Server 192.168.0.4

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

Src: 192.168.0.4


Query:

Blue 10.0.0.3

L2 Src: MAC(10.0.0.4)


ARP Who has

10.0.0.3?

192.168.0.4 is not

hosting any instances

in VPC Blue.

Mapping Denied

Alarm Raised

VPC isolation

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

…

L2 Src: MAC(10.0.0.4)

L2 Dst: MAC(10.0.0.3)

L3 Src: 10.0.0.4

L3 Dst: 10.0.0.3

ICMP/TCP/UDP/…

VPC: Blue

Src: 192.168.0.4

Dst: 192.168.1.4

Src: 192.168.1.4


Validate:

Blue 10.0.0.4 is at

192.168.0.4


Dst: 192.168.1.4

Mapping invalid!

192.168.1.4 does not

deliver the packet to

the instance.

Alarm Raised.

L3 - VPC

Server 192.168.0.3

Server 192.168.0.4

Server 192.168.1.3

Server 192.168.1.4

10.0.1.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

L2 Src: MAC(10.0.0.2)


ARP Who has

10.0.0.1?

L2 Src: MAC(10.0.0.1)

L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.1 is at

MAC(10.0.0.1)

Src: 192.168.0.3


Query:

Blue 10.0.0.1


Dst: 192.168.0.3

Reply:

Host: Gateway

MAC: MAC(10.0.0.1)

10.0.0.2

L3 - VPC

Server 192.168.0.3

Server 192.168.0.4

Server 192.168.1.3

Server 192.168.1.4

10.0.1.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

Src: 192.168.0.3


Query:

Blue 10.0.1.3


Dst: 192.168.0.3

Reply:

Host: 192.168.1.4

MAC: MAC(10.0.1.3)

10.0.0.2

L2 Src: MAC(10.0.0.2)

L2 Dst: MAC(10.0.0.1)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.1.4

Src: 192.168.1.4


Validate:

Blue 10.0.0.2 is at

192.168.0.3


Dst: 192.168.1.4

Mapping valid:

Blue 10.0.0.2 is at

192.168.0.3

L2 Src: MAC(10.0.1.1)

L2 Dst: MAC(10.0.1.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

Caching

Server 192.168.0.3

Server 192.168.0.4

Server 192.168.1.3

Server 192.168.1.4

…

10.0.0.2

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

L2 Src: MAC(10.0.1.1)

L2 Dst: MAC(10.0.1.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

10.0.0.0/18

172.16.0.0/16

10.0.0.0/24 10.0.1.0/24

10.0.0.7

10.0.0.8

10.0.0.9

10.0.1.12

10.0.1.51

VPC: Blue

Src: 192.168.0.3

Dst: ???

L3 Src: 10.0.0.7

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

Getting home (or anywhere, really)

Edges

Server 192.168.0.3

Server 192.168.0.4

Edge 192.168.4.3

Edge 192.168.4.4

10.0.1.3

10.0.0.4

10.0.0.2

Mapping Service

10.0.0.2

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

Host 10.0.0.4 192.168.0.4

Host 10.0.1.3 192.168.0.4

…

172.16.0.0/16 Edge 192.168.4.3

…

Edges (three different ones) – VPN

Edge 192.168.4.3VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

IPSEC Stuff

Src: 54.68.100.245

Dst: 205.251.242.54

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

Edges (three different ones) – AWS Direct Connect


Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

802.1Q VLAN Tag

Src: 54.68.100.245

Dst: 205.251.242.54

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

Edges (three different ones) – Internet


Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

L3 Src: 10.0.0.2

L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

54.148.157.46

Edges (three different ones)

VPNEdge 192.168.4.3

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

IPSEC Stuff

Src: 54.68.100.245

Dst: 205.251.242.54

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

Direct ConnectEdge 192.168.4.3

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

802.1Q VLAN Tag

Src: 54.68.100.245

Dst: 205.251.242.54

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

InternetEdge 192.168.4.3

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

L3 Src: 54.148.157.46

L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

Image credit: Wikipedia

https://en.wikipedia.org/wiki/1918_Eighth_Avenue

A brief diversion – Fun Fact

Back to our regularly scheduled program…

Amazon S3

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7 172.31.2.12

Amazon S3 endpoints

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7 172.31.2.12

Server 192.168.0.3

Server 192.168.0.4

Edge 192.168.4.3

Edge 192.168.4.4

10.0.1.3

10.0.0.4

10.0.0.2

10.0.0.2

L3 Src: 10.0.0.2

L3 Dst: 54.231.33.89

TCP/HTTP/…

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.4

L3 Src: 10.0.0.2

L3 Dst: 54.231.33.89

TCP/HTTP/…

EdgesMapping Service

Host 10.0.0.4 192.168.0.4

Host 10.0.1.3 192.168.0.4

…

172.16.0.0/16 Edge 192.168.4.3

S3.us-east-1 Edge 192.168.4.4

…

A new edge – S3 endpoint


Src: 192.168.0.3

Dst: 192.168.4.4

L3 Src: 10.0.0.2

L3 Dst: 54.231.33.89

TCP/HTTP/…

VPC Endpoint 1a2b3c4d

Src: 54.68.100.245

Dst: 54.231.33.89

L3 Src: 10.0.0.2

L3 Dst: 54.231.33.89

TCP/HTTP/…

Endpoints and policy

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7 172.31.2.12

{

"Statement": [

{

"Sid": "Access-to-specific-bucket-only",

"Principal": "*",

"Action": [

"s3:GetObject",

"s3:PutObject"

],

"Effect": "Allow",

"Resource": ["arn:aws:s3:::my_secure_bucket",

"arn:aws:s3:::my_secure_bucket/*"]

}

]

}

{

"Statement": [

{

"Sid": "Access-to-specific-VPC-only",

"Principal": "*",

"Action": "s3:*",

"Effect": "Deny",

"Resource": ["arn:aws:s3:::my_secure_bucket",

"arn:aws:s3:::my_secure_bucket/*"],

"Condition": {

"StringNotEquals": {

"aws:sourceVpc": "vpc-111bbb22"

}

}

}

]

}

Nov 10, 2010

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7

172.31.1.8

172.31.2.12

172.31.2.51

VPC as a platform

Simple Complex

Limited Flexible

EC2 VPC

VPC pricing

Cost per VPC: $0.00

Cost per subnet: $0.00

Upcharge per instance: $0.00

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7

172.31.1.8

172.31.1.9

172.31.2.12

172.31.2.51

Default VPC

Simple Complex

Limited Flexible

EC2 - VPC

VPC CIDR 10.1.0.0/16

Availability Zone A Availability Zone B

Public Subnet Public Subnet

Private Subnet Private Subnet

Instance A10.1.1.11 /24

Instance B10.1.2.22 /24

Instance C10.1.3.33 /24

Instance D10.1.4.44 /24

10.1.1.0/16

10.1.2.0/16

10.1.3.0/16

And Today…

Public Subnet

Remember to complete

your evaluations!

another day, another billion packets

Technology