(net403) another day, another billion packets

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Eric Brandwine, AWS Security

October 2015

NET403

Another Day, Another Billion

Packets

Deja Vu

We have the cloud

EBS

RDS ElastiCacheAmazon

Redshift

EC2 Elastic Load

Balancing

Customers have data centers

Whiteboard engineering

EBS

RDS ElastiCacheAmazon

Redshift

EC2 Elastic Load

Balancing

EC2 as it was

10.44.12.4 10.44.12.5

10.44.92.1710.44.12.27

10.108.6.4

Why that doesn’t work

192.168.0.0/16

Routing Table

• 192.168.0.0/16: stay here

• 10.44.12.4/32: AWS

• 10.44.92.17/32: AWS

• 10.108.6.4/32: AWS

10.44.0.0/16

10.44.12.4 10.44.12.5

10.44.92.1710.44.12.27

10.108.6.4

Requirements

Customer selected IP addresses

Route aggregation for external connectivity

Conformance with existing network designs

Virtual private cloud

172.31.0.0/18

192.168.0.0/16

Routing Table

• 192.168.0.0/16: stay here

• 172.31.0.0/18: AWS

172.31.1.0/24 172.31.2.0/24

172.31.1.7

172.31.1.8

172.31.1.9

172.31.2.12

172.31.2.51

This is just virtual networking!

Subnet ~= VLAN

VPC ~= VRF (virtual routing and forwarding)

But…

Scaling challenges

VLAN ID space is constrained

• 12 bits => 4096 total VLANs

VRF support is constrained

• Large routers => 1-2 thousand VRFs

Fixed ratio of VLANs:VRFs

Router and capacity dimensions

Big Router

Data Plane

Control

Plane

Big Router

Data Plane

Control

Plane

An example

Average router configuration line: 50 chars

Config per VPC: 10 lines

Subnets per VPC: 4

Config per subnet: 5 lines

Total VPCs: 2,000

Config size: 3 MB

But…

Doesn’t scale

• 12 bit VLAN ID = 4096 VLANs (not enough)

• BIG routers support 4000 VRFs ($200k+)

Large VLANs make NEs cry

Tied to vendor bugfix cycles (6 months +)

We want commodity, fungible network gear

• BIG virtual routers are built by few companies

• Interoperability of advanced features is marginal

Silos of capacity

A

C

B

FE

D

G

A AA

A

B

C

B B

B B

C

D

F FF

D

D

B

G G

/4 /4

/40 /40

0

0

0

0

1324 132

C

G G

3 27

D DD

9910

F F F F F

1815 40

BB B B B

BB B B B

BB B B B

B B

Implementation requirements

Scale to millions of environments the size of Amazon.com

Any server, anywhere in a region can host an instance

attached to any subnet in any VPC

Concepts

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

…

10.0.0.2

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

Server:

Physical host in an

Amazon datacenter

Instance:

Amazon EC2

instance owned by a

customer

VPC:

Amazon Virtual

Private Cloud

owned by a

customer

VPC ID:

Identifier for a VPC

such as vpc-

1a2b3c4d

Mapping Service:

Distributed lookup

service. Maps VPC

+ Instance IP to

server

L2 - Ethernet

10.0.0.2

10.0.0.3

L2 Src: MAC(10.0.0.2)

L2 Dst: ff:ff:ff:ff:ff:ff

ARP Who has

10.0.0.3?

The switch floods the

ARP request out all

ports

Ethernet Switch

L2 Src: MAC(10.0.0.3)

L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.3 is at

MAC(10.0.0.3)

The switch snoops the

ARP response and

learns the port for

MAC(10.0.0.3).

L2 Src: MAC(10.0.0.2)

L2 Dst: MAC(10.0.0.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.0.3

ICMP/TCP/UDP/…

L2 - VPC

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

L2 Src: MAC(10.0.0.2)

L2 Dst: ff:ff:ff:ff:ff:ff

ARP Who has

10.0.0.3?

L2 Src: MAC(10.0.0.3)

L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.3 is at

MAC(10.0.0.3)

Src: 192.168.0.3

Dst: Mapping Service

Query:

Blue 10.0.0.3

Src: Mapping Service

Dst: 192.168.0.3

Reply:

Host: 192.168.1.4

MAC: MAC(10.0.0.3)

10.0.0.2

Server 192.168.0.3

Server 192.168.0.4

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

…

L2 Src: MAC(10.0.0.2)

L2 Dst: MAC(10.0.0.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.0.3

ICMP/TCP/UDP/…

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.1.4

Src: 192.168.1.4

Dst: Mapping Service

Validate:

Blue 10.0.0.2 is at

192.168.0.3

Src: Mapping Service

Dst: 192.168.1.4

Mapping valid:

Blue10.0.0.2 is at

192.168.0.3

L2 - VPC

…

VPC isolation

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

Src: 192.168.0.4

Dst: Mapping Service

Query:

Grey 10.0.0.3

L2 Src: MAC(10.0.0.4)

L2 Dst: ff:ff:ff:ff:ff:ff

ARP Who has

10.0.0.3?

VPC isolation

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

Src: 192.168.0.4

Dst: Mapping Service

Query:

Blue 10.0.0.3

L2 Src: MAC(10.0.0.4)

L2 Dst: ff:ff:ff:ff:ff:ff

ARP Who has

10.0.0.3?

192.168.0.4 is not

hosting any instances

in VPC Blue.

Mapping Denied

Alarm Raised

VPC isolation

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

…

L2 Src: MAC(10.0.0.4)

L2 Dst: MAC(10.0.0.3)

L3 Src: 10.0.0.4

L3 Dst: 10.0.0.3

ICMP/TCP/UDP/…

VPC: Blue

Src: 192.168.0.4

Dst: 192.168.1.4

Src: 192.168.1.4

Dst: Mapping Service

Validate:

Blue 10.0.0.4 is at

192.168.0.4

Src: Mapping Service

Dst: 192.168.1.4

Mapping invalid!

192.168.1.4 does not

deliver the packet to

the instance.

Alarm Raised.

L3 – IP routing

10.0.0.2

10.0.1.3

L2 Src: MAC(10.0.0.2)

L2 Dst: ff:ff:ff:ff:ff:ff

ARP Who has

10.0.0.1?

Ethernet Switch

L2 Src: MAC(10.0.0.1)

L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.1 is at

MAC(10.0.0.1)

L2 Src: MAC(10.0.0.2)

L2 Dst: MAC(10.0.0.1)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

RouterEthernet Switch

L2 Src: MAC(10.0.1.1)

L2 Dst: MAC(10.0.1.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

L3 - VPC

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.1.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

L2 Src: MAC(10.0.0.2)

L2 Dst: ff:ff:ff:ff:ff:ff

ARP Who has

10.0.0.1?

L2 Src: MAC(10.0.0.1)

L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.1 is at

MAC(10.0.0.1)

Src: 192.168.0.3

Dst: Mapping Service

Query:

Blue 10.0.0.1

Src: Mapping Service

Dst: 192.168.0.3

Reply:

Host: Gateway

MAC: MAC(10.0.0.1)

10.0.0.2

L3 - VPC

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

10.0.1.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

Src: 192.168.0.3

Dst: Mapping Service

Query:

Blue 10.0.1.3

Src: Mapping Service

Dst: 192.168.0.3

Reply:

Host: 192.168.1.4

MAC: MAC(10.0.1.3)

10.0.0.2

L2 Src: MAC(10.0.0.2)

L2 Dst: MAC(10.0.0.1)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.1.4

Src: 192.168.1.4

Dst: Mapping Service

Validate:

Blue 10.0.0.2 is at

192.168.0.3

Src: Mapping Service

Dst: 192.168.1.4

Mapping valid:

Blue 10.0.0.2 is at

192.168.0.3

L2 Src: MAC(10.0.1.1)

L2 Dst: MAC(10.0.1.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

Caching

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

…

10.0.0.2

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

L2 Src: MAC(10.0.1.1)

L2 Dst: MAC(10.0.1.3)

L3 Src: 10.0.0.2

L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

Getting home (or anywhere, really)

10.0.0.0/18

172.16.0.0/16

10.0.0.0/24 10.0.1.0/24

10.0.0.7

10.0.0.8

10.0.0.9

10.0.1.12

10.0.1.51

VPC: Blue

Src: 192.168.0.3

Dst: ???

L3 Src: 10.0.0.7

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

Edges

Server 192.168.0.3

Server 192.168.0.4

…

Edge 192.168.4.3

Edge 192.168.4.4

10.0.1.3

10.0.0.4

10.0.0.2

Mapping Service

10.0.0.2

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

Host 10.0.0.4 192.168.0.4

Host 10.0.1.4 192.168.0.4

…

172.16.0.0/16 Edge 192.168.4.3

…

Edges (three different ones)

Edge 192.168.4.3VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

IPSEC Stuff

Src: 54.68.100.245

Dst: 205.251.242.54

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

VPN

Edges (three different ones)

Edge 192.168.4.3VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

802.1Q VLAN Tag

Src: 54.68.100.245

Dst: 205.251.242.54

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

Direct Connect

Edges (three different ones)

Edge 192.168.4.3VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

L3 Src: 10.0.0.2

L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

Internet

54.148.157.46

Edges (three different ones)

VPNEdge 192.168.4.3

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

IPSEC Stuff

Src: 54.68.100.245

Dst: 205.251.242.54

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

Direct ConnectEdge 192.168.4.3

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

802.1Q VLAN Tag

Src: 54.68.100.245

Dst: 205.251.242.54

L3 Src: 10.0.0.2

L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

InternetEdge 192.168.4.3

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.3

L3 Src: 10.0.0.2

L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

L3 Src: 54.148.157.46

L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

Image credit: Wikipedia

https://en.wikipedia.org/wiki/1918_Eighth_Avenue

A brief diversion

VPC pricing

Cost per VPC: $0.00

Cost per subnet: $0.00

Upcharge per instance: $0.00

Nov 10, 2010

VPC as a platform

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7

172.31.1.8

172.31.2.12

172.31.2.51

VPC as a platform

VPN and Direct Connect

Security group egress filtering

Network ACLs

Routing tables

Elastic Network Interfaces (ENIs)

Multiple IPs

Amazon S3 endpoints

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7 172.31.2.12

Amazon S3 endpoints

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7 172.31.2.12

Server 192.168.0.3

Server 192.168.0.4

…

Edge 192.168.4.3

Edge 192.168.4.4

10.0.1.3

10.0.0.4

10.0.0.2

10.0.0.2

L3 Src: 10.0.0.2

L3 Dst: 54.231.33.89

TCP/HTTP/…

VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.4

L3 Src: 10.0.0.2

L3 Dst: 54.231.33.89

TCP/HTTP/…

EdgesMapping Service

Host 10.0.0.4 192.168.0.4

Host 10.0.1.4 192.168.0.4

…

172.16.0.0/16 Edge 192.168.4.3

S3.us-east-1 Edge 192.168.4.4

…

A new edge

Edge 192.168.4.4VPC: Blue

Src: 192.168.0.3

Dst: 192.168.4.4

L3 Src: 10.0.0.2

L3 Dst: 54.231.33.89

TCP/HTTP/…

VPC Endpoint 1a2b3c4d

Src: 54.68.100.245

Dst: 54.231.33.89

L3 Src: 10.0.0.2

L3 Dst: 54.231.33.89

TCP/HTTP/…

S3 endpoint

Endpoints & policy

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7 172.31.2.12

{

"Statement": [

{

"Sid": "Access-to-specific-bucket-only",

"Principal": "*",

"Action": [

"s3:GetObject",

"s3:PutObject"

],

"Effect": "Allow",

"Resource": ["arn:aws:s3:::my_secure_bucket",

"arn:aws:s3:::my_secure_bucket/*"]

}

]

}

{

"Statement": [

{

"Sid": "Access-to-specific-VPC-only",

"Principal": "*",

"Action": "s3:*",

"Effect": "Deny",

"Resource": ["arn:aws:s3:::my_secure_bucket",

"arn:aws:s3:::my_secure_bucket/*"],

"Condition": {

"StringNotEquals": {

"aws:sourceVpc": "vpc-111bbb22"

}

}

}

]

}

Simple Complex

Limited Flexible

EC2 VPC

Default VPC

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7

172.31.1.8

172.31.1.9

172.31.2.12

172.31.2.51

Simple Complex

Limited Flexible

EC2 - VPC

Thank you!

Remember to complete

your evaluations!