lisa2017 big three cloud networking
TRANSCRIPT
![Page 1: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/1.jpg)
October 29–November 3, 2017 | San Francisco, CAwww.usenix.org/lisa17 #lisa17
The Ins-and-Outs of Networking in the Big Three Clouds
Chris "mac" McEniry
![Page 2: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/2.jpg)
Introduction
2
![Page 3: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/3.jpg)
Topics• Network Substrates
• Routing, Routing, Routing
• Access Control
3
![Page 4: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/4.jpg)
• Focus on the Big Three Cloud Service Providers (B3CSP)
• Slides are indicated with what Cloud Provider it's talking about
• Amazon Web Services
• Azure
• Google Cloud Platform
4
![Page 5: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/5.jpg)
What about Cloud X?This will not be covered...
5
![Page 6: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/6.jpg)
What's the best cloud?This will be covered... (5 seconds)
6
![Page 7: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/7.jpg)
"It Depends"That's always the answer to questions without requirements...
7
![Page 8: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/8.jpg)
Not Covered: "Legacy" Models• Focus is on current practices. Some legacy practices are not even available to new
accounts.
• Two Main Areas
• Organization: Some providers have a first pass on account or resource organization. Not Covered.
• Networking: Each provider has a first pass on their network offerings. Not Covered:
• EC2 Classic
• Virtual Network Classic
• Legacy Networking
8
![Page 9: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/9.jpg)
Not Covered: IPv6 Addresses• Changes assumptions about Internal/External IP allocation
• Not the majority of use cases (yet)
• Support
• AWS: Yes
• Azure: ?
• GCP: No
9
![Page 10: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/10.jpg)
Beware of Limits/Quotas• http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/
VPC_Appendix_Limits.html
• https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#networking-limits
• https://cloud.google.com/compute/docs/vpc/#quotas_and_limits
• https://cloud.google.com/router/quotas
10
![Page 11: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/11.jpg)
Design Exercises• Pen and paper exercises
• Goals
• Better understanding of concepts
• Compare/Contrast implementations in the B3CP
• Best to attempt same goal with all three providers, but one is sufficient for this tutorial.
• Some scattered throughout. More at the end
• Write down assumptions you make when designing
11
![Page 12: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/12.jpg)
Announcements since 9/15
12
![Page 13: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/13.jpg)
• 9/20 VPC Resizing: Adding secondary CIDR
• 11/1: Direct Connect Gateway, 3 more CloudFront POPs, Direct Connect Pricing Update
• (11/27: re:Invent)
• Early 2019: Bahrain Region
13
![Page 14: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/14.jpg)
• 9/21 Availability Zones
• 9/26 (Ignite) VNet Service Endpoints. DDoS Protection for VNets. NSG using Application Security Groups, Service Tags, more rule combinations. 1 Gbps VPN connection. Monitoring ExpressRoute, S2S VPN. ExpressRoute Public access for MS Services, Route Filters. P2S macOS support. P2S AD Authentication. More ExpressRoute Partners.
• 9/27 VNet Integration for Azure Storage and Azure SQL. Storage Firewalls. SQL Endpoints.
• 10/31 Triple capacity in China
14
![Page 15: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/15.jpg)
• 9/19 Sao Paulo Region
• (10/3 Custom IAM Roles)
• 10/7 Multiple (<=8) NICs GA.
• 10/31 Mumbai Region. Dedicate Interconnect GA, more POPs.
• 11/2 Faster SDN
15
![Page 16: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/16.jpg)
Organizing
16
![Page 17: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/17.jpg)
Organizing• Structure of CSP physical and logical/administrative build outs has impacts
on network capabilities and topology
• Examples
• What do you need to account for to build resiliency into the infrastructure?
• How do you maintain separation of administrative domains (e.g. account) while allowing traffic to traverse where needed?
• Does it make sense to do large shared networks, or smaller finely tuned networks?
17
![Page 18: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/18.jpg)
Physical Organization• All B3CSP have Regions
• Geographically separate facilities hosting resources
• All B3CSP have some sub-Region container (separate power, network hardware, etc)
• Availability Zones
• Fault Domains
• Zones
18
![Page 19: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/19.jpg)
Availability Zones, Zones• Separate Data Centers that are close to each other
• AWS: "less than 2 ms latency between each other"
• Directly exposed to customer
• You have to choose where to put resources
19
![Page 20: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/20.jpg)
Fault Domains, Paired Regions• Fault Domain is similar to AZ/Z but is not directly manipulated by customer
• Specify intent by putting resources into Availability Sets
• "I want my 5 web servers to not fail all at the same time"
• Availability Set distributed across Fault Domains*
• Paired Regions: Regions in the same geographic region but are kept separate (> 300 miles, not operated on at the same time, etc)
• East US<->West US, UK West<->UK South
20
![Page 21: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/21.jpg)
Locations
21
Regions
AWS 16 44 Availability Zones
Azure 26(not counting Gov)
60 Fault Domains(not counting Gov)
GCP 12 36 Zones
![Page 22: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/22.jpg)
22
![Page 23: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/23.jpg)
23
![Page 24: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/24.jpg)
24
![Page 25: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/25.jpg)
Logical/Administrative Organization• Each CSP has a way of administering users and resource permissions
• Comparisons:
• Where users are
• Where resources are
• Where permissions are set
• Fundamentally asking "Who can spin up a VM in this network? Who can stop a VM that is running in this network?"
25
![Page 26: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/26.jpg)
(Billing) Accounts• Single unified item
• Hard boundary which contains all resources associated with it - can't share
• Can be in an organization, but only resources are not shared across - only policies
• Can build IAM accounts inside of this (or roles which connect from other accounts)
• All permissions are primarily set for actions (not targets) on the account*
• Must change account/role into another account to operate on its resources
26
![Page 27: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/27.jpg)
Account / Tenant• Single unified item - Microsoft or Organizational Account
• Hard boundary which holds all resources associated with it
• Subdivided into Subscriptions, and then Resource Groups
• Access Control can be place on Subscriptions and/or Resource Groups
• Resources bound to one Resource Group/Subscription at a time
• Can move (with some restrictions) resources across Subscriptions/RGs
27
![Page 28: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/28.jpg)
Cloud Resource Hierarchy• Organization
• Root of hierarchy
• Projects
• Core organizational component
• Associated with Organizations (or stand alone)
• Can have own permissions (and inherit)
• Resources
• Associated with Projects
• Can have own permissions (and inherit)
• In some cases, can be shared with other projects
28
![Page 29: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/29.jpg)
Tags• Instances can be tagged
• Tags are used in selection items
• Firewall
• Routing
29
![Page 30: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/30.jpg)
Network Substrate
30
![Page 31: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/31.jpg)
Network Substrate• Virtual version of the traditional physical networks
• Handles the basic packet forwarding
• Organized into Subnets
• Supports resilience - spread over different resources
• Supports administrative separation - grouping of similar services
31
![Page 32: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/32.jpg)
Not like a physical network• All packet forwarding based on some sort of mapping or hidden
networking layer
• Unicast only: No support for Broadcast or Multicast
• No transit networking through the substrate
32
![Page 33: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/33.jpg)
Virtual Private Cloud (VPC)• A single CIDR allocated to a Region in one Account
• Subnets are smaller contained CIDRs assigned to an Availability Zone
• Subnet CIDRs can't overlap
• VPC CIDR can overlap regardless of Region/Account (as long as you don't want to connect them together)
• Default VPC per Region (172.31.0.0/16) with /20 subnets in each AZ
• Used for resilience and administrative separation
33
![Page 34: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/34.jpg)
VPC Addressing• Can be RFC1918 or other IP space
• AWS will not advertise space out to the Internet
• Each object is a contiguous CIDR
• Allocated at time of object (VPC, Subnet) creation
• Can't change without destroying old/creating new
34
![Page 35: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/35.jpg)
VirtualNet (VNet)• Single CIDR allocated to one Region in one Subscription
• Subnets are smaller contained CIDRs that span Fault Domains for that Region
• Subnet CIDRs can't overlap
• VNET CIDRs can overlap regardless of Region/Subscription (as long as you don't want to connect them together)
• Used for administrative separation
35
![Page 36: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/36.jpg)
VNet Addressing• Can be RFC1918 or other IP space
• Azure will not advertise space out to the Internet
• Single CIDR at time of creation for VNet
• Subnets can change CIDR if it's not in use at all
• Must support a couple of special subnets
• First contiguous CIDR of VNet : Used for Load Balancers
• "Gateway Subnet" : Used for putting gateway devices (i.e. VPN)
36
![Page 37: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/37.jpg)
Virtual Private Cloud Network (VPC Network)
• Global private communication space allocated to a Project
• Can be shared with other Projects
• Subnets are single CIDRs allocated to a Region, and can span Zones
• Two modes of allocation (one way switch from Auto -> Custom)
• Auto Mode: Allocate a subnet to each region. Can manually add your own.
• Custom Mode: Manually build subnets (recommended)
• Each subnet has primary CIDR
• (In Custom Mode) Can allocate secondary CIDR - typically for container networks
37
![Page 38: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/38.jpg)
VPC Network Addressing • RFC1918 space only
• Auto Mode: Predefined /20 to each region. E.g.
• 10.128/20 -> us-central1
• 10.132/20 -> europe-west1
• Instances have primary IP and can have alias IPs
• Alias IPs can come from primary CIDR or secondary CIDR
• Can use CIDR (not just host) for Alias IPs on secondary
• Secondary CIDR does not reserve network/gateway IPs
38
![Page 39: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/39.jpg)
AWS Azure GCP
Name Virtual Private Cloud Virtual Network Virtual Private Cloud Network
IP Addressing
RFC1918 or OtherCarving up CIDR of VPC
RFC1918 or OtherCarving up CIDR of VNet
RFC1918 onlyAccumulation of Subnet CIDRs
Locality One Region One Region Global
Subnet Locality One AZ Region Selection of Zones in a Region
CIDR Changes Fixed at creation Only if nothing is using it Can increase CIDR
Account Resource Sharing
NoUsers use multiple Subs.
Resources bound to one VNet inside one Sub at a time
Across Projects - YesAcross Organizations - No
![Page 40: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/40.jpg)
Resiliency• Build out a Network Substrate
• Capable of surviving a failure of one subregion area (AZ/FD/Z)
• Supports a 3 Tier application (web/app/db) with clear delineation
40
![Page 41: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/41.jpg)
Instance Properties
41
![Page 42: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/42.jpg)
Instance Properties• IP Forwarding / Source-Destination Checking
• Checks whether a packet headed to the instance matches the IP(s) of the instance, or whether a packet leaving the instance matches the IP(s) of the instance
• Instance NIC Properties
• Number of IPs per NIC
• Number of NICs per VM
• Locations of NICs
42
![Page 43: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/43.jpg)
AWS Azure GCP
Forwarding Property Source/Destination Check IP Forwarding
(enableIPForwarding) IP Forwarding (can-ip-forward)
Property Default On Off Off
NIC Name Elastic Network Interface Virtual Network Interface Cards Network Interface
IPs per NIC 6-50 50Unspecified
(Alias IP not supported with multiple NICs)
NICs per Instance 1-15 2-8 1-8
NIC Locations Same VPC Same VNet Each must be on separate VPC
Networks
![Page 44: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/44.jpg)
Routing - Inside
44
![Page 45: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/45.jpg)
Route Tables• Route Table (rtb) = Multiple sets of routes
• Default or Main Route Table is nothing specific is configured
• Each Subnet is associated with a single Route Table
• Routes are either static (manually configured) or propagated (from BGP connections)
• Priority
• Most specific match
• Static
• Propagated
45
![Page 46: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/46.jpg)
Route Tables• Route in a Route Table = Prefix + Next Hop
• Next Hop Types
• Instance NIC (Src/Dst Check)
• Peering Connection: For connections to other VPCs
• Virtual Private Gateway: For connections to VPNs or Private Circuits
• NAT Gateway: For egress NATing
• VPC Endpoint: For supported AWS Services
• Internet Gateway
46
![Page 47: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/47.jpg)
System Routes + BGP + UDR• All Subnets have System Routes
• Can't be removed, but can be shadowed by custom routes
• Can add additional (custom) User Defined Routes via a Route Table
• Priority
• Most specific match
• User Defined Route
• BGP System Route
• System Internet Route
47
![Page 48: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/48.jpg)
System Routes• 3 Default Route Sets automatically associated with VNet
• Local VNet's Subnets
• Associated Networks: Routes propagated from Peering / VPN / ExpressRoute
• Internet
48
![Page 49: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/49.jpg)
User Defined Routes• Customer configured routes that exist additionally to System Routes
• Prefix + Next Hop. Next Hop one of:
• (Local) VNet: For VNet CIDR Destinations
• Virtual Network Gateway: Site-to-Site Connection
• Virtual Appliance: VMs inside VNet (IP Forwarding)
• Internet
• None: Blackhole
• Can't route back into a subnet
49
![Page 50: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/50.jpg)
Routes• Global Routing Table for a VPC Network
• Custom Static Routes
• Priority
• Most specific match
• Highest priority (lowest by value)
• Multipath (Hash: Protocol, Src IP, Dst IP, Src Port, Dst Port)
50
![Page 51: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/51.jpg)
Routes• Route =
• Name + VPC network + Prefix + (optional) Instance Tags + Priority + Next Hop.
• Next Hop one of
• Instance
• IP - inside of network (must be a primary IP)
• Gateway - The internet gateway (though maybe more options in the future)
• VPN Tunnel
51
![Page 52: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/52.jpg)
AWS Azure GCP
Name Route Tables System Routes +User Defined Routes Routes
Route Selection
Most specific CIDR match,Static
Propagated
Most Specific CIDR match,User Defined Routes,
BGP Routes,System Routes
Most specific CIDR match,then by priority,
then mutlipath hash
Route Sharing
All subnets associated with same Route Table
All subnets associated with same User Defined Route
One shared route table;Specific route rules applied via
instance tag
Default Routes for New
VPC CIDRInternet
Peered ConnectionsVNet CIDR
InternetSubnet CIDRs
![Page 53: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/53.jpg)
Use an Instance as a Gateway• Connect two different network areas together
• Use an instance as a gateway
53
![Page 54: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/54.jpg)
Routing - Internet
54
![Page 55: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/55.jpg)
Ingress/Egress• For Instance to get in/out, it has to have
• Route to Internet ("Route - Inside")
• Policy to permit access to/from Internet ("Access Control")
• Allocated Public IP on the CSP's external network
• Or - something else can do the work for it
55
![Page 56: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/56.jpg)
CSP Public IP Selection• All Internet routed addresses come from the CSP's allocated Internet IP space
• Allocated to a Region
• No BYOIP
• Two types
• Ephemeral Addresses: allocated and released when VM starts/terminates
• Static Addresses: allocated ahead of time and remain attached to the Account even if the VM is terminated (called Elastic IP for AWS)
• Azure/GCP: Can promote an ephemeral public IP to static
56
![Page 57: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/57.jpg)
Public IP == Public NAT No Public IP == Possibly Only Egress Traffic*
• If a VM has a Public IP (ephemeral or static) associate with it, it gets a NAT for Internet traffic
• The remainder of this section talk mostly about the "No Public IP" cases
57
![Page 58: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/58.jpg)
Internet Gateway (IGW)• No default Internet Route
• Have to allocate an IGW to the VPC
• Have to add route to IGW
• Instance has to be configured with a public IP
58
![Page 59: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/59.jpg)
NAT Gateway (NGW)• AWS Managed Service which provides SNAT for egress traffic only
• Must provide AWS with designated Elastic IP (public side) and subnet which has an internet route
• Other subnets must use a different Route Table to send 0/0 to the NAT Gateway
• Assigned to an Availability Zone
• But can support multiple AZs --- impact on resilience
• Common patterns to see one ngw per AZ
• Dependent on an IGW+Route for the NGW's outside access
59
![Page 60: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/60.jpg)
Shared SNAT• Default Internet Route provided
• Default shared SNAT
• Shared == possibly with other accounts
• Prevent this with Access Control (NSG)
60
![Page 61: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/61.jpg)
Ephemeral IP• Default Internet Route provided
• Default ephemeral public IP provided on each instance
• can choose not to allocate
• can be limited with Organization Policy (beta)
61
![Page 62: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/62.jpg)
NAT?• No Managed Service
• You stand up your own instance which is performing NAT
• Add 0/0 to override routes
• Can use tags to decide which Instances get external access
• Can use tags to decide which Instances get NAT access
62
![Page 63: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/63.jpg)
AWS Azure GCP
Default Internet
Route for New Net
None Yes Yes
Default NAT None SNAT Ephermeral IP
![Page 64: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/64.jpg)
DMZ + Private• Want DMZ hosts which can get out, and Internal hosts which have to go
through the DMZ to get out
64
![Page 65: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/65.jpg)
Routing - CSP Networks
65
![Page 66: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/66.jpg)
Routing - CSP Networks• On their networks, CSPs have...
• Other Network Substrates that we may want to privately communicate with
• Other Cloud Services
• These services may exist in local or remote Regions
• CSPs have a lot of similar behaviors - differences highlighted
66
![Page 67: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/67.jpg)
VPC Peering (pcx)• Private connectivity between two VPCs in the same Region
• Same or different accounts
• Latency/bandwidth/cost is same as talking inside a VPC
• Non-overlapping IP space; Non-transitive
• Add routes to the Route Table(s) to be connected
• Can reference Security Groups in foreign VPC
67
![Page 68: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/68.jpg)
VPC Endpoints (vpce)• Private connectivity between VPC and AWS Services
• Limited to same region services only
• Non-transitive
• Only some services provide VPC Endpoints - S3, DynamoDB
• Given an identifier (pl-xxxx) to use in Route Tables
• In some service policies, can use VPC as a permission item (i.e. this VPC can access this service)
• Can reference VPC Endpoint (pl-xxxx) in Security Groups (but not NACL)
68
![Page 69: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/69.jpg)
Virtual Network Peering• Private connectivity between two VNets in the same Region
• Can be different Subscriptions but same Account*
• Non-overlapping IP space; Non-transitive, but can share VPN Gateway
• Latency/bandwidth/cost is same as talking inside a VPC
• System Routes are automatic for Peered VNet
• Can override with UDR to point to instances (on either side) as gateway
69* Being worked on, but in the interim, use VPN peering
![Page 70: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/70.jpg)
VPC Network Peering• Private connectivity between VPC Networks (in all Regions)
• Same or different projects/organizations
• Latency/bandwidth/cost is same as talking inside a VPC
• Non-overlapping IP space; Non-transitive
• Once peered, can't create conflicts (overlapping subnets, routes overlapping, etc) among the full set of VPC Networks that are peered (even between two that aren't direct)
70
![Page 71: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/71.jpg)
Private Google Access• Private connectivity (IP space exposed) to Google Services
• Can be enabled on a Subnet basis
• Must still use internet route to access (can be tag limited)
71
![Page 72: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/72.jpg)
AWS Azure GCP
Peering Name VPC Peering VNet Peering VPC Network Peering
Scope Across AWS Accounts Across Azure Subscriptions Across GCP Organizations
Limits 50/125 Peers 10/50 Peers 25 Peers7500 Instances Combined
CSP Service Peering Name
VPC Endpoint N/A Private Google Access
![Page 73: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/73.jpg)
Peers + Cloud Services• Peer Two Networks in the same CSP (same Account/Subscription/Project)
• Allow (preferably) private connections from both networks to CSP Cloud Services
73
![Page 74: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/74.jpg)
Routing - Private Gateways
74
![Page 75: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/75.jpg)
Private Gateways• Connection points for linking CSP Network to an outside CSP network
• VPN
• Physical
• Can be managed by a third party - Cloud Exchanges
• In all cases, you can run your own VPN software (third party VM appliances)
75
![Page 76: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/76.jpg)
CSP Managed VPN• Use of IPSec devices to connect privately to an on-premise (or other)
network
• Connect VPC/VNet/VPC Network to Corporate Office or Data Center
• Connect VPC/VNet/VPC Network to other CSP Network
• Inter-Region VPC/VNet network connectivity
76
![Page 77: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/77.jpg)
Physical• Private Circuit connecting your network at a CSP POP to an associated
CSP Region
• If not adjacent to POP, have to get tail circuit to connect your site to POP
• Can also be used to peer with Public side of CSP network
• Access to other CSP services
• Access to CSP management interfaces
77
![Page 78: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/78.jpg)
Limits• VPNs and Physical Connections have limits
• In general, can setup parallel connections and rely on multipath
• Multiple VPN tunnels working together
• Multiple Physical connections working together
• Combination of Physical and VPN - preference usually goes to Physical
78
![Page 79: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/79.jpg)
Virtual Private Gateway (VGW)• Logical router sitting outside of VPC
• Associated with single VPC; so limited to a Region
• All foreign networks come in via VGW
• Set routes in the VPC Route Tables to forward traffic to VGW
• Or allow routes to be propagated from the VGW peers
• Only the VPC CIDR is advertised to VGW and its peers
• Can't transit for VPC Peering or VPC Endpoints or Internet traffic - though can proxy
• Will transit for non-VPC peers - "CloudHub"
79
![Page 80: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/80.jpg)
Hardware VPN• Private IPSec connectivity between AWS and non-AWS managed VPN device (aka Customer
Gateway - CGW)
• Can't use between two AWS Regions
• Site-to-Site
• Always builds two tunnels (two devices are allocated on the AWS side) to one CGW
• If you want redundancy on the CGW side, you'll build four tunnels
• IP of AWS VPN side not designated until VPN Connection is configured
• Must supply static IP for CGW (can use same for multiple connections - NAT-T supported)
• Static or BGP
80
![Page 81: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/81.jpg)
Direct Connect (DX)• AWS Physical Connection (CON)
• 1Gbps, 10Gbps options (can do smaller through a Cloud Exchange)
• Can also use multiple (<=4) CON together with LACP
• BGP Peering Only
81
![Page 82: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/82.jpg)
Direct Connect (DX) - Sharing• Can be used with multiple accounts/VPCs.
• Each Direct Connect Connection (CON) is split into virtual interfaces (VIF).
• You assign the VIF information
• VLAN Assignment
• BGP Neighbor
• Hosting Account (one which has the CON) handles creation of VIF
• Guest Account attaches VIF to VGW
• Once attached, can't reattach (have to destroy/recreate)
82
![Page 83: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/83.jpg)
Virtual Network Gateway• Managed Gateway Devices
• Deployed as VMs inside of a dedicated "Gateway Subnet" (/27 or larger)
• Two types
• VPN Gateway
• ExpressRoute
• Can only have one of each type per VNet
• Multiple SKUs for sizing
83
![Page 84: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/84.jpg)
VPN Gateway• Site-to-Site
• IPSec from VPN Gateway to Local Network Gateway
• Allocated to a Region
• Static (Policy list of CIDRs) or BGP Route Based
• Limits/Sizing place preference on Route Based
• Gateway Transit: can extend reachability to Peered VNets
• Use this topology to get around some limits
84
![Page 85: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/85.jpg)
VPN Gateway• VNet-to-VNet
• (special case of Site-to-Site)
• Connect separate Regions and separate Accounts/Subscriptions together
• Point-to-Site
• Access via the in-box Windows VPN SSTP Client
85
![Page 86: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/86.jpg)
ExpressRoute• Private BGP Connections to Microsoft
• Handoff
• IP VPN (i.e. MPLS)
• Ethernet tail to POP
• Cloud Exchange Virtual Cross Connection
• Handoff affects location of on-prem and latency, but not necessarily Region access (depending on Subscription level)
• Offered as redundant pair of connections (no SLA without)
• Provides
• VNet Private Peering
• Azure Public Peering
• (add-on) Microsoft Peering for Office 365 et al
86
![Page 87: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/87.jpg)
ExpressRoute• For each Circuit, can setup one to all of the three peering types
• Peering Information:
• BGP IP/Neighbor IP
• VLAN
• Neighbor ASN - Public restrictions for Public/Microsoft
• For Microsoft: Routing Registry Name
• Restrictions on Peering Type for advertised routes (# and RFC1918)
87
![Page 88: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/88.jpg)
ExpressRoute to VNet• Owner Subscription handles Circuit creation and Peering configuration
• Owner Authorizes Other Subscriptions
• Other Subscription link VNets to ExpressRoute Circuit VNet Peering
• Must have a Virtual Network Gateway (ExpressRoute Type) configured ahead of time
• Standard vs Premium limits on # of VNet linkages and Region
• All VNets linked to the same ExpressRoute Circuit
• ExpressRoute and Point-to-Site are not supported together in the same VNet
88
![Page 89: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/89.jpg)
Cloud Router• Managed Router which handles all dynamic routing via BGP relationships with other
routers
• Makes changes to your Routes
• Scoped to local or global route changes (i.e. advertise remote Regions)
• You assign a private ASN to Cloud Router
• Create multiple interfaces (link-local addressing)
• Create multiple BGP neighbor configurations
• Unlike the other CSPs, even if you setup tunnel devices, the Cloud Router is the BGP neighbor
89
![Page 90: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/90.jpg)
Cloud VPN Gateway• Managed IPSec VPN Endpoint which can run multiple tunnels
• Allocated to a Region, but can forward any traffic on the VPC Network
• Can connect too another Project's or Organization's VPN Gateway
• You specify tunnels to create (so not always in pairs - but that is recommended)
• Typical Tunnel Configuration
• Local IP: Assigned from your pool of static external IPs
• (Static mode) Local subnets, IP ranges - Can't change after creating
• Peer IP, IKE, ESP, PFS, Shared Secret
• NAT-T not supported
90
![Page 91: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/91.jpg)
Cloud VPN Gateway• Static Mode
• Specify traffic selection via --remote-traffic-selector
• Add VPN Gateway as a NextHop in your route table
• Dynamic Mode
• Create an interface on Cloud Router that is associate to the tunnel
• Handle all traffic selection via BGP relationship with Cloud Router
• Cloud Router adds VPN Gateway as NextHop
• BGP Peer Information can be configured or have GCP generate it
91
![Page 92: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/92.jpg)
Dedicated Interconnect• GCP Physical Connection
• 10 Gbps
• Can use multiple with LACP
• BGP Established with Cloud Router (i.e. requires Cloud Router)
• Is divided up into VLAN Attachments
• GCP allocates the VLAN and BGP IPs to use (ASN specified by Cloud Router)
• Use that information to configure Cloud Router and on-prem router
92
![Page 93: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/93.jpg)
Dedicated Interconnect - Sharing• Can be shared among Projects in an Organization
• Even if not using a Shared VPC Network
• Two methods - both using IAM permissions
• Hosting Project grants permissions to Guest Project's Users to update Interconnect
• Guest Project grants permissions to Hosting Project to update Gust Project's Cloud Router
93
![Page 94: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/94.jpg)
AWS Azure GCP
Gateway Name Virtual Private Gateway (VGW) Virtual Network Gateway Cloud Router
Gateway Model
Router outside of VPCIs NextHop for all gateway traffic
VM Insides of "Gateway Subnet" which handle routing and
gateway
BGP Talker that updates routes. Separate gateways that forward
traffic.
Substrate Route Mgmt
You add routes to RTB, orsetup to propagate all routes
from VGW to RTBRoutes show up as System
Routes Cloud Router updates Routes
Gateway Peer Transit Yes
VPN: Yes including VNet PeeringExpressRoute: No Yes
![Page 95: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/95.jpg)
AWS Azure (VNet Peering) GCP
VPN Name
VPN Connection Virtual Network Gateway (Type=VPN) aka VPN Gateway
Cloud Interconnect - IPSec VPNaka Cloud VPN
Options Site-to-Site Point-to-SiteSite-to-Site Site-to-Site
Deploy Model
Attached to VGW Managed VMs deployed to Gateway Subnet as NextHop
Endpoint associated with VPC Network as NextHop
VPN Transit Yes (CloudHub) No Yes
Size Options
1 3 (4) 1
Advertised Routes
1 Prefix 1 Prefix Static: 128 PrefixesBGP: 100 Prefixes (CR limit)
Received Route limits
100 Prefixes . 100 Prefixes (CR limit)
Other Side Customer Gateway (CGW) Local Network Gateway Peer VPN Gateway
Route Selection
Static: What is specifiedBGP: VPC + All VGW BGP Peers
Static: What is specifiedBGP: VNet, Configured Peers
Static: What is specifiedBGP: By Cloud Router
![Page 96: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/96.jpg)
AWS Azure GCP
Physical Name Direct Connect Express Route Cloud Interconnect - Dedicated
Interconnect
Link Options 1Gbps, 10Gbps, LACP
IP VPN50,100,200,500M ; 1,2,5,10Gbps
Cloud Exchange10Gbps, LACP
Routing BGP BGP BGP(to Cloud Router)
VLAN/BGP Allocation You Pick You Pick GCP Picks
Sharing YesCentrally Managed
YesCentrally Managed Configuration
RBAC VNet Linking
YesRBAC Managed
Route Limits 100 Prefixes Depends on Size
VNet: 4000-10000 100 (CR limit)
![Page 97: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/97.jpg)
CSP Network as Data Center Extension• Setup CSP Network which only has connectivity from a Private Data
Center via a physical connection
• Any ingress or egress traffic goes via the Private Data Center
97
![Page 98: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/98.jpg)
Cross Region Network• Want to peer two Networks in the Same VPC
98
![Page 99: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/99.jpg)
Connect Multiple CSP Together• Have two CSP Networks
• One in each of two different CSPs
• "Peer" them together
99
![Page 100: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/100.jpg)
Cloud Storage from Corp Office• Setup protected path from corporate office to Storage offering of CSP
100
![Page 101: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/101.jpg)
Access Control
101
![Page 102: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/102.jpg)
AWS Access Control• Two types
• Security Groups (SG): For instances
• Network Access Control Lists (NACLs): For subnets
102
![Page 103: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/103.jpg)
Security Group (SG)• Stateful Packet Filters
• Separate lists for ingress and egress
• Applied on Network Interfaces
• By "Applied to Instance," it really means "Applied on the Instance's Primary NIC"
• Positive Control: Only ALLOW. No DENY
• Can apply multiple (5) SG on an NIC
• ALLOWs are cumulative
103
![Page 104: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/104.jpg)
Security Group (SG)• Structure:
• Protocol
• Port Range, or ICMP Type
• Source (ingress), or Destination (egress) CIDR, or SG
• New SG has default egress allows anything out
• "default" SG available when VPC created (and always available)
• Ingress: ALLOW from "default" SG to ANY
• Egress: ALLOW to ANY
104
![Page 105: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/105.jpg)
Security Group (SG)• Applied to Instances
• Context is focused on Instance (or groups of Instances) Rules
• But with tagging is hard to tailor to only your instances for modifying SG
105
![Page 106: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/106.jpg)
Network Access Control List (NACL)• Stateless
• Only one allowed per subnet
• Ordered priority
• Positive and Negative control (ALLOW/DENY)
106
![Page 107: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/107.jpg)
Azure Access Control• Just one
• Network Security Groups (NSG)
107
![Page 108: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/108.jpg)
Network Security Group (NSG)• 1 per NIC and/or subnet
• Applied in priority order (ascending)
• default rules for inbound and outbound on both (permit out, permit VNET)
• The NSG on the NIC, if present, and on the Subnet have to permit to permit traffic. One DENY on either will block traffic.
• NSG Tags : canned collections of CIDRs (e.g. VIRTUAL_NETWORK)
• Can do flow logging
108
![Page 109: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/109.jpg)
Network Security Group (NSG)• Structure
• Priority (Low numerically is Higher Priority)
• Name
• Protocol + Source IP/Tag + Source Port + Destination IP/Tag + Destination Port
• Allow/Deny
109
![Page 110: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/110.jpg)
Network Security Group (NSG)• Managing
• Action (Permissions): Microsoft.Network/networkSecurityGroup/*
• Roles: Network Contributor, Owner
• Context is global to VNet
110
![Page 111: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/111.jpg)
Firewall• Single managed firewall for VPC network
• Manages inbound and outbound
• Can limit sources/destinations using Target Tags
• Tag matched against Tags on Instance
111
![Page 112: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/112.jpg)
Firewall• Structure
• Priority (ascending, first match wins)
• (Ingress) Protocol + Destination Port + Source IP
• (Egress) Protocol + Destination Port + Destination IP
• Target Tags
• Allow/Deny
112
![Page 113: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/113.jpg)
Firewall• Managing
• Permissions: compute.firewalls.*
• Role: roles/compute.securityAdmin
• Context is global to Project/Share VPC Project
113
![Page 114: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/114.jpg)
AWS AWS Azure GCP
Name Security Group Network Access Control List Network Security Group Firewall
State Stateful Stateless Stateful Stateful
Additive Positive Only Positive/Negative Postive/Negative Positive/Negative
Target NIC Subnet NIC or Subnet VPC(target tags for instance)
Multiple Yes (5 on instance 250 in account) No No No
Ordered No Yes Yes Yes
![Page 115: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/115.jpg)
Setup Access Control for 3 Tier App• Given 4 VMs - web, app, db, bastion - setup access control such that:
• Anyone is allowed from any tcp port to "web" on 443/tcp
• "web" is allowed from any tcp port to "app" on 8009/tcp
• "app" is allowed from any tcp port to "db" on 3306/tcp
• "bastion" is allowed from any tcp port to all three instances on 22/tcp
• "bastion" is allowed from any udp port to all three instances on 161/udp
115
![Page 116: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/116.jpg)
Summary and Next Steps
116
![Page 117: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/117.jpg)
Locations
117
Regions
AWS 16 44 Availability Zones
Azure 26(not counting Gov)
60 Fault Domains(not counting Gov)
GCP 12 36 Zones
![Page 118: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/118.jpg)
AWS Azure GCP
Name Virtual Private Cloud Virtual Network Virtual Private Cloud Network
IP Addressing
RFC1918 or OtherCarving up CIDR of VPC
RFC1918 or OtherCarving up CIDR of VNet
RFC1918 onlyAccumulation of Subnet CIDRs
Locality One Region One Region Global
Subnet Locality One AZ Region Selection of Zones in a Region
CIDR Changes Fixed at creation Only if nothing is using it Can increase CIDR
Account Resource Sharing
NoUsers use multiple Subs.
Resources bound to one VNet inside one Sub at a time
Across Projects - YesAcross Organizations - No
Substrate
![Page 119: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/119.jpg)
AWS Azure GCP
Forwarding Property Source/Destination Check IP Forwarding
(enableIPForwarding) IP Forwarding (can-ip-forward)
Property Default On Off Off
NIC Name Elastic Network Interface Virtual Network Interface Cards Network Interface
IPs per NIC 6-50 50Unspecified
(Alias IP not supported with multiple NICs)
NICs per Instance 1-15 2-8 1-8
NIC Locations Same VPC Same VNet Each must be on separate VPC
Networks
Instance Properties
![Page 120: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/120.jpg)
AWS Azure GCP
Name Route Tables System Routes +User Defined Routes Routes
Route Selection
Most specific CIDR match,Static
Propagated
Most Specific CIDR match,User Defined Routes,
BGP Routes,System Routes
Most specific CIDR match,then by priority,
then mutlipath hash
Route Sharing
All subnets associated with same Route Table
All subnets associated with same User Defined Route
One shared route table;Specific route rules applied via
instance tag
Default Routes for New
VPC CIDRInternet
Peered ConnectionsVNet CIDR
InternetSubnet CIDRs
Routing - Inside
![Page 121: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/121.jpg)
AWS Azure GCP
Default Internet
Route for New Net
None Yes Yes
Default NAT None SNAT Ephermeral IP
Routing - Internet
![Page 122: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/122.jpg)
AWS Azure GCP
Peering Name VPC Peering VNet Peering VPC Network Peering
Scope Across AWS Accounts Across Azure Subscriptions Across GCP Organizations
Limits 50/125 Peers 10/50 Peers 25 Peers7500 Instances Combined
CSP Service Peering Name
VPC Endpoint N/A Private Google Access
Routing - CSP Networks
![Page 123: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/123.jpg)
AWS Azure GCP
Gateway Name Virtual Private Gateway (VGW) Virtual Network Gateway Cloud Router
Gateway Model
Router outside of VPCIs NextHop for all gateway traffic
VM Insides of "Gateway Subnet" which handle routing and
gateway
BGP Talker that updates routes. Separate gateways that forward
traffic.
Substrate Route Mgmt
You add routes to RTB, orsetup to propagate all routes
from VGW to RTBRoutes show up as System
Routes Cloud Router updates Routes
Gateway Peer Transit Yes
VPN: Yes including VNet PeeringExpressRoute: No Yes
Routing - Private Routes
![Page 124: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/124.jpg)
AWS Azure (VNet Peering) GCP
VPN Name
VPN Connection Virtual Network Gateway (Type=VPN) aka VPN Gateway
Cloud Interconnect - IPSec VPNaka Cloud VPN
Options Site-to-Site Point-to-SiteSite-to-Site Site-to-Site
Deploy Model
Attached to VGW Managed VMs deployed to Gateway Subnet as NextHop
Endpoint associated with VPC Network as NextHop
VPN Transit Yes (CloudHub) No Yes
Size Options
1 3 (4) 1
Advertised Routes
1 Prefix 1 Prefix Static: 128 PrefixesBGP: 100 Prefixes (CR limit)
Received Route limits
100 Prefixes . 100 Prefixes (CR limit)
Other Side Customer Gateway (CGW) Local Network Gateway Peer VPN Gateway
Route Selection
Static: What is specifiedBGP: VPC + All VGW BGP Peers
Static: What is specifiedBGP: VNet, Configured Peers
Static: What is specifiedBGP: By Cloud Router
Routing - VPN
![Page 125: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/125.jpg)
AWS Azure GCP
Physical Name Direct Connect Express Route Cloud Interconnect - Dedicated
Interconnect
Link Options 1Gbps, 10Gbps, LACP
IP VPN50,100,200,500M ; 1,2,5,10Gbps
Cloud Exchange10Gbps, LACP
Routing BGP BGP BGP(to Cloud Router)
VLAN/BGP Allocation You Pick You Pick GCP Picks
Sharing YesCentrally Managed
YesCentrally Managed Configuration
RBAC VNet Linking
YesRBAC Managed
Route Limits 100 Prefixes Depends on Size
VNet: 4000-10000 100 (CR limit)
Routing - On-Premise
![Page 126: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/126.jpg)
AWS AWS Azure GCP
Name Security Group Network Access Control List Network Security Group Firewall
State Stateful Stateless Stateful Stateful
Additive Positive Only Positive/Negative Postive/Negative Positive/Negative
Target NIC Subnet NIC or Subnet VPC(target tags for instance)
Multiple Yes (5 on instance 250 in account) No No No
Ordered No Yes Yes Yes
Access Control
![Page 127: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/127.jpg)
Additional "Network" Areas• Load Balancers - some affect SNAT and packet forwarding
• DNS - interacts with Substrate DNS view
• CDN
127
![Page 128: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/128.jpg)
Limits!!!• http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/
VPC_Appendix_Limits.html
• https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#networking-limits
• https://cloud.google.com/router/quotas
128
![Page 129: LISA2017 Big Three Cloud Networking](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a6e7a5d7f8b9a7a058b5da3/html5/thumbnails/129.jpg)
October 29–November 3, 2017 | San Francisco, CAwww.usenix.org/lisa17 #lisa17
Remember to fill in yourtutorial evaluation!
Thank You!
R6 - The Ins-and-Outs of Networking in the Big Three CloudsChris "mac" McEniry