Public Cloud Connection for R&E Network

Jin Tanaka APAN-JP/KDDI

45th APAN Meeting in Singapore 28th March 2018

Hyper Scale Public cloud and research & science data

US Fermilab Use AWS to analyze data generated by CERN's LHCNASA EOSDIS(Earth Observing System Data and Information

System are now running in AWS

RIKEN in Japan provides genome data analysis environment to Japanese R&E institutes via Microsoft Azure

World wide government and educational institutes use AWS

large research institute in Japan

Public cloud services on Global NRENs

GEANT Cloud Services

Internet2 NET+

SINET5 direct connect service AARNET CONNECT

NSF announced new collaboration with public cloudsNSF Adds $30M to BIGDATA Program; AWS, Google, and Azure Participate

https://www.nsf.gov/news/news_summ.jsp?cntn_id=244450

▸ The National Science Foundation (NSF) is providing nearly $30 million(3years) in new funding for research in data science and engineering through its Critical Techniques, Technologies and Methodologies for Advancing Foundations and Applications of Big Data Sciences and Engineering (BIGDATA) program.

▸ NSF's awards are paired with support from Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure, which have each committed up to $3 million in cloud resources for relevant BIGDATA projects over a three-year period, beginning with this year’s awards.

Cloud connection that the R&E network aims forImprove User Experience▸ R&E network is a high performance, low latency than public internet ▸ Google cloud is also aiming for high performance networking ▸ It is important to have high-speed access to Public cloud in order to

improve user experience ▸ Each NREN peers with public cloud at local IX ▸ If IX is not nearby, consider connecting to nearby global open exchange point

▸ Where is the Network PoP and edge router of each major public cloud? We need to investigate the physical location of public cloud within the region Public Internet

R&E Network

Cloud connection that can be realized NREN in AsiaNREN try to connects with public cloud as much as possible in Asian main PoP

▸ Mapping the topology of major public cloud PoPs and APAN/Asia connect network

▸ NREN connects to the nearest cloud PoP via IX and peer with them ▸ BGP based IP peering

▸ In particular, there are many clouds and commercial IXs, in Tokyo, Hong Kong, and Singapore are the best

▸ Commercial traffic not related cloud may flow, so NOC needs monitor traffic

NREN

NREN

NREN

NREN

NREN

Asia connect

Asia connect

Asia connect

NREN

NREN

NREN

NREN

NREN

Microsoft AzureAWS

Google GCP

NREN

IP Peering

How to connect to public cloud

INTERNET VPN DIRECT CONNECT

IPsec VPN tunnel connection between VPC and remote network •AWS VPC

- VPN Gateway •Azure Virtual Network

- VPN Gateway •Google Cloud Virtual

Network - VPN

• SD-WAN Options

Virtual machines can be set an Internet gateway to connect public Internet by user setting or by default. Mainly use for the providing a service with VMs in public subnet. Not suitable for sending important data from campus/lab

Can establish private/dedicate connections to public clouds consistent performance •AWS

- Direct Connect •Azure

- Express Route •Google

- Cloud Interconnect

Connecting models of public clouds are almost same

High Performance Secure Privacy

Reduce bandwidth cost

▸ Classified into 3 connection methods: Public internet, VPN, and direct connect ▸ Some skills of network technology are required for the VPN and direct connect ▸ To reduce the burden on researchers and information, NREN engineers should

understand the cloud connection and learn its skill ▸ Network component and definition for each cloud services ▸ Configuration of IPsec VPN, Q-in-Q, BGP of each vendors

How to connect to public cloud

INTERNET VPN DIRECT CONNECT

IPsec VPN tunnel connection between VPC and remote network •AWS VPC

- VPN Gateway •Azure Virtual Network

- VPN Gateway •Google Cloud Virtual

Network - VPN

• SD-WAN Options

Virtual machines can be set an Internet gateway to connect public Internet by user setting or by default. Mainly use for the providing a service with VMs in public subnet. Not suitable for sending important data from campus/lab

Can establish private/dedicate connections to public clouds consistent performance •AWS

- Direct Connect •Azure

- Express Route •Google

- Cloud Interconnect

Connecting models of public clouds are almost same

High Performance Secure Privacy

Reduce bandwidth cost

▸ Classified into 3 connection methods: Public internet, VPN, and direct connect ▸ Some skills of network technology are required for the VPN and direct connect ▸ To reduce the burden on researchers and information, NREN engineers should

understand the cloud connection and learn its skill ▸ Network component and definition for each cloud services ▸ Configuration of IPsec VPN, Q-in-Q, BGP of each vendors

VPN ConnectionIPsec via Internet▸ Encrypted tunnel connections between your campus/lab and public cloud ▸ User prepare IPsec hardware router in your campus/lab as VPN end point ▸ Cisco, Juniper, Fortinet, Palo Alto, etc.

▸ Main parts to the configuration ▸ IPsec, IKE, Tunnel ▸ AES 128-bit/256-bit encryption, SHA-1/SHA-2 hashing

▸ Supports static routing and BGP(Option), and redundant gateway ▸ SD-WAN enables manage the public cloud WAN like a branch ▸ Centralized configuration and policy management across on premise and cloud

end-points

AWS Direct connectEstablishes a secure, dedicated connection to AWS▸ Can reduce costs, increase bandwidth, and provide a more consistent

network experience than Internet-based connections ▸ Single AWS direct connection allow us to build multi-region services ▸ Options to connect

▸ Physically direct connect at AWS Direct Connect location (1G, 10G only) ▸ Network service by AWS Direct Connect partner who is a member of the AWS Partner

Network (APN) (10G, 1G, Sub-1G) ▸ Main parts to the configuration

▸ 802.1Q VLAN, BGP session and MD5 authentication, IPv4 and IPv6 ▸ BGP community to help control the scope(regional or global)and route preference

AWS Direct connectEstablishes a secure, dedicated connection to AWS▸ Can reduce costs, increase bandwidth, and provide a more consistent

network experience than Internet-based connections ▸ Single AWS direct connection allow us to build multi-region services ▸ Options to connect

▸ Physically direct connect at AWS Direct Connect location (1G, 10G only) ▸ Network service by AWS Direct Connect partner who is a member of the AWS Partner

Network (APN) (10G, 1G, Sub-1G) ▸ Main parts to the configuration

▸ 802.1Q VLAN, BGP session and MD5 authentication, IPv4 and IPv6 ▸ BGP community to help control the scope(regional or global)and route preference

AWS Direct connect location related to APAN

Global Switch, Singapore Equinix SG2, Singapore iAdvantage Mega-i, Hong Kong KINX, Seoul, South Korea Equinix TY2, TY6 - TY8, Tokyo, Japan GPX, Mumbai, India Equinix SY1 - SY4, Sydney, Australia Global Switch, Sydney, Australia

https://aws.amazon.com/directconnect/details/

Google Cloud InterconnectAccess to GCP over high speed and stable network▸ Dedicated Interconnect ▸ This solution allows you to directly connect your on-premises network to GCP ▸ Requires you to have a connection in a Google supported colocation facility ▸ The minimum deployment per location is 10 Gbps. ▸ Main parts to the configuration

▸ EBGP-4 with multi-hop, 802.1Q VLAN, RFC 1918 address space ▸ Direct peering ▸ Connect your campus/lab directly to Google at any of 100+ locations in 33 countries ▸ is the simplest!!

▸ Carrier peering ▸ If you cannot satisfy Google’s peering requirements, you can connect via a Carrier

Peering partners

Google Cloud InterconnectAccess to GCP over high speed and stable network▸ Dedicated Interconnect ▸ This solution allows you to directly connect your on-premises network to GCP ▸ Requires you to have a connection in a Google supported colocation facility ▸ The minimum deployment per location is 10 Gbps. ▸ Main parts to the configuration

▸ EBGP-4 with multi-hop, 802.1Q VLAN, RFC 1918 address space ▸ Direct peering ▸ Connect your campus/lab directly to Google at any of 100+ locations in 33 countries ▸ is the simplest!!

▸ Carrier peering ▸ If you cannot satisfy Google’s peering requirements, you can connect via a Carrier

Peering partners

GCP interconnect colocation facility location related to APAN

Global Switch, Singapore Equinix SG2, Singapore iAdvantage Mega-i, Hong Kong Equinix Hong Kong (HK2) Equinix TY2, Tokyo, Japan GPX, Mumbai, India Equinix SY3, Sydney, Australia NEXTDC S1, Sydney, Australia

https://cloud.google.com/interconnect/docs/concepts/colocation-facilities

Microsoft Azure Express routeProvides private network access to 3 collections of Microsoft Azure resources▸ More reliability, faster speeds, and lower latencies than Internet connections ▸ Express Route circuit consists of 2 redundant connections to Microsoft Edge ▸ Connectivity to Azure public, Azure private, and Microsoft (Office365,CRM) ▸ Global connectivity with ExpressRoute premium add-on ▸ Ports have an oversubscription ratio of 4:1 ▸ Options to connect

▸ CloudExchange Co-location, Point-to-point Ethernet Connection, IP-VPN connection ▸ Bandwidth 50M,100M ,200M, 500M, 1G, 2G, 5G, 10G

▸ Main parts to the configuration ▸ 802.1ad(Q-inQ), BGP(community, Local preference, AS path prepend), etc.

Microsoft Azure Express routeProvides private network access to 3 collections of Microsoft Azure resources▸ More reliability, faster speeds, and lower latencies than Internet connections ▸ Express Route circuit consists of 2 redundant connections to Microsoft Edge ▸ Connectivity to Azure public, Azure private, and Microsoft (Office365,CRM) ▸ Global connectivity with ExpressRoute premium add-on ▸ Ports have an oversubscription ratio of 4:1 ▸ Options to connect

▸ CloudExchange Co-location, Point-to-point Ethernet Connection, IP-VPN connection ▸ Bandwidth 50M,100M ,200M, 500M, 1G, 2G, 5G, 10G

▸ Main parts to the configuration ▸ 802.1ad(Q-inQ), BGP(community, Local preference, AS path prepend), etc.

Microsoft Azure Express route location and NW provider related to APAN

NW: AARnet, SINET, GEANT, Intenet2

Location : Singapore, Hong Kong Seoul, Mumbai, India, Tokyo Sydney, Australia

https://docs.microsoft.com/en-us/azure/expressroute/expressroute-locations-providers#locations

NREN

NRENAsia

connect

NREN

NREN

NREN

IP Peering

VPN Model

Connect to the nearest public cloud in Asian region with VPN

NREN

NREN

▸ Since there is not much difference from the current IP level connection, it is easiest to have secure connection to the cloud

▸ NREN don’t need to set virtual circuits or additional routing protocol, simple

▸ Science flows will traverse the public Internet unless steps are taken to ingress and egress onto R&E networks instead of cloud provider transit networks

▸ The public internet is highly fragmented and not engineered to support the large science data

Asia connect

NREN

NREN

NREN

We NREN NOCs are BGP Expert! We should provide high performance and high speed network

We NREN NOCs are BGP Expert!

Why don’t we challenge to provide direct connect services !

We should provide high performance and high speed network

Dedicated Direct connect Model

Establish high capacity connection with Direct Connect or Direct Peer

▸ Connect directly to main public cloud PoP that will be near your country

▸ NREN connects to the public cloud by creating direct connection to supports the campus research institute

▸ It is possible to connect with the public cloud with high capacity and low latency, and support the large science data

▸ NREN's designated switches at direct connect location of cloud provides VLAN connectivity with neighboring NREN router ▸ Tokyo, HongKong, Singapore

NREN

NREN

NREN

NREN

NREN

NREN

NREN

Dedicated VLAN

NREN

NREN

Asia connect

Asia connect

NREN

Designated Switch

Open Exchange Point ModelOpen Exchange Point may be responsible for cloud exchange of R&E Network

‣ OXP in Asian region should has a capability to support direct connect between the public cloud and NRENs (by stand-alone or each-other)‣ NREN ensures connectivity for user institutions (as

usual) ‣ NREN connects to one or more OXPs

‣ Public cloud has NREN connectivity through: ‣ Direct connections to NREN ‣ Connections to one or more OXPs

‣ Networking should meet the agility of cloud service ‣ Deploy a dynamically controlled switch on demand

and connect with the public cloud edge ‣ Provide flexible and scalable bandwidth between

NREN and cloud services for efficient use of network resources ‣ 10M…100M…200M…500M…1G…10G

NRENOpen

Exchange Point

NREN

NREN

Open Exchange Point

Asia connect

Backbone

NREN

OXP should be direct connect provider of public clouds

What’s issues?▸ Technical Point ▸ Definition of provisioning flow when OXP set VLANs ▸ API inter-working for matching the VLAN-IDs on OXP and

user-IDs on public cloud services ▸ Direct connection is best but commercial commercial

cloud exchange solves the difficulty of technology ▸ Commercial cloud exchange uses SDN technology for

agility, R&E GXP must also implement similar technology

User Interface

OXP Switch

API

API

DBSDN App

SDN Interface

▸ Partner ship ▸ How we APAN make collaboration model

between public cloud providers ▸ Collaboration between public cloud at the

global level is required

▸ Implementation ▸ In case of direct connection public cloud and

elaborate testing will be necessary ▸ First of all, we will start trial in GXP Japan

planned in Tokyo

SDN capability for cloud change service at OXP

Distribute Open Exchange Point in Tokyo(Planning)

References

▸ AWS https://aws.amazon.com http://www.iic.hokudai.ac.jp/pdf/20170905_07_Amazon-ML_Public.pdf

▸ Microsoft Azure https://azure.microsoft.com https://www.microsoft.com/ja-jp/casestudies/riken.aspx

▸ Google Cloud Platform https://cloud.google.com

▸ NSF https://www.nsf.gov/news/news_summ.jsp?cntn_id=244450