Enhancing Network Visibility Based On Open-Converged
Network Appliance
Dongheon Lee, Junho Suh SW-Defined Infra Lab, SK telecom
Telco Services
Software-Defined Infrastructure
Open Hardware and Software
Universal Platform for Diverse Applications
Software-Defined RAN
Network & Service Slicing
Next-Generation OSS (TANGO)
Ultra High Data Rate
Low Latency
Massive Connectivity
ATSCALE: Scalable, Cognitive, Automated, Lean, E2E COSMOS: Composable, Open, Scalable, Mission-critical Optimized System
All-IT Infrastructure
Operational Intelligence
E2E monitoring and analytics for global orchestration and
management
Disaster prediction and recovery
Security
Bandwidth-on-demand service
Data center interconnection - stretching SDN coverage to the
transport layer
Data center backup and virtual machine migration
Performance and reliability
Open source software performance and reliability
Provisioning and live migration speed
SDN controller and switch performance
Static Dynamic
Manual Automated
Cloud Core
Cloud RAN/Core
New RAT
On-Demand
Virtual Network Slice
Mobile Traffic Explosion
All-IT Network Architecture and Challenges
TiNA for COSMOS Network Monitoring
TiNA: SKT integrated Network Analyzer
Unified network monitoring and operating
solution for software-defined data center
You Can't Manage What You Can't See
Network visibility is extremely important to network providers for managing All-IT Infra.
Troubleshooting application performance
issues
Monitoring application performance and
reliability
Ensuring network scalability
Protecting and securing the network
Managing complex network infrastructure
Network Visibility
TiNA: SKT integrated Network Analyzer
Unified network monitoring and operating solution
which includes essential systems and tools for network health monitoring, traffic
monitoring, packet analysis, session analysis, and troubleshooting.
Tennant traffic monitoring
Physical/virtual network visualization
End-to-end flow monitoring
Pakcet-Level: Packet into flow aggregation Connection performance analysis
Flow-Level: Tennant traffic classification Top-N conversations E2E Flow Path Calculation
On-demand packet brokering
w/ filtering/replication/aggregation
SNMP
sFlow, NetFlow
Flow-aware high speed
packet dump
Fast Packet Processing
DPDK based I/O, support L2-L4, VXLAN, ERSPAN, ….
Service-Centric Network Monitor 3D Network Visualization
Network Analyzer
Network Probe Network Packet Broker Packet Capture Appliance
X86 Commodity Servers
T-CAP is a converged network appliance,
which integrates high performance Xeon server with data center switch
T-CAP: SKT-Converged Appliance Platform
TiNA Overview
TiNA consists of 6 systems in which each system can be deployed as a standalone appliance or systems can be bundled together according to the use-case.
Network Packet Broker
Brokering network traffic from multiple mirror or taps according to the demand of multiple monitoring devices
Network Probe
Supporting network and traffic monitoring protocol such as SNMP and sFlow
Network Analyzer
Packet-level: Aggregates packets into flows and measures various TCP statistics
Flow-level: Tennant traffic classification, Top-N Conversations, E2E flow path calculation
Packet Capture & Analysis Appliance
High speed packet capture appliance supporting up to 40Gbps traffic
3D-Based Network Monitor
Supporting intuitive network monitoring and operation by efficiently visualizing complex network topology and traffic flow
Service Centric Network Monitor
Classifying tenant traffic and collecting various network traffic statistics of cloud data center
Flow-level network analyzer
support analyzing millions of information of nodes, links, and traffic flows by using spark streaming engine
Packet-level network analyzer
TCP connection analyzer support up to 1million TCP flows
Packet into Flow Aggregation : support up to 200K new flow arrivals per second
Network Packet Broker
Virtual and physical NPB, vNPB supports 40Gbps packet filtering & forwarding performance
Packet Capture Appliance
40Gbps packet capture system with 24 HDDs
Low Cost
implemented on x86 commodity hardware by leveraging DPDK fast packet processing framework
TiNA Specifications
Rack #1
SW SW
Switch
(L3)
SW SW SW SW
Rack#N
SW SW SW
Switch
(L3)
F/W F/W L4 L4
Rack#2
Mirror, sFlow, SNMP
Mirror, sFlow, SNMP
T-CAP
2 6
Service-Centric Network Monitor
3D Network
Visualization
Packet-Level Network Analyzer
Network Probe
Network Packet Broker
Packet Capture Appliance
Flow-Level Network Analyzer
Network Monitoring with TiNA
Connection Analyzer
Packet into Flow Aggregation
TCP Connection Analysis
Packet-Level
Network Analyzer
Network Probe
Packet-Level
Network Analyzer
Network Probe
NPB
NPB Controller
NPB Fabric
x86
vNPB
SNMP/sFlow Collector
Packet into Flow Aggregation
TCP Connection Analysis
SNMP/sFlow Collector
Virtual network packet broker
OpenFlow based packet
broker
T-CAP
T-CAP
Commodity Server
Packet-Flow Analysis with T-CAP
TCP Connection Performance Analysis
TCP Connection Setup Time, Connection per Sec, SYN/FIN Count, Retransmission
Rate, Zero Window, App Response Time
Determine whether the network problem is the application or the network
Packet-Level Network Analyzer:
Connection Analyzer
Network
Interface
Network
Interface
PMD
PMD
Writer
Writer
Writer
Writer
High Speed Packet Capture and Retrieval System
Essential tool for troubleshooting and network forensics
Pipelined parallel packet processing based on DPDK
Support both packet and flow-based capture
Specifications
20 cores (Intel Xeon CPU E5-2650 v3 @ 2.30 GHz, 2ea)
128 GB RAM (DDR4 16 GB DIMM 2,133 MHz, 8ea)
192 TB HDD (SATA 6 Gb/s 3.5” 7.2K RPM 8 TB ENT HDD, 24ea)
Support 40Gbps line rate packet capture to disk
With 192 TB storage
• 40 Gbps * 11 hour
• 10 Gbps * 43 hour
• 1 Gbps * 427 hour
Packet Capture Appliance
Elephant Flows
SONA: Simplified Overlay Network Architecture presented in ONS ‘16
Consolidated physical/virtual network based on real-time flow stats
L3-based leaf-spine networking to maximize link utilization
OpenStack Neutron
SONA
Compute Nodes G/W Nodes Physical Switches
SONA Fabric
Packet-Level Network Analyzer
Use-Case: Traffic Engineering
with TiNA and SONA
Spine
Switch
Server
Server
Server
Server
Switch
Server
Server
Server
Server
Switch
Server
Server
Server
Server
Spine
SONA Fabric
Controller
TiNA
Router
Use-Case: Traffic Engineering
with TiNA and SONA
Spine
Leaf
...
Cloud Data Center Multi-Tenancy
VM VM VM
VM VM VM
VM VM
VM VM
VM
VM
VM VM VM
VM VM VM
Baremetal
Baremetal
Tenant A Tenant B Tenant C Tenant D
Cloud needs to be managed and
monitored based on application
Per-tenant traffic monitoring for SLA
management and pricing
Classifying tenant traffic based on IP
flow, VLAN, VXLAN
Mirror or sFlow
T-CAP
Service-Centric
Network Monitor
Aggregating packets
into flows
JSON format conversion
Flow-Level
Network Analyzer Classifying flows
into tenants
Register tenants and hosts
Monitor tenant traffic
TiNA Multi-Tenant Traffic Monitoring System
Service-Centric Network Monitor
Service traffic statistics monitoring per-tenant
Per service/host/flow traffic statistics, Top-N conversations
Service-Centric Network Monitor
Visualizing real-time network analysis results
Collect and analyze millions of information of nodes, links,
and traffic flows by using spark streaming engine
Diagnose network events with game animation within a few
seconds
End-to-end traffic flow path visualization
3D UI/UX
Display physical/logical/virtual network topology efficiently
Drill down from a global datacenter to a traffic flow in one
second
Support Android, Windows Client
Efficiently visualize multi-layer network topology
providing intuitive and interactive user interface
TiNA 3D-Based Network Monitor
Logical
Network
Topology
Physical
Network
Topology
Virtual
Network
Topology
E2E
Flow Path
Visualization
TiNA 3D-Based Network Monitor
Spine
Switch
Server
Server
Server
Server
Switch
Server
Server
Server
Server
Switch
Server
Server
Server
Server
Spine
SONA Fabric
Controller
TiNA
3DV
Router Network Operation
Traffic Engineering with TiNA, SONA, 3DV
DU
Pool
AS
AR(PE)
AR : Aggregation Router
AS : Access Switch
CO : Central Office
CO
PE PE SGW PGW
MME HSS
EPC Core )
IP Backhaul
AS
IP/MPLS Backhaul
Packet Capture
NPB Network Analyzer
3D/WEB UI
Packet Capture
NPB
Network Analyzer
3D/WEB UI
EPC Core
3rd party monitoring tool
NAT
IMS
Internet
GTP-U
Mirror Tap
60Gbps
3rd party monitoring tool
Subscriber and tunnel end-point based packet filtering
VoLTE Traffic Analysis: throughputs, packet loss, jitter, latency
H/W NPB Time-stamping by T-CAP S/W silicon
GTP Correlation
LTE Network Monitoring
and Troubleshooting
T-CAP is a converged network appliance,
which integrates high performance Xeon server with data center switch
T-CAP: SKT-Converged Appliance Platform
T-CAP Hardware Specification
36-port Ethernet with 4 PCIe Gen3 Ethernet Controller
Up to 200Gbps B/W toward hosts
Flexible Frame Processing Pipeline
L2/L3/L4/OpenFlow Forwarding
ACLs & NAT, Stateless Load Balancing
Tunneling Engine
64k Exact Match Table
NVGRE/Geneve/VXLAN-GRE/NSH
Unparalleled HW Latency
300 ns network latency
1000 ns host-network latency
http://files.opencompute.org/oc/public.php?service=files&t=6a542c4983c8180f5c6f6e7da198e60a
T-CAP Hardware Specification
http://files.opencompute.org/oc/public.php?service=files&t=6a542c4983c8180f5c6f6e7da198e60a
CPU: Intel Rangeley/Avoton
Memory: up to 4 GB DDR3/4
Storage
8 GB storage embedded
SATA-DOM
Peripherals
2x USB2.0
1x External Console(RJ45 Type)
1x GbE for Management
T-CAP Hardware Specification
Dual Intel Xeon E5-2600 v3 CPU (Haswell)
Up to 512GB memory
4X 2.5` SATA SSD
1+1 Redundant PSU (Power Supply Unit)
Front Loading Switch Port Module
10G 12 Port
10G 4 Port + 40G 2 Port
http://files.opencompute.org/oc/public.php?service=files&t=6a542c4983c8180f5c6f6e7da198e60a
T-CAP Software Stack
Linux Kernel
fm10k
SFP/QSFP+ LED PWD Sensors
i2c
fm10k Device
Networking Stack
Quagga OVS Linux
Bridge
3rd
Party
Net Device
Unified Network Config. Data Model & API (e.g., OpenConfig)
T-O&M
SKT’s Op. & Mgmt. Tool
SONA Fabric
SKT’s NW Fabric Auto. Tool
Platform Manager
Legend
FOSS
SKT maintained
Intel proprietary
SKT projects
SwitchD
IES SDK
Real deployment requirements for a single cluster
Performance for packet filtering and actions
~ # of policy rules
~ # of vantage points
Low Latency ~ 400ns
T-CAP Software Stack for Hardware Network Packet Broker
Aggregated Mirrored Traffic NPB OF Controller
Feature
Required Description
Packet
Filtering
L2 Filter(src/dst mac, ip, vlan id, eth type …), L3 Filter(src/dst ip, ip protocol
ToS …), L4 Filter (src/dst Port, VNI)
Action Forwarding, Aggregation, Replication, Load Balancing, Drop
Source Port
Labeling Tagging source port number in packet headers
Header
Stripping
Strip unnecessary packet headers to analyze traffic such as VLAN
Tagging/VXLAN,MPLS
Packet
Slicing Trim packets
HW
Timestamp
Time synchronization between analysis server farms and NPB H/Ws through
PTP
Tunneling Transfer to remote branch RRC based
L1~L4 Filtering and
Forwarding w/ OpenFlow
OpenFlow
Build and Package Management From Kernel to Applications
Yocto Project (DIY/Roll-Your-Own Linux Distro.)
CPP board
(Intel x86)
ONIE image
SKT T-CAP Test Lab
Mininet extension to support feature
validation test for physical switches
Scalability test for control-plane protocols
support on multiple physical hosts or cloud
Reusable test scenario regardless of vendors
Interop test between multiple NOS vendors by
plugins implemented
Host1 Host2
Mininet
Other NOS
Plugins
SKT NOS
Plugin
Wrapper Module
TCAP NOS
TCAP + OcN
OS
SW2
L2
Test
Scripts
L3
Test
Scripts
Other
Test
Scripts
…
Work Flow for NetDevOps
OpenSource
ori repo
SKT
repo
1. branch & sync
Local Devs
2. clone
Code Review
3. push
4. merge
Jenkins 5. build
Test Framework
6. test pass or fail
7. notification if failed
6. code quality check
SonarQube
NEXUS 8. release if pass
Production
Site
9. update noti.
&
package update
WAN
End-to-end Network Orchestration
Local NFV Orchestrator Transport Infra Orchestrator
Un
ified
-O
Edge DC Central DC
Open & Programmable H/W
Resource Abstraction Layer
Open & Programmable H/W
Virtualized Network Functions
Fronthaul
L1/L2
RF
Remote Unit
4G
5G SDRAN
RNF ESF
vCore
CNF CSF OSF
Transport Infrastructure
Transport
Open H/W POTN
Network Service Functions
TNF
uCTN
Mobile Connectivity Functions
uCTN
Service Orchestration and Exposure
Low Latency Service
Immersive Media
Telco Service
Virtualized Network Slice #1
#2 #N
NG
-OS
S
(E2
E R
eso
urc
e M
gm
t., Co
gnitiv
e &
Inte
lligent A
uto
matio
n)
SKT New Infra Architecture for 5G and Beyond COSMOS – All-IT Undelay Architecture