onos open network operating system an experimental open-source distributed sdn os pankaj berde,...
TRANSCRIPT
ONOSOpen Network Operating System
An Experimental Open-Source Distributed SDN OS
Pankaj Berde, Umesh Krishnaswamy, Jonathan Hart, Masayoshi Kobayashi, Pavlin Radoslavov, Pingping Lin, Sean Corcoran, Tim Lindberg, Rachel Sverdlov, Suibin Zhang, William Snow, Guru Parulkar
Agenda Overview of ONRC (Open Networking Research Center)
ONOS • Architecture• Scale-out and high availability • Network graph as north-bound abstraction• DEMO• Consistency models • Development and test environment • Performance • Next steps
Leadership
National Academy of EngineeringACM SIGCOMM Award Winners
Fellow of IEEE and ACMEntrepreneurs
Impact on practice of networking/cloud
Nick MckeownKP, Mayfield, Sequoia
Professor, Stanford
Larry PetersonBob Kahn Professor Princeton
Chief Architect, ON.LAB
Scott Shenker
Professor, UC Berkeley
Chief Scientist, ICSI
Stanford/Berkeley SDN Activities With Partners
2007 20112008 2009 2010
Ethane
Dem
oD
eplo
ymen
tPl
atfor
m D
evel
opm
ent
OpenFlow Spec v0.8.9 v1.0 v1.1
Reference Switch NetFPGA Software
Network OS NOX SNAC Beacon
Virtualization FlowVisor FlowVisor (Java)
Tools Test Suite oftrace MininetMeasurement tools
GENI software suite Expedient/Opt-in Manager/FOAM
Stanford University ~45 switch/APs ~25userIn McKeown Group
CIS/EE BuildingProduction Network
US R&E Community GENI: 8 Universities + Internet2 + NLRMany other campuses
Other countries Over 68 countries(Europe, Japan, China, Korea,Brazil, etc.)
VM Migration(Best Demo)
Trans-PacificVM Migration
Baby GENI Nation Wide GENI “The OpenFlow Show” – IT World
SDN Concept(Best Demo)
SIGCOMM08 GEC3 SIGCOMM09 GEC6 GEC9Interop
2011
+Broadcom
Scaling of SDN Innovation
Standardize OpenFlow and promote SDN~100 Members from all parts of the industry
Bring best SDN content; facilitate high quality dialogue3 successive sold out events; participation of ecosys
Build strong intellectual foundationBring open source SDN tools/platforms to community
SDN Academy
Bring best SDN training to companiesto accelerate SDN development and adoption
ONRC Organizational Structure
BerkeleyScott Shenker
Sylvia Ratnasamy
Open Network LabExec Director: Guru Parulkar
VP Eng: Bill SnowChief Architect: Larry Peterson
16-19 Engineers/Tech Leads(includes PlanetLab team)
Tools/Platforms for SDN community
OpenCloud demonstration of XaaS and SDN
PhD/Postdocs
Research
StanfordNick McKeownGuru ParulkarSachin Katti
7
Mission
Bring innovation and openness to internet and cloud infrastructure with open source
tools and platforms
Tools & Platforms
3rd partycomponents
Network OS
Apps Apps
Network OS
Apps Apps
Open Interfaces
Open Interfaces
Network Hypervisor
Forwarding
FlowVisor, OpenVirteX
MININET, Cluster Edition
ONOS
SDN-IP Peering
Test
ON
with
deb
uggi
ng s
uppo
rt
Net
Sigh
t
Open Network OS (ONOS)
Architecture
Scale-out and high availability
Network graph as north-bound abstraction
DEMO
Consistency models
Development and test environment
Performance
Next steps
ONOS: Executive Summary
ONOS
Status
Distributed Network OS
Network Graph Northbound Abstraction
Horizontally Scalable
Highly Available
Built using open source components
Version 0.1
- Flow API, Shortest Path computation, Sample
application
- Build & QA ( Jenkins, Sanity Tests, Perf/Scale Tests, CHO)
- Deployment in progress at REANNZ (SDN-IP peering)
Next
Exploring performance & reactive computation frameworks
Expand graph abstraction for more types of network state
Control functions: intra-domain & inter-domain routing
Example use cases: traffic engineering, dynamic virtual networks
on demand, …
ONOS – Architecture Overview
Routing TE
Network OS
PacketForwarding
PacketForwarding
PacketForwarding
Mobility
ProgrammableBase Station
Openflow Scale-outDesign
Fault Tolerance
Global network view
Open Network OS Focus(Started in Summer 2012)
Global Network View
Prior Work
Distributed control platform for large-scale networks
Focus on reliability, scalability, and generality
Scale-out NOS focused on network virtualization in data centers
State distribution primitives, global network view, ONIX API
ONIX
Other Work
Helios (NEC), Midonet (Midokura), Hyperflow, Maestro, Kandoo
NOX, POX, Beacon, Floodlight, Trema controllers
Community needs an open source distributed SDN OS
Host
Host
Host
Titan Graph DB
Cassandra In-Memory DHT
Instance 1 Instance 2 Instance 3
Network GraphEventually consistent
Distributed RegistryStrongly Consistent Zookeeper
OpenFlow Controller+
OpenFlow Controller+
OpenFlow Controller+
ONOS High Level Architecture
+Floodlight Drivers
Scale-out & HA
ONOS Scale-Out
Distributed Network OS
Instance 2 Instance 3
Instance 1
Network GraphGlobal network view
An instance is responsible for maintaining a part of network graph
Control capacity can grow with network size or application need
Data plane
Master Switch A = ONOS 1
Candidates = ONOS 2, ONOS 3
Master Switch A = ONOS 1
Candidates = ONOS 2, ONOS 3
Master Switch A = ONOS 1
Candidates = ONOS 2, ONOS 3
ONOS Control Plane Failover
Distributed Network OS
Instance 2 Instance 3Instance 1
Distributed Registry
Host
Host
Host
A
B
C
D
E
F
Master Switch A = NONE
Candidates = ONOS 2, ONOS 3
Master Switch A = NONE
Candidates = ONOS 2, ONOS 3
Master Switch A = NONE
Candidates = ONOS 2, ONOS 3
Master Switch A = ONOS 2
Candidates = ONOS 3
Master Switch A = ONOS 2
Candidates = ONOS 3
Master Switch A = ONOS 2
Candidates = ONOS 3
Network Graph
Cassandra In-memory DHT
Id: 1A
Id: 101, Label
Id: 103, Label
Id: 2C
Id: 3B
Id: 102, Label
Id: 104, Label
Id: 106, Label
Id: 105, Label
Network Graph
Titan Graph DB
ONOS Network Graph Abstraction
Network Graph
port
switch port
device
port
onport
port
port
linkswitch
on
device
host host
Network state is naturally represented as a graph Graph has basic network objects like switch, port, device and links Application writes to this graph & programs the data plane
Example: Path Computation App on Network Graph
port
switch port
device
Flow pathFlow entry
port
onport
port
port
link switch
inport
on
Flow entry
device
outportswitchswitch
host host
flowflow
• Application computes path by traversing the links from source to destination• Application writes each flow entry for the path
Thus path computation app does not need to worry about topology maintenance
Example: A simpler abstraction on network graph?
Logical Crossbar
port
switch port
device
Edge Port
port
onport
port
port
link switch
physical
on
Edge Port
device
physical
hosthost
• App or service on top of ONOS• Maintains mapping from simpler to complex
Thus makes applications even simpler and enables new abstractions
Virtual network objects
Real network objects
Network Graph Representation
Flow path
Flow entry
Flow entry
flow
flow
Vertex with 10 properties
Vertex with 11 properties
Vertex represented as Cassandra row
Property(e.g. dpid)
Property(e.g. state)
Property … Edge Edge
Edge represented as Cassandra
column
Column Value
Label id + direction
Primary key
Edge id Vertex id Signature properties
Other properties
Switch
Vertex with 3 properties
Row indices for fast vertex centric queries
Switch Manager Switch ManagerSwitch Manager
Network Graph: Switches
OFOF
OFOF
OFOF
Network Graph and Switches
SM
Network Graph: Links
SM SM
Link Discovery Link Discovery Link Discovery
LLDP LLDP
Network Graph and Link Discovery
Network Graph: Devices
SM SM SMLD LD LD
Device Manager Device Manager Device Manager
PKTIN
PKTIN
PKTINHost
Host
Host
Devices and Network Graph
SM SM SMLD LD LD
Host
Host
Host
DM DM DM
Path Computation Path Computation Path Computation
Network Graph: Flow Paths
Flow 1
Flow 4
Flow 7
Flow 2
Flow 5
Flow 3
Flow 6
Flow 8
Flow entriesFlow entriesFlow entries
Flow entriesFlow entriesFlow entries
Flow entriesFlow entriesFlow entries
Flow entriesFlow entriesFlow entries
Flow entriesFlow entriesFlow entries
Flow entriesFlow entriesFlow entries
Flow entriesFlow entriesFlow entries
Flow entriesFlow entriesFlow entries
Path Computation with Network Graph
SM SM SMLD LD LD
Host
Host
Host
DM DM DM
Flow Manager
Network Graph: FlowsPC PC PC
Flow Manager Flow ManagerFlowmod Flowmod
Flowmod
Flow 1
Flow 4
Flow 7
Flow 2
Flow 5
Flow 3
Flow 6
Flow 8
Flow entriesFlow entriesFlow entries
Flow entriesFlow entriesFlow entries
Flow entriesFlow entriesFlow entries
Flow entriesFlow entriesFlow entries
Flow entriesFlow entriesFlow entries
Flow entriesFlow entriesFlow entries
Flow entriesFlow entriesFlow entries
Flow entriesFlow entriesFlow entries
Network Graph and Flow Manager
Host
Host
Host
Titan Graph DB
Cassandra In-Memory DHT
Instance 1 Instance 2 Instance 3
Network GraphEventually consistent
Distributed RegistryStrongly Consistent Zookeeper
OpenFlow Controller+
OpenFlow Controller+
OpenFlow Controller+
ONOS High Level Architecture
+Floodlight Drivers
DEMO
Consistency Deep Dive
Consistency Definition
• Strong Consistency: Upon an update to the network state by an instance,
all subsequent reads by any instance returns the last updated value.
• Strong consistency adds complexity and latency to distributed data
management.
• Eventual consistency is slight relaxation – allowing readers to be behind
for a short period of time.
Strong Consistency using Registry
Distributed Network OS
Instance 2 Instance 3
Network Graph
Instance 1
A = Switch A
Master = NONE A = ONOS 1
Timeline
All instances Switch A Master = NONE
Instance 1 Switch A Master = ONOS 1Instance 2 Switch A Master = ONOS 1Instance 3 Switch A Master = ONOS 1
Master elected for switch A
Registry Switch A Master = NONE
Switch A Master = ONOS 1
Switch A Master = ONOS
1
Switch AMaster = NONE
Switch A Master = ONOS 1
Delay of Locking & Consensus
All instances Switch A Master = NONE
Why Strong Consistency is needed for Master Election
Weaker consistency might mean Master election on
instance 1 will not be available on other instances.
That can lead to having multiple masters for a switch.
Multiple Masters will break our semantic of control
isolation.
Strong locking semantic is needed for Master Election
Eventual Consistency in Network Graph
Distributed Network OS
Instance 2 Instance 3
Network Graph
Instance 1
SWITCH A STATE= INACTIVE
Switch AState = INACTIVE
Switch ASTATE = INACTIVE
All instances Switch A STATE = ACTIVE
Instance 1 Switch A = ACTIVEInstance 2 Switch A = INACTIVEInstance 3 Switch A = INACTIVE
DHT
Switch Connected to ONOS
Switch A State = ACTIVE
Switch A State = ACTIVE
Switch ASTATE = ACTIVE
Timeline
All instances Switch A STATE = INACTIVE
Delay of Eventual Consensus
Cost of Eventual Consistency
Short delay will mean the switch A state is not ACTIVE on
some ONOS instances in previous example.
Applications on one instance will compute flow through the
switch A while other instances will not use the switch A for
path computation.
Eventual consistency becomes more visible during control
plane network congestion.
Why is Eventual Consistency good enough for Network State?
Physical network state changes asynchronously
Strong consistency across data and control plane is too hard
Control apps know how to deal with eventual consistency
In the current distributed control plane, each router makes its
own decision based on old info from other parts of the network
and it works fine
Strong Consistency is more likely to lead to inaccuracy of
network state as network congestions are real.
Consistency learning
One Consistency does not fit all
Consequences of delays need to be well understood
More research needs to be done on various states
using different consistency models
Development & test environment
ONOS Development & Test cycle
Source code on github
Agile: 3-4 week sprints
Mostly Java and many utility scripts
CI: Maven, Jenkins, JUnit, Coverage, TestON
Vagrant-based development VM
Daily 4 hour of Continuous Hours of Operations (CHO) tests as
part of build
Several CHO cycles simulating rapid churns in network & failures
on ONOS instances
ONOS Development Environment
Single installation script creates a cluster of Virtual Box VMs
Test Lab Topology
ON.LAB ONOS Test implementation
ON.LAB team has implemented following aspects of automated tests• ONOS Unit Tests (70% coverage)• ONOS System Tests for Functionality, Scale,
Performance and Resiliency test (85% coverage)• White Box Network Graph Performance
Measurements All tests are executed nightly in Jenkins
Continuous Integration environment.
Performance
Key performance metrics in Network OS
Network scale (# switches, # ports) -> Delay and Throughput• Link failure, switch failure, switch port failure
• Packet_in (request for setting reactive flows)
• Reading and searching network graph
• Network Graph Traversals
• Setup of proactive flows Application scale (# operations, # applications)
• Number of network events propagated to applications (delay & throughput)
• Number of operations on Network Graph (delay & throughput)
• Parallelism/threading for applications (parallelism on Network Graph)
• Parallel path computation performance
46
Performance: Hard Problems
Off the shelf open source does not perform
Ultra low-latency requirements are unique
Need to apply distributed/parallel programming techniques to
scale control applications
Reactive control applications need event-driven framework
which scale
ONOS: Summary
ONOS
Status
Distributed Network OS
Network Graph Northbound Abstraction
Horizontally Scalable
Highly Available
Built using open source components
Version 0.1
- Flow API, Shortest Path computation, Sample
application
- Build & QA ( Jenkins, Sanity Tests, Perf/Scale Tests, CHO)
- Deployment in progress at REANNZ (SDN-IP peering)
Next
Exploring performance & reactive computation frameworks
Expand graph abstraction for more types of network state
Control functions: intra-domain & inter-domain routing
Example use cases: traffic engineering, dynamic virtual networks
on demand, …
www.onlab.us