Download - ONOS System Test - ONS2016
System Testing SDN- A Case Study on ONOS
Suibin Zhang, ON.LabJon Hall, Ciena
KnownUNKNOWNs
Known KNOWNs
UnknownUNKNOWNs
Design Considerations
Unit Test Considerations System Test Considerations
Quality Assurance on Software System
NASubjective Perception
Obj
ectiv
e Re
ality
✔
2#ONOSProject
3#ONOSProject
• Black/Grey Box Testing– Often mimic usage scenarios– Validity not influenced by internal design
• Product Verification/Qualification• Pre-Deployment Verification
System Test - Where Used
4#ONOSProject
System Test Environment
Testing SDN OS – Nontrivial
SDN Network Domain
Distributed
Controllers
SDNCTRL
1
SDNCTRL
2
SDNCTRL
n
APP APPAPP
Test Exec.
?
?
Legacy Network Domain
Dependencies
?
5#ONOSProject
• Technical– Accessing heterogeneous, evolving interfaces– Scaling test environment– Facilitate Debugging
• Arise from Community/Open Source – Run “anywhere”– Handling test dependencies– Contribution and merging tests
Challenges in Building System Tests
6#ONOSProject
A Case Study on ONOS
7#ONOSProject
• Functionality Suite– NB: Intents and Flow operations– SB: OF 1.0 &1.3, NETCONF, OVSDB, etc.
• High Availability Suite– Various scenarios of cluster failures
• Performance and Scale-out Suites– Intent, flow throughputs, latency as scaling-out– Topology discovery latency as scaling-out
• Longevity– Continuous operation under typical deployment scenarios
• Selected Use Case Suites– SDNIP App, Segment Routing App
ONOS System Test Suites
8#ONOSProject
Establish Test Handles Run-time Configuration Test Logic
Reset/Clean Test Env. Inject Run Params to
TestStation Start Test Cases
Jenkins Plugins Update Codes Publish Results
Onos System Test CI
9#ONOSProject
A Deeper Dive into TestON
Onos System Test CI
10#ONOSProject
TestON – Key Benefits• TestON
o SDN-focused Testing Frameworko Test case authoring, executiono Python-based
• Key Benefits:o Extensibility – Simplify case authoring by compartmentalizing
concernso Flexibility - Handle heterogeneous test env. OVS, HW, Quagga, etc.o Debuggability - Facilitate troubleshooting
11#ONOSProject
Test Case Files*.params*.py*.topo Dependency
Details on TestON FrameworkTestON Core Framework:
Logging, Utilities, …
Driver Files
Mininet Driver ONOS Rest Driver Quagga DriverONOS CLI Driver
Test Files.params File – XML file, used to pass variables into test, such as order of test cases.topo file – XML File, defines the components of the testDependency Folder – Config files or Mininet topology files used by the test.py file – Combine TestON and driver functions to define your test cases.
Test Cases are split into steps, each of which should have an assertion in them
• Driver Files• Can use REST, CLI via SSH, or other custom connection interfaces• Defined in a hierarchical structure• Creates a library of functions to be used in the test
12#ONOSProject
ONOS Intent Feature
• Set intent through NB REST interface• Validate intent and flows are correctly added• Test end-to-end connectivity in Mininet with the specific selectors
System Test Procedure
• Need a library to send packets with test-specified selector• Leveraging Python Scapy module
System Test Requirement
• High-level abstraction of setting end to end connectivity• Can be defined by “Selectors” of MAC, IP, PORT, MPLS label, etc.
Example: ONOS Intent Functionality Test
13#ONOSProject
Building a Scapy Driver(send/recv user-defined packets)
class ScapyCliDriver( Emulator ): def sendPacket( self, iface=None, packet=None, timeout=1 ): try:
sendCmd = 'srp( ' if packet: sendCmd += packet else:
... if iface: sendCmd += ", iface='{}'".format( iface ) sendCmd += ', timeout=' + str( timeout ) + ')' self.handle.sendline( sendCmd ) self.handle.expect( self.scapyPrompt ) except:
...
TestON |--drivers | |--common | |--api | |--cli | | |--emulator | | |--ScapyCliDriver.py |--tests
Example: ONOS’ Intent Functionality Test
14#ONOSProject
for hostName in [ “h1”,”h2” ]: # Create two host components with the scapy driver4 main.Scapy.createHostComponent( hostName )main.step( ”Login to scapy on these new components" )for host in [ main.h1, main.h2 ]: host.startHostCli() host.startScapy()
Using Scapy Library - in Mininet Host Handle
<Scapy> <host>192.168.1.101</host> <user>admin</user> <password></password> <type>ScapyCliDriver</type> <connect_order>6</connect_order></Scapy>
TestON |--drivers |--tests |--Example |--Example.py |--Example.params |--Example.topo
Example: ONOS’ Intent Functionality Test
15#ONOSProject
# Add connectivity from h1 to h2 (unidirectional)addResult = main.ONOS1.addPointIntent( ingress, egress, ethDst, ipDst) main.h2.startFilter() # Start packet filter on host2main.h1.buildEther( dst=main.h2.hostMac ) # build ethernet framemain.h1.buildIP( dst=main.h2.hostIp ) # Build IP framemain.h1.sendPacket() # Send built packetfinished = main.h2.checkFilter() # Check if host2 received any packets
Intent Test Case - Using Scapy Driver
TestON |--drivers |--tests |--Example |--Example.py |--Example.params |--Example.topo
Example: ONOS’ Intent Functionality Test
16#ONOSProject
• Parameterize Test Env & Run Var– Jenkins to inject at run time– Test cases deal with “pure” test
logic• Extensive Exception Handling in
Drivers– Take care of exceptions, timeout– Use “retry” method
• Test Dependencies– Run-time Activation– “Dockerize” Fixture (Future Work)
• Debug Utilities– Test pausing– Log collation– More (Future Work)
• Scale Test Network (Future Work)
Lessons Learned and Future Work
17#ONOSProject
For more Information• S3 Showcase: “SDN/ONOS System Test”• Wiki: https://wiki.onosproject.org/display/ONOS/System+Tests• Repo: https://gerrit.onosproject.org/#/admin/projects/OnosSystemTest• Github: https://github.com/opennetworkinglab/OnosSystemTest
Summary
Python-Based
Extensibility
DebuggabilityFlexibility
ONOS Test Suites:FunctionalityHigh AvailabilityPerformanceScale-out CapabilityLongevityUse Cases
ONOS System Test Highlights
Open Source
18#ONOSProjectJoin the journey @
onosproject.org
Software Defined Transformation of Service Provider Networks
Q&A