Network Debugger:A Unified Tool for Diagnosing Network

Controlling Applications

WTC 2012 Workshop on Software Defined Networks (SDN) and OpenFlow

March 7, 2012Yasunobu Chiba and Hideyuki Shimonishi

System Platforms Research LaboratoriesNEC Corporation

Outline

▐ Background and problem statement▐ High-level design of Network Debugger▐ Preliminary design and implementation results▐ Conclusion and future work

© NEC Corporation 2012Page 2

Background

▐ Software-Defined Network (SDN)

•Network elements are provided by hardware vendors and they are in

Current

•Features provided in networks can be flexibly defined or updated

SDN Era

vendors and they are in operation for 5-10 years

•Features cannot be added while users require new features or have new ideas

defined or updated

•Features can be added by users (not hardware vendors)

•New features can be easily evaluated and chosen for production use

© NEC Corporation 2012Page 3

Background – cont’d

▐ Open Networking Foundation says…

▐ Are you sure about its success?

https://www.opennetworking.org/media-room/press-releases/63-open-networking-foundation-formed-to-speed-network-innovation

SDN thus gives owners and operators of networks better control over their networks, allowing them to optimize network behavior to

best serve their and their customers’ needs.

at this moment…

© NEC Corporation 2012Page 4

Issues

▐ No sophisticated debugging environment for developing control plane application is provided

▐ Current/legacy networks are standards-based and they have been implemented/deployed by spending many years

▐ In SDN, various non-standards-based features are implemented/deployed in a timely manner based on user’s implemented/deployed in a timely manner based on user’s requirements� It is hard to develop purpose-built validation/diagnosis tools for each

specific feature due to short development period and cost constraint� If we develop purpose-built tools, it reduces flexibility, a notable

benefit of SDN

Providing a unified tool for diagnosing control plane applications is a key success factor of SDN

© NEC Corporation 2012Page 5

Goal and scope of work

▐ Provide Network Debugger – A unified tool for diagnosing various control plane applications

Control Plane Application / OpenFlow Controller

Software-Defined / OpenFlow Network

Module A Module B Module Z

Controller Platform

State / Event

Collection

State / Event

Analysis

Scope of Network DebuggerScope of rest of presentation

Switch

Host

Emulated / Real Network

Network Debugger Console

(UI)

Breaking

Single-stepping

Network Debugger provides similar functionality to debugger for standalone application software

© NEC Corporation 2012Page 6

Collecting states/events from network: Past and current

Log

LogLog

Log

Log

Log

Log

Log

DUT/SUT

Collect logs

•Collect states/events on network elements using various ways and then merge and serialize them at a certain place

•See if a network is in proper state by comparing the state/event series with an expectation constructed from product and protocol knowledge

•Multiple mechanisms depending on network elements are required for collecting states/events

Combine logs into a single event series

Sent packet A to …0.00Sent packet B to …0.12Sent packet A to …0.15Validate packet A …0.29Sent packet B to …0.52Receive packet A …0.81

Find out focusing events

Sent packet A to …0.00Sent packet A to …0.15Validate packet A …0.29Receive packet A …0.81

Sent packet A to …Sent packet A to …Validate packet A …Receive packet A …

Log

LogLog

Log Log

Log

Log

Log

DUT/SUT

Compare with expectation

Protocol Specification

Product Manual

collecting states/events

•Some state/event collection mechanisms seriously interfere in operation status (heavily consume CPU/memory resource etc.)

•Real-time state/event collection is hard

© NEC Corporation 2012Page 7

Collecting states/events from network: Proposed

State / Event Notifications w/

Control Plane Application /

OpenFlow Controller

Developer / Operator

Real-time / Off-line Monitor

•Centrally collect states/events from various network elements with a common format and a light-weight transport in real-time or offline

•Provide a tool (state/event viewer) to analyze a huge amount of collected states/events efficiently/easily

State / Event Viewer

File

Event Collector

Circular Buffer

State / Event Notifications w/ Common Notification Format

Network Interface / Tap

Switch

Host

Serialized State / Event Notifications

© NEC Corporation 2012Page 8

Effectiveness and issue

▐ Effectiveness� All states/events can be collected at a single place in real-time with a

common light-weight collection mechanism� Application developers/operators can focus on analysis of collected

states/events� State/event filter functions embedded in the state/event viewer helps

to find a focusing event series easily

▐ Issue� Finding a context (a series of events that need to be addressed)

should still be done with experienced application developers/operators

© NEC Corporation 2012Page 9

Proof-of-concept implementation: Design

Event

Notifications through Trema IPC (messenger)

Switch

Trema-based OpenFlowController

Host

Developer / Operator

Real-time / Off-line Monitor

SyslogRelay

Syslog

State / Event Viewer

IPC mechanism provided by Trema is reused as a light-weight

transport for our method

State/event notification mechanism is

implemented in Trema

Wireshark with a plug-in for our method is used as state/event viewer

Event Collector

Circular Buffer

Network Interface / Tap

Serialized Notifications

Relay

Packet Capture

StdinRelay

PcapFile

Wireshark w/ plugin

Any Text String

Bridges are provided for accommodating legacy state/event

collection mechanisms© NEC Corporation 2012Page 10

Proof-of-concept implementation: State/Event Viewer

Inter-function module events

Packet capture

Syslog message

Packet capture

OpenFlow Controller

learning_switch

switch.abc

27. Packet-In

28. Packet-In29. Packet-In

31. Packet-Out30. Packet-Out

32. Packet-Out

34. Error !!!

© NEC Corporation 2012Page 11

Evaluation of PoC implementation: Metrics

1. State/event collation performance� Measure events per second performance with our method and a

legacy event collection mechanism (i.e. syslog)

2. CPU usage� Assume a case that software developer or operator confirms that all

flow entries are properly installed from an OpenFlow controller into flow entries are properly installed from an OpenFlow controller into OpenFlow switches in an OpenFlow network

� Measure CPU usage in operation with our method and a legacy event collection mechanism (i.e. syslog)

© NEC Corporation 2012Page 12

State/event collation performance: Result

100,000

150,000

200,000

# o

f E

ve

nts

Co

llecte

d [e

ve

nt/

se

c]Event Collection Performance

Proposed Method Syslog (rsyslog) Syslog (syslog-ng)

Our method collects states/events with higher rate than syslog implementations

0

50,000

# o

f E

ve

nts

Co

llecte

d [e

ve

nt/

se

c]

# of Events Generated [event/sec]

© NEC Corporation 2012Page 13

CPU usage: Result

40

50

60

70

CP

U U

sage [%

]CPU Usage in Operation

W/o Event Collection Proposed Method Syslog (rsyslog) Syslog (syslog-ng)

Our method collects states/events with lower side effects than syslogimplementations in terms of operation status

0

10

20

30

100 300 500 700 900

CP

U U

sage [%

]

Flow Setup Rate [flow/sec]

© NEC Corporation 2012Page 14

Conclusion and future work

▐ Conclusion� Presented a vision for Network Debugger which should be a key

success factor of SDN� Proposed and implemented a mechanism for collecting states/events

from various network elements including control plane application, switches/routers, and etc. with a common and light-weight way

� Confirmed that our method is able to collect events with higher rate and lower CPU utilization than a legacy event collection mechanismand lower CPU utilization than a legacy event collection mechanism

▐ Future work� Usability test in real software development/operation cases� Integration with a network emulator� Design and implementation of a feedback mechanism from console UI

to emulated network

© NEC Corporation 2012Page 15

Conclusion and future work – cont’d

▐ Some parts of this work are shipped with Trema OpenFlowController Platform� http://trema.github.com/github� https://github.com/trema/trema

© NEC Corporation 2012Page 16

NEC Group Vision 2017

To be a leading global companyleveraging the power of innovation

to realize an information societyfriendly to humans and the earthfriendly to humans and the earth