a soft way for openflow interoperability testing

1

A SOFT Way for OpenFlow Interoperability Testing

Maciej Kuźniar, Peter Perešini, Marco Canini†, Daniele Venzano, Dejan Kostić‡

EPFL †TU Berlin/T-Labs ‡IMDEA Networks

2

Software-Defined Networking (SDN)

Third-partycontrol

programController

3

Release

Interoperability at Deployment Time

OpenFlowprogram

OpenFlowmessages

One OpenFlow API specification… Are OF switches interoperable?

Interop is critical for the success of SDN

4

Switch III1. Silently drops

packets

Switch I1. Responds with an

error message

Switch II1. Trims VLAN value

to 12 bits2. Installs the rule

Inconsistency - Example

FlowMod message1. Modify VLAN ID to 40972. Forward packet

Network in 3 different statesWhich state is assumed by the controller?

Where are packets forwarded?

5

Interop: How Hard Can It Be?

OF SwitchInputs

Hardware correctness is formally

verified

Packets

OpenFlowmessages

“Forwarding” interface

OpenFlowinterface

ASIC switch chip

OS

OpenFlow Agent

Likely source of OpenFlow

interop issues

Flow TableHardware Abstraction Layer

This work: Finding differencesbetween OpenFlow Agent implementations

6

OpenFlow Software Agent

Specifications• Rapid flux (3 revisions in ~ 1 year)• Ambiguities Specifications Implementation• Implementation freedom• Vendors may not follow the specs

Testing, testing and testing…

Switch software is not provably correct

7

Interoperability Event

• Gather various vendors• Hook up switches and controllers• Create and run test cases• See what breaks and fix it

• Very high manual effort• Test cases are not exhaustive• It is not a one time thing

[ONF Interop WG, March ’12]

8

Automating Interop Testing

Insight:systematically crosscheck OF implementations

9

The 10,000 foot view

OF Agent 1

Test inputs

Input-drivenexecution

Observablebehaviors

Inconsistency!

OF Agent 2

Problem I: What inputs should we use?

10

Symbolic Execution

port < 25?yes no

Symbolic messageport = ∗

Forward

port < 25

port == CTRL?yes no

Send to CTRL

port ≥ 25∧port = CTRL

Send ERROR

port ≥ 25∧port ≠ CTRL

port 1 24 25 ... 6553

5

...

CTRL

Forward Send to CTRL Send ERROR

Problem II: Path explosion

11

Challenges

• Manage test inputs and coverage efficiently

• Capture behaviors

• Avoid simultaneous access to all code

12

SOFT(Systematic OpenFlow Testing)

Phase 1

Phase 2?

OF Agent 1 OF Agent 2

• Automated solution to interop testing• Systematic code coverage• No simultaneous access to all agents

13

* * * *C3 C2C1

Structuring Inputs

* * * * * * * *FLOW MOD LE

N1 STAT

REQ LEN

2

1.0 1.0

Further reductions• Some messages are independent• Many inputs are entirely concrete• Small number of messages• Concrete values at cost of coverage

14

Benefit of Concretizing

0 0.5 1 1.5 2 2.5 3 3.50%

10%

20%

30%

40%

50%

Concrete Actions Concrete

Match

Coverage

Time [h]

0%

10%

20%

30%

40%

50%

FullySymbolic

28h

15

Capturing Behaviors

Externally observable outputs• OpenFlow reply messages• Data plane packets• Normalize harmless nondeterminism (e.g., Buffer IDs)

Internal state changes affect successive inputs• Use concrete probe packets

16

ExampleAgent 1 Agent 2

17

Finding InconsistenciesAgent 1 Agent 2

18

Is there an input thatcauses two distinct behaviors?

Finding Inconsistencies

No false positives

Agent 1:

Agent 2:

port 1 24 25 ... 6553

5

...

CTRL

19

Limitations

• Short sequences of inputs• Unable to find problems with a complex state

• Is an inconsistency harmless?• Can it affect the controller?

• How to test all initial configurations?• Agent’s behavior depends on initial config

20

Prototype & Evaluation

• SOFT prototype built on top of Cloud9/Klee

• Compared• OpenFlow 1.0 Reference Switch (55k LoC)• Open VSwitch 1.0.0 (80k LoC)

• Input Sequences containing 1 - 4 messages

21

Does SOFT Work?Found 7 classes of inconsistencies:Packets dropped when action is invalidDifferent ports considered invalidLack of error messagesDifferent order of message validationSilently ignored statistics requestMissing featuresSwitch terminates with an error

Mostly related to message validation

Result of underspecification• No expected behavior in the specification• Inconsistent interpretation of the specification

22

Summary

SOFT automates interoperabilitytesting of OpenFlow Agents

Also useful for:• regression testing• specification improvements