metamorphic, model-based testing of …metamorphic, model-based testing of autonomous systems mikael...
TRANSCRIPT
Metamorphic, Model-Based Testing of Autonomous Systems
Mikael LindvallAdam Porter
Gudjon Magnusson Christoph Schulze
Testing Autonomous Systems
§ Increasing interest in using autonomous systems to perform safety-critical missions in the real-world
§ Can’t test every possible scenario§ Difficult to determine correct test behavior
§ Systems designed to make their own decisions§ Non-deterministic control algorithms
§ Uses sense and avoid algorithm to autonomously navigate an unknown environment to deliver packages
§ Uses LIDAR and traditional cameras to avoid obstacles and identify landing pads
§ Used a simulation environment and a testing framework to generate numerous test cases
Testing an Autonomous Drone
Model-Based, Metamorphic Testing
§ Manually created test cases for a small set of interesting scenarios
§ Use model-based testing to generate a large number of variants from each manual scenario
§ Instrumented drone software to emit information about its internal state
Model-Based, Metamorphic Testing
§ Used metamorphic testing principles to identify anomalous behaviors in “equivalent” test cases
§ Assume small changes in test cases should lead to small changes in behavior
§ Flag anomalies to detect unsafe/unstable behavior for certain configurations
Metamorphic Equivalences
§ Behavior should be relatively independent of:§ Repeated runs§ Geographic rotation§ Geographic translation§ Obstacle locations § Obstacle formations
Comparing Behaviors
§ Flight path shape§ Experimental
§ Safety property violations§ Time in unsafe internal states§ Control state sequences
Example Test Scenario
Starthere(LandingpadA)
Landhere(LandingpadB)
Drone
Avoidobstacles
Analyzing a Test Run
Identifying an Anomaly
Original Rotated1º
Identifying Another Anomaly
360 Test Case Variants
Preliminary Findings
§ Model-based, metamorphic testing was practical and useful for safety testing of an autonomous system§ Identified unstable/unpredictable behaviors§ Identified corner cases with catastrophic failure and
landing problems§ Confirmed usefulness of safety-related features for
identifying anomalies (implemented as safety cages)
Future Work
§ Higher fidelity simulation§ Smarter (adaptive) test generation§ Automatically learn relationships between
changes in environment and changes in behavior
§ Define autonomy requirements