Download - Random Testing of Interrupt-Driven Software
![Page 1: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/1.jpg)
Random Testing of Interrupt-Driven Software
John Regehr
University of Utah
![Page 2: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/2.jpg)
Randominterrupttesting
Source-sourcetransformation
Staticstack
analysis Geneticalgorithms
Semanticsof interrupts
Deltadebugging
Integrated stress testing and debugging
![Page 3: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/3.jpg)
Goal: Stress testing and debugging for interrupt-driven embedded software
Why? Interrupts hard to get right Regular testing typically exercises
small part of state space Stress testing tends to improve
software quality Interrupt-driven software used in
safety-critical applications
![Page 4: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/4.jpg)
Specific case: Sensor network nodes running TinyOS Strongly interrupt-driven Application code runs in interrupt
mode Highly resource constrained Distributed and opaque –
magnifies effects of bugs
![Page 5: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/5.jpg)
Obvious stress testing technique: Random interrupt testing – fire
interrupts at random times Potential show stoppers:
Random interrupts can violate application semantics
Interrupts can reenter and overflow the stack
![Page 6: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/6.jpg)
time
request ADC
ADCint.
randomADCint.
aberrant interrupt
cras
h!
![Page 7: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/7.jpg)
time
randomnetwork
interrupts
cras
h!
stack overflow
Many embedded systems permit reentrant interrupts
![Page 8: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/8.jpg)
Problem: Interrupts arriving at inconvenient times break applications
Solution: Restrict interrupt arrivals
First classify each interrupt vector Requested – arrives in response to
an action taken by the system Spontaneous – may arrive at any
time
![Page 9: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/9.jpg)
Restricted Interrupt Discipline (RID): Requested interrupts – only permit
when a request is outstanding Spontaneous interrupts – only
permit when the interrupt isn’t already running
![Page 10: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/10.jpg)
Implementing RID
1. Annotate interrupt requests
2. Ensure that device initialization code leaves each interrupt disabled
3. Run system through a source-to-source translator Enable interrupt upon request Disable requested interrupts
upon interrupt Suppress reentrant interrupts
![Page 11: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/11.jpg)
RID in TinyOS
Implemented RID for five interrupt vectors
Only bottom-level device driver files modified A few LOC modified per vector Normal developers don’t touch
these files Use custom CIL extension for
src-src translation of C code output by nesC compiler
![Page 12: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/12.jpg)
Wit
ho
ut
RID
Wit
h R
ID
![Page 13: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/13.jpg)
RID Benefits
Enables random testing by suppressing aberrant and reentrant interrupts
Hardens embedded system with respect to unexpected interrupts after deployment SW bugs can cause these So can loose wires, EMI, or other
HW problems
![Page 14: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/14.jpg)
Back to Random Testing
Generate interrupt schedule
Cycle accurate simulation withinterrupt scheduling support
Problem? Debug!No Yes
![Page 15: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/15.jpg)
Interrupt Schedules
List of pairs (vector #, firing time)
Schedule generator parameterized by density for each interrupt vector
![Page 16: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/16.jpg)
Simulator Support
We hacked Avrora – sensor net simulator from UCLA Our interrupt scheduling patches
now included in the distribution
![Page 17: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/17.jpg)
Detecting Failure
1. Ask the application – See if it responds to network packets
2. Ask the simulator – Avrora reports illegal memory access and illegal instructions
![Page 18: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/18.jpg)
TinyOS Oscilloscope Bug
time
ADCrequestand int.
dataTask
Interrupt stores data into array dataTask resets buffer pointer No interlock between interrupt
and task
![Page 19: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/19.jpg)
TinyOS Oscilloscope Bug
time
random ADCrequests
and interrupts
cras
h!
Buffer overrun kills the system unless dataTask runs on time
![Page 20: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/20.jpg)
Original interrupt schedule that triggers bug is > 300,000 interrupts Hard to tell what went wrong!
Used “delta debugging” algorithm to minimize schedule Can trigger bug with just 75
interrupts Bug much easier to find now
Fixing the bug: Easy – add array bounds check
![Page 21: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/21.jpg)
Problem: Stack overflow kills sensor network programs
Solution: Compute WC stack depth through static analysis of binaries
Lingering questios: Is the bound actually
conservative? If so, how pessimistic is
the bound? Answer: Testing
data,BSS
stack
cras
h!
![Page 22: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/22.jpg)
Stack Depth w/o Random
![Page 23: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/23.jpg)
Stack Depth w/Random
![Page 24: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/24.jpg)
Finding Deep Stacks Pure random testing doesn’t cut it
Program behavior surprisingly sensitive to interrupt schedule density and structure
Even running overnight did not find schedules that make deep stacks
Solution: Genetic algorithm evolves better interrupt schedules About 100 generations to find
deepest stack 3 hours CPU time
![Page 25: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/25.jpg)
Revising a Stack Depth Bound
![Page 26: Random Testing of Interrupt-Driven Software](https://reader030.vdocuments.us/reader030/viewer/2022032709/568131a1550346895d981234/html5/thumbnails/26.jpg)
Conclusions
Random interrupt testing: Good Restricted Interrupt Discipline
makes it work Src-src transformation makes RID
easy to implement GA does directed search for
interesting schedules Delta finds interesting subsets of
large interrupt schedules