Contingency Software in Autonomous Systems

Stacy Nelson, Nelson Consulting/QSS

Robyn Lutz, JPL/Caltech & ISU

SAFE

Terminate Flight

This research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, and at NASA Ames Research Center, under a contract with the National Aeronautics and Space Administration. The work was sponsored by the NASA Office of Safety and Mission Assurance under the Software Assurance Research Program led by the NASA Software IV&V Facility. This activity is managed locally at JPL through the Assurance and Technology Program Office

OSMA Software Assurance Symposium July 20-July 22, 2004

• Overview– Goals– Technology Readiness Level– Availability of Data

• Approach• Preliminary Results• Work-in-progress• Benefits

– Potential Applications– Barriers to Research or Application

• Future Work

Topics

Contingency Software in Autonomous Systems

Video from Camcorder

Video from Color Camera

Video from tracking camera on trailer

Virtual rotorcraftfollowing APEX plan(green bar)

Apex plan

DART DEMO

• Adding intelligent diagnostic capabilities by supporting incremental autonomy

• Responding to anomalous situations currently beyond the scope of the nominal

fault protection

• Contingency planning using the SAFE (Software Adjusts Failed Equipment)

approach

Unique Research Relevant to NASA

Contingency Software in Autonomous Systems

• Mitigate failures via software contingencies resulting in safer, more reliable autonomous vehicles in space and in FAA national airspace– Enhance diagnostic techniques to identify failures – Provide software contingencies to mitigate failures– Perform tool-based verification of contingency software– Apply results to ARP (Years 1 & 2) and MSL (Years 2 & 3)

• Status: Year 1 of planned 3-year study (1/04 start)

Overview

Contingency Software in Autonomous Systems

CurrentPractice

SW ContingencyPlanning Full Autonomy

• Current technology readiness level = 2+– 2: “Technology concept and/or application formulated”

– completed 6/04– 3: “Analytical and experimental critical function and/or

characteristic proof-of-concept” – in-progress (12/04 completion)

• Current penetration factor = 8– Data passed back to project

Contingency Software in Autonomous SystemsTechnology Readiness Level

Contingency Software in Autonomous SystemsAvailability of Data: High

Contingency Software in Autonomous SystemsProblem

Failure

WHAT FAILED?

Autonomous vehicles have limited capacity to identify/mitigate failures

Contingency Software in Autonomous Systems

• Enhance diagnostic techniques to identify failures • Provide software contingencies to mitigate failures • Perform tool-based verification of contingency software and • Apply results to ARP (and MSL) to pave the way to more resilient, adaptive unmanned systems

Approach

SAFE Vehicle(Software AdjustsFailed Equipment)

Flight Critical Parameters

Failure Diagnosis

Failure1

2

3

ARP Functional Requirements:

CurrentPlanned

Contingency Analysis:SFMECA

SFTA

Contingency Planning:Available indicatorsContingency triggers

Contingency responses2-Level (recover/predict)

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Contingency Process Overview

Customized the IEEE/EIA 12207.2 Annex I Evolutionary/Spiral Methodology

1. Brainstorm with UAV team to uncover candidates for software contingenciesReview UAV literature and project reportsLead brainstorming sessions with domain expertsWork with team to identify and prioritize high-concern candidatesSelect top priority candidates

2. Model unit of interest (i.e. cameras, communications systems…)Model system including: Architecture & State diagramVerify models with UAV team

3. Contingency requirements verificationPerform SFMECA

4. Analyze testabilityIdentify how each contingency can be detectedPerform SFTAExperiment with assignment of measure of uncertainty

5. Develop recovery strategyDetermine candidate strategies for contingency responses (prevent/respond/safe)Determine availability of data needed to determine/execute appropriate contingency

6. Prototype contingency in progressively higher fidelity testbeds 7. Monitor contingency performance

Design of Hybrid Mobile Communication Networks for Planetary ExplorationRichard Alena, John Ossenfort, Charles Lee, Edward Walker, Thom Stone

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMSRelated WLAN Work

• RF signal strength measurements can be normalized to

theoretical values and used to predict range

( Good correlation and repeatability of signal strength

measurements using different antenna configurations

and test distances)

• Network throughput is reasonably predictable for

single hop links at short distances (WLAN link runs

under nominal conditions with no packet loss)

• However, network throughput is not predictable for

complex WLANs consisting of multiple repeater hops or

long distances. WLAN links run under conditions of

varying packet loss. Packet loss significantly reduces data

pipelining by introducing highly variable packet transfer

latencies due to packet re-transmission

• RF signal strength measurements can be normalized to

theoretical values and used to predict range

( Good correlation and repeatability of signal strength

measurements using different antenna configurations

and test distances)

• Network throughput is reasonably predictable for

single hop links at short distances (WLAN link runs

under nominal conditions with no packet loss)

• However, network throughput is not predictable for

complex WLANs consisting of multiple repeater hops or

long distances. WLAN links run under conditions of

varying packet loss. Packet loss significantly reduces data

pipelining by introducing highly variable packet transfer

latencies due to packet re-transmission

• Packet loss due to multi-path, low signal

strength, interference significantly disrupt the timing

of packet transfers due to packet re-transmission.

• MAC layer uses packets for many purposes such as

node authentication, data flow management and data

transfer. Packet loss can affect any of these functions

resulting in a wide variety of failures.

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMSPerception (Cameras)

•Perception is a critical function in systems requiring obstacle

avoidance, threat detection, science missions and

“opportunistic” discovery.

•Optical flow systems use contrasts in the surrounding

imagery to determine position. If a vehicle using optical flow

flies, for instance, over a very regular terrain such as a grassy

field or an empty parking lot, it may crash.

RotorcraftControl

Center(“Trailer”)

Rotorcraft

Comm. Range(varies)

Not to Scale

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS

RadioModem

802.11b PCMCIA card

OnboardAntenna

GPS Autonomous flight (Nominal Case) (RC pilot standing by in case of emergency)

Equipment

New: Critical communications over radio modem and other communications via WiFi. Reason: Security and bandwidth

CLAWFlight Control Laws

DOMSDistributed Messaging

System

GPS

APEXReactive Planner

Telemetry

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMSPartial Onboard Architecture

*domsD – DOMS transport daemon

*

Yamaha System

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMSPerception (Cameras)

•Perception is a critical function in systems requiring obstacle

avoidance, threat detection, science missions and

“opportunistic” discovery.

•Optical flow systems use contrasts in the surrounding

imagery to determine position. If a vehicle using optical flow

flies, for instance, over a very regular terrain such as a grassy

field or an empty parking lot, it may crash.

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Cameras Onboard Rotorcraft

Gray scale wing tip (stereo vision)

Color Camcorder

Color Camerafor situationalawareness

Firewire Hub

Image Processing System

Firewire

Left Wing

Right Wing

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMSOther Perception Components Onboard Rotorcraft

• SIC (K) – Fast & accurate scanning laser

• Laser range finder – returns single point used for precision autonomous landing if GPS signal is lost

• Sonar (or Ultrasonic) range finder to determine distance to ground

Sonar Range Finder

Laser Range Finder (coming soon)

GPS

Scanning Laser Range Finder (SICK) (coming soon)Cameras

Cases in which the cameras are a critical system:1. Cameras assigned responsibility during nominal ops

• No line of sight -> Camera provides position info

2. Cameras are backup when other subsystems fail• Failed/degraded GPS -> Camera provides position info• Failed/degraded ARP -> Camera provides landing-site data

3. Images as mission objective (surveillance)• Failure of cameras can jeopardize success

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMSCamera Criticality

• Collaborating with Autonomous Rotorcraft Project to experimentally apply approach

• Project provides feedback on our models, guidance on future plans– Feasibility check– Reviewed ARP architecture including communications &

perception– Proposed initial SW contingencies for communication and

perception failures

• ARP team including us in team meetings • PM has agreed to try contingencies appearing

viable• Finalized SW contingencies for communications

& perception with ARP team– ARP team considers further investigation & simulation high priority

for 4 identified SW Contingencies

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Preliminary Results

• Loss of Communication:• Detect loss of communication revise mission plan:

– Reroute– Fly to rally point

• Interference with Communication:• WiFi Security• Throttle back communication

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Preliminary Results

• Loss of Perception:• Detect camera failure and reconfigure to use another camera

– If color camera used for situational awareness fails, then switch to one of the gray scale cameras.

– If left wing camera fails then reconfigure to use left wing color camera for stereo vision.

• Degradation of Perception:• Change image-acquisition configuration or parameters

– If need to lower resource usage, reduce image size

• Change image-transmission configuration or parameters– If need to lower bandwidth, drop color, drop frame rate, compress image

more (trade off with CPU cycles)

• Paves the way to more resilient, adaptive unmanned systems

• Supports spectrum of project adoption of autonomy– Flexible: project determines how much autonomy– Incremental requirements (evolutionary process model)

• Considers contingencies beyond failures:– Environmental changes that threaten mission (e.g., surveillance)– Changes in resource needs vs. availability that impact mission

success (e.g., will need high-bandwidth)– Mobility capabilities that create tradeoffs with communication,

imaging optimizations• NASA Experience: Will demonstrate on NASA projects• Anticipated cost savings for projects with evolving

autonomy needs• Equips us with a methodology to continue to move toward autonomy

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMS Benefits

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMSTowards MSL Risk Assessment for SW Contingencies

Example Using DDP tool (fault treeApproach) to assess risk of SW Contingency Plans(collaborationbetween CSAS &Dr. Martin Feather)

Note: example risknumbers relative not absolute – more workrequired

• Autonomous Rotorcraft Project (ARC)• Mars Science Laboratory (JPL)• Other autonomous vehicles• Other mobile imaging systems

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMSPotential Applications

• Challenge 1: ARP is moving target (rapid evolution)Approach: Track planned & unplanned changes via weekly telecons

• Challenge 2: Planning for MSL application Approach: Demo benefits on ARP first; select ARP functionalities also important to MSL (communication, perception)

• Challenge 3: Tech transfer will depend on ease of reuseApproach: Provide results both in terms of (1) improved verification techniques for contingencies and (2) reusable designs for common contingency applications

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMSBarriers to Research or Application

• Tool-based verification on NASA project

• Advance NASA’s information about communications and perception systems for autonomous vehicles

CONTINGENCY SOFTWARE in AUTONOMOUS SYSTEMSFuture Work