ft4 data analysis summary (ssi-arc) · ft4 data analysis summary (ssi-arc) doug isaacson, chester...

1

FT4DataAnalysisSummary(SSI-ARC)DougIsaacson,ChesterGong,ScottReardon,ConfesorSantiago12DEC2016

AbstractStandardsforUnmannedAircraftSystem(UAS)Detect-and-Avoid(DAA)systemsarecurrentlybeingdevelopedundertheauspicesoftheRTCASpecialCommittee228(SC-228).Tosupportthedevelopmentofthesestandards,aseriesofflighttestshasbeenconductedatNASA’sArmstrongFlightResearchCenter(NASA-AFRC).Thefourthinthisseriesofflighttestactivities(FlightTest4,orsimplyFT4)wasconductedduringtheSpringandSummerof2016.FT4supportedtheobjectivesofnumerousorganizationsworkingtowardUASDAAMinimumOperationalPerformanceStandards(MOPS)andUASDAARadarMOPS.Thesummaryprovidedhereinislimitedtotheobjectives,analysisandconclusionsoftheNASAAmesResearchCenter(NASA-ARC)SSIteamtowardtherefinementofUASDAAMOPS.Thisdocumentprovidesahigh-leveloverviewofFT4andtheSSI-ARCobjectives,asummaryofthedataanalysismethodologyandrecommendationsforUASDAAMOPSrefinementsbasedonthedataanalysisresults.Atotalof72encounterswereflowntosupportSSI-ARCobjectives.TestresultsweregenerallyconsistentwithacceptableUASDAAsystemperformanceandwillbeconsideredinbroaderSC-228requirementsvalidationefforts.ObservedalertleadtimesindicatedacceptableUASDAAalertingperformance.EffectiveinteroperabilitybetweentheUASDAAsystemandtheTrafficAlertandCollisionAvoidanceSystem(TCAS)wasobservedwithonenotableexception:TCASResolutionsAdvisories(RA)wereobservedintheabsenceofanyDAAalertontwooccasions,indicatingtheneedforalertparameterrefinement.FindingsfurtherindicatedtheneedforcontinuedworkintheareasofDAAWellClearRecoverylogicandalertstabilityforMode-C-onlyintruders.Finally,resultsdemonstratedahighlevelofcompliancewithasetofevaluationcriteriadesignedtoprovideanecdotalevidenceofacceptableUASDAAsystemperformance.

SystemUnderTest:JADEMFT4wasconductedatNASA-AFRCbetweenAprilandJuneof2016.NASA’sIkhanaresearchaircraftandGroundControlStation(GCS)wereequippedwiththenecessaryhardware,displaysandsoftwaretoevaluatethreeprototypeUASDAAsystems:NASADaedalus,NASAJavaArchitectureforDetectandAvoidExtensibilityandModeling(JADEM)andGeneralAtomicsAeronauticalSystems,Inc.ConflictPredictionandDisplaySystem(CPDS).WhileeachsystemwasdevelopedtobeasconsistentwiththedevelopingSC-228MOPSaspractical,therewerekeydifferencesintheirimplementationthatprecludeaggregatingFT4resultsacrossthesystemsundertest(SUTs).ThisdocumentprovidestheanalysismethodologyandresultsforflighttestingoftheNASASSI-ARC-developedJADEMsystem.

AlertingCriteriaTheFT4JADEMimplementationemployedthesamealertingcriteriaanddisplaysymbologyutilizedintheNASAPartTask6(PT6)study,asshowninTable1.Alertinghysteresis,asdefinedinthedraftDAAMOPS,wasimplementedforFT4(minimum4-secondalertpersistenceunlesssupersededbyhigherpriorityalert).

2

OmnibandsGuidance

TheJADEMsystemprovidesguidancetotheUASpilotintheformofheading/altitudebands,thecolorofwhichindicatesthelevelofalertthatwouldresultiftheownshipweretoinitiateaturn/altitude-changetotheheading/altitudedepicted.Greenbandingindicatesaturn/altitudechangetothedepictedheading/altitudeisnotpredictedtoresultinaLossofWellClear(LoWC).Yellowbandingindicatesaturn/altitude-changeispredictedtocauseaDAACorrectiveAlert(i.e.potentialLoWCinthenext25-55s).Lastly,redbandingindicatesaturn/altitude-changeispredictedtocauseaDAAWarningAlert(i.e.potentialLoWCinthenext25sorless).Figure1depictssampleOmnibandsguidanceaswouldbedisplayedintheUASGCS;a“Corrective”DAAAlertisdisplayedfortheonlydisplayedintruder.TheguidanceillustratedinFigure1indicatesviablemaneuverstoremainwellclearareavailableinboththeheading(~15degreesleftorright)andaltitude(+600ftor-400ft)bands.

Table1:JADEMFT4AlertingCriteria

3

Figure1:SampleJADEMOmnibandsGuidance

Omnibandsguidanceformultipleintrudersisproducedbyformingtheunionofindividualomnibandsguidanceforeachintruderwithinthefieldofregard,withhigherpriorityalertsoverridinglowerprioritieswhenalertsdifferbetweenintrudersforagivenheading/altitudelevel.Additionally,thefollowingassumptionsandparameterswereusedintheJADEMguidanceandalertinglogicforFT4:

1. Ownshipassumedtoexecuteturnsataconstantrateof3deg/s2. Ownshipassumedtoclimbordescendatconstantrateof1000ft/min3. Omnibandsheadingguidancelimitedwithin+/-100degreesofcurrentownshipheading4. Omnibandsheadingguidanceprovidedwith1degreeresolution/discretization5. Omnibandsaltitudeguidancelimitedwithin+/-3000feetofcurrentownshipaltitude6. Omnibandsaltitudeguidanceprovidedwith100ftaltitudediscretization

WellClearRecoveryGuidance

WellClearRecovery(WCR)guidancereferstoguidancethatisprovidedtotheUASpilottoregainWellClearfromallintruders.JADEMwasconfiguredforFT4toprovideWCRguidancewhennoallowableownshipmaneuverispredictedtoremainwellclearofallintruders.ItshouldbenotedthatthisimplementationexceedstheminimumMOPSrequirements(whichonlyrequiresWCRguidancewhenaLoWChasalreadyoccurred).Figure2illustratesWCRguidanceasimplementedinJADEMforFT4;theredhorizontalbandingindicatesnoviablemaneuversarepredictedtoremainWellClear,andthegreenWCR“wedge”indicatestheadviseddirectionandroughmagnitudeofhorizontalmaneuverthatwillresultinregainingWellClearinasafeandtimelymanner.Determinationofa“safeandtimely”maneuverisdictatedbyevaluationofaWCRcostfunction.

HeadingBands

AltitudeBands

4

TheJADEMWCRcostfunctionforFT4considersanumberoffactors,butismostinfluencedbytheproximityofthepredictedClosestPointofApproach(CPA)totheacceptedNMACvolumeandsecondarilybythedesiretomaintainconsistentWCRguidance(changesinturndirectionincuracostpenalty).Finally,verticalJADEMWCRguidancewasnotprovidedforFT4topreventtheUASpilotfrommaneuveringverticallynearthecollisionavoidanceboundaryagainstcooperativeintruders(whichmaydegradeTrafficAlertandCollisionAvoidanceSystem(TCAS)IIperformance)andduetoverticalstateestimationuncertaintiesforintruderstrackedsolelybytheradar(i.e.non-cooperativeintruders).

Figure2:SampleWCRGuidance

TrafficAlertandCollisionAvoidanceSystem(TCAS)InteroperabilityImplementation

OneoftheprimaryobjectivesofFT4wastoevaluatetheUASDAA-TCASInteroperabilityconceptdevelopedsinceFT3.ThefollowingfourheuristicssummarizetheDAATCASInteroperabilityConceptasimplementedinJADEMforFT4:

1. AnyintruderwithanactivecorrectiveResolutionAdvisory(RA)isremovedfromallDAAguidancecalculations

a. HorizontalDAAguidancewillbeshownfornon-RAaircraftb. AllDAAverticalguidanceissuppressedduringacorrectiveRA

2. DuringapreventiveRA,TCASguidanceisaninputtotheDAAverticalguidancetoensureanyverticalDAAguidanceisconsistent(e.g.DONOTCLIMB)withtheRAguidance

3. Anytimeownship’scompliancewithacorrectiveRAleadstoasecondaryDAAWarningalert(maneuvernow),DAAguidanceshallreverttowellclearrecoveryinordertobemoredirectwithguidance,e.g.:

a. CompliancewithTCAS‘DESCEND’RAleadstoasecondaryDAAWarningb. RatherthanshowthepilotfullOmnibandssuggestiveguidance,limitedsuggestive

guidanceisdisplayed(e.g.maneuverleft)

HighBound

LowBound

5

FT4ObjectivesandMethodologyforJADEMAnalysisEachoftheFT4systemsundertest(SUTs)wasevaluatedindependently,andeachaccordingtoobjectivesandanalysismethodologiestailoredtotheirspecificimplementationofthedraftSC-228UASDAAMOPSrequirements.ThissectiondescribestheFT4objectives,scenariodevelopmentandanalysismethodologydevelopedandemployedfortheJADEMsystemundertest.

SSI-ARCFT4ObjectivesandScenarioGeneration

Asetofhigh-leveltestobjectivesweredevelopedtosupportthedevelopmentoftheSC-228UASDAAMOPS.Thefollowingfourhigh-leveltestobjectiveswereusedtoguidetheFT4planning,conductandanalysisofJADEMalertingandguidancelogic:

1. ValidateDAArequirementsinstressingcasesthatdriveMOPSrequirements,including:High-speedcooperativeintruder,Low-speednon-cooperativeintruder,highverticalclosurerateencounter,andModeC/S-onlyintruder(i.e.withoutADS-B).

2. ValidateTCAS/DAAalertingandguidanceinteroperabilityconceptinthepresenceofrealisticsensor,trackingandnavigationalerrorsandinmultiple-intruderencountersagainstbothcooperativeandnon-cooperativeintruders.

3. Validate“WellClearRecovery”guidancerequirementsinthepresenceofrealisticsensor,trackingandnavigationalerrors.

4. ValidateDAAalertingandguidancerequirementsinthepresenceofrealisticsensor,trackingandnavigationalerrors.

Insupportoftheseobjectives,aseriesofencounterswerescriptedtoaddresseachobjective.Thedevelopmentofeachscenariorequiredidentificationof:theprimaryhigh-levelobjectivebeingaddressed,theencountergeometry,intruderequipage,datacollectionrequirements,encountermethodology(includingpilotinstructions),checkstoensuredataquality,draftMOPSreference,andevaluationcriteriatodeterminewhetherornotthetestobjectivesweremet.AsamplescenariotemplateresultingfromthisprocessisincludedasFigure3.TheSSI-ARCteamsubsequentlycoordinatedwithAFRCflighttestpersonneltorefinethescenariosoutlinedinthetemplatesforproductionofflight“testcards”providingthelevelofdetailnecessarytotrainownshipandintruderpilotsandexecutethescriptedscenarioonthedayoftheflight.Intotal,72testcardsweredevelopedandflownforFT4JADEManalysis;asampletestcardisincludedasFigure4.

6

Figure3:Sam

pleFT4ScenarioTemplate

7

Figure4:Sam

pleFT4TestCard

8

AnalysisMethodology

Theanalysismethodologywasdesignedtoaddressthe“evaluationcriteria”identifiedforeachscenariotemplate.Assuch,thefollowingkeyeventdatawereextractedfromtherecordedflighttestdataforeachtestcard:

• FirstTrack:Firstrecordedtrackforeachintruderintheencounter• DAAPreventiveAlert:Onset(s)ofDAAPreventivealert(s)(foreachnewoccurrence)• DAACorrectiveAlert:Onset(s)ofDAACorrectivealerts(s)(foreachnewoccurrence)• DAAWarningAlert:Onset(s)ofDAAWarningalerts(s)(foreachnewoccurrence)• LoWC:OnsetofeachnewLossofWellClear(maybemultipleLoWCsand/orLoWCfor

multipleintruders)• RegainWellClear:firsttimeatwhichWellClearisregainedafteraLoWCevent.• ClosestPointofApproach:time(s)atwhichminimumslantrangeisachievedduringthe

encounterforeachintruder• TCASEvent:TimeofinitialTCASPreventiveorCorrectiveAlertandanyassociated

guidance(orTCASallclearmessage)• WellClearRecovery:OnsetofWCRguidanceandassociateddirection(e.g.turnright)as

wellasanychangestodirectionofguidance• FinalAlert:Thelastrecordedalertforeachintruderintheencounter• LastTrack:Lastrecordedtrackforeachintruderintheencounter

Theseeventswereloggedtoadatabasethatwasusedintheanalysistodetermineifeventsoccurredinthepropersequence(e.g.TCASCorrectiveRAswereprecededbyDAAcorrectiveorwarningalerts)andwithappropriatetiming,asdictatedbyeachencounter’sevaluationcriteria.Table2includestheextractedeventsfromasingleFT4JADEMencounter.

Table2:SampleEncounterEventTableCycleCount Intruder Time ClockTime Event

191 N3GC 1461942437.39 15:07:17 FIRST_TRACK

198 N3GC 1461942445.37 15:07:25 DAA_PREVENTATIVE

203 N3GC 1461942450.38 15:07:30 DAA_CORRECTIVE

208 N3GC 1461942455.41 15:07:35 DAA_PREVENTATIVE

215 N3GC 1461942462.35 15:07:42 DAA_CORRECTIVE

233 N3GC 1461942480.40 15:08:00 DAA_WARNING

257 n/a 1461942504.39 15:08:24 WCR_TURN_RIGHT

258 N3GC 1461942505.38 15:08:25 LOWC

270 N3GC 1461942518.39 15:08:38 TCAS_CLIMB

291 N3GC 1461942539.39 15:08:59 CPA

291 N3GC 1461942539.39 15:08:59 TCAS_LEVEL_OFF

296 N3GC 1461942544.36 15:09:04 TCAS_AA_CLEAR

297 N3GC 1461942545.39 15:09:05 DAA_PREVENTATIVE

297 N3GC 1461942545.39 15:09:05 REGAIN_WC

300 N3GC 1461942548.40 15:09:08 FINAL_ALERT

308 N3GC 1461942556.36 15:09:16

LAST_TRACK

9

Alsosupportingtheassessmentofevaluationcriteriawereaircraftstatedata,alertingthreatparameterdata,intruderrelativestatedata,andomnibandsguidancedata.ExamplesofdataplotsforeachoftheseareshowninFigures5-8.

Figure5:SampleAircraftStateDataPlots

Figure6:SampleAlertingThreatParameterDataPlots

10

Figure7:SampleIntruderRelativeStateDataPlots

Figure8:SampleOmnibandsDataPlots

11

Ineachoftheaircraftstateplots(Figure5),relativestateplots(Figure6)andalertingthreatdataplots(Figure7),thecolorofthefilled-circlemarkerstransposedonthedataplottraceindicatesthetypeofDAAalert:DAApreventive(black),DAAcorrective(orange),andDAAwarning(red)alerts.Additionally,thebackgroundcolorcodingontheOmnibandsdataplots(refertoFigure8)isconsistentwiththeOmnibandscoloringpredictingnoLoWC(green),LoWCwithin25-55s(yellow)andLoWCwithinthenext25s(red)attheindicatedheading/altitude.Lastly,theOmnibandsdataplotsalsoindicatewhenWCRguidance(yellowtrianglemarkers)and/orTCASRAs(magentatrianglemarkers)arepresent.OmnibandsdataplotswereusedtoassessDAA-TCASinteroperability.

Finally,videorecordingsoftheUASDAAJADEMdisplaywerereviewedtoprovideanyadditionaldetailsnecessarytoassesstheperformanceoftheDAAsystemagainstthescenarioevaluationcriteria.

ResultsScenarioEvaluationCriteriaCompliance

ThefirstelementoftheanalysisfocusedonevaluationoftheBooleanevaluationcriteriaforeachscenario.Eightevaluationcriteriawereassessedaspartofthisanalysis(FT4JADEMcompliancewitheachcriterionisprovidedinparenthesis)

1. ADAACorrectiveAlertisissuedtotheUASPilot(94%)2. OwnshipremainsWellClearofintruders(90%)3. DAAAlertsareremovedonceclearofthreat(100%)4. DAAAlertsandGuidanceareprovidedtotheUASPilotfollowingtheexpirationofaTCAS

RA,ifappropriateaccordingtotheDAAalertingthresholds(100%)5. DAAAlertsandGuidanceareremovedforintruderswithCorrectiveTCASRAGuidance

(90%)6. WellClearRecoveryGuidanceisprovidedwhennoviablemaneuversarepredictedto

remainwellclearofallintruders(100%)7. UASPilotmaneuversinresponsetoWCRguidanceandregainsWellClear(100%)8. UASPilotmaneuversinresponsetoandconsistentwithcorrectiveTCASRAguidance

(100%)

Dataanalysisdemonstratedagenerallyhighlevelofcompliancewiththeevaluationtestcriteria,withafewnotableinstancesofnon-compliance.Thesenon-complianceswereevaluatedonacase-by-casebasisasfollows:

FailuretoProvideCorrectiveAlert:AllcasesinwhichaDAACorrectivealertwasnotissuedpriortohigherpriorityalertsoccurredduetotheDAACorrectiveandWarningAlertshavingthesameHMDthreshold.ScenariosdevelopedtoevaluateTCASinteroperabilitywerescriptedwithintrudersmaneuveringtointerceptthecourseoftheUASownshipwithinclosetemporalproximitytothepredictedCPA.TheresultofthisisthatthepredictedmodifiedtauoftheseencounterswasgenerallybelowtheDAAWarningAlertthresholdpriortotheinterceptmaneuver,whilethepredictedHMDdidnotmeettheHMDcriterionforDAACorrective.AstheintruderturnedtointercepttheUASownship,thepredictedHMDrapidlydecreasedinvalueuntilitmettheDAAWarningalertHMDthresholdandaDAAWarningAlertwasissuedwithoutapriorcorrectivealertduetothe

12

lowmodifiedtauvalueatthetimeofthemaneuver.WhileincreasingtheDAACorrectiveAlertHMDvalueinthesecaseswouldleadtothepreferredalertsequencing(DAACorrectiveAlertprecedingDAAWarningAlert),itisunclearifsuchachangeisnecessary.IncreasingtheDAACorrectiveAlertHMDvalueshouldbeconsideredinthecontextofthetradeoffbetweenincreasedDAACorrectiveAlertrateandtherateofoccurrenceofencountersinactualoperationsthatwouldotherwiseleadtotheundesirablealertsequence.Thesenon-compliancesarebelievedtobeanartifactofacarefullyplannedtestmaneuverandareexpectedtobeuncommoninactualoperations;ananalysisofencountersanticipatedinactualoperationsisnecessarytocorroboratethisexpectation.

FailuretoRemainWellClear:TheevaluationcriteriatoremainwellclearonlyappliedtomitigatedencountersinwhichpilotswereinstructedtofollowDAAguidance.Forsuchencounters,UASpilotswereinstructedto:1)postponemaneuversuntilaDAAWarningAlerthadbeenissued,2)maneuvertotheedgeofthegreenguidancebandinthedirectionoftheirchoice(ifmorethanoneoptionwasprovided),and3)toonlymaneuveronce(nocorrectionstoinitialmaneuverallowed).TheseinstructionswereprovidedwiththeintentionofisolatingJADEMperformancefromthepilot’sabilitytoaccommodateshortcomingsinDAAsystemperformanceinremainingWellClear(e.g.byaddinghis/herownexcessmaneuverbuffersormakingadjustmentstotheinitialmaneuvertocounterpoorJADEMtrajectorypredictionsorguidanceinstabilities).SuchpilotresponsetoaDAACorrectiveAlertinoperationaluseofJADEMwouldbeconsistentwithapilotassessingthepredictedLoWC,anddelayingmaneuveringpendingfurtherdevelopmentoftheencounteroruntilahigherprioritytaskiscompleted.WhiledelayingUASmaneuveringtoavoidLoWCuntilaDAAWarningAlertisreceivedshouldgenerallynotleadtoLoWC,suchdelayswouldresultinhigherrateofLoWC(acceptableratesofLoWCinthiscasearebeyondthescopeofFT4).Thepilotinstructiontoonlymaneuveronceforagivenencounterremovesthepilot’sabilitytoreacttotheuncertaintiesofdevelopingencountersandartificiallyincreasestherateofLoWC;refinementstoUASmaneuverstoremainWellCleararetobeexpectedinactualuseofaDAAsystem.GiventhepilotinstructionsintheuseofJADEMinFT4,itissomewhatsurprisingthatLoWCsonlyoccurredin2ofthe20mitigatedencounters.Bothinstancesweretheresultofmodestintruderaccelerations:increasedgroundspeedinonecaseandaslowlyarcingturntowardtheUASinthesecond.ItislikelythatbothLoWCswouldhavebeenavoidedwithearlierownshipmaneuvering(inresponsetotheinitialDAACorrectivealerts)andtheallowanceofmaneuveradjustmentstoaccountformaneuveringintruders.

FailuretoRemoveDAAGuidanceforIntruderswithCorrectiveRAGuidance:TheTCASinteroperabilityconceptimplementedforFT4requiredthatintrudersinvolvedinaTCAScorrectiveRAberemovedfromconsiderationforDAAAlertingandGuidanceconsideration.WhilethiswasimplementedintheJADEMsoftware,3instancesofconcurrentJADEMDAAandTCASRAguidancewereobservedduringFT4.Eachoftheseinstancesweredeterminedtobetheresultofasoftwareissueunrelatedtotheinteroperabilityconceptandisconsideredatestartifactnotindicativeofsystemperformance.

DAAAlertTiming

DAAalerttimingwasanalyzedtoassesstheefficacyofJADEMDAAalertsinarealisticenvironment,includingstressingencountergeometriesandsurveillancesensorerror.TheDAAalerttiminganalysiswaslimitedtounmitigatedencounterstoremovetheinfluenceofpilotactionsontheresultsandbecausesomeofthemetricsarereferencedtoinitialLossofWellClear.

13

DAAAlertLeadTime:Figures9and10providehistogramsofDAACorrectiveAlertleadtimeandDAAWarningAlertleadtime,respectively.Leadtimeisdefinedastheelapsedtimebetweentheinitialalertofagivenpriority(e.g.firstDAAWarningalertissuedtotheUASpilotduringanencounter)andtheinitialLoWC.TheobservedmeanDAACorrectivealertleadtimewas46sandthemeanDAAWarningalertleadtimewas23s.TheleadtimeforDAACorrectivealertsintheabsenceofaDAACorrectivealertwasassignedthesameleadtimeastheDAAWarningalert;thatis,thefirsttimeatwhichallthealertingthresholdsfortheDAACorrectivealertaremet.Whentheseencountersareexcludedfromtheanalysis,themeanDAACorrectivealertleadtimeishigher:49s.

LeadTime,tLEAD(s)

Figure9:DAACorrectiveAlertLeadTimeHistogram

LeadTime,tLEAD(s)

Figure10:DAAWarningAlertLeadTimeHistogram

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80

Frequency

Frequency

14

DAAAlertTransitionTime:Alerttransitiontimeisdefinedastheelapsedtimebetweenagivenalertprioritylevelandthenexthighestalertprioritylevel(e.g.transitiontimebetweeninitialDAACorrectivealertandinitialDAAWarningalert).Table3providesthemeantransitiontimesbetweenalertprioritylevels;theresultsareconsistentwithexpectationsgiventhemodifiedtauvaluesforeachalertlevel,asspecifiedinTable1.NofirmconclusionscanbedrawnfromthemeantransitiontimesinTable3duetothelimitedsamplesizeandoperationalcoverageoftheFT4encounterset.

Table3:DAAAlertTransitionTime

AlertTransition MeanTransitionTime(s)

DAAPreventivetoDAACorrective 3.2s

DAACorrectivetoDAAWarning 24.9s

DAAWarningtoWellClearRecovery 19.3s

WellClearRecoverytoTCAS 9.7s

DAATCASInteroperability

Asdiscussedintheprecedingevaluationcriteriacompliancesection,compliancewiththerequiredelementsoftheDAA-TCASinteroperabilityconceptswaslargelymet,withfewexceptionsbeingtheresultofJADEMsoftwareimplementationissues.Ofnote,twoscenariosresultedincorrectiveTCASRAswhileJADEMdeemedtheownshipWellClearoftheintruderwithgreenbandingguidanceatthetimeofthecorrectiveTCASRA.FurtheranalysisbySC-228personnelandTCAS-IIexpertsconcludedthatsuch“WellClearRAs”observedinFT4aretheresultofincompatibilitybetweenDAACorrectiveAlertHMDthreshold(0.75nm)andhowHMDisused(orinthiscase,NOTused)tofilterTCASRAs.Inthesetwocases,itwasconcludedthatTCASdidnotconsiderHMDinissuingacorrectiveRAduetothecloseproximityoftheintruderandtheexpectederrorintheHMDprediction.BasedonthisfindingandconsistentobservationswithallsystemsundertestinFT4,itisrecommendedthattheMOPSAlertingrequirementsberefinedtoconsiderTCASemploymentoftheHMDfilterandtomitigateitsimpactonalertingperformance.

ModeCIntruderGuidanceStability

AnalysisofJADEMguidancestabilityforintruderslackingADS-Bequipmentwasconsideredasecondarypriorityofthedataanalysis.WhenModeCsurveillanceistheonlyavailablebearingsourcefortheDAAsystem,ahighdegreeofbearinguncertaintyistobeexpected.Unfortunately,onlythreeFT4JADEMencountersincludedintruderswithtranspondersbutwithoutADS-Bequipage.Further,theJADEMguidancefortheseencounterswasbasedonafusiontrackthatincludedradarsurveillance(i.e.,haslowbearingerrorwithintheradardetectionrange).Thus,whiletheobservedguidancedemonstratedexcellentstabilityforthesethreeencounters,moreanalysisisneededtodrawanyconclusionsaboutthestabilityofJADEMDAAguidanceforModeCIntruders.However,itisunclearifthisshouldbeconsideredapriorityforfurtherinvestigationgiventheexpectationthatintruderswillgenerallybewithinradarrangepriortotheprescribedalertingthresholds(i.e.bearingerrorisnotexpectedtoinfluenceDAAalerting).

15

WCRPerformance

WhileWCRperformancemettheobjectivecriteriaprescribedinthescenarioevaluationcriteria,FT4UASpilotparticipantssubjectivelyassessedWCRperformanceaslackinginanumberofareas.First,WCRguidancestabilitywasdeemedpoorforintruderslackingADS-Bequipage;suchencountersdemonstratedmultiplechangesintheWCRguidancedirectionthatpilotsfounddistractingorlackinginformativevalue.Second,UASpilotsobjectedtoWCRguidancethatprescribedturnstoheadingsaftofthecurrentUAScourse;suchturnswouldrequiresignificantdeviationfromplannedrouteofflight,wouldresultinsignificanttimeincloseproximityforhead-onencountergeometries,andoftenplacenon-cooperativeintrudersoutsidethesurveillancevolume.

ConcludingRemarksandRecommendationsFT4providedauniqueopportunitytoevaluatetheperformanceofaprototypeUASDAAsystem(NASAJADEM)acrossarangeofencounterconditionsandinarealisticflightenvironmentincludingsurveillancesystemerrors.However,aswithallflighttests,FT4wasatightlycontrolledexperimentwithpriorpilotknowledgeofencountergeometryandintruder“escape”procedures.Assuch,whiletheperformanceoftheUASDAAsystememployedinFT4isevaluatedforthepurposeoffurtheringMOPSdevelopment,theresultsmustbeconsideredinthecontextofFT4andincombinationwithresultsfromotherUASDAAexperiments(e.g.NASAPT5andNASAPT6,wheretheUASpilotshadnoadvanceknowledgeofencountersandintruderactions).However,whereashuman-in-the-loopsimulationsusemodelsofaircraftdynamics,surveillancesensorsandatmosphericphenomena,flighttestoffersaglimpseattheperformanceoftheprototypeUASDAAsystemunderreal-worldconditions;itisthroughthislensthattheperformanceoftheprototypeUASDAAsystemisassessed.DAAsystemperformancewasassessedacrossfivecategoriesofmetrics:AlertTiming,UASDAA-TCASInteroperability,ModeCIntruderAlertStability,WellClearRecoverPerformanceandabroadsetofScenarioEvaluationCriteria.

ObservedalertleadtimesindicatedacceptableUASDAAalertingperformance;meanDAACorrectiveandDAAWarningalertleadtimeswere46and23secondsrespectively.Excludingencountersthatincludedturnstowardtheintruder(withtheexplicitpurposeofcreatingimmediateTCASRAs),theDAACorrectiveAlertleadtimeincreasedto49s.Thesemeanleadtimesareconsistentwith(albeitslightlybelow)theMOPSaveragealertleadtimes(Section2.2.4.3.4.),andareindicativeofacceptablealertingthresholdsimplementedwithinJADEMforFT4.

FT4included29JADEMscenariostoinvestigatetheUASDAA-TCASinteroperabilityconcept.Resultsindicateeffectiveinteroperabilityacrosstherangeoftestconditionswithonenotableexceptioncase:the“WellClearRA”.Observedforasmallnumberofscenarios(andacrossallsystemsundertest),UASpilotswerepresentedwithaTCASRAwhiletheUASDAAsystemhaddeterminedtheintrudertobeWellClear.ItwasdeterminedthatsuchRAswereduetosuppressionoftheHorizontalMissDistance(HMD)criteriontestwhenTCASdeemedtheintruderbearinginformationtobeofinsufficientquality.BecauseTCASdoesnothaveaqualitybearingsurveillancesource(e.g.ADS-Borairborneradar),suchcasesaretobeexpected,anditisrecommendedfurtheranalysisbeconductedtodeterminetherateofoccurrenceofsuchRAsintheNASandpotentialmitigations,ifneeded.

Observationsonthestabilityofalertsfor“ModeC-onlyIntruders”werelimitedtoasinglescenarioinFT4.Assuch,moredataisnecessarytodrawfirmconclusionsontheacceptabilityofUASDAA

16

systemperformanceforsuchintruders.TheconcernregardingalertinstabilityforMode-C-onlyintrudersstemsfromthepoorbearinginformationderivedfromtransponderrepliesalone.However,giventheairborneradarequipagerequirementandtypicalradardetectionrange,itisunclearifsuchintrudersrepresentacredibleconcernregardingalertingstability;bearingerrorshouldbeanon-issueoncetheintruderiswithinradardetectionrange.FurtheranalysisofModeCintruderalertstabilityshouldthusconsiderradardetectionrangeandthebearingperformanceofanintegratedintrudertrack.

NinescenarioswereconductedtoassesstheperformanceofUASDAAWellClearRecovery(WCR)guidancetotheUASpilot.WhileobservationsindicatedeffectiveWCRguidanceforADS-Bequippedintruders,guidancewasdeemedlargelyineffectiveforotherintrudersforanumberofreasons.First,UASpilotsindicated(anddataconfirmed)somedirectionalinstabilityinWCRguidance;itisrecommendedthatadditionalheuristicsand/orhysteresisbeincludedinWCRlogictopreventfrequentchangesinthedirectionalWCRguidance.Second,WCRguidanceoccasionallyincludedturnswellbeyond90degreesfromthecurrentcourse;UASpilotsfoundthisobjectionableandineffective.ItisrecommendedthatWCRdirectionalguidancebelimitedtoturnsoflessthansomereasonablebound(e.g.90degreesfromcurrentcourse).

Finally,abroadsetofscenarioevaluationcriteriawerecollectedtoassesshigh-levelUASDAAsystemperformance.CompliancewithscenarioevaluationcriteriaisconsideredindicativeofacceptableUASDAAperformance.FT4JADEMscenarioevaluationcriteriacomplianceexceeded90%andnon-complianceswereassessedtobeeithertestartifactsortheresultsofa(sincecorrected)softwarecodingerror.Whilenoacceptancethresholdforevaluationcriteriacompliancewasestablished,theobservedhighlevelofcomplianceprovidesanecdotalevidenceforrequirementsvalidationofthealertingparameters,WCRguidancelogicandTCASinteroperabilityconceptimplementedwithinJADEMforFT4.Finally,itisimportanttoreiteratethattheresultspresentedhereinarebasedonalimitedsetofscriptedscenariosexecutedinatightlycontrolledflighttestenvironmentwithwell-rehearsedandlimitingpilotproceduresforuseofthesystemundertest;theresults,conclusionsandrecommendationsincludedinthisdocumentprovidekeyinsightsandanecdotalevidence,butrepresentasmallfractionoftheanalysesnecessarytofullyvalidatetheSC-228UASDAAMOPSrequirements.