list of figures - eurocontrol · no part should be reproduced in any form without the agency’s...

EUROPEAN ORGANISATIONFOR THE SAFETY OF AIR NAVIGATION

EUROCONTROL EXPERIMENTAL CENTRE

SAPPHIRE RAIM VALIDATION

EEC Note No.15/01

Project GNS-Z-E2

Issued: June 2001

The information contained in this document is the property of the EUROCONTROL Agency andno part should be reproduced in any form without the Agency’s permission.

The views expressed herein do not necessarily reflect the official views or policy of the Agency.

EUROCONTROL

REPORT DOCUMENTATION PAGE

Reference:EEC Note No.15/01

Security Classification:Unclassified

Originator:IfEN, University FAF Munich

Originator (Corporate Author) Name/Location:Institute of Geodesy and NavigationUniversity FAF MunichD-85577 NeubibergGERMANYTelephone : + 49 (0)89 6004-3425

Sponsor:EEC

Sponsor (Contract Authority) Name/Location:EUROCONTROL Experimental CentreCentre de Bois des BordesBP1591222 Brétigny-sur-Orge CEDEXFRANCETelephone : +33 (0)1 69 88 75 00

TITLE:SAPPHIRE

- RAIM VALIDATION -

AuthorsTheodor Zink

Jürgen Pielmeier

Date06/2000

Pagesxii+31

Figures12

Tables18

Appendix-

References12

TaskSpecification

-

ProjectGNS-Z-E2

Task No. Sponsor

-

Period03/1999 – 05/2001

Distribution Statement:(a) Controlled by: EEC – Head of the EATMP GNSS Programme(b) Special Limitations: None

Descriptors (keywords):

SAPPHIRE, DUAU, RAIM, Detection, Identification, Validation, Statistics, Sample, Comparison, SatelliteNavigation

Abstract:

This report presents the results of the comparison of the RAIM results of the SAPPHIRE DUAU softwarewith the RAIM results of the `RAIM Validation` software, which was independently developed by theInstitute of Geodesy and Navigation (IfEN) at the University FAF Munich and which is also denoted as`RAIM Validation Software Tool` (RV). The objective of this project was thereby to validate two selectedRAIM algorithms, which are implemented in the SAPPHIRE DUAU, with this independently developed`RAIM Validation` software. In general, the results of this SAPPHIRE DUAU RAIM algorithm validationwere based on the comparison between statistical SAPPHIRE and RV RAIM results. In the case of out-of-normal-behaviour of these statistical RAIM results, the RAIM results of the SAPPHIRE DUAUsoftware and of the `RAIM Validation` software were compared on sample-by-sample basis.

SAPPHIRE – RAIM VALIDATIONEUROCONTROL

v

FOREWORD

This report presents the results, which were provided by the Institute of Geodesy and Navigation,University FAF Munich (IfEN), of the RAIM availability, detection and identification performancecomparison between the EUROCONTROL ‘SAPPHIRE’ and the IfEN ‘RAIM Validation’ (RV) tool. Theobjective of this project was to validate two selected SAPPHIRE Database Update and Access Unit(DUAU) RAIM algorithms. This independent validation represented a major key point for SAPPHIRE tobe used as intended: As a system giving statistical evidence about the performance of satellitenavigation systems in the airborne environment so that they may be approved for operational use.

This programme was initiated in 1998, when the SAPPHIRE DUAU had received the EUROCONTROLacceptance. The objective then was to validate this acceptance of the SAPPHIRE RAIM algorithms byindependent means.

The programme started spring 1999 with establishing a detailed User Requirements Document.

The implementation of a life-cycle approach and adopting a general `GNSS Studio` softwareenvironment for developing the special RAIM Validation `processes` formed the baseline of theactivities to independently implement developed RAIM algorithms.

The main part of the project was the comparison of about 46 SAPPHIRE flights, which were processedwith the SAPPHIRE DUAU software and with the `RAIM Validation` software, as well as their detailedevaluation on the different required levels.

We would like to thank MM Theodor Zink and Jürgen Pielmeier of IfEN for their effort to conduct thestudy and provide EUROCONTROL with this report, Carsten Butzmuelhen of TU Braunsweig for thefruitful discussions, and all our colleagues from the GNSS Programme for their valuable inputs andtheir support to this project. We also would like to express our thanks to all our partners in theSAPPHIRE project for their continuing support.

With respect to the original project report, some editorial changes were necessary to accommodate theEUROCONTROL reporting format.

Aline LHERMITEBernd TIEMEYER

EUROCONTROL Experimental CentreEATMP GNSS Programme


vi


vii

TABLE OF CONTENTS

LIST OF FIGURES ...............................................................................................................................VIII

LIST OF TABLES .................................................................................................................................VIII

TABLES OF ACRONYMS...................................................................................................................... IX

SUMMARY..............................................................................................................................................XI

1. INTRODUCTION.............................................................................................................................. 11.1 General ...................................................................................................................................... 11.2 Overview.................................................................................................................................... 1

2. PROJECT ACTIVITIES.................................................................................................................... 2

3. VALIDATION APPROACH .............................................................................................................. 43.1 RAIM Validation software Test Approach.................................................................................. 43.2 SAPPHIRE Validation Test approach........................................................................................ 4

4. RESULTS......................................................................................................................................... 74.1 Background................................................................................................................................ 74.2 RAIM integrity Parameter Definition .......................................................................................... 84.3 Step by Step Verification............................................................................................................ 9

4.3.1 Comparison of Position ...................................................................................................... 94.3.2 Comparison of RAIM Algorithm Statistical Results .......................................................... 17

4.4 Output Analysis........................................................................................................................ 204.4.1 Evaluation of the normal flights (without detected incorrect measurements)................... 204.4.2 Evaluation of normal flight with detected incorrect measurements .................................. 254.4.3 Evaluation of error flights.................................................................................................. 27

5. CONCLUSIONS............................................................................................................................. 28

6. ACKNOWLEDGEMENTS.............................................................................................................. 29

7. REFERENCES............................................................................................................................... 30


viii

LIST OF FIGURES

Figure 4-1: Horizontal SAPPHIRE/RV Position Deviation from Onboard Position (FK_32)............ 9Figure 4-2: Horizontal SAPPHIRE/RV Position Deviation from Onboard Position (FK_32).......... 10Figure 4-3: Horizontal Position Difference between RV and SAPPHIRE Position (FK_32) .......... 11Figure 4-4: Horizontal Position Difference between RV SAPPHIRE Position (FK_32) .................. 11Figure 4-5: Horizontal SAPPHIRE/RV Position Deviation from Onboard Position

(FK_2000003) ...................................................................................................................... 12Figure 4-6: Horizontal SAPPHIRE/RV Position Deviation from Onboard Position

(FK_2000003) ...................................................................................................................... 13Figure 4-7: Horizontal RV Position Deviation from Onboard Position (FK_2000003) ................... 14Figure 4-8: Horizontal SAPPHIRE Position Deviation from Onboard Position

(FK_2000003) ...................................................................................................................... 14Figure 4-9 Horizontal Position Difference between RV and SAPPHIRE Position

(FK_2000003) ...................................................................................................................... 15Figure 4-10 RV and SAPPHIRE Satellite visibility for (FK_29, NPA flight phase) .......................... 23Figure 4-11 RV HPL using the Brenner Algorithm (FK_29, NPA flight phase) ............................... 24Figure 4-12 SAPPHIRE PMD using the Brenner algorithm (FK_29, NPA flight phase) ................. 24

LIST OF TABLES

Table 4-1: SAPPHIRE/RV RAIM Results Conversion Matrix .............................................................. 7Table 4-2: RAIM Integrity and Related Parameters............................................................................. 8Table 4-3: SAPPHIRE Positioning Module compared with RV Positioning/RAIM FDI

Results (FK_2000003) ........................................................................................................ 16Table 4-4: FK_2000003 RAIM Differences For Departure Phase Of Flight (273 Samples) ............ 18Table 4-5: FK_2000003 RAIM Differences For En Route Phase Of Flight (1230 Samples) ........... 18Table 4-6: FK_2000003 RAIM Differences For Terminal Phase Of Flight (433 Samples) .............. 19Table 4-7: FK_2000003 RAIM Difference For Initial Approach Phase Of Flight (129

Samples) ............................................................................................................................. 19Table 4-8: FK_2000003 RAIM Differences For Final Approach (NPA) Phase Of Flight (182

Samples) ............................................................................................................................. 19Table 4-9: FK_91 Statistical RAIM Result Differences For Departure Phase Of Flight (362

Samples) ............................................................................................................................. 20Table 4-10: FK_29 RAIM Availability For Final Approach (NPA) (174 Samples) ........................... 21Table 4-11: FK_49 RAIM Availability For Departure (493 Samples) ................................................ 21Table 4-12: FK_79 RAIM Availability For Departure (603 Samples) ............................................... 21Table 4-13: FK_35 RAIM Availability For Initial Approach (172 Samples) ...................................... 22Table 4-14: FK_77 RAIM Availability For Departure (494 Samples) ................................................ 22Table 4-15: FK_48 RAIM Availability For Final Approach (NPA) (178 Samples) ............................ 23Table 4-16: FK_76 RAIM Differences For En Route (23374 Samples)............................................. 25Table 4-17: FK_76 SAPPHIRE/RV RAIM Results On Sample Level (FDI Algorithm Of

Brenner) .............................................................................................................................. 26Table 4-18: FK_76 SAPPHIRE/RV RAIM Results On Sample Level (FD Algorithm Of

Sturza/Brown Combined With FI Algorithm Of Sturza) .................................................. 26


ix

TABLES OF ACRONYMS

CFAR Constant False Alarm RateDD Design DocumentDO DocumentDUAU Database Update and Access UnitEC EUROCONTROLEEC EUROCONTROL Experimental CentreESA European Space AgencyFAF Federal Armed ForcesFD Fault DetectionFDI Fault Detection and IdentificationFI Fault IdentificationFR Final ReportGNSS Global Navigation Satellite SystemGPS Global Positioning SystemICD Interface Control DocumentID IdentifierIfEN Institut für Erdmessung und NavigationNPA Non-Precision ApproachPRN Pseudo Random NoiseRAIM Receiver Autonomous Integrity MonitoringRTCA Radio Technical Commission for AeronauticsRV RAIM Validation Software ToolSAPPHIRE Satellite and Aircraft Database Project for System Integrity ResearchSNA Satellite Navigation ApplicationsSRD Software Requirements DocumentSTP Software Test PlanSUM Software User ManualSV Space VehicleURD User Requirements Document


x


xi

SUMMARY

A comprehensive database of real measurements is used by the SAPPHIRE Data Update and AccessUnit (DUAU) for analyses yielding statistical statements about the accuracy, integrity, availability andcontinuity of the service of GPS onboard commercial aircraft. The objective of the activities describedin this report was the validation of two selected RAIM algorithms, which are implemented in theSAPPHIRE DUAU, in order to provide statistical evidence of the integrity, availability and continuity ofthe service of GPS.

The note reports on the project activities performed to validate the two selected SAPPHIRE DUAURAIM algorithms. This comprises a description of the test approach for the validation of the tool (RAIMValidation software) to be used in the project, and a description of the project method. In addition,comparison results between the two different implementations of the RAIM algorithms are part of thereport.

A lifecycle approach was adopted for this project in order to achieve a high level of confidence in thedevelopment of the RAIM Validation software. This formal approach subdivided the RAIM Validationsoftware development into different phases where the output of each phase was documented. TheRAIM Validation software was finally tested before it was used for validation purposes.

The RAIM Validation software results were compared with the SAPPHIRE DUAU software results inorder to validate the two selected SAPPHIRE DUAU RAIM algorithms. This comparison of the RAIMValidation software results with the SAPPHIRE DUAU software results was performed at the positionsolution level as well as at the RAIM results level. The RAIM algorithm performance could be evaluatedusing a unique source for position solutions. The ones computed by the SAPPHIRE DUAU software,were used by the RAIM Validation software for the determination of the RAIM results since the specificdata quality check of the SAPPHIRE positioning module (GPSALL) already excluded erroneoussatellite information in the positioning part of this SAPPHIRE DUAU software.

A total of 40 flights with real GPS pseudorange measurements and 6 flights with manipulated GPSpseudorange measurements was processed with both the SAPPHIRE DUAU software and the RAIMValidation software in order to validate the correct functioning of the two selected RAIM algorithmsimplemented in the SAPPHIRE DUAU.

The comparison has clearly validated the SAPPHIRE RAIM algorithms.


xii


1

1. INTRODUCTION

1.1 GENERAL

This report presents the results of the comparison of the SAPPHIRE position and RAIMresults with the independent RAIM Validation software results. This comparison was doneusing the same integrity parameters and the same flights, both part of the SAPPHIREDUAU, as input to the RAIM Validation software. It must be noted that the RAIM Validationsoftware is also denoted as `RAIM Validation Software Tool` (RV).

1.2 OVERVIEW

This document is divided in the following chapters:

• Chapter 2, Project activities. In this chapter the general programme developmentapproach and the detailed activities are described.

• Chapter 3, Test approach. In this chapter the general approach to test the RAIMValidation software itself and the approach to validate the two selected SAPPHIREDUAU RAIM algorithms are described.

• Chapter 4, Results. This chapter contains the numeric results of the comparison ofposition and RAIM results between SAPPHIRE and RV.

• Chapter 5, Conclusions. In this chapter the lessons learned and the final SAPPHIREevaluation will be summarised.


2

2. PROJECT ACTIVITIES

The project was carried out according to the lifecycle approach of the ESA standard [3] forsmall projects. The objective of this very formal approach was to establish a clear and commonunderstanding of the expectations (in form of requirements) and the implementation aspects(in form of ICD, Design Definition) by all team partners.

The basic approach to develop the software was to use an existing general-purposeenvironment and to incorporate the necessary RAIM algorithms. This framework was the‘GNSS Studio’ software, developed by IfEN in 1998.

In detail, the following phases with documents and software deliveries were part of thelifecycle:

• User Requirements Phase: URD• Software Requirements Phase: SRD, ICD• Software Development Phase: DD, SUM• Test Phase: STP, FR

The User Requirements Phase was essential as a starting point to have a commonunderstanding of the objectives of the programme.

The Software Requirements Phase was carried out to show that the intended developmentwas in line with the objectives. At this phase it was also essential to generate an ICD, whichensured that:

• The necessary input data were available and clearly defined.• The required output data were suited to cover the objectives.

The ICD represented the detailed proof of having a common understanding of inputs andoutputs of the programme. The importance of the ICD was reflected by the fact that 7 versionsof the ICD were issued until the final ICD.

The Software Development Phase was the core development part. It included the detaileddesign of the algorithms as well as of the software and the way to verify that theimplementation met the requirements. Therefore, the documents produced in this phase werevery detailed. This phase ended with delivery of the software including the SUM describing thesoftware for the users.

Finally the Test Phase verified the implementation of the RAIM Validation software and wasused to compare the RAIM Validation software with the SAPPHIRE DUAU software.


3

The lessons learned for these project activities were:

• An open software environment allowed incorporating new options into the softwarewithout changing the old parts. This was a vital function necessary to cover especiallythe evaluation process at the end of the project.

• The most important point to achieve a common understanding was the ICD. Therequirement to clearly define all input and output variables required thinking aboutevery variable and its content. The ICD represented the most important document forthe development.

• The SRD should be merged together with the DD, because the SRD is generally verysimilar to the URD. By merging both documents development time can be saved.

Finally it turned out, that even for such a small project it was very useful to follow projectlifecycle guidelines. On the other side the generation of the documents was very timeconsuming. However, this is often the only way to clarify certain issues by clear definitions inthe documents. Due to the large amount of time necessary for generating the documents andthe usually limited time available for a programme, it is necessary to carefully tailor thedocumentation and the project phases to the real needs of the project.


3. VALIDATION APPROACH

The validation approach was twofold (see Figure 3-1):• First it was necessary to verify the RAIM Validation software itself. This approach is

described in summary in chapter 3.1• Then this verified RAIM Validation software was used to validate the two selected

SAPPHIRE DUAU RAIM algorithms. This was achieved by comparing the RAIMValidation results with the SAPPHIRE DUAU results. This final report covers in detailthe results obtained by this comparison. The detailed approach is described in chapter3.2.

RTCA DO208

Test Specification 1 Test Specification 2

Test Specification 3 Test Specification 4

SAPPHIRE DUAU

EUROCONTROL

RAIM Validation

Comparison of Results

Comparison

Results

Figure 3-1: Validation Approach

4

3.1 RAIM VALIDATION SOFTWARE TEST APPROACH

To test the RAIM Validation software itself, two basic approaches were used:

• Comparison of positioning results with other positioning software.• Proof of RAIM algorithms using RTCA/DO-208 tests.

The objectives of the RTCA/DO-208 tests were the evaluation of the RAIM Validation softwareperformance with respect to the false alarm probability, the missed detection probability as wellas the time to alarm.GPS pseudorange measurements were generated by adding GPS pseudorange measurementerrors (obtained by the error models specified by the RTCA Special Committee 159 documentRTCA/DO-208 [5]) to the geometrical distances between the 24 specified geographic locationsand the GPS satellite positions. The RAIM Validation software performance was subsequentlyevaluated on a. large number of snapshot random trials (Monte Carlo). This was performed bythe aid of these simulated GPS pseudorange measurements, which were the basis for thedecision whether or not this RAIM Validation software has passed the RTCA DO208performance test cases.

3.2 SAPPHIRE VALIDATION TEST APPROACH

The general approach used to test the two selected SAPPHIRE DUAU RAIM algorithms was astep-by-step validation approach. This means that the results between the SAPPHIRE DUAUsoftware and the ‘RAIM Validation’ software were compared on different levels:

• Comparison of the satellite positions.• Comparison of the satellites used for the design matrix.• Comparison of the estimated user positions; the given onboard position represented

the reference position to be compared to the SAPPHIRE (GPSALL) position and the‘RAIM Validation’ position. Additionally the SAPPHIRE (GPSALL) position was alsodirectly compared with the ‘RAIM Validation’ position, with the ‘RAIM Validation’position used as reference position.


• Comparison of the RAIM statistical results (RAIM availability andDetection/Identification) of the flights per phase. Here the reference values were theSAPPHIRE statistical RAIM results per flight phase; the ‘RAIM Validation’ RAIM resultsobtained from SAPPHIRE positions were compared with the SAPPHIRE RAIM results;

• Comparison of the RAIM results on a sample per sample base if necessary. If thestatistical comparison showed obvious differences, a detailed comparison of values ona sample per sample base has enabled in depth analysis of the RAIM algorithms.

He next figure shows the relations between the different algorithmic parts of the comparison ofresults approach.

Error Budget Model

Clock/Ephemeris 2.3 Ionosphere 7.0 Troposphere 2.0 Multipath 1.5 Receiver Noise 0.6

Signal Failures

Single Event Failure! Distribution unknown!

Constellation

Fault-Case Detection

NSE d b AL Bias

a c Statistic

Feared Events

Ho H1

Signal Quality Monitoring

Pseudorange

Positioning GWG Geometry/Weight Matrix

GT W G

RAIM Availability

XPL AL

Pseudorange Domain Pseudorange to Position Connection Position Domain PR Residual Error

Integrity Parameter

P(missed detection) 10-3

P(false alarm) 10-5

Time to Alarm 30 s Alarm Limit 550m

Figure 3-2: Positioning/RAIM logic

5

Both RV and SAPPHIRE calculate first the Design Matrix (which connects measurements withthe position), taking the actual geometry of the GPS constellation and the GPS error budgetmodel into account.

This ’Design Matrix’ (Geometry/Weight matrix) was the major input for positioning and theRAIM availability calculation.The RAIM availability calculation needs additionally to the Design Matrix also the RAIM integrityparameters as input.

The positioning (LSQ) calculation additionally uses the pseudorange measurements.Finally the detection and identification of faulty measurements (so called feared events), takesplace using the pseudorange residuals from the positioning, if RAIM was available.The big difference here between RV and SAPPHIRE was the Signal Quality Monitoring of thepseudorange measurements. RV has no Signal Quality Monitoring implemented. SAPPHIREhowever uses this pseudorange monitoring for early detection of measurement failures. Due tothis Signal Quality Monitoring SAPPHIRE is able to exclude satellites/measurements from theDesign Matrix. Therefore different positioning results can occur and also a different RAIMbehaviour was possible.


6

To overcome this difference, all RAIM computations (availability/detection/identification) wereusing the SAPPHIRE Design Matrix (and hence SAPPHIRE satellite position, pseudorangeresiduals and constellation of satellites).

In general the comparison was divided into two main parts:

a) Comparison of positioning results, using onboard, RV and SAPPHIRE position.b) Comparison of RV and SAPPHIRE RAIM results, using the SAPPHIRE constellation of

satellites (position and number) and pseudorange residuals from a) as input.


7

4. RESULTS

4.1 BACKGROUND

The two RAIM algorithms to be compared between the SAPPHIRE DUAU software and theRAIM Validation software were:

1. The CFAR RAIM FD algorithm of Sturza/Brown [11] (fixed threshold RAIM algorithmonly) in combination with the CFAR RAIM FI algorithm of Sturza [10].

2. The CFAR RAIM FDI algorithm of Brenner [1].

The RAIM results of the RAIM Validation software with respect to a RAIM algorithm weresplit up into 6 possible result types (see also

Table 4-1 and ICD [4]):

• 0: Detection of a faulty satellite was impossible due to insufficient number of visiblesatellites;

• 1: Fault detection of the RAIM algorithm was not reliable due to bad geometry;• 2: Fault detection of the RAIM algorithm was reliable, but no detection of a faulty

satellite had occurred (everything was ok);• 3: Fault detection of the RAIM algorithm was reliable, a detection of a faulty satellite

had occurred and the faulty satellite was identified;• 4: Fault detection of the RAIM algorithm was reliable and a detection of a faulty

satellite had occurred, but the identification of the faulty satellite had not occurreddue to bad geometry;

• 5: Fault detection of the RAIM algorithm was reliable and a detection of a faultysatellite had occurred, but the identification of the faulty satellite was impossible dueto insufficient number of visible satellites.

The cases 0, 1 and 2 determine the RAIM availability performance.The cases 3, 4 and 5 determine the RAIM detection/identification performance.

The next table describes the mapping from the SAPPHIRE RAIM results flags to the RVRAIM results values.

SAPPHIRE RAIM Result FlagsRV RAIM ResultFlag Number DETIMP DETNREL EWS IDOCC IDNOCC IDIMP

0 (FD impossible, i.e. number ofused SVs < 5) x

1 (FD not reliable, i.e. badgeometry) x

2 (Everything withinspecification) x

3 (Detection occurred andidentification occurred) x

4 (Detection occurred andidentification not occurred, i.e.bad geometry)

x

5 (Detection occurred and FIimpossible, i.e. number ofused SVs < 6)

x

Table 4-1: SAPPHIRE/RV RAIM Results Conversion Matrix


8

The SAPPHIRE RAIM result flags INT_D, INT_I, DETREL as well as DETOCC wereincorporated into the RV RAIM result flag number and were not explicitly considered by theRAIM Validation software.

To validate all aspects (availability, detection and identification) of the RAIM algorithmsoptimally, two types of flights were used:

• 40 Normal Flights (Flight data as collected for SAPPHIRE), mainly to compare theintegrity monitoring availability performance.

• 6 Error Flights (Normal Flights with artificial errors incorporated), mainly to comparethe detection and identification performance.

In order to perform the validation of the two selected SAPPHIRE DUAU RAIM algorithms, theresults for the following phases of flight of these normal and error flights were compared:

• Departure (Phase ID: 2)• En Route (Phase ID: 3)• Terminal (Phase ID: 4)• Initial Approach (Phase ID: 5)• Final Approach, NPA only (Phase ID: 6)

The ‘Ground’ flight phase was not part of the validation itself.

4.2 RAIM INTEGRITY PARAMETER DEFINITION

The integrity and RAIM related parameters, which were used to validate the SAPPHIRE DUAURAIM algorithms, are given in Table 4-2.

Parameter Departure En Route Terminal InitialApproach

Final Approach(NPA)

Horizontal Alert Limit [m] 555 1850 1850 555 555False Alarm Rate [h-1] 10-5 10-5 10-5 10-5 10-5

Time to Alarm [s] 1 1 1 1 1Missed Detection

Probability 10-3 10-3 10-3 10-3 10-3

Standard Deviation OfPseudorange Noise [m] 33.3 33.3 33.3 33.3 33.3

Correlation Time OfPseudorange Noise [s] 1 1 1 1 1

Elevation Mask Angle [°] 0 0 0 0 0

Table 4-2: RAIM Integrity and Related Parameters

The RAIM algorithms were working on real flight data. As these flight data were recordedbefore GPS SA was turned off, the pseudorange error budget assumed was 33.3 meter.


9

4.3 STEP BY STEP VERIFICATION

4.3.1 Comparison of Position

Before the SAPPHIRE/RV RAIM results were compared, the differences of the position resultsbetween the three available position solutions (i.e. between the SAPPHIRE position solution,the RV position solution and the onboard position solution) were analysed as the first part ofthe step-by-step validation approach. The results of the position difference analysis of oneflight (once with additional errors on the pseudoranges and once without artificial errors) will begiven in the following. The used flight was the normal flight FK_32. The same flight with errorsincorporated on the pseudoranges was the error flight FK_2000003.

The following pictures show the horizontal position differences for the following three caseswhere either the horizontal onboard position, which was determined by the onboard positionestimated directly by the receiver, or the horizontal RV position was the horizontal referenceposition:

1. The horizontal position calculated by the SAPPHIRE GPSALL module was subtractedfrom the horizontal onboard position.

2. The horizontal position calculated by the RV positioning module was subtracted fromthe horizontal onboard position.

3. The horizontal position calculated by the SAPPHIRE GPSALL module was subtractedfrom the horizontal RV position.

It must be noted that the horizontal SAPPHIRE/RV position deviations from the horizontalonboard position will be shown in the same graphs. This enables a direct comparison of theSAPPHIRE and RV position estimation algorithm performance.

0 500 1000 1500 2000 2500 300025

30

35

40

45

50

FK32 horizontal position comparison

Hor

izon

tal d

iffer

ence

from

onb

oard

pos

ition

[m]

Sample ID

ONBOARD-RV_DIF ONBOARD-SAP_DIF

Figure 4-1: Horizontal SAPPHIRE/RV Position Deviation from Onboard Position (FK_32)


10

340 342 344 346 348 35025

30

35

40

45

50

FK32 horizontal position comparison

Hor

izon

tal d

iffer

ence

from

onb

oard

pos

ition

[m]

Sample ID


Figure 4-2: Horizontal SAPPHIRE/RV Position Deviation from Onboard Position (FK_32)

As can be seen from Figure 4-1, the horizontal RV (blue line) and SAPPHIRE (red line) positiondeviations from the horizontal onboard position were very similar for the normal flight FK_32.Therefore, the pseudorange residuals, which were used by the SAPPHIRE/RV RAIMalgorithms, were approximately the same in this case.

On the other hand, a more detailed look at the horizontal FK32 SAPPHIRE/RV positiondeviations from the horizontal onboard position showed that there could be significantdifferences at some sample IDs (see Figure 4-2). The horizontal position deviations of the RVand SAPPHIRE solution from the horizontal onboard were very similar at the start and at theend of Figure 4-2. However from sample ID 344 to 347 in Figure 4-2 there was an obviousdifference.

Figure 4-3 shows the direct position difference between RV and SAPPHIRE for the samesample ID’s as in Figure 4-2 and additionally the number of satellites used by the RV andSAPPHIRE-positioning module.

The big horizontal position difference, which was reflected by the significant differences shownin Figure 4-2, between the horizontal RV position and the horizontal SAPPHIRE positionregarding the normal flight FK_32 was caused by the rising of a new satellite above the virtualhorizon (elevation mask angle) which was acquired by the receiver (cf. Figure 4-3). TheSAPPHIRE-positioning module has used this new satellite three epochs (i.e. three sample IDs)later in order to perform signal quality checks concerning the new satellite within these threeepochs, while the RV positioning module used this new satellite immediately. Note that anepoch was represented by the corresponding sample ID. Therefore, for these three epochsthere were obvious horizontal position differences between the horizontal RV positions and thehorizontal SAPPHIRE positions up to 20 meters.


11

340 341 342 343 344 345 346 347 348 349 350

0

5

10

15

20

SAPPHIRE: 7 SVRAIM VAL: 8 SV

8 Satellites7 Satellites

RV

- SAP

PHIR

E Po

sitio

n D

ista

nce

SAMPLE ID

FK_32

Figure 4-3: Horizontal Position Difference between RV and SAPPHIRE Position (FK_32)

To look for all such ‘Satellite-Rising-Above-Elevation-Mask-Angle’ conditions and for thegeneral horizontal position difference between the horizontal RV position and the horizontalSAPPHIRE position, the same comparison (horizontal RV position minus horizontal SAPPHIREposition) was done for the complete normal flight FK_32 as shown in Figure 4-4.

0 500 1000 1500 2000 2500 3000

0

5

10

15

20

RV

- SAP

PHIR

E po

sitio

n di

ffere

nce

[m]

Sample ID

FK_32

Figure 4-4: Horizontal Position Difference between RV SAPPHIRE Position (FK_32)


12

Every time a new satellite rose, there was an obvious horizontal position difference betweenthe horizontal RV and SAPPHIRE position for three epochs (i.e. for three sample IDs) withregard to the normal flight FK_32 as can be seen from Figure 4-4. This condition occurred fourtimes in this normal flight FK_32.

The horizontal position difference between the horizontal RV position and the horizontalSAPPHIRE position for the normal flight FK_32 was up to 50 cm nominally, i.e. withoutconsidering the ‘Satellite-Rising-Above-Elevation-Mask-Angle’ conditions.

What was very interesting was the increasing position accuracy during the normal flight FK_32.For the nominal condition (i.e. without considering the ‘Satellite-Rising-Above-Elevation-Mask-Angle’ conditions), the horizontal position difference between the horizontal RV position and thehorizontal SAPPHIRE position decreased from 0.5 m down to 0.05 m in this case. For the‘Satellite-Rising-Above-Elevation-Mask-Angle’ conditions with respect to the normal flightFK_32, the horizontal position difference between the horizontal RV position and the horizontalSAPPHIRE position decreased from 18.5 m (first occurrence of such a ‘Satellite-Rising-Above-Elevation-Mask-Angle’ condition) down to 1.5 m (fourth occurrence of such a ‘Satellite-Rising-Above-Elevation-Mask-Angle’ condition).

The ‘Satellite-Rising-Above-Elevation-Mask-Angle’ was a clear indicator that through theSAPPHIRE ‘signal quality monitoring’ approach, different position solutions can arise (RV hasno ‘signal quality monitoring’ on the pseudoranges). Therefore to study more ‘signal qualitymonitoring’ related effects, also the error flight FK_2000003 was studied in detail.

Error flight FK_2000003 was the same flight as normal flight FK_32, but with artificially inducedpseudorange measurement errors (e.g. steps, ramps, oscillations, etc.), clearly visible in theFigure 4-5.

0 500 1000 1500 2000 2500 3000

0

500

1000

1500

2000

2500

FK_2000003: Comparison of horizonal position difference

Hor

izon

tal d

iffer

ence

from

onb

oard

pos

ition

[m]

Sample ID


Figure 4-5: Horizontal SAPPHIRE/RV Position Deviation from Onboard Position(FK_2000003)


13

580 600 620 640 660 680 700 7200

25

50

75

100

125

150

175

200

225

250

275

300FK_2000003: Comparison of horizonal position difference

Hor

izon

tal d

iffer

ence

from

onb

oard

pos

ition

[m]

Sample ID


Figure 4-6: Horizontal SAPPHIRE/RV Position Deviation from Onboard Position(FK_2000003)

From Figure 4-5 it can be seen that the horizontal RV (blue line) and SAPPHIRE (red line)position deviations from the horizontal onboard position were generally very similar, reflectingthe different pseudorange error types. Only the SAPPHIRE/RV position deviations from theonboard position regarding the sample IDs from 600 to 720 have shown a clear differentbehaviour (see Figure 4-6).

In order to obtain a more detailed look at the SAPPHIRE/RV position deviations from theonboard position, the horizontal RV position deviation from the onboard position (Figure 4-7)and the horizontal SAPPHIRE position deviation from the onboard position (Figure 4-8) wereseparately plotted to see the clear differences.


14

600 610 620 630 640 650 660 670 680 690 700 710 720

0

50

100

150

200

250

300

Onb

oard

min

us R

V Po

sitio

n [m

]

Sample ID

RV_POS_DIF

Figure 4-7: Horizontal RV Position Deviation from Onboard Position (FK_2000003)

600 610 620 630 640 650 660 670 680 690 700 710 720

0

50

100

150

200

250

300

Onb

oard

min

us S

APPH

IRE

Posi

tion

[m]

Sample ID

SAP_POS_DIF

Figure 4-8: Horizontal SAPPHIRE Position Deviation from Onboard Position(FK_2000003)

In Figure 4-7 and 4-8 an oscillation error type (one second, there is an error on thepseudorange; in the next second there is no error on the pseudorange and so on) withincreasing amplitude (pseudorange) was induced into the measurements.


15

From Figure 4-8 it can be seen, that the SAPPHIRE ‘signal quality monitoring’ has excludedthe faulty pseudorange (beyond a certain threshold), here the SV PRN number 24 (‘Raw DataCheck not OK’), from the position solution. Therefore after sample ID 635, the SAPPHIREposition again was smooth and correct.

From Figure 4-7 it can be seen, that the induced pseudorange error completely haspropagated into the RV position solution. This was due the fact that, the RV has no ‘signalquality monitoring’ checks of the pseudoranges implemented.

Due to the fact that the RV used all visible satellites (including the corrupted satellite with PRNnumber 24) for the determination of the RV position, while the SAPPHIRE positioning modulehad excluded this faulty satellite, the horizontal position difference between the RV and theSAPPHIRE position from sample 640 to 710 is obvious and reasonable (see also Figure 4-9).

In Figure 4-9 an additional effect of the SAPPHIRE ‘signal quality monitoring can be seen. Atsample 651, the error was removed from PRN 24. However the SAPPHIRE uses two additionalseconds to recheck the excluded SV. And then in the next epoch (sample 653), the PRN 24 isagain used for the position computation. That meant even when there was no error present atsample 653, the positions have an obvious difference for two additional seconds (SAPPHIRE:7 satellites, RV: 8 satellites) at sample 651 and 652. Obviously this ‘Check-excluded-SV-before-reuse’ was implemented in SAPPHIRE to smoothly exclude oscillation error types.

638 640 642 644 646 648 650 652 654 656 658 660

0

10

20

30

40

50

60

70

80

SAPPHIRE: 7 SV (640-652)RAIM VAL: always 8 SVR

V m

inus

SAP

PHIR

E Po

sitio

n [m

]

Sample ID

RV minus SAPPHIRE

Figure 4-9 Horizontal Position Difference between RV and SAPPHIRE Position(FK_2000003)

This ‘signal quality monitoring’ capability of SAPPHIRE with

• 3 epochs of signal checking before a new satellite is used• Exclusion of faulty SV capability before positioning!• 2 epoch signal check before an excluded satellite is reused

has to be taken carefully into account when comparing the RAIM results.


16

The next table clearly shows what could happen, if RAIM results from RV (using RV position)and SAPPHIRE (using SAPPHIRE) are compared.

In Table 4-3 different RAIM results can be seen due to the ‘signal quality monitoring’ capabilityof SAPPHIRE:

• The SAPPHIRE ‘signal quality monitoring’ has excluded the PRN 24 permanentlyduring the oscillation errors. Therefore the SAPPHIRE RAIM has detected/excludednothing, because the faulty SV was already excluded.

• The RV has used all satellites (including the faulty PRN 24) in the positioning. Then theRAIM algorithms have detected and excluded the faulty PRN 24.

Despite the fact that the RAIM results were different here, both SAPPHIRE and RV behavecorrect (the SV PRN 24 excluded by SAPPHIRE using the ‘signal quality monitor’ was alsoexcluded by the RV RAIM detection/identification). Only the ‘place’ where the error wasdetected and then excluded was different.

SAPPHIRE Positioning Module Results RV Positioning/RAIM FDI Results

SampleID

Number OfSatellitesUsed For

Positioning

Number OfSatellites

Excluded BySAPPHIREPositioning

Module

PRN NumberOf SV

Excluded BySAPPHIREPositioning

Module

Number OfSatellitesUsed For

Positioning

PRN NumberOf Faulty SVDetected And

Identifiedby Brenner

RAIMAlgorithm

PRN NumberOf Faulty SVDetected And

Identifiedby Sturza/

Brown RAIMAlgorithm

704 7 1 24 8 24 24705 7 1 24 8 - -706 7 1 24 8 24 24707 7 1 24 8 - -708 7 1 24 8 24 24709 7 1 24 8 - -710 7 1 24 8 24 24711 7 1 24 8 - -712 7 1 24 8 - -713 8 0 - 8 - -

Table 4-3: SAPPHIRE Positioning Module compared with RV Positioning/RAIM FDIResults (FK_2000003)

Therefore in the next sections, where the RAIM performances were compared, only theSAPPHIRE position was used as input to the RV/SAPPHIRE RAIM algorithms, to avoiddifferences based on the SAPPHIRE ‘signal quality monitoring’ capability.

As a consequence, from now on only the satellites used by the SAPPHIRE algorithmsand their SAPPHIRE-provided position will be used for positioning as input for thecomparison of the RAIM results!


17

4.3.2 Comparison of RAIM Algorithm Statistical Results

The RAIM results of the RAIM Validation software were compared with the RAIM results of theSAPPHIRE RAIM algorithms in order to validate these SAPPHIRE DUAU RAIM algorithms.

Two different RAIM algorithms were used for comparison/validation:

• CFAR RAIM FDI algorithm of Brenner [1]• CFAR RAIM FD algorithm of Sturza/Brown [11], i.e. the fixed threshold RAIM algorithm

only, in combination with CFAR RAIM FI algorithm of Sturza [10]

This comparison of the SAPPHIRE/RV RAIM results was performed for every phase of flight ofthe considered:

• 40 normal flights (mainly measuring the RAIM availability performance)• 6 error flights (mainly measuring the RAIM detection/identification performance)

Generally the comparison was based on a statistical base. Only if a significant difference of theresults appeared on the statistical level, that could not be explained, detailed analysis of thisdifference was necessary (see chapter 4.4).

The results of this statistical comparison of the SAPPHIRE/RV RAIM results were presented inthe following paragraphs.

The comparison of the 40 normal flights yielded the following results (All percentages valuesgiven represent absolute percentage values):

a) 31 normal flights showed identical or nearly identical behaviour of the SAPPHIRE/RVRAIM algorithms. Usually the RAIM availability results were identical, rising up to 2%difference (e.g. RV RAIM availability: 98%, SAPPHIRE RAIM availability: 100%) in rarecases. However, these rare events could be easily explained with satellite/usergeometry yielding RAIM FDI availability parameters, which were close to their largestadmissible values. Some of these rare events occurred, for instance, when thehorizontal protection level of the CFAR RAIM FDI algorithm of Brenner [1] was close tothe horizontal alarm limit and/or when the detection geometry parameter of the CFARRAIM FD algorithm of Sturza/Brown [11] was close to the ceiling value for thisdetection geometry parameter.

b) 8 normal flights showed evident differences (up to 36% for a phase of flight, e.g. RVRAIM availability: 0%, SAPPHIRE RAIM availability: 35.9%) in the SAPPHIRE/RVRAIM availability. Usually such a difference occurred only in one of the 5 phases of aflight (In general, a SAPPHIRE/RV RAIM availability result difference only appeared inone of the more demanding flight phases Departure, Initial Approach, and FinalApproach, because there the integrity requirements are often at the RAIM availabilityboundary). So even if one phase of flight has shown an evident difference in theSAPPHIRE/RV RAIM availability performance, the SAPPHIRE/RV RAIM availabilityresults for the other phases of the same flight behaved like type a) results, with no realdifference. These flights with significant SAPPHIRE/RV RAIM availability differenceswere further evaluated in the output analysis section (see chapter 4.4).

c) Normal flight FK_76 was the only normal flight with detected incorrect measurements.This case was also considered in detail as described in the output analysis section(see chapter 4.4).


18

The comparison of the 6 error flights yielded the following results:

The SAPPHIRE/RV RAIM availability results were identical or very similar (differencebelow 2% of absolute RAIM availability). In addition to this, these SAPPHIRE/RV RAIMresults also have shown that the detection and identification of faulty satellites inSAPPHIRE/RV had approximately the same RAIM FDI performance, even if manyartificial errors were included in these error flights. In general, the detection andidentification parts of the SAPPHIRE/RV RAIM algorithms were surprisingly similar in spiteof the different implementations of these SAPPHIRE/RV RAIM algorithms. Moreover, theRV RAIM algorithms have shown a marginal more conservative RAIM availabilitybehaviour.

The following tables show the statistical differences between the statistical RV and SAPPHIRERAIM results for all RAIM result flag numbers. Every phase of flight of the error flightFK_2000003 is assigned one table. A statistical difference di for the ith RV RAIM result flagnumber was given by di = ri − si. ri was the ith RV RAIM result flag number. si was the ithSAPPHIRE RAIM result flag number. Therefore the SAPPHIRE RAIM results in the followingtables are absolute values. The RV RAIM results a relative value (difference) to the SAPPHIRE(absolute) values.

FDI Algorithm Of Brenner FD Algorithm Of Sturza/Brownwith FI Algorithm Of Sturza

RV RAIM Result FlagNumber/Description

StatisticalSAPPHIRE

RAIM Results[%]

StatisticalDifferences [%]

of RV RAIMResults

StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

0 (FD impossible � SV < 5) 0.000 0.000 0.000 0.0001 (FD not reliable � bad geometry) 23.443 0.000 23.443 0.0002 (Everything within Specification) 76.557 0.000 76.557 0.0003 (Failure Detect./Ident. Occurred) 0.000 0.000 0.000 0.000

4 (Fail. Detect. Occ./Ident. Not occ.) 0.000 0.000 0.000 0.0005 (Fail. Det. Occ./Ident. Impossible) 0.000 0.000 0.000 0.000

Table 4-4: FK_2000003 RAIM Differences For Departure Phase Of Flight (273 Samples)

Result of Table 4-4: Both algorithms in SAPPHIRE and RV show identical results!



StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults



Table 4-5: FK_2000003 RAIM Differences For En Route Phase Of Flight (1230 Samples)

Result of Table 4-5: SAPPHIRE and RV show identical results! The algorithmic performancebetween Brenner/Brown is slightly different.


19



StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

0 (FD impossible � SV < 5) 0.000 0.000 0.000 0.0001 (FD not reliable � bad geometry) 0.000 0.000 0.000 0.0002 (Everything within Specification) 18.014 0.000 18.245 0.0003 (Failure Detect./Ident. Occurred) 9.700 -1.848 81.755 0.000

4 (Fail. Detect. Occ./Ident. Not occ.) 72.286 1.848 0.000 0.0005 (Fail. Det. Occ./Ident. Impossible) 0.000 0.000 0.000 0.000Table 4-6: FK_2000003 RAIM Differences For Terminal Phase Of Flight (433 Samples)

Result of Table 4-6: SAPPHIRE and RV show identical availability results! The algorithmicidentification performance between for the Brenner is slightly different (at 8 samples) betweenSAPPHIRE/RV.



StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults



Table 4-7: FK_2000003 RAIM Difference For Initial Approach Phase Of Flight (129Samples)

Result of Table 4-7: SAPPHIRE and RV show identical availability/detection/identificationresults! The identification performance between Brenner/Brown is slightly different due to thedifferent algorithmic approaches (If the Sturza/Brown detects a faulty satellite, the satellite canbe always identified! This is not the case in the Brenner approach.).



StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

0 (FD impossible � SV < 5) 0.000 0.000 0.000 0.0001 (FD not reliable � bad geometry) 0.000 0.549 0.000 0.5492 (Everything within Specification) 41.758 -0.549 39.011 -0.5493 (Failure Detect./Ident. Occurred) 17.582 0.000 60.989 0.000

4 (Fail. Detect. Occ./Ident. Not occ.) 40.659 0.000 0.000 0.0005 (Fail. Det. Occ./Ident. Impossible) 0.000 0.000 0.000 0.000Table 4-8: FK_2000003 RAIM Differences For Final Approach (NPA) Phase Of Flight (182

Samples)

Result of Table 4-8: SAPPHIRE and RV show identical results for detection/identification! Theavailability performance between Brenner/Brown is slightly (only 1 sample) different.The SAPPHIRE/RV RAIM results were identical or nearly identical (absolute statisticaldifferences below 2% for availability/detection/identification) as can be seen from the tables au-dessus.


20

It can be noted that the RV algorithms are slightly more conservative considering the detectionand identification availability.

Generally the SAPPHIRE/RV RAIM algorithms showed a very high level of similarity, despite ofthe fact that they were developed completely independently. All significant differences of theSAPPHIRE/RV RAIM results, which could not be solved by the above statistical view on theseresults, were evaluated on a sample per sample basis, as described in the following, in order toachieve a clearer picture of these differences.

4.4 OUTPUT ANALYSIS

The results of a detailed analysis of the significant SAPPHIRE/RV RAIM result differencesregarding the normal flights and the error flights are presented in the following chapters.

4.4.1 Evaluation of the normal flights (without detected incorrect measurements)

The statistical RAIM results of RV and SAPPHIRE for the normal flights have shown evidentSAPPHIRE/RV differences for availability and are listed in the following tables. No incorrectmeasurements were detected during the processing of these normal flights. Therefore thefollowing table only consider the cases:

• 1: RAIM not available due to bad geometry• 2: RAIM detection available and no detection occurred (everything within specification)

Additionally is must be noted that from the complete flights, always only in one phase of flight asignificant availability difference appeared. Generally four different cases can be noted:

Case 1: Both RAIM algorithm implementations (RV and SAPPHIRE) were internally (betweenthe RAIM algorithm types) consistent.

The following table shows the SAPPHIRE/RV availability differences for the departure phase offlight (362 samples) of the normal flight FK_91 with evident SAPPHIRE/RV RAIM resultdifferences:



StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

0 (FD impossible � SV < 5) 0.000 11.050 0.000 11.0501 (FD not reliable � bad geometry) 100.000 -11.050 100.000 -11.050

Table 4-9: FK_91 Statistical RAIM Result Differences For Departure Phase Of Flight (362Samples)

For FK_91, the RV algorithms were 40 epochs less available compared with SAPPHIRE.


21

Case 2: The next three flights show the same behaviour. The two RV RAIM algorithms showthe same availability results, so they are internally consistent. The SAPPHIRE RAIM availabilityresults were different between the SAPPHIRE algorithms itself and also different to the RVresults. The reason for difference (RV RAIM availability is more conservative) are the hardclearly the Non-Precision-Approach/Departure RAIM requirements, which lead to a boundaryproblem (The Protection Level is most of the time in the range of the Alarm Limit for NPA). Soin fact even very small differences in the Protection Levels between the algorithms andbetween SAPPHIRE/RV can lead to case 1 (Not available) or case 2 (Available).

The next tree tables show the SAPPHIRE/RV availability differences for different phases offlight of different flight with an evident SAPPHIRE/RV difference:

1.



StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

1 (FD not reliable � bad geometry) 2.299 15.517 8.621 9.1952 (Everything within Specification) 97.701 -15.517 91.379 -9.195

Table 4-10: FK_29 RAIM Availability For Final Approach (NPA) (174 Samples)For FK_29, the Brenner algorithm was 27 epochs different between SAPPHIRE/RV. For theSturza/Brown algorithm, 16 epochs were different between SAPPHIRE/RV.

2.



StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults


Table 4-11: FK_49 RAIM Availability For Departure (493 Samples)For FK_49, the Brenner algorithm, 159 epochs were different between SAPPHIRE/RV. For theSturza/Brown algorithm 20 epochs were different between SAPPHIRE/RV.

3.



StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults


Table 4-12: FK_79 RAIM Availability For Departure (603 Samples)For FK_79, the Brenner algorithm was 76 epochs different between SAPPHIRE/RV. For theSturza/Brown algorithm, 25 epochs were different between SAPPHIRE/RV.


22

Case 3: The next two flights again show a similar behaviour:

Here the SAPPHIRE results are internally consistent and also consistent with the RV Brenneralgorithm. Only the RV Sturza/Brown algorithm shows a higher unavailability.

The following two tables show the SAPPHIRE/RV availability differences for different phases offlight of different flights with an evident SAPPHIRE/RV difference:

1.



StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

1 (FD not reliable � bad geometry) 0.000 0.000 0.000 12.2092 (Everything within Specification) 100.000 0.000 100.000 -12.209

Table 4-13: FK_35 RAIM Availability For Initial Approach (172 Samples)

Only the RV Sturza/Brown algorithm shows a higher unavailability during 21 epochs.

2.



StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

1 (FD not reliable � bad geometry) 0.000 0.000 0.000 7.0852 (Everything within Specification) 100.000 0.000 100.000 -7.085

Table 4-14: FK_77 RAIM Availability For Departure (494 Samples)

The RV Sturza/Brown algorithm shows a higher unavailability during 35 epochs.


23

Case 4: Here the RV results are consistent internally and also consistent with the SAPPHIRESturza/Brown algorithm. Only the SAPPHIRE Brenner possesses a higher availability.

The next table shows the SAPPHIRE/RV availability differences for the final approach (NPA)phase of flight (178 samples) of the normal flight FK_48 with an evident SAPPHIRE/RVdifference:



StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

1 (FD not reliable � bad geometry) 64.607 35.393 100.000 0.0002 (Everything within Specification) 35.393 -35.393 0.000 0.000

Table 4-15: FK_48 RAIM Availability For Final Approach (NPA) (178 Samples)

In that case, the SAPPHIRE Brenner possesses a higher availability during 63 epochs.

To visualise the reason for the availability differences of these 6 flights, the protection level(RV) and the PMD (SAPPHIRE of the NPA phase of flight for FK_29 were shown in Figure 4-11 and 4-12. Figure 4-10 shows the satellite visibility for the same sample ID’s.

34450 34460 34470 34480 34490 34500 34510 34520 34530

5

6

7

8

Num

ber o

f visi

ble

sate

llites

dur

ing

NPA

Sample ID

FK_29_NPA_Used_Satellites

Figure 4-10 RV and SAPPHIRE Satellite visibility for (FK_29, NPA flight phase)


24

34450 34460 34470 34480 34490 34500 34510 34520 34530400

425

450

475

500

525

550

575

600

Alarm Limit for NPA (555m)H

oriz

onta

l Pro

tect

ion

Leve

l [m

]

Sample ID

FK_29_NPA HPL RV-Brenner

Figure 4-11 RV HPL using the Brenner Algorithm (FK_29, NPA flight phase)

34450 34460 34470 34480 34490 34500 34510 34520 345301E-8

1E-7

1E-6

1E-5

1E-4

1E-3Missed Detection for NPA (10-3)

Prop

abilit

y of

Mis

sed

Det

ectio

n

Sample ID

FK_29_NPA_SAPPHIRE_PMD

Figure 4-12 SAPPHIRE PMD using the Brenner algorithm (FK_29, NPA flight phase)


25

The RV Brenner RAIM availability shows a clear dependence on the number of satellites (seeFigure 4-12).

As for the RV Brenner only HPL values are available and for the SAPPHIRE Brenner only PMDvalues are available, a direct value comparison is impossible. However together with the AlarmLimit and the Missed Detection (Risk), both figures can be compared.

Until sample ID 34500, 8 satellites were available. Here both, SAPPHIRE and RV are available.The protection level of RV was below the Alarm Limit (555m) and the PMD (Probability ofMissed Detection) of SAPPHIRE was clearly (~ 1.0E-10) below the required PMD of 10-3.

From sample 34500 on, only 7 satellites are available. Here the RV was about 30 meter abovethe Alarm Limit (Integrity not available due to bad geometry), while the SAPPHIRE RAIM PMDis slightly below (~ 1E-04) the allowed 10-3 (Integrity available).

From sample 34514 until 34517, only six satellites are available. The change in HorizontalProtection Level for RV (increase of about 10 meter) and the PMD (increase from 10-4 to 10-3)is only very small. However now also the SAPPHIRE RAIM was not available due to badgeometry.

This results in about 27 sample epochs with different results between SAPPHIRE and RV(SAPPHIRE RAIM detection was 27 epochs more available than RV).

This shows that the differences in availability remarked are most often due to boundary valuesissues (values near to the Alarm Limit).

4.4.2 Evaluation of normal flight with detected incorrect measurements

The flight FK_76 is the only normal flight with a failure detection (and for SAPPHIRE alsoexclusion through signal quality monitoring) occurred. And this has happened during En-routephase of flight (phase of flight 3), with the weakest integrity requirements.The next table shows the differences between the RV and the SAPPHIRE RAIM results for theen route phase of flight (23374 samples) of the normal flight FK_76.

FDI Algorithm of Brenner FD Algorithm of Sturza/Brownwith FI Algorithm of Sturza


StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

StatisticalSAPPHIRE

RAIM Results[%]


of RV RAIMResults

0 (FD impossible � SV < 5) 0.004 0.000 0.004 0.0001 (FD not reliable � bad geometry) 0.013 -0.013 0.000 0.0002 (Everything within Specification) 99.974 0.013 99.974 0.0003 (Failure Detect./Ident. occurred) 0.000 0.000 0.021 -0.013

4 (Fail. Detect. Occ./Ident. Not occ.) 0.009 0.000 0.000 0.0005 (Fail. Det. Occ./Ident. impossible) 0.000 0.000 0.000 0.013

Table 4-16: FK_76 RAIM Differences For En Route (23374 Samples)

From this table it can be seen that failure detection and identification has occurred for a normalflight. This means that this flight was the only flight with a real error case!The differences in the availability were a consequence of detection/identification or exclusionthrough signal monitoring and were there not further considered here for evaluation.


26

For a detailed analysis the samples were the detection/identification occurred were given in thenext 2 tables (1 table for the Brenner and 1 for the Sturza/Brown algorithm).

In addition to the RAIM result flag (from 0 to 5), the PRN number of the SV, which wasidentified by a SAPPHIRE/RV RAIM algorithm, was given. If no fault detection occurred or thePRN of the faulty satellite cannot be identified, the PRN was set to 0.

RAIM Results (FDI Algorithm of Brenner)Sample

IDPhase ID/Visible SV

RAIM Result FlagNumbers Of

SAPPHIRE RAIM

RAIM Result FlagNumbers Of RV

RAIM

PRN NumberOf SV IdentifiedBy SAPPHIRE

PRN NumberOf SV Identified

By RV15588 3 / 6 4 4 0 015589 3 / 6 4 4 0 015590 3 / 5 1 2 0 015591 3 / 5 1 2 0 015592 3 / 5 1 2 0 0

Table 4-17: FK_76 SAPPHIRE/RV RAIM Results On Sample Level (FDI Algorithm OfBrenner)

RAIM Results (FD Algorithm of Sturza/Brown with FI Algorithm ofSturza)Sample

IDPhase ID

Visible SV RAIM Result FlagNumbers Of

SAPPHIRE RAIM

RAIM Result FlagNumbers Of RV

RAIM

PRN NumberOf SV IdentifiedBy SAPPHIRE

PRN NumberOf SV Identified

By RV15588 3 / 6 3 3 17 1715589 3 / 6 3 3 23 2315590 3 / 5 3 5 17 015591 3 / 5 3 5 9 015592 3 / 5 3 5 21 0

Table 4-18: FK_76 SAPPHIRE/RV RAIM Results On Sample Level (FD Algorithm OfSturza/Brown Combined With FI Algorithm Of Sturza)

As can be seen from Table 4-17 and from Table 4-18, incorrect measurements were detectedby the SAPPHIRE/RV RAIM algorithms from sample 15588 to sample 15592.

The Brenner algorithm was only able to detect that there is a failure for samples 15588 and15589, but unable to identify the faulty satellite. In the next 3 samples (15590-15592), only 5satellites were used for the positioning. Here the RV was available, while the SAPPHIREBrenner was unavailable due to bad geometry.

The Sturza/Brown algorithm detected and additionally identified the satellite failures for thesamples 15588 and 15589. In the next 3 sample (15590-15592), where only 5 satellites wereused for the positioning, the RV states that identification is impossible (only 5 satellitesavailable), while the SAPPHIRE Sturza/Brown detects and identifies faulty satellites. Here theSAPPHIRE Sturza/Brown algorithm has identification performance with only 5 satellites. Thisidentification usually requires at least 6 satellites (according to the approach of Sturza [10]implemented in the RV, requiring at minimum two redundant measurements for identification)to be available. Here the SAPPHIRE Sturza/Brown clearly behaves contrary to the usual Sturzaidentification algorithm [10]. A different interpretation of the literature could then be identified.But this had no consequences on the conduction of the project and the results obtained.


27

However from the PRN’s identified (many different PRN’s identified), it was clear that thiscannot be a single satellite failure.

If looking into the pseudo ranges it can be seen that all pseudo ranges were affected. Thisindicated that there was a hardware/software problem at the receiver for some epochs,affecting all receiver channels and therefore all pseudo ranges.

The usual RAIM algorithms are clearly not built to cover such a receiver-feared event. TheRAIM algorithms generally assume a single failure-hypothesis with a priori known probabilitydistribution (usually Gaussian) of the error.

The receiver error (with absolutely unknown probability distribution) does clearly not fall underthese RAIM assumptions. This means that the RAIM algorithms used here were simply notfitted to cover this type of errors.

The only way to cover this kind of error is to implement a special signal quality monitor, whichcovers this special feared event case.

4.4.3 Evaluation of error flights

The statistical differences in the RAIM results of the two selected RAIM algorithms, which wereimplemented in the SAPPHIRE DUAU as well as in the RV, with respect to the six error flightswere less than 2% (absolute percentage difference) for detection and identification. Thismeans that these six error flights have shown thereby identical or nearly identical behaviour ofthese two selected RAIM algorithms implemented in the SAPPHIRE DUAU and in the RV.Thus, no significant difference, which could not be explained, of these results appeared on thestatistical level.

The analysis of six simulated flights with artificial errors incorporated led to the followingconclusions can be drawn:

• The SAPPHIRE and RV Sturza/Brown algorithm show complete identical detection andidentification performance.

• Conceptually, the Sturza/Brown approach always is able to identify a faulty satellite incase detection was possible and has occurred (regardless the number of satellite).

• For the Brenner algorithm approach, the RV/SAPPHIRE results differ up to 1.848%(absolute percentage). But this is again due to boundary issues for identificationavailability (geometry of satellites good enough/not good enough). Here again the RVBrenner seems to be slightly more conservative concerning the availability of theidentification capability.

Generally, this confirms the high quality for detection of the RAIM algorithms implemented inSAPPHIRE, even under severe error conditions.

Additionally, the study of these error flights enabled to test the performances of identification,as the faulty satellite is known. The study shows that the SAPPHIRE and RV RAIM algorithmsalways have identified the correct faulty satellite. The identification of faulty satellites ofFK_200003, FK_200004 and FK_200005 were compared with the raw measurements(pseudoranges) of FK_32 (FK_200003, FK_200004 and FK_200005 were using the originalFK_32 pseudoranges, but with changed pseudorange values for one satellite).

By comparison of the PRN of the identified ‘faulty’ satellite with the PRN’s of the manipulatedpseudorange, the correctness of the identification of the right ‘faulty’ satellite was clearlyproved. The correctness of identifying the right ‘faulty’ satellite was 100%.


28

5. CONCLUSIONS

All comparisons performed in the frame of the SAPPHIRE RAIM validation lead to the followingfinal results:

• The SAPPHIRE detection and identification performance in case on an error leads tovery good results and can be considered as validated.

• The SAPPHIRE availability performance is very similar to the RV availabilityperformance. The SAPPHIRE seems to be more optimised for availability whilekeeping the detection/identification performance at the same level. Consequently thetrade-off between availability and integrity (detection/identification) is better optimisedin the SAPPHIRE DUAU.

• FK_76 clearly turned out that the RAIM algorithms only could cover the failure casesthey were fitted for. Other special failure cases (especially in the receiver) need special‘signal quality monitors’, fitted for the special feared event case.

• The SAPPHIRE ‘signal quality monitor’ capability has shown its potential to deal at avery early stage (in the pseudo-range domain before positioning) with certain fearedevents.

• Both considered algorithms are very similar (the Brenner algorithm is a little bit moreoptimised towards availability) in their performance. However one should be aware ofthe conceptually differences (the Sturza/Brown always can identify a faulty satellite incase of detection whereas the Brenner algorithm cannot always identify a faultysatellite in case of a detection).

Generally the SAPPHIRE RAIM algorithms can be considered as validated concerning theirRAIM availability, detection and identification capability and performance.


29

6. ACKNOWLEDGEMENTS

On behalf of IfEN, University FAF Munich, we would like to thank EUROCONTROL for theirsupport, effort and contributions to the SAPPHIRE RAIM validation activities.

Theodor ZINKJürgen PIELMEIER

Institute of Geodesy and NavigationUniversity FAF Munich


30

7. REFERENCES

[1] Brenner, M., Implementation of a RAIM Monitor in a GPS Receiver and an IntegratedGPS/IRS, Proceedings of The Third International Technical Meeting of The SatelliteDivision of The Institute of Navigation, ION GPS-90, Colorado Springs, CO, September19-21, 1990, pp. 397-406

[2] Design Document for SAPPHIRE RAIM Validation, Institute of Geodesy and Navigation(IfEN), University FAF Munich, Reference: EC-IFEN-RV-DD, Issue: 1, Revision: B,Neubiberg, March 6, 2000

[3] Guide to applying the ESA software engineering standards to small software projects,Document Title: BSSC(96)2, Issue No. 1, Revision No. 0, Prepared by: ESA Board forSoftware Standardisation and Control (BSSC), European Space Agency / Agencespatiale européenne, Paris, May 8, 1996

[4] Interface Control Document for SAPPHIRE RAIM Validation, Institute of Geodesy andNavigation (IfEN), University FAF Munich, Reference: EC-IFEN-RV-ICD, Issue: 2,Neubiberg, April 12, 2000

[5] Minimum Operational Performance Standards for Airborne Supplemental NavigationEquipment Using Global Positioning System (GPS), RTCA Document No. RTCA/DO-208, prepared by RTCA Special Committee 159 (RTCA SC-159), RTCA, Inc.,Washington, D.C., July 12, 1991

[6] Minimum Operational Performance Standards for Airborne Supplemental NavigationEquipment Using Global Positioning System (GPS), Change 1 to RTCA/DO-208,RTCA Paper No. 479-93/TMC-106, prepared by RTCA Special Committee 159 (RTCASC-159), RTCA, Inc., Washington, D.C., September 21, 1993

[7] Software Requirements Document for SAPPHIRE RAIM Validation, Institute ofGeodesy and Navigation (IfEN), University FAF Munich, Reference: EC-IFEN-RV-SRD, Issue: 1, Revision: A, Neubiberg, July 20, 1999

[8] Software Test Plan for SAPPHIRE RAIM Validation, Institute of Geodesy andNavigation (IfEN), University FAF Munich, Reference: EC-IFEN-RV-STP, Issue: 1,Revision: C, Neubiberg, March 6, 2000

[9] Software User Manual for SAPPHIRE RAIM Validation, Institute of Geodesy andNavigation (IfEN), University FAF Munich, Reference: EC-IFEN-RV-SUM, Issue: 1,Revision: C, Neubiberg, April 27, 2000

[10] Sturza, M. A., Navigation System Integrity Monitoring Using RedundantMeasurements, NAVIGATION, Journal of The Institute of Navigation, Vol. 35, No. 4,Winter 1988-89, pp. 483-501

[11] Sturza, M. A., and Brown, A. K., Comparison of Fixed and Variable Threshold RAIMAlgorithms, Proceedings of The Third International Technical Meeting of The SatelliteDivision of The Institute of Navigation, ION GPS-90, Colorado Springs, CO, September19-21, 1990, pp. 437-443

[12] User Requirements Document for SAPPHIRE RAIM Validation, Institute of Geodesyand Navigation (IfEN), University FAF Munich, Reference: EC-IFEN-RV-URD, Issue: 1,Revision: D, Neubiberg, May 20, 1999

list of figures - eurocontrol · no part should be reproduced in any form without the agency’s...

Documents