1

Unified Simulation and Test Platform for ControlSystems of Unmanned Vehicles

Xunhua Dai, Chenxu Ke, Quan Quan and Kai-Yuan Cai

Abstract—Control systems on unmanned vehicles are safety-critical systems whose requirements on reliability and safetyare ever-increasing. Currently, testing a complex autonomouscontrol system is an expensive and time-consuming process, whichrequires massive repeated experimental testing during the wholedevelopment stage. This paper presents a unified simulation andtest platform for vehicle autonomous control systems aiming tosignificantly improve the development speed and safety level ofunmanned vehicles. First, a unified modular modeling frameworkcompatible with different types of vehicles is proposed with meth-ods to ensure modeling credibility. Then, the simulation softwaresystem is developed by the model-based design framework, whosemodular programming methods and automatic code generationfunctions ensure the efficiency, credibility, and standardization ofthe system development process. Finally, an FPGA-based real-time hardware-in-the-loop simulation platform is proposed toensure the comprehensiveness and credibility of the simulationand test results. In the end, the proposed platform is applied toa multicopter control system. By comparing with experimentalresults, the accuracy and credibility of the simulation testingresults are verified by using the simulation credibility assessmentmethod proposed in our previous work. To verify the practica-bility of the proposed platform, several successful applicationsare presented for the multicopter rapid prototyping, estimationalgorithm verification, autonomous flight testing, and automaticsafety testing with automatic fault injection and result evaluationof unmanned vehicles.

Index Terms—Unmanned vehicles, Modeling, Safety testing,Control system, HIL, UAV.

I. INTRODUCTION

Unmanned vehicles (e.g., cars, boats, fixed-wing aircraft,multicopters, robotics, and helicopters) are becoming increas-ingly popular in both civil and military fields [1]. For allunmanned vehicles, safety is always the most basic require-ment, and the concern over potential safety issues remainsthe biggest challenge for the commercialization of unmannedvehicles. For most commercial unmanned vehicles, there isusually not enough space or payload to carry more hardwareredundancy (such as backup engines, actuators or motors) dueto the limitation of cost and performance, so the softwareredundancy is often applied in control systems to ensure safety.As a result, among all components on an unmanned vehicle,the control system is the most complex and important compo-nent that undertakes responsibilities for both reliable operationunder nominal conditions and safety decision under failurescenarios. Efficient simulation and test methods [2] for controlsystems of unmanned vehicles are urgently needed for theever-increasing system complexity and safety requirements.

The authors are with School of Automation Science and Electrical Engi-neering, Beihang University, Beijing 100191, China.

According to [3], more than 80% development tasks of anautonomous control system are in the middle level (see Fig. 1)to guarantee safety under various possible faults. However,most faults are rare to be encountered in practice, so massiverepeated experimental tests are essential to ensure that controlsystems can correctly detect and handle unexpected faults.Although different unmanned vehicles have different shapes,configurations, or running environment, they have a similarsystem structure presented in Fig. 1 and share many commonmodel features and fault modes. The common faults forunmanned vehicles include actuator faults (e.g., blocked, failedor unhealthy), sensor and communication faults (e.g., lossof signal, delays, GPS failed, and transmitting interference)[4], environment faults (e.g., obstacles, collisions, and winddisturbances) [5] and vehicle model faults (e.g., vibration andloss of weight). Thus, a unified simulation and test platformcompatible with different types of vehicles will be beneficialto share fault mode information and safety design experienceto improve the safety level of the whole unmanned vehiclefield. Besides, it can help to increase the exchange of safetydesign experience among different companies, manufacturersand certification authorities, and decrease the repetitive workduring testing and assessment processes, which is also benefi-cial to the rapid development requirements and better responseto the rules and regulations of governments.

Control Software SystemSensors Actuators

Vehicle Body Environment

Fixed-wing Multicopter VTOL

Ship Car Helicopter

Operation SystemDriver Interfaces

Low Level: Control

Top Level: AI

Middle Level:

Safety AssuranceIMU Barometer

Vision GPS

Control

Signals

Sensor

Data

Force &

Moment

Kinematics&Dynamics Rotation&Thrust

Propeller

Battery

Wheel

Ground

Water

Air

Wind

Obstacles

Moto

rRudder

Fig. 1. System structure of unmanned vehicles.

Currently, experimental testing is widely adopted becauseit can reflect real situations to the utmost extent. Besides, thesafety problems of control systems are usually highly coupledwith the actual situations. Since there is still no widely rec-ognized safety assessment standard published for unmannedvehicle systems (both unmanned cars and aircraft), many pre-researches are proposed to comprehensively test and assess the

arX

iv:1

908.

0270

4v1

[ee

ss.S

Y]

7 A

ug 2

019

2

safety of unmanned systems based on experimental methods.For example, in [6], the experimental testing and assessingmethod for the safety of control systems of Unmanned AerialVehicles (UAVs) are studied inspired by the testing methodsfrom the airworthiness of manned aerial vehicles. However, forunmanned vehicles, the experimental testing methods are usu-ally high-cost, inefficient, dangerous, and regulatory restricted.With the ever-increasing complexity of control systems, theexperimental testing methods become increasingly inefficientin revealing potential safety issues and covering the testingcases. Besides, in experimental testing, it is usually hard toobtain the true states of unmanned vehicles (usually estimatedby sensors or human perceptions) and to ensure consistencyof environment variables (e.g., wind, temperature, and roadcondition). For example, the sensor failures may cause thevehicle state estimation inaccuracy and unreliable to reflectthe true states of vehicles, so the test results in these casesare not suitable for quantitative assessment of vehicle safety.As a result, the experimental testing results usually need tobe analyzed by experienced engineers to evaluate the safetylevel of a control system, which is not efficient, precise, andautomatic enough for modern complex control systems. Dueto the above disadvantages of experimental testing methods,new simulation and test methods (e.g., the real-time simulationmethods [7], high-precision modeling and system identifi-cation methods [8], model-based safety assessment methods[9]) are becoming the trend for both manned and unmannedvehicles. Although experimental testing cannot be completelyabandoned, simulation testing techniques are taking on moreand more safety testing and assessment tasks [10].

Simulation methods for vehicle control systems canbe divided into Software-In-the-Loop (SIL) simulationand Hardware-In-the-Loop (HIL) simulation. As shown inFig. 2(b), by running the control algorithms in the same com-puter with the vehicle simulation model, SIL simulation canquickly test the control algorithms with the simulation speedmuch faster than the real world. However, the software andhardware operating environment of the SIL simulation is dif-ferent from the real vehicle system, whose control algorithmsare running on specific hardware (e.g., embedded systems, andindustrial computers). In summary, SIL simulations can accel-erate the test speed, but it is based on the expense of losingsimulation credibility. To improve the simulation credibility, asshown in Fig. 2(c), HIL simulation is proposed to reflect thereal operating environment of the control algorithms by usingreal control systems and real-time simulation computers. TheHIL simulation requires the simulation model runs in real-time with the control systems, which makes it convenient tocommunicate with other external hardware. Currently, thereare many popular simulation software supporting HIL sim-ulations for accelerating the development efficiency, such asAirsim, SwarmSim, and other systems for UAVs [11], [12],[13], and Carsim, Apollo, and other systems for autonomouscars [14], [15], [16]. Besides, HIL simulation systems arealso widely used in robotics studies to verify their algorithms[17], [18] or train the autonomous vehicle control systems[19]. These HIL simulation systems have been proven tobe convenient and efficient in accelerating the development

speed of control systems of unmanned vehicles, but there arestill some problems. First, these HIL systems focus more onproviding testing environments for the upper-level algorithmssuch as control algorithms, Computer Vision (CV) algorithmsand Artificial Intelligence (AI) algorithms of control systems;they usually cannot simulate the low-level software (e.g.,operating system, and drivers) and hardware (e.g., sensor chipsand high-speed analog circuits) due to computing capabilitylimitations. Secondly, they require to modify the code ofcontrol systems to disable the original hardware drivers andadd interface programs to exchanging data with the simulationcomputer, which may affect the operating environment andperformance of the original system (unstable and unreliable).Thirdly, the lack of widely recognized assessment methodsmakes it hard for people (e.g., users, manufacturers, and certi-fication authorities) to believe the credibility of the simulationtesting results for assessments and certifications.

Control SystemReal World

Environment

Vehicle Body

Actuators

Processor

Core Algorithms

Drives

Sensors&Communication Circuits

(a) Real Vehicle System

Control System

Processor

Core Algorithm

Drivers

Sensors&Communication Circuits

(d) FPGA-based HIL Simulation

High-performance Computer

Vehicle Simulation

System (body,environment,act

uators)

Core Algorithms

(b) Software-in-the-loop Simulation

Vehicle Simulation

System

Interface Program

Control SystemReal-time Computer

Processor

Sensors&Communication Circuits

(c) Hardware-in-the-loop (HIL) Simulation

Vehicle Simulation Subsystem

CPU 1kHz

Sensors Simulation Subsystem

FPGA 100MHz

Drivers

Core Algorithms

Drives

Force Moment

Actuator States

Interface Program

Actuator Control Outputs

Control Electronic Signals

Vehicle Motion States

Sensor Electronic Signals

CPU 1kHz

Real-time Computer

Fig. 2. Comparisons between common simulation methods.

In the past, limited by the real-time response performanceof simulation computers, it is hard for Central ProcessingUnits (CPUs) [20] on simulation computers to simulate sensorsor other electronic chips with high-speed interfaces or high-frequency analog circuits. For example, a nanosecond-levelreal-time update frequency (100MHz) is required to reliablysimulate a sensor chip with the high-speed Serial PeripheralInterface (SPI), which is a difficult task for traditional CPU-based Commercial-Off-The-Shelf (COTS) simulation com-puters whose reliable update frequencies are usually withinmicrosecond level. In recent years, with the utilization ofField Programmable Gate Array (FPGA) [20], COTS FPGA-based simulation computers (e.g., OPAL-RTr/OP series andNIr/PXI series) start to have the simulation performance ofnanosecond-level real-time update frequency [7], [21]. Thismakes it possible to simulate almost everything (includingvehicle motion, sensor chips, electric circuits, and high-speedinterfaces) outside the main processor of the control systemsas presented in Fig. 2(d). Besides, the tested control systemscan be treated as black boxes in HIL simulation systemswith no need for accessing the code or adding interfaceprograms. Therefore, by simply replacing the sensor models,

3

the HIL simulation system can also be applied to performcomprehensive tests for different brands of control systems.

For all simulation methods, the primary challenge is toensure simulation credibility [22], namely making peoplebelieve the simulation results are as real as experiments inthe real world. The simulation credibility (compared withreal systems) mainly determined by two aspects: platformcredibility and model credibility. The platform credibility canbe further divided into hardware credibility and softwarecredibility. As previously mentioned, the hardware credibilitycan be guaranteed by the FPGA-based HIL simulation method,while the software credibility and the model credibility are stillchallenges for simulation methods. The Model-Based Design(MBD) [23] method is an effective means to solve the abovecredibility problems by using modular visual programmingtechnology and automatic code generation technology to stan-dardize the modeling, developing and testing procedures ofcomplex control systems. The MBD method can ensure thesoftware credibility by eliminating the disturbance factors suchas manual programming negligence and nonstandard develop-ment process. For example, the MathWorksr/MATLAB (themost widely used MBD software) can ensure the generatedcode meet the requirements of standards and guidelines suchas DO-178C. In MBD methods, the whole simulation systemscan be divided into many small subsystems (modules), suchas kinematic modules, GPS modules, ground modules, andpropeller modules. Certification authorities can verify andvalidate these modules to build a standard product modeldatabase for companies to develop the vehicle prototype andthe corresponding vehicle simulation system. Then, the modelcredibility can be guaranteed by using well-validated standardcomponent models. What is more, with MBD method, thesame component model can be applied to different types ofvehicles and control systems, which may significantly improvethe design, verification, validation, and certification process ofthe unmanned vehicle filed. For ensuring model credibility, inour previous research [24], a credibility assessment method isproposed based on experiments to assess the model credibilityof HIL simulation platforms from multiple aspects, such askey performance indices, time-domain characteristics, and thefrequency-domain characteristics. By combining the abovemethods, the credibility of the simulation platform can beguaranteed from the model, development process, and platformhardware aspects.

In this paper, a unified simulation and test platform isproposed for control systems of unmanned vehicles based onFPGA-based HIL simulation and MBD methods, aiming tosignificantly improve the test efficiency and safety level ofcontrol systems on unmanned vehicles. The main researchcontents and the corresponding contributions are listed asfollows.

(i) Unified Modeling Method. There are so many in commonamong different types of unmanned vehicles. They should notbe treated separately as more and more composite vehicles(e.g., multicopter + fixed-wing and car + fixed-wing) emerged.Therefore, a unified modeling method is proposed for all typesof unmanned vehicles along with parameter measurement andidentification methods to validate the obtained model and

ensure simulation credibility. The corresponding content ispresented in Section II.

(ii) Real-time HIL Test Platform with MBD. A real-time HILtest platform is built for the testing and assessment of controlsystems. The platform is capable of simulating any real-world situation outside the control software with advantages inobtaining the true states and controlling the testing variables.The utilization of MBD methods can ensure that the testingresults are credible and standard-compliant. Besides, the HILsimulation method and MBD method ensure the platformcan be easily applied to different brands of control systems,which is convenient for both companies and manufacturers.The corresponding content is presented in Section III.

The significant advantages of the proposed unified HIL sim-ulation test platform are reflected in four aspects: extensibility,comprehensiveness, verification, standardization.

(i) Extensibility. By changing the parameters (e.g., weight,size, and aerodynamic coefficients) of specific subsystemmodules, it is easy to extend a simulation system to othervehicle systems with similar structures. Moreover, by replacinga whole subsystem module (e.g., a propeller module to a tiremodule), the vehicle simulation system can be extended toother types of vehicles.

(ii) Comprehensiveness. The current simulation systemsmainly focus on functional testing, i.e., whether the vehiclescan work properly in normal situations. However, unmannedvehicles are safety-critical systems, and most of the effects arefocused on safety testing, i.e., whether the vehicles can worksafely when accident and faults happen. With the modularprogramming method, the fault modes, the aging process, andthe probabilistic reliability property can be modeled for eachsubsystem module to improve the comprehensiveness of thesimulation platform. Mathematically, the fault injection simu-lations (or other safety simulations) can be realized by onlinechanging the module parameter or functional expressions of asubsystem module while the simulation program is running.

(iii) Verification. In practice, it is difficult to verify andvalidate the simulation accuracy and credibility of a complexsimulation system. However, it is relatively simple to verify asmall subsystem. Therefore, the modular programming methodcan divide a complex simulation system into many smallsubsystems, and verify it from lower levels to the top level.More importantly, if all subsystems used in a simulationsystem are well-verified modules from certification authorities,the verification efficiency can be significantly improved.

(iv) Standardization. A standard certification frameworkis urgently needed for unmanned vehicles to improve thetesting and certification efficiency. The modular programmingmethod is a feasible way to solve this problem with thecertification framework presented in Fig. 3. In this framework,the manufactures should provide the product hardware alongwith a simulation model which should be fully verified andcertificated by authority agencies based on the simulation dataand experimental data. That coincides with the idea of DigitalTwin [25] for the efficient design and testing of complexsystems. Then, the vehicle companies can use the certifiedmodels for simulation system development and prototypedesign. Finally, the simulation results and experimental results

4

Product

HardwareSimulation

Model

Manufacturers

Vehicle CertificationModel Certification

Authority Agencies

Experimental Testing

Design and HIL

Simulation

Vehicle Companies

Verified

Models

Simulation

DataExperimental

DataSimulation

Results

Experimental

Results

Products

Certification

Report

VehiclePrototype

Fig. 3. Certification framework for unmanned vehicles.

can be applied for the certification of the unmanned vehicle.In Section IV, the successful application of the testing

platform on multicopter vehicles with the PX4 autopilot sys-tem indicates the effectiveness of the proposed modeling andsimulation methods. To verify the practicability of the pro-posed platform, several successful applications are presentedfor the multicopter rapid prototyping, estimation algorithmverification, autonomous flight testing, and automatic safetytesting with automatic fault injection and result evaluationof unmanned vehicles. In the end, Section V presents theconclusions.

II. UNIFIED MODELING METHOD

As shown in Fig. 1, different types of vehicles (e.g., cars,aircraft, and boats) have many common features in modelingand simulation. To make the maximum utilization of thesecommon features, a unified simulation testing system is de-veloped with the system structure presented in Fig. 4 forall unmanned vehicle systems. In this section, the unifiedmodeling methods for the simulation system in Fig. 4 will beintroduced in detail. The hardware structure and developmentprocess for the simulation system in Fig. 4 will be introducedin Section III.

A. Overall Vehicle Model1) Model Abstraction: In practice, a complex dynamic

system usually consists of many small subsystem systems(e.g., body system and propulsion systems). Every subsystemincludes input signals u∗, output signals y∗, parameters Φ∗,dynamic states x∗, and and dynamic state and output functionsf∗ (·) and h∗ (·), which can be mathematically described bythe following dynamic equations{

x∗ = f∗ (x∗,Φ∗,u∗)y∗ = h∗ (x∗,Φ∗,u∗)

. (1)

For simplicity, a dynamic subsystem in Eq. (1) is furthersimplified to the following input/output form as

y∗ = S∗ (u∗) . (2)

The system description form in Eq. 2 will be applied in thefollowing content to describe the connection relationshipsamong different subsystems. The subscript symbol “∗” inEqs. (1)(2) can be replaced by different abbreviation wordsto represent different dynamic systems, such as the controlsoftware system Sctrl and the sensor simulation subsystemSsens.

2) Main Framework of Simulation System: As presentedin Fig. 4, the whole simulation system can be divided intothree main subsystems: the vehicle simulation subsystem Svehi(generating vehicle states according to the control signals), the3D environment simulation subsystem S3d (generating visiondata according to the vehicle states), and the sensor simulationsubsystem Ssens (generating sensor signals according to thevehicle states and vision data). Besides, there is a controlsystem Sctrl (generating control signals according to the sensordata) to be tested. The above connection relationships amongthe above four systems are mathematically described by

yctrl = Sctrl (uctrl) , uctrl = ysensyvehi = Svehi (uvehi) , uvehi = {yctrl,y3d}y3d = S3d (u3d) , u3d = yvehiysens = Ssens (usens) , usens = {yvehi,y3d}

(3)

which is consistent with the connection relationships in Fig. 4.In the following, the unified modeling methods for the abovethree main subsystems will be introduced sequentially.

B. Vehicle Simulation Subsystem

The vehicle simulation subsystem Svehi in Fig. 4 can befurther divided into four main subsystems: the actuator subsys-tem Sact, the environment subsystem Senv, the force&momentsubsystem Sfm, and the vehicle body subsystem Sbody. Asshown in Fig. 5, the connection relationships of the foursubsystems are mathematically described as

ybody = Sbody (uctrl) , uctrl = yfmyfm = Sfm (ufm) , ufm = {ybody,yact,yenv}yenv = Senv (uenv) , uenv = {ybody,y3d}yact = Sact (uact) , uact = {yctrl,ybody,yenv}

(4)

where ybody contains vehicle motion states (e.g., position,velocity, and attitude), yfm denotes all the forces and momentsacting on the vehicle, yenv includes environment parameters(e.g., gravity, air density, terrain, and obstacle distribution) ,yact denotes actuator states (e.g., rotating speed of rotors, anddeflection angle of control surface). By combining the outputsignals in Eq. (4), the output set yvehi for the vehicle simulationsubsystem Svehi is given by

yvehi , {ybody,yfm,yenv,yact} .

The key modeling methods for the four subsystems in Eq. (4)will be introduced as follows.

1) Vehicle body subsystem: As shown in Fig. 5, the vehiclebody subsystem Sbody computes the vehicle states ybody ac-cording to the force and moment yfm acting on the vehicle.In practice, based on the flat-earth assumption (ignoring thecurvature of the earth in a small range) and the rigid-bodyassumption (the body is rigid and not flexible), most vehiclebody subsystem Sbody can be described by Six-Degree-of-Freedom (6-DOF) equations [26, pp. 25-54],[27, pp. 99-143]

ep = Reb · bv

bv = −bω × bv + (1/m) · bF

Reb = Re

b ·[

bω]×

J · bω = −bω ×(J · bω

)+ bM

(5)

5

Control Software System

Force & Moment Subsystem

Actuator Control Force&Monent

Vehicle Aerodynamics

Ground Support & Physical Collision

Gravity

Vehicle Body Subsystem

Velocity

Position

Quaternion

Angular Velocity

Dynamics

Environment Subsystem

Gravity Air Density

Wind

Magnetic

Terrain&Obstacle

...

Propulsion Unit Subsystem

Battery

ESC

Motor

Propeller

...

Actuator Subsystem

...

...

Vehicle Simulation Subsystem (CPU)

Vehicle States

Control Signals

Sensor Data Subsystem

GPS

Barometer Gyroscope

Accelerometer

Vision ...

3D Environment Subsystem

Sensor Product Subsystem

Ublox N8N

MS5611 LiDAR

MPU6050

Camera ...

Communication Subsystem

...

Radar

Obstacles

Background Light

View Points

...

Sensor Simulation Subsystem (FPGA)

Other External Hardware Systems

Testing Programs

Model Parameters

Fault Injection

Safety Assessment

3D Display

Real-Time Simulation Computer

...

Actuator Subsystem

#1 #2

...

Servo Systems Engine&Tires

#3

#1 #2

...#3

#1 #2

...#3

Propulsion Units

Host Computer

Sensor Signals

SPI I2C

CAN PWM

AD/DA UART

3D Data

Health States

Vehicle States

External Signals

Mission Commands

3D Data

Control System

Fig. 4. System structure of the simulation test platform for unmanned vehicles.

Steady State Function

1sDynamics

ss,1v1v

Control Signal Vector

ns ss,nv

nv

� ...is

iv

ctrly

Actuator Subsystem

acty

Actuator State

Vector

Steady State Function

Dynamics

Actuator Control Force

Aerodynamic Force

Force of Gravity

Contact Force

å

grouF

aeroF

actF

gravF

fmy

Force&Moment Subsystem

Total Force and

Moment

Vehicle Body SubsystemEnvironment

Subsystem bodyy

Vehicle States

Environment ParametersAnd States envy

bodyy

fmy

envy

bodyy

Fig. 5. The structure of the vehicle simulation subsystem.

where the right-superscript symbols “e” and “b” denote theNorth-East-Down (NED) earth frame and the Head-Right-Down body frame [26, pp. 25-54] respectively; ep ∈ R3 isthe position vector defined in the earth frame; bv ∈ R3 andbω ∈ R3 are the velocity vector and the angular velocity vectordefined in the body frame; bF ∈ R and bM ∈ R are theforce vector and the moment vector defined in the body frame;Re

b ∈ R3×3 is the rotation matrix to transform a vector frombody frame to earth frame; J ∈ R3× 3 and m are the momentof inertia matrix and the mass of the vehicle. The 6-DOFdynamic equations in Eq. (5) can be applied to describe thevehicle body module ybody = Sbody (yfm) in Eq. (1), where thestate set is xbody ,

{ep, bv,Re

b,bw}

, the input set is ubody =

yfm ,{

bF, bM}

, the parameter set is Φbody , {J,m}, andthe output set is ybody ,

{xbody,

ev, bv, bω,Reb, · · ·

}.

2) Environment subsystem: As shown in Fig. 5, the environ-ment subsystem Senv generates environment parameters yenv(e.g., air density, temperature, terrain, wind, and magneticfield) based on the position of the vehicle ep ∈ ybody. Inpractice, the World Geodetic System (WGS84) model [28]is widely used to describe the shape of the earth, which canconvert the position vector ep to the earth Latitude-Longitude-Altitude (LLA) global position epg , [µ ι h]

T, where µ, ι(unit: degree) are the latitude and longitude, and h (unit:m) is the altitude. Then, the acceleration of gravity g canbe estimated by the WGS model [28] based on the vehicleglobal position epg. Similarly, the air density and temperatureare estimated by the International Standard Atmosphere (ISA)model [29], and the magnetic field vector is estimated by

the World Magnetic Model (WMM) [30]. Besides, accordingto the Military Specification MIL-F-8785C [31], the windvelocity disturbance vector evwind ∈ R3 (defined in the earthframe) can be described by the following superposition form

evwind = evturb +evcons +

evsheer +evgust (6)

where evturb denotes the atmospheric turbulence field, evconsdenotes the prevailing wind field, evsheer denotes the windshear field, and evgust denotes the wind gust field. There manywidely used mathematical models for the wind componentsin Eq. (6). For example, the wind turbulence evturb can bedescribed by the Dryden Wind Turbulence Model [31].

3) Actuator subsystem: As shown in Fig. 5, the actuatorsubsystem Sact outputs actuator state yact according to thecontrol input yctrl from the control system. In practice, itis difficult to obtain the mathematical model of an actuatorsystem because it is usually composed by complex mechanicalcomponents along with programmable control units, such asthe Electronic Speed Controller (ESC) for UAV brushlessmotors, and the Electronic Control Unit (ECU) for car en-gines and steering systems. These control units have feedbackcontrol to ensure that the actuator steady output δss,i satisfy thepreprogrammed function of the input control signal σi ∈ yctrlunder different operating environment. According to [27], acomplex actuator system can be linearized to a steady-stateprocess fss,i (·) and a dynamic response process Gss,i (s)around the rated operation condition. For example, a motor-propeller system with an ESC can be simplified as a first-orderor second-order inertial process Gss,i (s) and a steady-statefunction fss,i (σi) as

δi = Gss,i (s) · fss,i (σi) . (7)

Noteworthy, fss,i (·) can be measured by static testing, andGss,i (s) can be measured by system identification methodsthrough frequency-response testing [32]. By using Eq. (7), itis easy to obtain the actuator output signal δi (t) (propellerrotating speed) under the given control signal σi (t) (throttlecontrol signal). Then, the control force and torque generatedby the state of an actuator δi can be obtained by the groundfriction model, aerodynamic model, or other mechanical mod-els [27], [33], [34].

6

4) Force & moment subsystem: As shown in Fig. 5, theforce&moment subsystem Sfm outputs the force and momentyfm ,

{bF, bM

}to the vehicle body subsystem Sbody. The

total force bF and moment bM acting on a vehicle can bedivided into many components from different sources. Takingthe force vector bF ∈ yfm (defined in the body frame) as anexample, it can be described by the following superpositionform

bF = bFaero +bFgrav +

bFcont +∑

bFact,i (8)

where bFaero ∈ R denotes the aerodynamic force vector,bFgrav ∈ R denotes the force of gravity vector, bFcont ∈ R

denotes the contact force vector from ground supporting orphysical collision, and bFact,i ∈ R denotes the control forcevector generated by an actuator. Noteworthy, the above forcevectors should be all transformed to the body center andprojected to the body frame.

The aerodynamic force vector bFaero is a nonlinear functiondetermined by the relative speed of the surrounding air evrelas

evrel ,evwind − ev

where evwind is the wind speed from the environment sub-system in Eq. (6) and ev is the vehicle speed from thebody subsystem in Eq. (5). The high-precision aerodynamicmodeling method has been well studied in [26], which iscompatible with all types of vehicles such as multicopters [27],helicopters [33] and cars [34].

The contact force bFcont caused by the ground supporting orphysical collision can be modeled by simplifying the vehiclebody shape to a cuboid or a cylinder and simplifying thecontact surface to a spring-loaded system. By adjusting thespring stiffness, it is convenient to simulate physical contacton objects with different surface hardness. The terrain andobstacle information comes from the environment subsystemoutput yenv, which further comes from the 3D environmentsubsystem S3d.

The actuator force vector bFact,i can be unified describedby the following nonlinear expression

bFact,i = fact,i (Φfm,yenv,ybody, δi) (9)

where δi ∈ yact is the instantaneous state of an actuator(the rotating speed of a propeller, deflection angle of a servosystem, or driving torque of a tire) from the actuator moduleSact. Noteworthy, the expression of fact,i (·) is also related tothe vehicle state ybody and the environment state yenv, and themethods to obtain the force model fact,i (·) have been wellstudied in [27], [33], [34].

As shown in Fig. 6, the different types of vehicle simulationmodels are mainly distinguished by the actuator types andconfigurations. In a similar way, the actuator force models fordifferent types of vehicles in Fig. 6 can be easily obtained withproper actuator system modeling methods.

C. 3D Environment Simulation Subsystem

The 3D environment subsystem S3d in Eq. (3) aims togenerate vision data y3d based on the vehicle states yvehi from

Propeller

Elevator #2

Elevator #1 Rudder

Aileron #1

Aileron #2

Propeller #3 Propeller #1

Propeller #2Propeller #4

Tire #3

Tire #1

Tire #4

Tire #2Propeller #1 Propeller #2

Fig. 6. Actuator force models for different types of unmanned vehicles.

the vehicle simulation subsystem. The vision data will be sentto the sensor subsystem to generate data for vision sensorssuch as camera, radars, laser range finder, etc.

Currently, many widely used 3D environment engines canbe applied for vehicle vision modeling. For example, theSimulink 3D Animation Toolbox provides interfaces to con-veniently access the video stream for image process andcontroller design in Simulink; the Airsim [11] is developedby Microsoftr to generate high-fidelity visual and physicalsimulation environment using Epic Gamesr/Unreal Engine 4(UE4). Both Simulink 3D Animation Toolbox and Airsim canbe applied to different types of vehicles, including aircraftand cars. There are also many 3D simulation environmentsexclusively developed for specific types of vehicles. For ex-ample, the Gazebo HIL simulator [16] for visual simulationof autonomous cars, and the FlightGear [35] for aircraftsimulations.

These 3D simulation engines can be applied to develop the3D environment subsystem to generate vision data (cameras),obstacle distance information (rangefinders, radars), or pointcloud data (Lidar). With the development of GPU performanceand 3D modeling technology, the obtained vision data y3d willbe more and more high-fidelity and realistic in the future,which will significantly improve the credibility of visualsimulations.

D. Sensor Simulation Subsystem

The sensor subsystem Ssens in Eq. (3) describes the processto transform the vehicle state yvehi and vision data y3d tothe electric signals ysens for the control system. Concretely,it can also be divided into three small subsystems, includingthe sensor data subsystem Sdata, the sensor product subsystemSprod and the communication subsystem ysens. As shownin Fig. 4, their connection relationships are mathematicallydescribed as

ydata = Sdata (udata) , udata = usens = {yvehi,y3d}yprod = Sprod (uprod) , uprod = ydataysens = ycom = Scom (ucom) , ucom = yprod

(10)

where ydata contains ideal data for sensors (e.g., the accelera-tion of accelerometers, the magnetic field of magnetometers,and the image or point cloud data of vision sensors), and yprod

7

is the sensor signals after adding detailed product features(e.g., noise level, temperature drift, failure mode, and cameradistortion), and ysens is the binary electrical signals transmittedto the control system for position and attitude estimation.

1) Sensor Data Subsystem: As described in Eq. (10), thesensor data subsystem Sdata generates the sensor data ydata(applicable for a class of sensor products) based on thevehicle state yvehi and vision data y3d. Transformations usuallyrequired to obtained the sensor data from the vehicle statesand vision data. For example, accelerometers measure thespecific force (the difference between the acceleration of theaircraft and the gravitational acceleration) [36, p. 122] insteadof the vehicle acceleration bv ∈ yvehi. Similar computationexpressions are applied to other types of sensors, such asthe GPS Modules (longitude and latitude obtained from thevehicle position), electronic compasses (the magnetic fieldintensity obtained from the attitude and global position of thevehicle), optical flow sensors (relative velocity obtained fromimage stream), etc.

2) Sensor Product Subsystem: The sensor product subsys-tem Sprod is developed to add product features (e.g., noise,vibration, and calibration) to the sensor data ydata obtainedfrom the above sensor data subsystem Sdata. Given the samesensor data, different sensor products may obtain differentresults due to product features, so the senor product subsystemis necessary.

In most cases, given the ideal sensor data xm ∈ ydata, thenoise feature is mainly reflected in the noise na and bias driftba of the measured value x′m, which can be described as [27,p. 151] {

x′m = xm + ba + na

ba = nb(11)

where na ∼ N(0, σ2

a

)and nb ∼ N

(0, σ2

b

)are zero-mean

Gaussian noise vectors for inertial sensors. The standard devi-ation parameters σa, σb can be found in the datasheet documentof a sensor product or obtained by system identification withthe actual sensor output signals.

If the system is not affected by vibrations, σa can bemodeled as a constant value. When the vibration feature ofa sensor is considered, the measuring noise na may also beaffected by the vibration from may sources (e.g., engines,motors, and fuselage).Therefore, the standard deviation σa isnot always constant value, which should be modeled based onthe actual system characteristics.

The calibration feature is mainly determined by the workingenvironment of the installation configuration a sensor, whichcan be described as [27, p. 149]

x′′m = TeKe (x′m + pe) (12)

where pe is a constant vector for the position installationdeviation, Te is a rotation matrix for the installation deviation,Ke is a diagonal matrix for the scale deviation, and x′′m ∈ yprodis the final output data of a sensor. Eqs. (11)(12) are applicableto most types of sensors to simulate the properties of realsensor products.

There are also many methods to add product features forvision sensors. For example, the methods to add camera

SCK

CS

Clock signal

Transmission enable signal

...

...

Query command received

Recognition, computation,

data preparation

Communication model simulates the

SPI protocol (8-bit per exchange)

Sensor data

returned

MOSI ...Q12~7Q11 Q18 Q21 Q28 Q31Q22~7

MISO ...R11 R18 R21 R28 R31R22~7R12~7

Control system query commands

Sensor returned

data

Fig. 7. Communication subsystem model for SPI buses.

features (e.g., blurs, distortions, and noises) to an ideal imageis introduced in [37]. Other environment factors (e.g., lighting,reflection, and fogging) can be simulated by the latest 3Dengines, such as UE4.

3) Communication Subsystem: The communication subsys-tem is developed to transform the sensor data with productfeatures yprod to binary electronic signals ysens for the controlsystem. The outputs yprod of the above sensor product sub-system Sprod are decimal numerical signals, but binary elec-tronic signals are required for the communication requirementsbetween the control system and other hardware. There aremany communication interfaces and protocols widely usedin the vehicle control systems, such as SPI, Inter-IntegratedCircuit (I2C) [38], Controller Area Network (CAN), UniversalAsynchronous Receiver-Transmitter (UART), and Pulse-WidthModulation (PWM).

The mathematical model for a communication interface isusually simple, but it is hard to be simulated by a CPU-based simulation computer due to the extremely high real-timeupdate frequency and bandwidth requirements. Taking the SPIinterface as an example, the SPI interface uses four signalwires to exchange information between the master device (themain processor of the control system) and the slave device (on-board sensors). As shown in Fig. 7, the sensor has to finish thecommand recognition, measured data computation, and outputdata preparation within a small interval (after the previous bytedata received and before the following byte data to be sent).In a real sensor chip, the above process is instantaneouslyrealized by analog circuits whose time consumption can betreated as infinitely small. However, for a real-time simulationsystem, it usually requires a real-time update frequency at thenanosecond level to ensure that the sensor signals are correctlycomputed, prepared, and transmitted. For the performancerequirements, this paper uses FPGA system to simulate allsensor communication features (e.g., data transmission, chiprecognition, programmable setting functions), which ensuresall the sensor hardware related low-level test cases can besimulated by the simulation platform.

E. Model Identification and Validation

The mathematical model of a subsystem can be classifiedinto three types: the static models, the dynamic models, andthe statistical models. (i) The static models are described by

8

constant parameters that can usually be precisely measured byprofessional equipment. These parameters include the mass,moment of inertia, air density, aerodynamic coefficients, etc.(ii) The dynamic models are described by dynamic equationsthat are usually difficult to be measured by equipment directly.System identification methods [8], [32] are widely used inobtaining dynamic parameters in aircraft and vehicle systems.(iii) The statistical models are described by statistical valuesor stochastic processes (e.g., the sensor noise, and the atmo-spheric turbulence). The statistical models can be obtained byspectral analysis or statistical analysis of the output signalswith enough sample data. For example, the expectation andvariance of the White Gaussian noise of a sensor product inEq. (11)(12) can be obtained by its output signals [39].

In practice, by using advanced modeling methods withenough experimental data, any vehicle system can be modeledarbitrarily close to the real system. However, a high-fidelitymathematical model usually indicates more computationalcomplexity and cost (higher dimension and nonlinearity), soabstraction and simplification methods are important for simu-lation systems. The simulation credibility is the most importantindex in the trade-off between precision and complexity. Inour previous research [24], by feeding the same signals toboth the simulation system and the real system and analyz-ing the simulation errors, a simulation credibility assessmentmethod is proposed to assess the simulation credibility ofa simulation system from multiple aspects, such as systemperformance, time-domain response, and frequency-domainresponse. A normalized assessment index is proposed to scorethe simulation credibility of a subsystem under a uniformassessment criterion. The proposed method is efficient toverify and validate the simulation effect of each subsystemand the whole simulation system, which is used in this paperto ensure the credibility of the obtained vehicle subsystemmodels.

III. Real-time HIL Test Platform with MBD

This section presents the hardware structure and develop-ment framework of the proposed HIL test platform in Fig. 4successively.

A. Hardware Structure of the HIL Platform

The platform hardware consists of three parts.1) Real-time Simulation Computer: A real-time simulation

computer (also called real-time simulator) is a special typeof computer, running Real-Time Operation System (RTOS)to ensure the simulation systems run at the same speed as theactual physical system. The latest COTS simulation computersusually provide a CPU-based system and an FPGA-based sys-tem for different requirements of real-time update frequency;the CPU-based system is better at running complex simulationmodels with moderate frequency requirements (usually smallerthan 100KHz); the FPGA-based system is better at runningsimple simulation models with extremely high frequency (usu-ally larger than 100MHz). By combining the advantages of theabove two systems, the hardware structure of the proposed testplatform is presented in Fig. 4, where the CPU-based system

is applied to run the vehicle simulation subsystem presentedin Section II-B and the FPGA-based system is applied to runthe sensor simulation subsystem presented in II-D.

2) Control System: The control system is the test objectof the proposed test platform. The control system computescontrol signals for driving the actuators according to thevehicle states measured and estimated by different sensors. Toensure the control system can work normally in the proposedtest platform, the original sensors should be blocked (orremoved), and the sensor pins on the control system shouldbe reconnected to the FPGA-based system of the real-timesimulation computer. Then, the control system can receive thesimulated sensor chip signals for full-function operation. Theabove process only needs to know the brands and models ofsensors used in the control system, and has no requirementto access the source code or internal hardware structure.Therefore, it is practical to perform black-box testing fordifferent control system products from different unmannedvehicle companies.

3) Host Computer: A high-fidelity 3D simulation environ-ment is also essential for training or testing the top-levelalgorithms, including computer vision, machine intelligence,and decision making systems. Therefore, a host computerwith high-performance Graphics Processing Units (GPUs) andrealistic 3D engines are used in this paper to generate visionsignals to the real-time simulation computer. The latest high-end consumer GPUs (such as NVIDIA GTX2080) have beenpowerful enough to generate high-resolution (larger than 4K)video streaming with update frequency more than 100Hz,which is capable of simulating most vision sensors. Besides,the host computer also takes responsibility for running otherauxiliary programs such as model parameter configurationprogram, 3D display program, ground control program, etc.Noteworthy, for different data bandwidth and real-time re-quirements, the connection and communication among thesimulation computer and the host computer can be realizedby optical fibers, network cables, serial ports, etc.

B. Development Framework with MBD

1) Modular Programming: In practice, developing a com-plex simulation software through hand-coding is a difficultand unreliable task due to too many mathematical operations.Any coding mistake, logical mistake, or unknown vulnerabilitymay lead to wrong or inaccurate simulation results. The typesand amounts of unmanned vehicles will be far more thanmanned vehicles, and the development cycles are requiredto be much shorter, so the traditional hand-coding methodsare no longer suitable for developing simulation software forunmanned vehicles. As a result, modular (also described asgraphical or visual) programming methods have been widelyused in many MBD tools (e.g., MathWorksr/Simulink andNIr/LabVIEW). The whole simulation system presented inSection II has been divided into many simple and independentsubsystems, and each subsystem can be easily realized by avisual module or block in the above MBD tools, which makesit easy to develop and test a complicated vehicle simulationsystem.

9

Simulink Environment

Vehicle Subsystem

Code Generation

Sensor Subsystem

LabVIEW Environment

Vehicle States

Control Signals

Sensor Subsystem

Vehicle Subsystem

SPI/I2C(10MHz)

Serial(100Hz)

PWM(250Hz)

Processor (Control

Software)

Control System

(FPGA 100mHz) (CPU 1kHz)

Real-Time Simulation Computer

Network

Vision

Vehicle Subsystem

Host Computer

Model Parameter Interfaces

Fault Injection Program

3D Environment Model

...

Vision

UE4 Environment

Vehicle Control

System

3D Environment Subsystem

Code Generation

Code Generation

StatesControls

Sensor Data

Code Generation

Fig. 8. Code generation and deployment framework for simulation softwareof HIL testing platforms.

2) Automatic Code Generation: Modular programming andautomatic code generation are two of the most significantfeatures of MBD tools such as Simulink, LabVIEW, andUE4. Simulink is better at developing complex simulationsystems (such as unmanned vehicle simulation systems), andLabVIEW is better at developing hardware-closed simulationsystems (such as sensors, circuits and communication signals)and deploying the simulation program to the real-time simu-lator. For example, the Aerospace Blockset in Simulink [40]provides many demos for quickly developing vehicle simula-tion systems, such as aircraft and multicopter. There are many3D engines for developing high-fidelity vehicle 3D simulationenvironments. Among these engines, UE4 provides modularvisual programming functions, so the UE4 environment isselected to develop 3D simulation scenes for the HIL platform.

In order to take full advantages of both MBD tools, thedevelopment process for simulation system software is shownin Fig. 8. The development process is divided into the follow-ing steps: (i) developing and verifying the vehicle simulationmodel in Simulink Environment; there are many powerful ver-ifying tools in Simulink such as requirements traceability, codecoverage check, document generation, etc., which guaranteethe simulation software meets the standards and guidelinessuch as DO-178C; (ii) compiling the vehicle simulation sub-system into code, and importing it to the LabVIEW environ-ment; (iii) building the sensor subsystem in LabVIEW andbuilding the interfaces to communicate with other systems; (iv)generating code and executable files to deploy them to the real-time simulation computer; (v) developing the 3D simulationenvironment in UE4 and deploy it to the host computer. Thewhole development process in Fig. 8 is efficient and reliablebecause all coding and deploying operations are automaticallyfinished by MBD tools without much human intervention.Therefore, it is convenient to replace some models and rebuildthe simulation system for different types of vehicles or controlsystems.

IV. VERIFICATION AND APPLICATION

In this section, a real-time HIL test platform is first de-veloped for multicopter UAVs based on the proposed testplatform. Then, the simulation accuracy is verified througha series of experiments. In the end, several successful applica-tions are presented to verify the feasibility and practicabilityof the proposed methods in this paper.

Pixhawk Autopilot

Ground Station

Program

Unreal Engine 4

3D Environment Model

Fault Injection Safety

Testing Program

Sensor Model

FPGA | 100MHz

Vehicle Model

CPU | 5kHz

Sensors Blocked

CPU:NI® PXIe-8133FPGA:NI® PXIe-7846R High-performance

Workstation PC

Simulink + LabVIEW

Pixhawk® Autopilot

Futaba® Radio Control

Vehicle Simulation States

Test Cases | Vehicle Parameters | 3D Vision

Sensor Signals

Con

trol

Sig

na

ls

External Device Signals

Control Commands

Controller States

Fig. 9. Hardware composition of the real-time HIL test platform

Four videos have been published to demonstrate the devel-opment, experimental verification, and application process inthis section for the proposed platform. The first video givesan overall introduction of the proposed platform along withseveral applications on different multicopter control systems:

https://youtu.be/nXAoLdPzz IThe second video presents several demos of applying theproposed HIL platform to different types of vehicles (cars,copters, aircraft) with multi-vehicle traffic environment simu-lation:

https://youtu.be/xQUnkqH29qUThe third video presents the automatic fault injection, safetytesting, and safety assessment demos:

https://youtu.be/MHieyE3hbHYThe fourth video introduces the MBD modeling and devel-opment process of the platform with experiments to quantita-tively verify its simulation credibility:

https://youtu.be/ChNtkb5rrQsWe have also published the MATLAB source code and thedetailed modeling tutorial to Github:

https://github.com/XunhuaDai/CopterSimReaders can use it to rapidly develop SIL or HIL simulationsystems for different types of unmanned vehicles by modifyingthe aerodynamic model and the actuator model.

A. Platform Implementation

1) Hardware Composition: Based on the hardware struc-ture presented in Fig. 4, a real-time HIL test platform is de-veloped as shown in Fig. 9. The real-time simulation computeradopted in the platform is the NIr/PXI simulator with CPUboard: PXIe-8133 (Intel Core I7 Processor, PharLap ETS Real-Time System) and FPGA I/O Module: PXIe-7846R. The hostcomputer is a high-performance workstation PC with profes-sional GPU. The control system is Pixhawk autopilot, whichis one of the most popular open-source autopilot hardwaresystems for small unmanned vehicles. All the onboard sensors(IMU, magnetometer, barometer, etc.) and external sensors(GPS, rangefinder, camera, etc.) of the Pixhawk hardware areblocked, and the sensor pins are reconnected to the FPGA

10

I/Os to receive the simulated sensor signals through interfacesincluding SPI, PWM, I2C, UART, UBX, etc. In the real-time simulation computer, the update frequency of the vehiclesimulation model is up to 5 kHz, and the update frequencyof sensor simulation model is up to 100 MHz, whose perfor-mance is fast enough for HIL simulations of most commercialcontrol systems. The communication among between the hostcomputer and the real-time simulation computer is realized bynetwork cables with TCP and UDP protocols.

2) Software Development: The simulation software of thereal-time HIL test platform is developed based on the MBDmethod in Section III-B. The Simulink is selected to developthe vehicle simulation model because it is the most profes-sional and widely used software for vehicle dynamic systemdevelopment; the LabVIEW is selected to develop the sensorsimulation model because it is efficient and convenient in real-time simulation system development; the UE4 (version: 4.22)is selected to develop the 3D environment model because itis one of the most realistic 3D engines in the development ofsimulation systems, games and VR systems. The Simulink,LabVIEW, and UE4 developing environments all provideconvenient modular visual programming environments (seeFig. 10) and automatic code generation technology for thedevelopment of simulation systems, which are perfect forthe implementation of the model-based design method. Afterthe simulation models are all developed, the code generationand deployment framework in Fig. 8 is applied to deploythe simulation software to the real-time HIL test platformpresented in Fig. 9.

3) High-fidelity 3D Simulation Environment: The fidelity ofthe 3D simulation model is important for testing the vision-related functions of control systems, such as visual data pro-cessing, obstacle avoidance, safety decision-making, etc. WithUE4, it is easy to develop high-fidelity 3D scenes for differenttypes of vehicle in different environments. For example, asshown in Fig. 11, we have developed several 3D simulationscenes in UE4 for the HIL test platform. According to thecomparisons with experiments, the display effect presented inFig. 11 has been realistic enough for most unmanned vehiclesystems to simulate the real indoor or outdoor scenes.

B. Experiments and Verification

1) Platform Feature Verification: The Pixhawk autopilotsupports for running different types of flight control softwaresystems (e.g., PX4, Ardupilot, and other embedded controlsoftware systems) in it, and also supports for controllingdifferent types of unmanned vehicles (e.g., multicopters, smallcars, and fixed-wing aircraft). Besides, the Pixhawk autopilothas a series of available hardware configurations for certainperformance requirements, such as Pixhawk 1, Pixhawk 4, etc.To verify the proposed modeling method and test platform, wefirst apply the proposed unified modeling method in SectionII to develop the simulation models for different types of un-manned vehicles, including multicopters, small cars, and fixed-wing aircraft. Then, these simulation models are deployed tothe HIL test platform with the MBD framework presentedin Section III. Finally, a series of tests are performed for

ControlInput

Terrain & Obstacle Input

Actuator Subsystem

Force & Moment

Subsystem

Environment Subsystem

Fault Injection

Subsystem

Vehicle Body

Subsystem

OutputInterface

Subsystem

(a) Simulink graphical programming for vehicle model

(b) LabVIEW graphical programming for sensor model

(c) UE4 graphical programming for 3D environment Model

Vehicle States

GPS Chip

Signals

GPS Data Subsystem

Fig. 10. Modular visual programming environments in Simulink, LabVIEW,and UE4.

(a) Plateau ice lake scene for quadcopters (b) Forest scene for hexacopters

(c) Neighborhood street scene for cars (d) Mountain scene for fixed-wing aircraft

Fig. 11. 3D simulation scenes developed by UE4 for different types ofunmanned vehicles.

the Pixhawk systems with various combinations of hardwareconfigurations, software systems, and vehicle types. In ourexperimental tests, the four advantages of the proposed method(including extensibility, comprehensiveness, verification, andstandardization) concluded in Section III-B are verified withthe proposed modeling method and simulation test platform.

2) Experimental Setup: For simplicity, a quadcopter controlsystem will be selected as the representative tested objectin this section to perform quantitative verification for theproposed methods. The control system is the most widelyused open-source autopilot system for small-scale unmanned

11

(a) F450 quadcopter components (b) Indoor test equipment for attitude response identification

(c) Bifilar pendulum method for moment of inertia measurement

(d) Measuring devices for propulsion systems of small aircraft

Fig. 12. Verification equipment for the proposed HIL test platform.

vehicles, and the detailed configuration is Pixhawk 1 hardware(MCU: STM32F427, sensor: MPU6000, MS5611, LSM303D,L3GD20H, Ublox-M8N, etc.) with the PX4 control software.The quadcopter UAV is selected because it is the most repre-sentative unmanned vehicle type that covers the model char-acteristics (e.g., aerodynamics, ground collision, kinematics,and dynamics) and operating environments (e.g., near-ground,mid-air, indoor, outdoor, hovering flight, and forward flight)of most unmanned vehicles.

The experimental setup is presented in Fig. 12, wherean F450 quadcopter airframe (diagonal length: 450mm,weight: 1.4kg, propulsion system: DJI E310, battery: LiPo3S 4000mAh) is selected as the experimental subject whosecomponent diagram is shown in Fig. 12(a). In order to testthe attitude dynamics and aerodynamics of the quadcopter,an indoor test bench (see Fig. 12b) is developed with thequadcopter fixed to a stiff stick (through the center of mass)with smooth bearings to minimize friction. The quadcopteris free to smoothly rotating along one axis, which makesit possible to perform sweep-frequency testing for systemidentification and uniform rotation testing for roll dampingcoefficient measurement. Figs. (c)(d) present the test benchesto measure the moment of inertia and the propulsion systemparameters of small-scale unmanned vehicles, and the detailedmeasuring methods can be found in our previous work [27,pp. 121-143]. Besides, lots of outdoor flight tests are alsoperformed to obtain the aerodynamic coefficients of the testedquadcopter and verify the platform simulation results withactual flight results. The experimental process and resultshave also been presented in the video mentioned in the lastsubsection.

3) Simulation validation: A series of comparative experi-ments are performed with the experimental test benches (seeFig. 12) and the HIL simulation test platform (see Fig. 9)

Acc

ele

rati

on

z-a

xis

(m

/s2)

Ang

ula

r sp

eed x

-ax

is (r

ad/s

)

Time (s) Time (s)

(a) Accelerometer noise&vibration test (b) Gyroscope noise&vibration test

Control SignalMoto

r st

eady s

peed

(ra

d/s

)

Motor speed (rad/s)

Pro

pell

er

thru

st (

N)

(c) Motor&ESC input/output model test (d) Propeller aerodynamic model test

Mag

nit

ud

e (d

B)

(e) Bode magnitude plot for pitch

channel sweep frequency test

Ph

ase (

deg)

Experimental results Simulation results

Frequency (rad/s) Frequency (rad/s)

(f) Bode phase plot for pitch channel

sweep frequency test

Model: y = 1148x - 141.4 Model: y =0.000011x2

Fig. 13. Comparison experiments to verify the simulation accuracy of theHIL test platform.

to verify proposed modeling method and test platform, andseveral representative comparison results are presented inFig. 13. In Figs. 13(a)(b), the accelerometer and gyroscopedata obtained from the IMU sensor (MPU 6000 in Pix-hawk) are presented to verify the sensor modeling methodsin Section II-D. In the simulation and the experiment, themotor throttle is stepped to 50% at 21.5s to test the vi-bration effect caused by the high-speed rotating motors andpropellers. In Figs. 13(c)(d), the propeller actuator modelingmethod in Section II-B is verified with the propulsion systemtest benches in Fig. 12(d). Besides, sweep frequency testsare performed for the quadcopter pitch channel with the testbench presented in Fig. 12(c), and the magnitude and phasecurves of the Bode plots obtained by the CIFERr software[32] are presented in Figs. 13(e)(f), respectively. From thecomparative results in Fig. 13, the following conclusions canbe obtained. (i) The FPGA-based HIL test platform allowsusing the same control system in both experiments and sim-ulations, which minimizes the disturbing factors for resultanalysis and makes it convenient to acquire control systemdata for comparison and assessment. (ii) From the perspectiveof qualitative analysis, the obtained HIL test results are closeto the experimental results in Fig. 13 from both time-domain

12

Calculate

Total Weig

kg

Frame Size mm

Altitude

m

Aero Design

medium

Min. Battery Capacity

15%

Max. Takeoff Throttle

85%

FCU Max. Tilt Limit

No Limit

FCU & Attaches Current

A

Motor Brand: Model:

Propeller Brand Model:

ESC Brand Model

Battery Brand Model

Quadcopter- mm- kg- min-

Mathematical Model (Doc)

0.5

Air Temperature

25 °C

CFP 9x4.7

E310

DJI 2312 KV960

DJI

DJI

ACE LiPo 3S-1 1.1V-25C-4000mAh

: g = 9.8 m/s²

: Jxx = 1.704e-2 kg.m²

Jyy = 1.704e-2 kg.m²

Jzz = 3.176e-2 kg.m²

: d = 0.225 m

Propeller Thrust Coef. (Tp/!²) : CT = 9.286e-6 N/(rad/s)²

Propeller Moment Coef. (Mp/!²) : CM = 1.189e-7 N.m/(rad/s)²

Throttle " to Motor steady speed!ss

( !ss=CR*"+!b )

Motor-Propeller Inertia

: CR = 691.33 rad/s

: !b = 162.59 rad/s

: Jm = 8.18e-5 kg.m²

: Tm = 0.0119 sMotor Response Time Constant

Air-Drag D Coef. (D=Cd*V²) : Cd = 6.697e-2 N/(m/s)²

Air-Torque M Coef. (M=Cm*w²) : Cm = 9.174e-3 N.m/(rad/s)²

Multicopter Mass

Acceleration of Gravity

MultiCopter Inertia Matrix

J=diag(Jxx, Jyy, Jzz)

Distance of Motor to Center

: m = 1.4 kg

Fig. 14. Screenshot of the online toolbox flyeval.com.

and frequency-domain aspects, which verify the simulationaccuracy and credibility of the proposed modeling methodand test platform. (iii) The quantitative simulation credibilityassessment method proposed in our previous work [24] isapplied in this paper to assess and improve the simulationcredibility of the HIL platform, which ensures that a highmatching degree (the credibility index in [24]) larger than 90%(where 60% presents the minimum accuracy requirement, and100% presents a perfect match) is obtained by analyzing theresults between the test platform and real experimental systemfrom the quantitative perspective.

C. Method Applications

This subsection presents several successful applications withthe proposed test platform to increase the development, testing,and validation efficiency of multicopters.

1) Rapid Prototyping: To take maximum advantage ofthe proposed test platform with MBD method, a componentmodel database is developed for the rapid development ofelectric multicopters. The database covers the common prod-ucts on the market for multicopter propulsion systems withmodel parameters obtained by their product specificationsand experimental data. With this model database, an onlinetoolbox is released (URL: https://flyeval.com/) based on ourprevious studies [41], [42], [43] for the automatic design andperformance estimation of multicopter UAVs, and a screenshotof the online toolbox is presented in Fig. 14. Users can selectcomponent products from the database to quickly assemblea multicopter to estimate its flight performance and modelparameters. The toolbox has been released for more than twoyears, and the user feedback indicates that it can significantlyimprove the multicopter model development efficiency withdecent simulation accuracy.

(a) Real F450 Quadcopter

Time (s)

Pit

ch a

ngle

ste

p r

esp

onse

(ra

d)

(c) High-precision Model

(b) Website Estimated Model

Fig. 15. Level flight testing results for simulation validation.

The level flight testing is an effective way to assess thesimulation accuracy of the whole simulation model becauseit is the joint effect of all component models in the HILtest platform. Figs. 15(a)(b)(c) present the representative levelflight testing results from a real quadcopter, an estimatedmodel from the toolbox in Fig. 14, and a high-precision modelcalibrated with experimental data in Fig. 13. The quadcopteris commanded to step from hovering mode to level flightmode (the desired pitch angle is 15 degree, i.e., 0.262 rad)at 0.2s. The curve data are collected from the log data inPixhawk after flight tests. It can be observed from Fig. 15that the high-precision model curve almost coincides withthe real experimental curve, and the estimated model curve isslightly different from the experimental curve, but the error isacceptable because it reveals most dynamic and aerodynamiccharacteristics of the quadcopter.

2) Algorithm Comparison: Another important advantageof the proposed HIL test platform is that it can obtain thetrue states of the simulated vehicle, which is significantfor comparing performance difference of control algorithms.In Fig. 16, two simulations are performed on the HIL testplatform with two different estimation filter algorithms inthe Pixhawk autopilot. They are the Extended Kalman filteralgorithm in Fig. 16(a) and the complementary filter algorithmin Fig. 16(b), respectively. It can be observed from the resultin Fig. 16 that the extended Kalman filter has a better estima-tion effect than complementary, which is consistent with thetheoretical analysis. This conclusion is hard to obtain throughexperiments because a higher more precise external measuringdevices (e.g., differential GPS or visual positioning systemswith centimeter-level precision) is required to measure the truestates of the vehicle. These external measuring devices areusually expensive and restrained. For example, the differentialGPS is easy to be disturbed by flight environment factors, andits data frequency is too low (usually 5Hz), and the visualpositioning system cannot be used outdoors. Therefore, theproposed test platform is a better way to acquire the truestates of the vehicle for comparative analysis and performanceassessment.

3) Normal Testing: The proposed HIL test platform makesit possible to comprehensively test the control system onlywith computers, which is significant in reducing cost andtime relative to outdoor experiments. Fig. 17 presents anautonomous mission flight test with the proposed HIL test

13

Estimation value in PixhawkTrue value from HIL system

Time (s) Time (s)

Veh

icle

po

siti

on

x-a

ixs

(m)

Veh

icle

po

siti

on

x-a

ixs

(m)

(a) Flight evaluation for Pixhawk with Extended Kalman filter

(b) Flight evaluation for Pixhawk with complementary filter

Fig. 16. Comparing estimation performance of different filter algorithms inturning flight stage.

(c) Automatic Mission Flight with Ground Control Program

(b) Front Camera View from 3D Engine(a) Simulation Computer & Pixhawk

Fig. 17. Autonomous mission flight testing with the proposed HIL testplatform.

platform, which usually should be performed by outdoor flighttests in traditional test methods. Besides, more comparativesimulation tests and experiment demonstrate that all flight tests(indoor or outdoor, manual or automatic) can be tested on theHIL test platform if the vehicle modeling is comprehensiveand accurate enough.

4) Automatic Safety Testing: With the fault modes (e.g.,sensor failure, wind disturbances, and motor fault) being wellmodeled, the proposed HIL test platform can also be appliedto perform automatic safety testing for control systems. Wehave developed an automatic safety testing framework (seeFig. 18) based on the HIL test platform. First, a test casedatabase should be developed to store the vehicle commandscript, the fault injection triggering time, the test stop time, andthe desired vehicle flight performance. In each testing case, thecontrol system and the simulation models are automaticallyreinitialized to default states, and then the vehicle controlsystem automatically controls the vehicle model in the HILtest platform to the desired state. At this moment, a fault caseis injected to the HIL test platform to simulate a vehicle failure.After a period of time, the vehicle and control system states areautomatically analyzed and recorded to a report. Finally, the

Test Case Database

Control System

Sensor Data& Commands

Real-time Simulation Computer

Control Signals& State Info

Evaluation

Report

Evaluation Program

Test Case

Test Results

Next Test Case

Fig. 18. Automatic safety testing and validation framework.

next testing case is tested in the same way until all the testingcases are tested and evaluated. Some automatic test exampleshave been presented in the attached video in the previoussubsection. The test results demonstrate that the platform cansimulate the vehicle failure situations realistically, and the testefficiency is significantly improved with the automatic safetytest method.

V. CONCLUSION AND FUTURE WORK

This paper presents a unified simulation and test platform,aiming to significantly improve the development speed andsafety level of unmanned vehicle control systems. A unifiedmodular modeling framework is proposed by abstracting thecommon features of different types of unmanned vehicles. Theapplication examples demonstrate this framework is efficientin developing a vehicle simulation model with compatibilityfor the future safety assessment and certification standards.Another key problem solved in the paper is to develop a HILsimulation test platform to ensure simulation credibility. Withthe model-based design method, the developing process ofthe simulation software can be automatized and standardized,which ensures different developers can obtain the same simula-tion software with credibility guaranteed by the automatic codegeneration tools. With the FPGA-based real-time hardware-in-the-loop simulation technology, the operating environmentof the control algorithms is guaranteed by using the samecontrol system in both experiments and simulations. After thecredibility of the simulation test platform is well guaranteed,we can focus on verifying and validating the simulation modelswith quantitative assessment method proposed in our previouswork. The proposed test platform is applied to a multicoptercontrol system, where the accuracy and fidelity of the simula-tion testing results are verified by comparing with experiments.The successful applications present the advantages in the mul-ticopter rapid prototyping, estimation algorithm verification,autonomous flight testing, and automatic safety testing withautomatic fault injection and result evaluation of unmannedvehicles.

Since the test platform can provide high-fidelity and cred-ibility simulation results, it will help to improve the trainingefficiency of artificial intelligence algorithms. Besides, the air-worthiness for unmanned aerial vehicles require more formal,quantitative, and efficient testing methods to assess the safety

14

level, and we will apply the proposed test platform for thesafety assessment of unmanned vehicle systems.

REFERENCES

[1] H. N. Nguyen, S. Park, J. Park, and D. Lee, “A novel robotic platformfor aerial manipulation using quadrotors as rotating thrust generators,”IEEE Transactions on Robotics, vol. 34, no. 2, pp. 353–369, 2018.

[2] O. Goury and C. Duriez, “Fast, generic, and reliable control and sim-ulation of soft robots using model order reduction,” IEEE Transactionson Robotics, vol. 34, no. 6, pp. 1565–1576, 2018.

[3] H. Lipson and M. Kurman, Driverless: intelligent cars and the roadahead. Mit Press, 2016.

[4] R. Mebarki, V. Lippiello, and B. Siciliano, “Nonlinear visual control ofunmanned aerial vehicles in gps-denied environments,” IEEE Transac-tions on Robotics, vol. 31, no. 4, pp. 1004–1017, 2015.

[5] K. Cole and A. M. Wickenheiser, “Reactive trajectory generation formultiple vehicles in unknown environments with wind disturbances,”IEEE Transactions on Robotics, vol. 34, no. 5, pp. 1333–1348, 2018.

[6] C. M. Belcastro, D. H. Klyde, M. J. Logan, R. L. Newman, andJ. V. Foster, “Experimental flight testing for assessing the safety ofunmanned aircraft system safety-critical operations,” in 17th AIAAAviation Technology, Integration, and Operations Conference. AIAA2017-3274, 2017.

[7] S. S. Noureen, V. Roy, and S. B. Bayne, “An overall study of a real-time simulator and application of RT-LAB using MATLAB simpower-systems,” in 2017 IEEE Green Energy and Smart Systems Conference,2018, pp. 1–5.

[8] M. B. Tischler, “System identification methods for aircraft flight controldevelopment and validation,” in Advances In Aircraft Flight Control,2018, pp. 35–69.

[9] O. Lisagor, T. Kelly, and R. Niu, “Model-based safety assessment:Review of the discipline and its challenges,” ICRMS’2011 - Safety First,Reliability Primary: Proceedings of 2011 9th International Conferenceon Reliability, Maintainability and Safety, pp. 625–632, 2011.

[10] D. Jung and P. Tsiotras, “Modeling and hardware-in-the-loop simulationfor a small unmanned aerial vehicle,” in AIAA Infotech@Aerospace 2007Conference and Exhibit. AIAA 2018-2768, May. 2007.

[11] S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: High-fidelity visualand physical simulation for autonomous vehicles,” in Field and servicerobotics, 2018, pp. 621–635.

[12] D. B. Worth, B. G. Woolley, and D. D. Hodson, “SwarmSim: a frame-work for modeling swarming unmanned aerial vehicles using hardware-in-the-loop,” The Journal of Defense Modeling and Simulation, pp. 1–20,2017.

[13] A. I. Hentati, L. Krichen, M. Fourati, and L. C. Fourati, “Simulationtools, environments and frameworks for UAV systems performanceanalysis,” 2018 14th International Wireless Communications & MobileComputing Conference (IWCMC), pp. 1495–1500, 2018.

[14] P. Sarhadi and S. Yousefpour, “State of the art: hardware in the loopmodeling and simulation with its applications in design, developmentand implementation of system and control software,” InternationalJournal of Dynamics and Control, vol. 3, no. 4, pp. 470–479, 2015.

[15] J. R. Sylnice and G. H. Alferez, “Dynamic evolution of simulatedautonomous cars in the open world through tactics,” in Proceedingsof the Future Technologies Conference, 2018, pp. 257–268.

[16] Y. Chen, S. Chen, T. Zhang, S. Zhang, and N. Zheng, “Autonomousvehicle testing and validation platform: Integrated simulation systemwith hardware in the loop,” in 2018 IEEE Intelligent Vehicles Symposium(IV), 2018, pp. 949–956.

[17] A. Franchi, C. Secchi, H. I. Son, H. H. Bulthoff, and P. R. Giordano,“Bilateral teleoperation of groups of mobile robots with time-varyingtopology,” IEEE Transactions on Robotics, vol. 28, no. 5, pp. 1019–1033, 2012.

[18] Y. Liu, J. M. Montenbruck, D. Zelazo, M. Odelga, S. Rajappa, H. H.Bulthoff, F. Allgower, and A. Zell, “A distributed control approach toformation balancing and maneuvering of multiple multirotor UAVs,”IEEE Transactions on Robotics, vol. 34, no. 4, pp. 870–882, 2018.

[19] W. Li, C. W. Pan, R. Zhang, J. P. Ren, Y. X. Ma, J. Fang, F. L. Yan, Q. C.Geng, X. Y. Huang, H. J. Gong, W. W. Xu, G. P. Wang, D. Manocha,and R. G. Yang, “AADS: Augmented autonomous driving simulationusing data-driven algorithms,” Science Robotics, vol. 4, no. 28, 2019.

[20] H. Saad, T. Ould-Bachir, J. Mahseredjian, C. Dufour, S. Dennetiere, andS. Nguefeu, “Real-time simulation of MMCs using CPU and FPGA,”IEEE Transactions on Power Electronics, vol. 30, no. 1, pp. 259–267,2015.

[21] S. Mikkili, A. K. Panda, and J. Prattipati, “Review of real-time simulatorand the steps involved for implementation of a model from MAT-LAB/SIMULINK to real-time,” Journal of The Institution of Engineers(India): Series B, vol. 96, no. 2, pp. 179–196, 2015.

[22] U. B. Mehta, D. R. Eklund, V. J. Romero, J. A. Pearce, and N. S.Keim, “Simulation credibility: Advances in verification, validation, anduncertainty quantification,” NASA Ames Research Center, Moffett Field,CA United States, Tech. Rep. JANNAF/GL-2016-0001, Nov. 01, 2016.

[23] I. MathWorks, “Model-based design - MATLAB & Simulink,”https://www.mathworks.com/solutions/model-based-design.html,Accessed May 24, 2019.

[24] X. Dai, C. Ke, Q. Quan, and K.-Y. Cai, “Simulation credibilityassessment methodology with FPGA-based hardware-in-the-loopplatform,” arXiv e-prints, vol. arXiv:1907.03981, pp. 1–8, 2019.[Online]. Available: https://arxiv.org/abs/1907.03981

[25] A. M. Miller, R. Alvarez, and N. Hartman, “Towards an extendedmodel-based definition for the digital twin,” Computer-Aided Designand Applications, vol. 15, no. 6, pp. 880–891, 2018.

[26] B. L. Stevens and F. L. Lewis, Aircraft Control and Simulation 2ndEdition. Wiley, New Jersey, 2004.

[27] Q. Quan, Introduction to Multicopter Design and Control. Springer,Singapore, 2017.

[28] R. W. Smith, Department of Defense World Geodetic System 1984:its definition and relationships with local geodetic systems. DefenseMapping Agency, 1987.

[29] M. Cavcar, “The international standard atmosphere (ISA),” AnadoluUniversity, Turkey, vol. 30, p. 9, 2000.

[30] A. Chulliat, S. Macmillan, P. Alken, C. Beggan, M. Nair, B. Hamilton,A. Woods, V. Ridley, S. Maus, and A. Thomson, The US/UK worldmagnetic model for 2015-2020. BGS and NOAA, 2015.

[31] D. J. Moorhouse and R. J. Woodcock, “Background information and userguide for MIL-F-8785C, military specification-flying qualities of pilotedairplanes,” U.S. Air Force Wright Aeronautical Labs, Wright-PattersonAFB, OH, Tech. Rep. AFWAL-TR-81-3109, July. 1982.

[32] R. K. Remple and M. B. Tischler, Aircraft and rotorcraft systemidentification: engineering methods with flight-test examples. AmericanInstitute of Aeronautics and Astronautics, 2006.

[33] G. Cai, B. M. Chen, and T. H. Lee, Unmanned rotorcraft systems.Springer Science & Business Media, 2011.

[34] R. Rajamani, Vehicle dynamics and control, Second Edition. SpringerScience & Business Media, 2011.

[35] A. I. Hentati, L. Krichen, M. Fourati, and L. C. Fourati, “Simulationtools, environments and frameworks for UAV systems performanceanalysis,” 2018 14th International Wireless Communications and MobileComputing Conference, IWCMC 2018, pp. 1495–1500, 2018.

[36] R. W. Beard and T. W. McLain, Small unmanned aircraft: Theory andpractice. Princeton university press, 2012.

[37] M. Kucis and P. Zemcık, “Simulation of camera features,” Proceedingsof the 16th Central European Seminar on Computer Graphics, pp. 117–123, 2012.

[38] F. Leens, “An introduction to I2C and SPI protocols,” IEEE Instrumen-tation & Measurement Magazine, vol. 12, no. 1, pp. 8–13, 2009.

[39] N. El-Sheimy, H. Hou, and X. Niu, “Analysis and modeling of inertialsensors using allan variance,” IEEE Transactions on instrumentation andmeasurement, vol. 57, no. 1, pp. 140–149, 2008.

[40] S. Gage, “NASA HL-20 lifting body airframe modeled with Simulinkand the aerospace blockset,” https://www.mathworks.com/help/aeroblks/nasa-hl-20-lifting-body-airframe.html, Accessed May 24, 2019.

[41] D. Shi, X. Dai, X. Zhang, and Q. Quan, “A practical performanceevaluation method for electric multicopters,” IEEE/ASME Transactionson Mechatronics, vol. 22, no. 3, pp. 1337–1348, 2017.

[42] X. Dai, Q. Quan, J. Ren, and K.-Y. Cai, “An analytical designoptimization method for electric propulsion systems of multicopterUAVs with desired hovering endurance,” IEEE/ASME Transactions onMechatronics, vol. 24, no. 1, pp. 228–239, 2019.

[43] X. Dai, Q. Quan, J. Ren, and C. Kai-Yuan, “Efficiency optimization andcomponent selection for propulsion systems of electric multicopters,”IEEE Transactions on Industrial Electronics, vol. 66, no. 10, pp. 7800–7809, 2019.