a human centric framework for the analysis of automated ...€¦ · adepartment of transport &...

Delft University of Technology

A human centric framework for the analysis of automated driving systems based onMeaningful Human Control

Calvert, Simeon; Heikoop, Daniël; Mecacci, Giulio; van Arem, Bart

DOI10.1080/1463922X.2019.1697390Publication date2019Document VersionFinal published versionPublished inTheoretical Issues in Ergonomics Science

Citation (APA)Calvert, S., Heikoop, D., Mecacci, G., & van Arem, B. (2019). A human centric framework for the analysis ofautomated driving systems based on Meaningful Human Control. Theoretical Issues in ErgonomicsScience, 1-29. https://doi.org/10.1080/1463922X.2019.1697390

Important noteTo cite this publication, please use the final published version (if applicable).Please check the document version above.

CopyrightOther than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consentof the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Takedown policyPlease contact us and provide details if you believe this document breaches copyrights.We will remove access to the work immediately and investigate your claim.

This work is downloaded from Delft University of Technology.For technical reasons the number of authors shown on this cover page is limited to a maximum of 10.

https://doi.org/10.1080/1463922X.2019.1697390

TheoreTical issues in ergonomics science

A human centric framework for the analysis of automated driving systems based on meaningful human control

Simeon C. Calverta , Daniël D. Heikoopa , Giulio Mecaccib and Bart van Arema

aDepartment of Transport & Planning, Delft university of Technology, Delft, The netherlands;bsection of ethics and Philosophy of Technology, Delft university of Technology, Delft, The netherlands

ABSTRACTThe future adoption of automated vehicles poses many challenges, with one of the more important being the preservation of control over vehi-cles that are no longer (fully) operated by drivers. There is consensus that vehicles should not perform actions that are unacceptable to humans. In this paper, we introduce the concept of Meaningful Human Control (MHC) as a function of a framework of the Automated Driving System (ADS). This framework is constructed through the core compo-nents that make up the ADS, primarily considered within the categories of the vehicle and driver. Identification of these components and the chain of control allow traceability of MHC to be performed, and aids vehicle manufacturers, software developers, other vehicle component designers, and vehicle- and driver licensing authorities to address many challenges related to the design and preservation of human control in automated vehicles. Operationalisation of MHC is discussed in the paper including a suggested approach that should aid understanding and the application of the concept. Four application examples are given and recommendations are made in regard to vehicle design, human machine interaction, transition of control, driver training, vehicle approval, and other topics. The framework and presented concept also allow researchers to identify areas to perform more explicit and relevant research and develop models that can be applied to perform projec-tions of future impacts.

Relevance to human factors/Relevance to ergonomics theory

The preservation of control over vehicles that are no longer (fully) operated by drivers is of major importance and a highly relevant topic in human factors and ergonomics research. This paper introduces the concept of Meaningful Human Control (MHC) as a function of a framework of the Automated Driving System (ADS) to address some of these challenges. The framework is essential for the construction of the theory of MHC and the operation-alisation in various field connected to human factors, HMI, automated vehicle software design. This aids vehicle manufacturers, software developers, other vehicle component

© 2019 The author(s). Published by informa uK limited, trading as Taylor & Francis group

CONTACT simeon c. calvert [email protected] Department of Transport & Planning, Delft university of Technology, Po Box 5048, 2600gaDelft, The netherlands

https://doi.org/10.1080/1463922X.2019.1697390

This is an open access article distributed under the terms of the creative commons attribution-noncommercial-noDerivatives license (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way.

ARTICLE HISTORYReceived 25 October 2018Accepted 21 November 2019

KEYWORDSAutomated driving systems; meaningful human control; vehicle automation; human behaviour; vehicle control; traffic flow

show [zaq no="AQ33"]http://orcid.org/0000-0003-4379-072Xhttps://doi.org/10.1080/1463922X.2019.1697390http://www.tandfonline.comhttp://crossmark.crossref.org/dialog/?doi=10.1080/1463922X.2019.1697390&domain=pdf&date_stamp=2019-12-9http://creativecommons.org/licenses/by-nc-nd/4.0

2 S. C. CALVERT ET AL.

designers, and vehicle- and driver licensing authorities to address many challenges related to the design and preservation of human control in automated vehicles.

1. Introduction

The transition towards vehicle automation with increasingly more advanced driver assis-tance systems (ADAS) has reached a point where these systems are no longer merely assis-tance systems, but are able to take on certain driving tasks, such as complete lateral or longitudinal control of the vehicle. Initially, lower level automated vehicles (SAE level 1 or 2) execute certain driving tasks such as lane-keeping and/or car following, while a driver ultimately remains in control and is also responsible for monitoring the environment and should retake complete control if and when required (SAE 2018). As the transition towards higher levels of automation is made, a driver will increasingly have less operational control and their tasks will be more of a supervisory nature, focussed on monitoring the Automated Driving System (ADS)’s proper execution of the driving task, until in the highest levels of automation (SAE level 4 and 5), the driver is relieved from that monitoring task as well. Relinquishment of operational control and eventually supervisory control places high demands on the capability and design of a vehicle. These vehicles are expected to act in a ‘correct’ and sometimes ‘human-like’ manner (Wang et al. 2018). Achieving this is not a trivial task, as it is entangled in many ethical and practical questions, such as ‘is driver retraining required to drive automated vehicles?’, ‘what additional demands are required from vehicle authorities for vehicle type and design approval?’, ‘what are the design consid-erations for control?’ and ‘should we even be trying to make vehicles drive like humans?’. These are just a small selection of uncertainties to be addressed by vehicle designers and manufacturers, vehicle approval authorities, driver licencing authorities, highway author-ities, and governmental bodies, amongst others.

In 2015, influential scientists, entrepreneurs, policy-makers and Non-Governmental Organisations (NGO’s) called for a ban on ‘offensive autonomous weapons beyond mean-ingful human control’ (Future of Life Institute 2015). Although vehicles are not designed as weapons, they are systems that have a critical degree of danger if not correctly designed and controlled. Not only maintaining control, but also human control is vital in a traffic system that relies and will rely heavily on interactions between humans and technology, and in which we demand of automation that it also acts in a way deemed acceptable to humans. Therefore, consideration of what Meaningful Human Control (MHC) entails for automated driving is imperative, where MHC is generically described as: humans, not computers and their algorithms, should ultimately remain in control of, and thus morally responsible for relevant decisions about operations’. (paraphrase from Future of Life Institute (2015)).

At present, there is insufficient structure to many discussions on the wider implications of automated driving. At the heart of many of the discussions lies the problem of explicit and acceptable control, which we argue should be defined as Meaningful Human Control. In this paper, we introduce the concept of Meaningful Human Control (MHC) as a function of the core components framework of the ADS. To be able to understand and plot how MHC plays a role in the ADS, it is required to first define what the main core components of the ADS are and how they interact. This allows trackability and traceability, two main conditions of MHC, to be applied to the ADS (Santoni de Sio and Van den Hoven 2018). Broadly speaking, trackability is to have the ability to track human moral reasoning, and

THEORETICAL ISSuES IN ERGONOMICS SCIENCE 3

traceability the ability to understand and trace consequences and therefore the chain of control (this is explained in more detail in Section 3). The identification of the core com-ponents framework in the ADS is based on extensive literature review and the consideration of what influences control. The framework forms the basis to allow MHC to be applied in the development of ADS design, automated driving simulation design and various control- and ethical discussions, and use by vehicle- and driving licence authorities as they propose and test new avenues of approval, inspection and testing, and therefore forms a crucial contribution to the state of the art in the domain of ADS.

To this extent, we first describe in Section 2, the core components framework of ADS, which is based on a multi-domain literature review. Therein, a categorisation is made, distinguishing between driver, vehicle, infrastructure, and environment components. In Section 3 we discuss the influence and control of ADS in regard to MHC, both from the perspective of the driver and the vehicle’s ADCS. The role and importance of MHC in ADS is discussed, as well as the main conditions and opportunities of MHC through trackability and traceability. This sets the scene to allow further application of the MHC concept to a wide variety of areas. Section 4 describes consideration of MHC operationalisation and in Section 5, we give examples of areas in which the framework can be applied, and demon-strate its relevance based on three different use cases.

2. Core components framework of ADS

Identification of the chain of control in ADS and of the core components of the ADS is per-formed through extensive categorisation based on a literature review and knowledge of the main categories. Four categories are defined to aid the classification: Driver, Vehicle, Infrastructure and Environment. These are high level categories, which each contain a further classification of their underlying components. The core components are defined as the main components of importance for that category based on the current accepted state of the art. In some cases, choices are made where there is no clear consensus in literature, which allow the classification to be made and fit with the natural chain of control in ADS. The criteria for the in-/exclusion of certain components are further made based on potential interactibility and relevance to the overall framework, such that future connections can be made between the (sub)components. These choices in themselves are also made based on literature and expert knowledge, and are substantiated in the text. We have further classified ‘traffic’ as a concatena-tion of the Driver and Vehicle categories, as traffic is the ensemble of vehicles controlled by drivers (of the vehicles themselves) and vehicles, although time-dependant traffic-dynamic variables are not explicitly considered, such as speed, flow, etcetera. Also note that the type of control we consider in this paper is operational control of a vehicle. Of course, tactical and strategic control are important, but require different or wider sets of components (Michon 1979), and are not the initial focus of this framework. In the upcoming sections, we discuss the selection procedure of the (contents of the) four component categories in more detail.

2.1. Driver components

Traditionally, vehicles have been and still are controlled by a human driver. On a strategic, tactical and operational level, the human driver forms an inherent part of the driver-vehicle


Figure 1. Driver core components of control in aDs.

symbiosis, and therefore inevitably serves as a major component within the control loop (Michon 1979). A human (driver) can be said to have a certain set of (static) traits (physical, such as having eyes and hands, and mental, such as their personality), and to be in a certain (dynamic) state (e.g., being fatigued or stressed). This internal state is commonly expressed as the driver’s behaviour (Pentland and Liu 1999). By definition, driver behaviour has a direct influence on traffic flow and most of the resulting traffic flow phenomena. Driver behaviour is, however, a domain that is not an exact science, and many different interpre-tations and constructs exist to describe the driver system. In this paper we adhere to the core components of the driver that are involved with (driving with) ADS. For this reason, driver trait is considered next to driver state, which is expressed through behaviour, that can be dispersed into perceptual, cognitive and action skills (see e.g., Theologus and Fleishman (1971) and Chandrasiri, Nawa, and Ishii (2016)). When driving, these behavioural components are continuously revisited and revised based on new perceptions, decisions and actions. Attempts have been made to model how drivers are affected by vehicle auto-mation in terms of psychological constructs (Heikoop et al. 2016; Stanton and Young 2000), which allow us to make decisions in regard to the structure of the driver core components. The core components for the Driver are given in Figure 1 and further explained hereafter.

A driver’s trait or attribute is described as anything that defines one’s personality and physique. Personality traits are commonly categorised into five categories –the Big Five–, namely Extroversion, Agreeableness, Conscientiousness, Neuroticism, and Openness to Experience (Goldberg 1990). In the driving domain, much attention is paid to drivers’ proneness to aggression (Deffenbacher et al. 2001) or stress (Hill and Boyle 2007; Matthews 2002), as those trait factors are thought to play an important role in traffic safety (Matthews 2002; Lajunen, Parker, and Stradling 1998). With the introduction of automated driving, other trait factors, such as proneness to fatigue, have increased concerns among scientists, as that is thought to alter the driver’s state negatively (Körber and Bengler 2014; Matthews 2002). A commonly applied descriptor of behaviour is found in the Locus of control (LoC)


of a person (Rotter 1966). This entails the extent to which a person maintains a feeling of control (internal LoC) or lacks a feeling of control (external LoC) and can in turn influence various aspects of a drivers’ behaviour.

The driver’s state is described as the driver’s current condition, which can be either mental or physical and can be influenced by external factors, such as the complexity or duration of the task that is to be performed. The extent to which, and how fast the driver’s state is affected by these factors is dependent on the driver’s trait. An example of a driver’s time-spe-cific state could be tiredness (Körber et al. 2015), which would influence a driver’s mental workload and attention, and could lead to longer reaction times or deteriorated situation awareness. Also situational statuses such as intoxication influence a driver’s state combined with their sensitivity thereof from their personal traits (McMillen et al. 1992).

In driving, a driver makes use of their perception to observe and process external infor-mation in order to be able to act upon a given situation safely. Perception can be split into two processes; that of processing sensory input, and that connected to a person’s personal concepts and traits to process the information (Bernstein 2010). On the first level, a clas-sification is made between the types of sensing in: Visual, Auditory, Tactile, Olfactory, and Taste. The processing of these signals convenes either consciously or sub-/non-consciously, which indicates an explicit distinction between a process that is allowed to be cognitively processed in a slightly longer time or one that leads to a subconscious reaction without cognitive processing, often in a very short time span (LeDoux 1998; Wickens and Hollands 2000). For clarity, conscious perception could be the visual observation of a pedestrian approaching a zebra-crossing, while sub-conscious perception could lead to a reactive brak-ing manoeuvre for an emergency stop. In the second case, the driver has ‘no time to think’ and reacts intuitively. In the first case, a driver has time to process the information cogni-tively and make a decision to slow down and stop.

Cognition relates to the mental processing of input, and can be referred to as ‘thinking’ in layman’s terms. We make a distinction between the Information-analysis step and the Decision selection step (Parasuraman, Sheridan, and Wickens 2000). The expansive avail-ability of different constructs and cognitive models forces us to make a choice that is in agreement with literature, and serves our purpose in this research. We define the Cognitive process as the conscious interpretation, comprehension and projection of perceived input, which is supported by literature (Seeber 2011; Wickens et al. 2015). The following Decision process forms the connection between the Cognitive process and actions, in which a driver comes to a conclusion and sets the process to undertake the decision in motion. Decisions can be taken on different levels, with higher level decisions often resulting in a chain of multiple time-extended actions, while lower level operational decisions may result in a more immediate and limited set of actions.

2.2. Vehicle components

In the domain of automotive systems, at the generic component level we are considering here, there is a generally accepted classification, which can be subdivided into the subcat-egories: Sensing, Control and Actuation. In this contribution, we mainly reference from a small subset of standard concepts (Emadi 2005; Robert Bosch GmbH, Reif, and Dietsche 2014; Ibañez-Guzman et al. 2012; McKnight and Adams 1970) for the basic components


of classical automotive vehicles; however, many other works can be referenced in this regard. We also make a clear distinction between the manual vehicle systems and automated vehicle systems. See Figure 2 for the core vehicle components of the framework.

Manual and classical driving systems: In classical vehicles, the primary sensing systems and interfaces offer an overview of the current status of a vehicle. That could involve the dynamic performance of a vehicle, such as the speed or engine revolutions, or more static information, such as the oil level. Some information is used by the vehicle itself for stabil-isation, especially where advanced driver assistance systems are present (e.g., cruise control, ABS, etc.). However, most of the sensing is relayed to the driver, who then takes actions based on the vehicle sensing in combination with the driver’s perception of the environment and infrastructure to control the vehicle (Amditis, Lytrivis, and Portouli 2012). The primary control of the vehicle available to a driver exists of the pedals (brake, clutch, and accelerator), the steering wheel, and the gear selector. The driver’s control actions translate to physical vehicle movement through the sub-category level of actuation. These involve components related to the power plant of a vehicle (i.e., engine, etc.), drive train (i.e., transmission, driveshaft, etc.), steering system, and the braking system (Amditis, Lytrivis, and Portouli 2012; Pucher, Cachón, and Hable 2012). Each of the described systems and their components here may be electrical, pneumatic, hydraulic, thermal, or of a different type (Pucher, Cachón, and Hable 2012). The type of sensing, control, or actuation does not significantly affect the classification (Amditis, Lytrivis, and Portouli 2012). Also note that we describe primary components here keyed towards the control of movement and therefore exclude secondary components, such as climate control, lighting, etcetera.

Automated driving systems: For vehicles with at least some level or type of automated driving systems (i.e., SAE level 1), additional components are present, and the chain of control and interaction between components can be different. Firstly, additional sensors will often be present in vehicles with automation. These will mainly fall under the classification of

Figure 2. Vehicle core components of control in aDs (red is manual control, blue is automated control).


perception sensors, which can assist or take-over certain monitoring tasks from a driver, and include the likes of radar, laser, lidar, etc. (Amditis, Lytrivis, and Portouli 2012; Ibañez-Guzman et al. 2012; Pendleton et al. 2017). Also for higher level automation and cooperative automation, virtual sensors may be present to aid positioning and external information generation (Andrews 2012). Automated control can relate to any level of vehicle automation, from simple ADAS to full automation. Automated control is fed by all relevant sensing components, and can give feedback and interact with human control through human-ma-chine interfaces (HMI). A further distinction is sometimes made between automation hard-ware and software (Pendleton et al. 2017; SAE 2018), where the software is primarily present in the control subcategory, and the hardware in the sensing and actuation. The actuation from automated control is in practice no different from that of manual control on the generic level (Amditis, Lytrivis, and Portouli 2012), although on a more detailed level, there are additional connections from the control to the underlying subcomponents that are not shown in Figure 2.

2.3. Infrastructure components

Although the core of control in a driving system lies with the driver and the vehicle, the surrounding infrastructure and environment have a large influence on the ADS perfor-mance, and will partially dictate the chain of control. For this reason, it is important to also consider the core components of infrastructure and environment within the framework. We start with a description of the core components of road infrastructure for which an overview is given in Figure 3.

With increasing digitalisation of society, and consequentially road infrastructure com-ponents, a common and logical distinction must be made between physical and digital subcategories of road infrastructure (Farah et al. 2018). The digital infrastructure refers to systems that involve information technology (both hard- and software), while the physical infrastructure consists of the more traditional infrastructure (SAE 2018).

The structural design of physical infrastructure focusses on the physical materials and how they are composed to create infrastructure. In the first instance, this is the ground bed on which the road infrastructure is built, which could also include tunnels and bridges. The

Figure 3. infrastructure and environment core components of control in aDs.


structure of the road surface and the underlying layers is considered, which also includes drainage and other services. A final subcomponent is the roadside features, which includes the like of crash barriers, hard shoulders, and artificial lighting. The physical infrastructure contains three main areas of components, namely the Geometric design, Structural design, and Signage and control (McKnight and Adams 1970). The geometric design of infrastructure considers the design of how infrastructure can be used by vehicles, and sets the limitations for vehicles. This starts with the alignment of the infrastructure, both horizontal and vertical. Classification is used to determine the level of use and purpose, which determines the number of carriageways and lanes and their characteristics, how these interact with other road sec-tions, such as junctions or weaving sections, and access restrictions. Although much signage could be considered as road-side infrastructure, we choose to group it under the subcategory ‘Signage and Control’, as this group is an important one for interaction with automated and cooperative vehicles (Roncoli, Papageorgiou, and Papamichail 2015). Road signs are a main component of signage, and include mandatory, warning, and informative signage of different types, which can also be either static or dynamic. Especially dynamic signage can be used for traffic control, through the likes of Variable Message Signs (VMS) of traffic lights (Tsugawa et al. 2001). Lane markings are also a method of control over vehicles. Traditionally these have been permanent and static; however, there are increasingly more options to use dynamic road markings. Automated vehicles also have the potential to indirectly influence the physical infrastructure design. The initial starting point is that automated vehicles should adapt to current infrastructure, while future roads may be designed in such a way that they are opti-mised for the use of automated vehicles (Ghosh and Lee 2010).

Digital infrastructure has taken leaps forward in past decades and is seen as vital for vehicle automation and cooperation (Sanchez, Blanco, and Diez 2016). Extensive research exists on digital infrastructure which goes beyond the scope of this paper; see for example Farah et al. (2018). Here we merely highlight the main components to aid further discussion and asso-ciation with other categories of vehicle automation and control. Main components of the digital infrastructure include sensing, mapping, connectivity & communication protocols and systems, and the underlying services (Sanchez, Blanco, and Diez 2016). Sensing is a more traditional part of the digital infrastructure, where cameras and induction loops have been present for a long time, increasingly more innovative sensing options are coming into exis-tence, with floating devices offering a major opportunity, through GPS, Bluetooth, etcetera (Rebsamen et al. 2012). Digital mapping is a crucial component for many systems, and certainly for automated vehicles, both on a micro-as well as a macro-level. For communica-tion, cellular networks (4G, 5G) and WiFi-p are currently the state-of-the-art, but with much research ongoing, further development should be expected. These will allow for all types of communication and connectivity: V2V, V2X, V2I and I2V (Hayeri, Hendrickson, and Biehler 2015). Data management in itself is not a component of the system, but is required over all the described digital systems to ensure safety and their proper use and operation.

2.4. Environment components

To complete the framework, we discuss the main environmental components. These com-ponents are themselves not related to ADS development, but do have a strong influence on the systems, and therefore need to be included to allow a complete picture to be constructed


of relevant components that influence ADS design and control. The environment category of the framework is also shown in Figure 3.

Weather is an example of one variable that has a strong exogenous influence on vehicles and traffic flow. This influence applies to both the infrastructure and the perception ability of a driver or ADS. Precipitation and temperature extremes influence the quality of the road surface and can lead to reduced vehicle performance due to limitations in the resistance offered by the road surface. Extreme temperatures can also affect electrical and digital systems in vehicles in some cases. Wind produces a physical force on a vehicle, while cloud and mist directly affect general visibility and are therefore also mentioned in the visibility category (Hamdar and Schorr 2013). Both human drivers and perception sensors can be adversely affected by poor visibility. These conditions can be cyclical due to the time of day (ToD), day of the week (DoW, e.g., due to smog), or week of the year (WoY), or can be permanent due to local conditions surrounding a road, such as in tunnels or sunken car-riageways. While some perception sensors are not overly affected by poor visibility (e.g. Lidar), others are and will reduce the overall level of perception. Finally, the geographic and demographic surroundings of a road can play a role (McKnight and Adams 1970). On one hand, this can physically influence a vehicle through certain infrastructure character-istics, for example change of gradient on hills, or reduced grip due to dirt on the road surface. On the other hand, the traffic composition is affected by different local characteristics. In a busy urban area, one may expect a high degree of commuting traffic, while in a more industrial area, a larger number of trucks may be expected. While all of the components in themselves do not need to be major issues, their effects on a human driver and on ADS are significant and should be considered when designing ADS and investigating the quality and ability to maintain and assert control.

3. Influence and control of automated driving systems

Investigating how control is influenced in ADS is not straightforward and certainly not generic, as there are different design approaches that can be taken and different levels of automation which can be setup in different ways. Therefore, it is necessary to take a tran-sitional view of the influence on, and the control of, ADS dependent on the level of auto-mation. The first subsection here considers the various levels of automation in regard to the core components, while in the second subsection we consider the chain of control and what this means for Meaningful Human Control. Again, the main focus here is in regard to operational control of a vehicle, rather than tactical or strategic.

3.1. Transition of monitoring and control

Current classification of automated vehicles defines three main areas of control, generically classified as operation, monitoring and fall-back. The SAE (2018) classification describes operation as ‘execution of steering and acceleration’, monitoring ‘of the driving environment’ and a fall-back for ‘performance of dynamic driving tasks’, as well as giving a system design domain. In fully manual vehicles, the driver is directly responsible for all three aspects of operation, monitoring and fall-back; they must observe the surrounding infrastructure, environment, and their vehicles’ own sensor feedback, make a perception thereof, process


(a)

Figure 4. a) influence of control for low automation sae l0-l1. red is human control and blue is aDs control. b) influence of control for medium automation sae l2. red is human control and blue is aDs con-trol. c) influence of control for high automation sae l4-l5. red is human control and blue is aDs control.


(b)

(c)

Figure 4. continued


the information, and take actions based on the decisions that follow their cognitive pro-cessing. This process is shown in regard to the core components in Figure 4, where red shows connections to the manual driving task. The figure shows a strong manual influence, while the automated control (in blue) is (nearly) non-existent. In the vehicle category, the perception and virtual sensors, and the automated driving system also do not play a role for manual driving. As the level of automation rises to SAE level 1, the main control of operation, monitoring, and fall-back remain with the driver; however, certain tasks are aided by the ADS, and the automated components of the vehicle will start to play a more prominent role.

The intermediate levels of vehicle automation (i.e., SAE level 2–3) see a shift occur from human operation to automated operation, while the driver is still responsible for some or most of the monitoring tasks, and certainly as fall-back (see Figure 4b). The infrastructure and environment are observed and perceived by both the driver and the vehicle’s perception sensors, while much of the control transitions from the driver to the ADS. The level of interaction between driver and vehicle also fades in comparison to manual driving. For higher level vehicle automation (i.e., SAE level 4–5), it is now the vehicle’s ADS that is responsible for operation, monitoring, and even fall-back. The driver plays a very limited role in the chain of operational control, and the interaction with the vehicle is minimal, which is seen in Figure 4c by near non-existent red and blue arrows indicating that the ADS is in operational control.

The main control concerns occur especially for the intermediate levels of automation, where the question of who is in (meaningful) control arises. From the perspective of the vehicle’s ADS, control lies within the technical capabilities of the automated driving system. This can range from the longitudinal driving task in SAE level 1 ACC systems, up to the entire dynamic driving task in its operational design domain for highly automated driving. One of the main control issues when a transfer of control to the human driver is required is that of regaining situation awareness. Situation awareness requires perception, compre-hension, and projection (Endsley 1995) and will also differ dependant on the level of auto-mation. With low level automation (i.e. SAE L1), this may only go as far as the leading vehicles and the road markings, while highly automated systems will perceive a much greater area and level of detail. However, nominal situation awareness for a human driver is deemed to remain at a steady level for all the lower three SAE levels of driving, as they are ultimately expected to take over control whenever necessary, and therefore are also expected to monitor the road ahead, the vehicle’s surroundings, the status of the ADS, and several other things.

On the other hand, the driver’s driving tasks rapidly diminish as automation increases. Thus, as physical operational control is transferred from driver to ADS, a lower cog-nitive demand is placed on a driver, which is commonly known to lead to inattention and drop in driving performance (Louw et al. 2015; Parasuraman, Mouloua, and Molloy 1996). With increasing levels of automation, the situation awareness of an ADS rises through its various perception sensors (e.g. radars, sensors, and camera’s), and increas-ingly powerful on-board computers. As stated, this can lead to inattention for drivers who should be, but may not be able to, remain suitably aware at all times. This already forms a problem at SAE level 1, because regaining situation awareness also takes time. For example, for the appropriate comprehension and projection of a vehicle’s speed difference with your own can take over 20 seconds (Lu, Coster, and de Winter 2017).


This indicates that a SAE level 1 ADS may require a take-over request time of at least 20 seconds in some cases. While other studies show different required time intervals for drivers to take over control (on average approximately 6 seconds; see Eriksson and Stanton (2017b) for a review), such high times appear inappropriate. It is with regard to partial automation, in which drivers are deemed to maintain levels of control from a viewpoint of monitoring, that there may be concerns over real meaningful control in practice. This has previously been identified and discussed in other works (Calvert et al. 2020) and will be carried forward in consideration of the operationalisa-tion of MHC.

3.2. Traceability of meaningful human control

3.2.1. Description of MHCA relatively new path of thought has arisen in the past years in regard to vehicle automation that focusses on the general human ability to maintain control over any level of vehicle automation: Meaningful Human Control (MHC) (Santoni de Sio and Van den Hoven 2018). The concept of MHC appeals to the intuition that when autonomous systems are deployed in unstructured, dynamic and potentially unpredictable environments, simply having a human agent involved at some point in the decisional chain (sometimes called ‘the kill chain’) may not be sufficient to prevent unwanted mistakes and so-called accountability gaps; human persons must maintain a role that is as prominent as possible (Human Rights Watch 2014; Article 36 2013).

A generic paraphrase of the principle of MHC is that systems must preserve MHC over actions, that is: humans not computers and their algorithms should ultimately remain in control of, and thus morally responsible for relevant decisions about operations. The tran-sition of the concept to vehicle automation is a logical one, as with vehicles humans must also maintain generic control over a system that is there to aid mobility, but also has the potential to cause undesirable, unsafe, or even dangerous situations. MHC is relevant because it lies at the heart of how ADS are designed, or at least should be designed, to allow humans to impose control even when they are not necessarily physically performing the driving task (Brookhuis, De Waard, and Janssen 2001). MHC does not entail that a driver must maintain operation, monitoring and fall-back control, but that the co-existence of driver operation and the human-orientated design of the ADS must maintain levels of acceptable behaviour initiated by humans. Santoni de Sio and Van den Hoven (2018) define two main conditions that must be met to allow MHC: tracking and tracing. They define tracking as:

‘Demonstrably and verifiably being responsive to the human moral reasons relevant in the circumstances – no matter how many system levels, models, software or devices of whatever nature separate a human being from the ultimate effects in the world, some of which may be lethal. That is: decision-making systems should track (relevant) human moral reasons’.

This basically entails that the system should be designed such that using all available input, decisions should follow a path of logic that is generally acceptable to human reasoning and acceptance of potential risks. These are processes that are more allied to ethical and psychological discussions in regard to control, rather than physical entities that we are focussing on here. Tracing is defined as:


Actions/states should be traceable to a proper moral understanding on the part of one or more relevant human persons who design or interact with the system, meaning that there is at least one human agent in the design history or use context involved in designing, programming, operating and deploying the autonomous system who a) understands or is in the position to understand the possible effects in the world of the use of this system; b) understands or is in the position to understand that others may have legitimate moral reactions towards them because of how the system affects the world and the role they occupy.

This entails that the place of control in the system should explicitly be identifiable and traceable to eventual actions that the system carries out. This condition requires the ability to perform tracing through different areas of the ADS, and can easily be performed with help of the identified components in this paper. To understand how MHC can be maintained under different levels of vehicle automation, the chain of control must first be constructed. The first steps for this were performed in Section 3.1.

In synthesis, the MHC approach adopts a notion of a (driving) system made of diverse and numerous elements, and suggests that all of them should be optimized to respond to the intentions and plans of its driver as well as to some intentions and plans of the traffic system’s designers, the policy makers or even to some general norms of a society. Admittedly, this may be seen as something that makes control more demanding than the traditional engineering notion of control, insofar as more design requirements potentially enter the picture. However, this approach also allows for an original combination of higher level of autonomy (i.e. less human driving) with a higher human control on a driving system (via technical and institutional infrastructures); in fact, according to MHC, in principle, control can be achieved also through agents that are not directly related to the driving task as drivers or supervisors, provided the vehicle is designed to respond to the relevant intentions and plans of these other relevant agents: designers, policy-makers, and the society as a whole. This will also encompass a wide range of control aspects and principles from many different related domains, such as cognitive psychology, automotive engineering, ergonomics, human-machine-interfacing (HMI), and so on. Without going into each of these areas in detail, we have already presented some of the main components that help link these areas in the framework given in Section 2.

The concept of MHC has remained under-defined in the political debate on autonomous weapon systems before Santoni de Sio and Van den Hoven (2018) recent comprehensive philosophical account of it. Alternative related approaches explored by other scholars in the recent literature, often appeal to the intuition that when autonomous systems are deployed in unstructured, dynamic and potentially unpredictable environments, simply having a human agent involved at some point in the decisional chain may not be sufficient to prevent unwanted mistakes and so-called accountability gaps. One of the first operation-alisations can be found in the seminal work by Horowitz and Scharre (2015). They identified general conditions for MHC over autonomous weapons systems. Specifically, they stressed the importance of human operators’ full decisional awareness and deep knowledge of the context of action. Moreover, the authors stressed that weapons and users should be, respec-tively, designed and trained to allow decisions to be taken under such optimal conditions. Santoni de Sio and Van den Hoven develop those intuitions and provide a theory that explains what the relation between human controller and the controlled system should look like in order for control to be meaningful. Inspired by Fischer and Ravizza (2000) theory


of responsibility and control, their conditions for MHC describe a controlling relation based on human moral reasons to act, and on a system that can adequately and seamlessly respond to those.

Santoni de Sio and van den Hoven’s approach is original in two ways. First, the authors produced an encompassing notion of control, one that applies not just to intelligent devices, but also to the entire ‘socio-technical system’ of which these are part. In their notion of intelligent system, devices themselves play an important role but cannot be considered without accounting for the numerous human agents, their physical environment, and the social, political and legal infrastructures in which they are embedded. Second, in line with the so-called Value-Sensitive Design approach (Van den Hoven 2013), Santoni de Sio and Van den Hoven’s work is meant to ultimately provide not just political and legal regulation but also general design guidelines – applicable to devices and (social) infrastructures alike – to achieve and maintain a meaningful form of control over autonomous systems in the military domain as well as in civilian domains like transportation. In this paper, we pick up that suggestion in regard to transportation with a focussed application towards devel-oping an account of MHC over automated driving systems.

3.2.2. MHC for levels of automationIn line with the given definition and conditions for MHC, it is reasonably clear how the chain of MHC is maintained for lower levels of vehicle automation (SAE level 1) in which primarily driving assistance systems or ADAS are present. As described, operation, moni-toring, and fall-back are all the responsibility of the driver. Therefore, it is clear that in all cases MHC for these levels of automation remains unchanged compared to manual driving, in which the driver is involved in every part of driving (even if some tasks are delegated) and remains responsible. This also means that the driver is responsible for safe and appro-priate use of any ADAS implemented in their vehicle.

High levels of vehicle automation (i.e., SAE level 4–5) on the other hand are quite the opposite to the low levels. All the control influencing aspects are maintained by the ADS, and the driver is at this point expected to be predominantly ‘out-of-the-loop’. This places demands on the design of the ADS to be designed with trackability and traceability in place. Design of these systems must therefore consider what acceptable logic decisions and actions may be taken, and the reasoning behind this process. Furthermore, a clear line of traceability must be present from human intervention or design in the ADS to actions taken on the road by a vehicle. These can be assisted by the chain of control in this paper. As both gen-erally accepted rules for tracking and satisfactory design structures for fully ADS are not yet in existence or in the public domain, we cannot retrospectively comment on these in any more detail at the moment. We recommend future effort in ADS design to consider the line of thought and core components from this paper.

While the two extremes of ADS are more clear-cut from a MHC point of view, interme-diate levels of vehicle automation (i.e., SAE level 2) are more complicated. While some monitoring tasks are still the responsibility of the driver, a driver is always responsible as fall-back in these levels of automation. This is a somewhat disputable responsibility, as many researchers have questioned a driver’s ability to remain focused as fall-back in cases where monitoring and operation is limited or no longer required (Louw et al. 2015; Vlakveld 2016). A driver’s task demand is reduced and leaves room for additional tasks that in turn can lead


to distraction and inattention. Therefore the first point of concern lies with the ability of a driver to perform their assigned tasks.

While the traceability of control for fall-back and monitoring would lie in part or fully with the driver, the trackability may not be acceptable due to general limitations of a driver’s ability to maintain attention. A second concern lies with an unclear ability to trace control in the driving system, where the task of monitoring is shared. Less so for SAE level 2 and more so with SAE level 3: the driver can let the ADS monitor some or many driving tasks. The ADS, while being able to monitor up to certain limits, is not designed for these levels of automation to be able to observe and perceive its’ environment, and requires the driver’s interaction. It therefore becomes difficult to identify where and when responsibility lies. This becomes even harder when ADS are designed using ‘black-box’ algorithms, such as machine learning or neural networks, which make explicit traceability nearly impossible.

3.2.3. Necessity of considering MHCWithin various domains, much discussion has taken place and continues to take place in regard to where responsibility should be placed over control of automated vehicles. In this paper, we are not interested in nominal responsibility, i.e. what has been agreed upon, but rather on where real control over an AV lies and how this can be described and maintained.

Various concepts of control exist and in various different domains. We do not mean to dig into each area of control and describe them, but we do need to state what control entails and why adding the term ‘meaningful human’ to it as a concept is important for automated driving. In regard to human machine cooperation, Flemisch et al. (2016) described control as ‘the power to influence the course of events’ and applied to human machine systems, which ADS is, control is described as the ability ‘to influence the situation so that it develops or stays in a way preferred by the controlling agent’. A major argument of MHC is that the controlling agent should always be a human or should relate to a human. However, just stating that a human should be involved is not enough, therefore the control should be meaningful human control, which means that the control that a human has also meets with conditions that relate to their accountability and their ability to exert control. Pacaux‐Lemoine (2015) describes know-how (to operate) and know-how–to- cooperate as the ability to control a part of a process and an agent’s ability to cooperate with other agents concerned by the process control. But knowing how, does not guarantee the ability to actu-ally exert that control, which has been strongly argued in regard to automated driving by many from within the human factors and behavioural domains (Calvert et al. 2020). By stating that the control should be meaningful, MHC explicitly includes an agent’s ability and moral involvement to understand and operate, which can be clearly seen from the two conditions: traceability and trackability.

Accordingly, human factors plays an important role in determining what is meaningful. Flemisch et al. (2016) also mentions that various forms of Situation awareness (SA) have been defined to support cooperation in the perception and the understanding of the situ-ation (Endsley 1995; Salmon et al. 2008; Shu and Furuta 2005). And indeed, SA is a key aspect of attaining MHC, but in itself is not control but part of the control sequence. Marberger et al. (2017) highlight this nicely in their description of driver availability, which is applied to assess driver states before or during transition of control processes. They state


that driver availability is not just an absolute description of the current driver state but a relative assessment that is influenced by the situational demands as it take the requirements of control transition into account and also maintains a focus on the relation between the occupation of attentional resources and the requirements for safe driving (Marberger et al. 2017). Based on (explicit or implicit) understanding of good or bad situations, actors per-ceive the world and influence the situation by using their abilities to act, thereby forming control loops (Flemisch et al. 2016). This is what makes MHC unique, but all the more necessary. It considers the control system in its entirety from human driver, with their knowledge and abilities, to ADCS and human designers (of hardware and software), and does this from a philosophical human viewpoint rather than a legal or technical one. Obviously, these and other domains must be considered, but we choose to present the framework with the core components openly in regard to MHC here to encourage the broader communities from these various domains to contribute to the concept.

The tracing condition for MHC specifically requires from a controller not just to merely possess the right skills and capacities, but a moral understanding of their role and respon-sibility. A genuine moral involvement, as seminally expressed e.g. in Santoni de Sio and Van den Hoven (2018), promotes active responsibility, i.e. proactive involvement in developing skills and making sure of their right execution, thereby also improving safety. This is one of the main takeaways of MHC, as opposed to the situational awareness, which is a necessary condition, but not a sufficient one to achieve a control of a more meaningful nature.

As previously stated, following trackability in regard to human moral reasons is a topic that requires extensive ethical consideration, among others, as is not readily captured by the framework. The presented framework is designed such that the traceability conditions can be analysed using the identified components. The way these have been structured based on literature, allows one to follow the chain of control and consider if the traceability con-ditions is met. To give a demonstration of this, we present a number of example cases in the following section.

4. Operationalisation of MHC for evaluation and design

Although the main purpose of this paper is to introduce the generic framework of MHC to the application area of automated road vehicles and make the link with the main com-ponents of the system that are most relevant, we also need to touch upon operationalisation of MHC in such a setting. The main work regarding operationalisation is still on-going and goes beyond the scope of this paper. Here, we give some considerations on how MHC may be operationalised as a prelude to this later work and to give insights into this process. We firstly propose and discuss a potential approach for traceability, which considers the aspects most closely related to the core components framework. We also give some considerations in regard to trackability. In Section 5 of this paper, a number of illustrative examples are given to make this process more tangible.

4.1. Suggestion for traceability operationalisation

The application of MHC is aimed at system design and evaluation of existing systems. In both cases, an a-priori evaluation of a system design should align with an a-posteriori


evaluation of an existing system if both are evaluated for the same aspects and under the same conditions. This is what we consider here as we give a potential approach for opera-tionalisation of MHC in regard to traceability. To do this, we propose the construction of a cascade evaluation approach. This approach considers the main aspects that affect MHC traceability in sequential order for all potential human agents that are involved in control over an automated vehicle. An overview of the process and equations is shown in Table 1.

Table 1 clearly shows how we define four main aspects that need to be considered: the exertion of operational control, the involvement of a human agent, the ability of that agent to understand and use the system, and the ability of the agent to understand moral respon-sibility over the system. The first two aspects focus mainly on the presence of operational control and the involvement of humans. Without vehicle operation, there can be no control and without identifying human involvement, MHC can by definition not be present. Aspects C and D address the explicit conditions for tracing. For each aspect, each potential agent is given a score that reflects that aspect for the agent. We have arbitrarily chosen a scale from 0 to 5. It is important to remember that exertion of MHC is not binary and that we must consider the extent to which control is exerted. The critical score for each aspect (A, B, C, D) is the highest score from all human agents, as if one agent can achieve perfect performance for that aspect, other aspects are less important as control may already be guaranteed. However, each aspect needs to be considered as part of assessing overall MHC from the point of traceability. Therefore, the aspects are set out in a cascade in which the score from the proceeding aspect and the current aspects are compared to determine the critical score for that aspect. In this case, we take the minimum score from that aspect and the previous aspect to determine the current critical score up to that point. Therefore, the critical scores for aspects B, C and D, are influenced by all the aspects that have preceded them. The final score of the system in regard to MHC for the traceability condition is the critical score produced by D. Also note that the third aspect (C) has two sub-aspects; These indicate the ability of an agent to understand how the ADS works (i), but also how to suitably act in the occurrence that they need to exert control over the system (ii).

This initial suggestion for an approach to MHC operationalisation for the tracing con-dition is just a start and further details need to be worked through, such as determination of the scoring or further analysis in regard to the comprehensiveness of the considered aspects. And we must also stress that the tracking condition needs to be met as well for MHC. Further approaches to operationalisation in regard to the tracing condition of MHC are of course certainly possible. Further developments may also aim at design recommen-dations for a system prior to the completion of a system design.

4.2. Considerations for trackability operationalisation

Giving an approach for operationalisation for MHC in regard to trackability is not as straightforward as for traceability, as trackability encompasses a much greater degree of human reasons and societal values and is therefore much more complex and prone to change. The aspects of human reasoning and values as well as how these fit with the prox-imity to agents goes beyond the goals of this paper. Nevertheless, we will give some con-siderations on this, but will stop sort of proposing an approach at this time.

The main considerations for the tracking condition relates to the extent that a system responds to (i) reasons or values of (ii) specific human agents that are connected to the


system. Important considerations within this need consider the prioritising of human rea-sons and values, for example this could entail privileging safety over individual freedom and flexibility, and therefore prioritising responsiveness to societal rules and regulations. Different policies might privilege individual freedom and independence, therefore priori-tizing (some) proximal reasons over distal ones, granting more control to final users. Any operationalisation of tracking must take a careful consideration of conflicting reasons and values in a way that is socially acceptable, but also allows implementational. Much discussion is possible in regard to how well ADS should respond to human reasons. Even from this short discussion here, it becomes evident that this goes beyond what we have resented in regard to the core components in this paper.

In the following section, we give an example of how this might unfold. However, this is just one example, and, to be clear, we are not necessarily campaigning for vehicles that track specific reasons. Our general claim is that vehicles that are under MHC should also respond to some distal reasons of their owners/drivers as well as to some (distal) reasons of other agents in society, as reflected in some moral and legal norms. Which of these reasons a specific systems should track remains a normative question on which reasonable persons and policy-makers may disagree.

5. Example cases and recommendations

The core components framework for ADS with consideration of MHC allows various dif-ferent areas of automated driving to explicitly consider the effects and consequences of the chain of control from various different angles. This could be from the perspective of the vehicle or interface design choices, of driver ability and control, or of policy decisions in regard to authorisation. Below, we give three examples of how the framework and chain of control for traceability can be considered, as well as one example for trackability. In the first example, we also give an illustrative application of the cascade approach for operationali-sation that was described in Section 4. These examples are given for different use cases and

Table 1. suggested cascade evaluation approach for mhc tracing operationalization.Tracing score (0-5) critical score

is operational control exerted by… Driver a A = max (a,b,c)aDcs bother, namely… c

is there a human agent involved irt… Driver d B = min (max (d,e,f ),A)aDcs eother, namely… f

Does or should this person understand the system in the sense that:

i) They have a propositional knowledge of the system’s functions (know-that)

ii) They have the right capacities to exercise a control task (know-how)

Driver g = min (gi, gii) C = min (max (g,h,k),B)

aDcs designer h = min (hi,hii)other, namely… k = min (ki,kii)

Does or should this person understand their own moral responsibility for the consequences of the actions of a system

Driver m D = min (max (m,n,p),C)

aDcs designer nother, namely… p


show how the framework and MHC concept can be further implemented in future works. Note that the cases are dependent on the wider definition and consensus of trackability of MHC, while currently (ethical) consensus on what MHC details is yet to be reached, and will be the topic of much discussion in the coming years.

5.1. Driver training for dual mode vehicles

It is likely that it will be ‘dual mode’ vehicles that will dominate the lion’s share of automated vehicles on the current infrastructure for the time being (Calvert, Schakel, and van Lint 2017). These are SAE level 1–3 vehicles that are partially automated in which the driver remains in the loop, either operationally or in a monitoring role. Much research has been performed on a driver’s ability to perform this role appropriately (Eriksson and Stanton 2017a; Louw et al. 2015) with many doubting the desirability or suitability with current driver training, legislation, and vehicle design (Regan, Lee, and Young 2008). Driver retrain-ing is a possibility or even necessity for drivers in control of these ‘dual mode’ vehicles. Figure 4b shows a mixed chain of control for this level of automation. As mentioned in Section 3.2, the chain of control is not clear for these vehicles. Legislators and driving licence authorities can consider the chain of MHC from the framework to derive a set of acceptable tasks that a driver must be able to carry out. As these tasks are currently not practiced, experimental research would be required to judge the suitability. Based on these tasks and the results of experiments, it may become evident that a driver would require retraining to deal with these new tasks and situations. This could refer to how a driver interacts with the vehicle control, for instance when to retake control and when to trust the vehicle, the driver could be trained to maintain focus and what exactly to monitor when not in operational control, and so on. The core components framework together with a definition of acceptable MHC allows these choices to be made explicit and based on sound reasoning. For example, referencing Figures 1 and 2, a clear control scheme can be drawn up to determine when and over what the driver is in control, and when the vehicle is in operational control. In regard to the vehicle, one can then test the vehicle’s capabilities to achieve suitable control for the desired level of control. For the driver, one can also consider if the desired role of the driver (e.g., to monitor) is acceptable, for which much literature exists that puts this in doubt.

Much concern has been highlighted in regard to a driver’s ability to retake control in SAE level 3 systems due to driver inattention. If we presume inattention and apply the suggested approach for operationalisation from Section 4.1, the evaluation cascade may resemble Table 2.

Using the cascade approach shows that MHC is present, but that it is not optimal for an arbitrary case of the tracing condition for a SAE L3 vehicle with an inattentive driver. The main aspects that negatively influence the score of the system relate to the third aspect in which the driver may only understand the system to a certain extent (it has been shown many times that drivers are not fully aware of what their vehicle’s technology can do), but mainly does not appreciate what is required to retake control if that is required (as we stated that the driver was inattentive). We expect that such an analysis can be performed in greater detail by explicitly using the core components as described in Section 2 once full opera-tionalisation tools become available. Even with a simplified example here using a suggestive approach, the benefits are already evident.


5.2. MHC design for driverless passenger vehicles

Driverless passenger vehicles (DPV) have been in use for decades within very restricted operational design domains, and can generally be defined as SAE level 4 vehicles (Lazarus et al. 2018). These vehicles transport passengers usually making use of dedicated infrastruc-ture (Lazarus et al. 2018). Increasingly, DPVs are being employed in places where they have to make use of infrastructure and interact with other road users, and may increase transit speed. As DPVs do not have an (operational) driver who is able to exert control over the vehicle, much of the control aspects must be integrated in the design phase of the vehicle. Traceability in such a vehicle is therefore exerted through the design of the software and control algorithms, making use of the capabilities of the vehicle hardware design. Figure 4c shows the chain of control, which relies heavily on the vehicle technology. The overview of this chain of control can allow vehicle approval organisations to set explicit demand for the performance of each component. These demands may go further into subcomponents level than the main components included in this paper. Execution of top-down and bottom-up MHC can be made explicit by considering the consequences of vehicle decisions and actions. Top-down, this involves designing the vehicle (both software and hardware) with predefined actions in mind, not too different from current practice. Bottom-up, this includes consid-eration of (yet undefined) conditions of MHC, and how the vehicle should act. The chain of control can be followed through actions, to actuation, to automated and primary control, all the way to the vehicle (perception) sensors (see Figure 2). A mismatch between the top-down and bottom-up chain of MHC would indicate a required design change.

5.3. Transition of control in truck platooning

The final example looks at the transition of control in truck platooning. Truck platooning makes use of SAE level 1 or 2 cooperative and automated technology to allow two or more trucks to drive at very small time- and space headways to reduce air resistance and, in turn, fuel consumption and costs (Calvert, Schakel, and van Arem 2018). Currently, truck pla-tooning requires a driver to maintain an active role, even if this is not in operational control (note: this may change in the future). Part of this reason is that the driver must be able to retake operational control if this is required, for example, due to platoon break-up or com-plicated road sections. Previous studies have shown that control transition requires a nec-essary amount of time that varies per driver and per moment (Merat et al. 2014), and the level of awareness for the driver is also variable at the moment of control transition (Hjälmdahl, Krupenia, and Thorslund 2017). This poses questions in regard to safety at the point of the transition of control (Axelsson 2017), and also the design of the ADS to allow a driver this time while still maintaining an acceptable level of control. Further consideration can be given to the legislative aspect of where responsibility lies during a control transition and design conditions that may be set by the vehicle approval authority. A further interesting aspect that arises here is that of ‘shared’ control or exogenous MHC. In a truck platoon, the leading vehicle and/or driver exerts operational control, which directly influences operational actions of the following platooning trucks. The manner and chain of MHC therein exceeds the chain of MHC of a singular vehicle and becomes inter-vehicular. This means that the framework would be expanded to consider multiple drivers and vehicles, with control for a second vehicle possibly being exerted by a first vehicle, driven by the first driver. The structured


approach of the framework allows us to make these connections and identify possible points of concern in regard to traceability of MHC. This poses further questions which are not answered here, but are given as a challenge and recommendation for further research. For further details of the application of MHC to truck platooning, see (Calvert et al. 2018).

5.4. Prioritising of moral action by a system

This example considers the case of two individuals faced with similar circumstances, but who deal with it differently to illustrate a system that is designed for tracking. The example considers different reasons of the drivers, but also some reasons of some other agents potentially involved in road traffic.

The first case sees Lucy as the protagonist. Lucy is driving home on a winter night. There is poor visibility, due to dense fog. However Lucy doesn’t have much experience with the technology in her car, therefore she does not want to use the available autopilot function and therefore drives in manual mode. At some point, a fastidious beeping signal interrupts her journey. The vehicle cuts the engine and swerves gently, dodging what seems to be a crashed car, still smoking on the road. When the vehicle has past the accident site, Lucy slows down and pulls over at a safe location to check the situation.

The second case is that of John. John lives near Lucy and is also heading back home around the same time as Lucy. He is using the autopilot feature in his vehicle. As he gets to the location of the accident, his dashboard starts beeping and the car slowly drives around the accident site. However, he is not as willing as Lucy to stop and check. He knows it is his legal duty to pull over and offer help if required, but he really does not feel like it that night. He notices that his car slows down dramatically, and seems to be pulling over. He imme-diately grabs the wheel and pushes on the accelerator pedal, trying to avoid that annoyance. However, his dashboard warns him that an emergency procedure is about to be deployed. In a matter of seconds, the steering wheel becomes loose, while the pedals seem to have been disengaged from the vehicles powertrain. The vehicle slows down and pulls off the road automatically, with its hazard lights on. A voice message compels John to wear his

Table 2. example case evaluation of mhc using cascade approach.Tracing score (0–5) critical score

is operational control exerted by… Driver 1 5aDcs 5other, namely… 0

is there a human agent involved irt… Driver 5 4aDcs 4other, namely… na

Does or should this person understand the system in the sense that:

iii) They have a propositional knowledge of the system’s functions (know-that) They have the right capacities to exercise a control task (know-how)

Driver min(3,2) 2

aDcs designer min(5,0)other, namely… na

Does or should this person understand their own moral responsibility for the consequences of the actions of a system

Driver 4 2

aDcs designer 2other, namely… na


safety vest, leave his car and offer help, while an emergency call is being automatically dispatched. Left with no choice, and starting to understand the potential severity of the situation, he leaves the car to offer his help.

That night, Lucy wanted to drive home. She wanted to do it herself though. The system allowed that. It allowed her reasons to influence the system, allowing her to drive manually. However, if the car would have kept realising those proximal intentions, she would probably have driven straight into the crashed vehicle, unaware of the wreck that occupied her lane due to the dense fog. Fortunately, her intentions were conflicting with her more general plan, which was to get home safely, and the car was programmed to prioritize those kinds of more distal reasons. It can be observed how in certain situations driving can be safer with an automated driving system permanently in control. However, this becomes relevant only if this is in some way specified within the system. That would have been the case if, for instance, a regulation was in place establishing that manual driving is unsafe and there-fore not allowed.

John’s plan was also to go home that night, but his case is different. His car not only denied his intentions to push the accelerator and keep driving, but also refused to comply with his more general plan to go home, pulling off the road instead. The car prioritised the interest of the victims of an accident over the driver’s will. Rescue (legal) obligations are one of the possible reasons that lead driving behaviour, and it is more general and further away in time than John’s plan to leave the site of the accident without checking. No matter what John’s plans were, the system was designed for cars to comply with this obligation.

5.5. General recommendations

Defining MHC, considering which components that affects, and what that means for vehicle design and wider road legislation and policy is just a start. As the ethical discussion on what is humanly acceptable for automated vehicles continues, the second main condi-tion of MHC, trackability, can be addressed. This involves the extent to which the ADS can track human moral reasoning, either explicitly or implicitly. This condition is the main underpinning aspect of what ‘meaningful’ in MHC actually entails. These discussions are being undertaken in collaboration with experts from various different fields, such as philosophy, ethics, automotive, behavioural psychology, and many others. In the meantime, the presented framework of core components of the ADS in this paper could be further elaborated to give greater levels of detail in each of the relevant categories, especially in those of driver-vehicle interaction. Increasing the level of subcomponents, categorising existing systems, and identifying existing and possible connections between various (sub-) components of the different categories can aid an increased understanding of the chain of control within the ADS based on MHC. Deeper analysis of components can yield insights into the acceptability control and the alignment with MHC. This will also allow greater detailing of ADS, HMI, and vehicle design from a MHC point of view. As part of this endeavour, the framework could be applied to help identify current gaps in under-standing and behaviour that should require additional research. Much of the additional research may include behavioural experiments, which focus on human driver interaction with technology in ADS and their ability therein. This goes for all the levels of automation, but with special attention required for the intermediate levels in which a driver is required to interact and work together with the ADS, as has been previously highlighted.


We have further considered avenues for the operationalisation of the MHC concept, especially in regard to the tracing condition. The initial proposal of a cascade evaluation approach gives insights into how the condition can be used to evaluate a driving system. It is part of an on-going effort to further construct a full evaluation framework for MHC operationalisation, which would include the tracing and tracking conditions. Tracking is more complex in this regard as it includes an expansive ethical discussion that must be performed with care. We do encourage other scholars to take up this challenge to address MHC operationalisation in an effort to achieve responsible concepts of control that can be effectively applied.

6. Conclusions

With little doubt that vehicle automation is here to stay, a safe, ethical, and responsible implementation of automated driving systems on roads is essential. There are major concerns in regard to transition of control, ADS design, driver training, and vehicle licencing, among others. The concept of Meaningful Human Control (MHC) acts as a safeguard that any type of vehicle automation should comply to acceptable and ethical human consensus on vehicle control. While consensus is still to be reached, consideration of MHC in the design and legislation of ADS can already be considered. To make this possible, the chain of MHC must be established. Trackability and traceability are two key conditions that must be met for MHC, and are served by the framework. Therefore, in this contribution we have proposed a framework of the core components of the ADS that can be used to trace control in auto-mated vehicles, while trackability is left to the domain of ethics to take further steps in. Transition of monitoring and control are considered for the transition from lower to higher levels of automation based on the framework. The influence, relevance and role of MHC is discussed, and example cases are given in which the framework and MHC are relevant and should be applied. In this regard, challenges exist and are identified for vehicle manufacturers, software developers, and other vehicle component designers, and for vehicle and driving licence authorities. We have also discussed and put forward suggestions in regard to the operationalisation of MHC that should aid understanding and the application of the concept. Recommendations are made to consider vehicle design, human machine interaction, tran-sition of control, driver training, vehicle approval, and other topics based on the chain of MHC and MHC conditions, and can be approached with the presented framework. Furthermore, the framework also allows researchers to identify areas to perform more explicit and relevant research, and develop models that can be applied to perform projections of future impacts.

Disclosure statement

We report that there is no potential conflict of interest.

Funding

This work is part of the research programme Meaningful Human Control over Automated Driving Systems with project number MVI.16.044, which is (partly) financed by the Netherlands Organisation for Scientific Research (NWO).


Notes on contributors

Simeon C. Calvert received the MSc and PhD degrees in Civil Engineering, specialized in Transport & Planning form the Delft University of Technology, The Netherlands, in 2010 and 2016, respectively. He concluded his PhD in the area of stochastic macroscopic traffic flow mod-elling. He is now employed as coordinator and researcher at data and simulation lab DiTTlab at Delft University of Technology. Between 2010 and 2016, he worked as a Research Scientist at TNO, Netherlands Organization for Applied Scientific Research. There, his research has focused on ITS, impacts of vehicle automation, traffic management, traffic flow theory and network analysis. Much of his recent research has involved various roles in leading national and European research projects involving the application and impacts of vehicle automation and cooperation.

Daniël D. Heikoop obtained his MSc in Applied Cognitive Psychology at the University Utrecht, after which he started a PhD within the Marie Curie-Skłodowska Actions funded project called HFAuto. Between 2014 and 2017 he performed his PhD on Driver Psychology during Automated Platooning as an external student from Delft University of Technology at the University of Southampton (UK). He now works at the Delft University of Technology, on the project called ‘Meaningful Human Control over Automated Driving Systems’, in which he actively collaborates with psychologists, traffic engineers, and philosophers.

Giulio Mecacci received his MA in Philosophy of Mind from the University of Siena, Italy. He obtained a PhD from Radboud University Nijmegen, at the Donders Institute for Brain, Cognition and Behavior, in the field of ethics of neurotechnology. He is now post-doctoral researcher at the Delft University of Technology, working together with psychologists and engineers on the multidis-ciplinary project ‘Meaningful Human Control over Automated Driving Systems’. He is also assistant professor with tenure in the department of Artificial Intelligence, at Radboud University Nijmegen, dealing with ethical and societal implications of AI and intelligent technologies.

Bart van Arem received the MSc and PhD degrees in applied mathematics from the University of Twente, Enschede, The Netherlands, in 1986 and 1990, respectively. From 1992 and 2009, he was a Researcher and a Program Manager with TNO, working on intelligent transport systems, in which he has been active in various national and international projects. Since 2009, he has been the Chair Professor of Transport Modeling with the Department of Transport and Planning, Delft University of Technology, Delft, The Netherlands, focusing on the impact of intelligent transport systems on mobility. His research interests include transport modelling and intelligent vehicle systems.

ORCID

Simeon C. Calvert http://orcid.org/0000-0002-1173-0071Daniël D. Heikoop http://orcid.org/0000-0003-4379-072X

References

Amditis, Angelos, Panagiotis Lytrivis, and Evangelia Portouli. 2012. “Sensing and Actuation in Intelligent Vehicles.” In Handbook of Intelligent Vehicles, edited by Azim Eskandarian, 31–60. London: Springer.

Andrews, Scott. 2012. “Vehicular Communications Requirements and Challenges.” In Handbook of Intelligent Vehicles, edited by Azim Eskandarian, 1091–1120. London: Springer.

Article 36. 2013. Killer Robots: UK Government Policy on Fully Autonomous Weapons. London: United Kingdom Government document. http://www.article36.org/wp-content/uploads/2013/ 04/Policy_Paper1.pdf

Axelsson, Jakob. 2017. “Safety in Vehicle Platooning: A Systematic Literature Review.” IEEE Transactions on Intelligent Transportation Systems 18 (5): 1033–1045. doi:10.1109/TITS.2016. 2598873.

http://orcid.org/0000-0002-1173-0071http://orcid.org/0000-0003-4379-072Xhttp://www.article36.org/wp-content/uploads/2013/04/Policy_Paper1.pdfhttp://www.article36.org/wp-content/uploads/2013/04/Policy_Paper1.pdfhttps://doi.org/10.1109/TITS.2016.2598873https://doi.org/10.1109/TITS.2016.2598873


Bernstein, Douglas. 2010. Essentials of Psychology. Independence, KY: Cengage Learning.Brookhuis, Karel A., Dick De Waard, and Wiel H. Janssen. 2001. “Behavioural Impacts of Advanced

Driver Assistance Systems–An Overview.” European Journal of Transport and Infrastructure Research 1 (3): 245–253.

Calvert, S. C., G. Mecacci, D. D. Heikoop, and F. Santoni de Sio. 2018. “Full Platoon Control in Truck Platooning: A Meaningful Human Control Perspective.” In 2018 21st International IEEE Conference on Intelligent Transportation Systems-(ITSC), Maui, USA.

Calvert, S. C., G. Mecacci, B. van Arem, F. Santoni de Sio, D. D. Heikoop, and M. Hagenzieker. 2020. “Gaps in the Control of Automated Vehicles on Roads.” IEEE Intelligent Transportation Systems Magazine. https://www.researchgate.net/publication/333566108_Gaps_in_the_control_of_auto-mated_vehicles_on_roads

Calvert, S. C., W. J. Schakel, and B. van Arem. 2018. “Evaluation and Modelling of the Traffic Flow Effects of Truck Platooning.” Transportation Research Part C: Emerging Technologies 105: 1–22. doi:10.1016/j.trc.2019.05.019.

Calvert, S. C., W. J. Schakel, and J. W. C. van Lint. 2017. “Will Automated Vehicles Negatively Impact Traffic Flow?” Journal of Advanced Transportation 2017: 1. doi:10.1155/2017/3082781.

Chandrasiri, Naiwala P., Kazunari Nawa, and Akira Ishii. 2016. “Driving Skill Classification in Curve Driving Scenes Using Machine Learning.” Journal of Modern Transportation 24 (3): 196–206. doi:10.1007/s40534-016-0098-2.

Deffenbacher, Jerry L., Rebekah S. Lynch, Eugene R. Oetting, and David A. Yingling. 2001. “Driving Anger: Correlates and a Test of State-Trait Theory.” Personality and Individual Differences 31 (8): 1321–1331. doi:10.1016/S0191-8869(00)00226-9.

Emadi, Ali. 2005. Handbook of Automotive Power Electronics and Motor Drives. Boca Raton: CRC Press.

Endsley, Mica R. 1995. “Toward a Theory of Situation Awareness in Dynamic Systems.” Human Factors: The Journal of the Human Factors and Ergonomics Society 37 (1): 32–64. doi:10.1518/001872095779049543.

Eriksson, Alexander, and Neville A. Stanton. 2017a. “Driving Performance after Self-Regulated Control Transitions in Highly Automated Vehicles.” Human Factors: The Journal of the Human Factors and Ergonomics Society 59 (8): 1233–1248. doi:10.1177/0018720817728774.

Eriksson, Alexander, and Neville A. Stanton. 2017b. “Takeover Time in Highly Automated Vehicles: Noncritical Transitions to and from Manual Control.” Human Factors 59 (4): 689–705.

Farah, Haneen, SandraM. J. G. Erkens, Tom Alkim, and Bart van Arem. 2018. “Infrastructure for Automated and Connected Driving: State of the Art and Future Research Directions.” In Road Vehicle Automation, edited by Gereon Meyer and Sven Beiker, Vol. 4, 187–197. Springer.

Fischer, John Martin, and Mark Ravizza. 2000. “Précis of Responsibility and Control: A Theory of Moral Responsibility.” Philosophy and Phenomenological Research 61 (2): 441–445.

Flemisch, Frank, David Abbink, Makoto Itoh, Marie-Pierre Pacaux-Lemoine, and Gina Weßel. 2016. “Shared Control Is the Sharp End of Cooperation: Towards a Common Framework of Joint Action, Shared Control and Human Machine Cooperation.” IFAC-PapersOnLine 49 (19): 72–77. doi:10.1016/j.ifacol.2016.10.464.

Future of Life Institute. 2015. “Autonomous Weapons: An Open Letter from AI & Robotics Researchers.” Future of Life Institute. https://futureoflife.org/open-letter-autonomous-weapons/.

Ghosh, Sumit, and Tony S. Lee. 2010. Intelligent Transportation Systems: Smart and Green Infrastructure Design. Boca Raton: CRC Press.

Goldberg, Lewis R. 1990. “An Alternative “Description of Personality”: The Big-Five Factor Structure.” Journal of Personality and Social Psychology 59 (6): 1216. doi:10.1037/0022-3514.59.6.1216.

Hamdar, Samer H., and Justin Schorr. 2013. “Interrupted versus Uninterrupted Flow: A Safety Propensity Index for Driver Behavior.” Accident Analysis & Prevention 55: 22–33. doi:10.1016/j.aap.2013.01.017.

Hayeri, Yeganeh Mashayekh, Chris T. Hendrickson, and Allen D. Biehler. 2015. “Potential Impacts of Vehicle Automation on Design, Infrastructure and Investment Decisions–A State DOT Perspective.” Transportation Research Board 94th Annual Meeting, Washington, DC, United States, Paper No. 15-2474.[

https://www.researchgate.net/publication/333566108_Gaps_in_the_control_of_automated_vehicles_on_roadshttps://www.researchgate.net/publication/333566108_Gaps_in_the_control_of_automated_vehicles_on_roadshttps://doi.org/10.1016/j.trc.2019.05.019https://doi.org/10.1155/2017/3082781https://doi.org/10.1007/s40534-016-0098-2https://doi.org/10.1016/S0191-8869(00)00226-9https://doi.org/10.1518/001872095779049543https://doi.org/10.1177/0018720817728774https://doi.org/10.1016/j.ifacol.2016.10.464https://futureoflife.org/open-letter-autonomous-weapons/https://doi.org/10.1037/0022-3514.59.6.1216https://doi.org/10.1016/j.aap.2013.01.017https://doi.org/10.1016/j.aap.2013.01.017


Heikoop, Daniël D., Joost C. F. de Winter, Bart van Arem, and Neville A. Stanton. 2016. “Psychological Constructs in Driving Automation: A Consensus Model and Critical Comment on Construct Proliferation.�

a human centric framework for the analysis of automated ...€¦ · adepartment of transport &...

Documents