available online at sciencedirecte-navigation.kr/down/2-jaeyong oh.pdf · 2020-01-22 · available...
TRANSCRIPT
Available online at www.sciencedirect.com
ScienceDirect International Journal of e-Navigation and Maritime Economy 5 (2016) 021 – 031
21
Original article
Advanced Navigation Aids System based on Augmented Reality*
Jaeyong OH 1, Sekil PARK 2, Oh-Seok KWON 3† 1 Korea Research Institute of Ships and Ocean Engineering, Korea, [email protected] 2 Korea Research Institute of Ships and Ocean Engineering, Korea, [email protected]
3† Dept. of Computer Engineering, Chungnam National University, Korea, [email protected], Corresponding Author
Abstract
Many maritime accidents have been caused by human-error including such things as inadequate watch keeping and/or mistakes in ship handling. Also, new navigational equipment has been developed using Information Technology (IT) technology to provide various kinds of information for safe navigation. Despite these efforts, the reduction of maritime accidents has not occurred to the degree expected because, navigational equipment provides too much information, and this information is not well organized, such that users feel it to be complicated rather than helpful. In this point of view, the method of representation of navigational information is more important than the quantity of that information and research is required on the representation of information to make that information more easily understood and to allow decisions to be made correctly and promptly. In this paper, we adopt Augmented Reality (AR) technologies for the representation of information. AR is a 3D computer graphics technology that blends virtual reality and the real world. Recently, this technology has been widely applied in our daily lives because it can provide information more effectively to users. Therefore, we propose a new concept, a navigational system based on AR technology; we review experimental results from a ship-handling simulator and from an open sea test to verify the efficiency of the proposed system.
Keywords: navigational aids system, augmented reality, IBS(Integrated Bridge System)
Copyright ⓒ 2016, International Association of e-Navigation and Ocean Economy. Hosting by Elsevier B.V.
This article is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/3.0/). Peer review under responsibility of Korea Advanced Institute for Intelligent Maritime Safety and Technology.
* This is a revised version presented at the 3rd Ai-MAST held at Rivera Hotel in Daejeon, Korea, November 12-14, 2015.
OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality
22
I. Introduction
The marine traffic environment has recently witnessed significant increases in traffic and trends
toward larger and faster ships, resulting in a greater risk of marine accidents. Data from the Korean
Maritime Safety Tribunal has revealed a steady rise in marine accidents over the last five years, with
80% of such accidents being caused by errors during operation. Among all accidents, collision
accidents were highest in number and mainly caused by human error such as negligence, and the
number of marine accidents continues to increase along with the resulting damages (Ministry of
Oceans and Fisheries, 2015). In order to prevent or help mitigate this, real-time support is provided
to ship navigators through the Vessel Traffic Services System (VTS) and the Aids to Navigation
System, and related legislation around the world is being carried out to prevent marine accidents.
Additionally, navigation systems with advanced IT technologies are being developed to provide
various data to navigators for safer operation; however, such complex equipment could be more of a
burden for navigators and interfere with safe operation. It cannot be assumed that all navigation
equipment provides useful information to navigators, and thus there is a need to analyze the current
limitations of different types of navigation equipment and solutions to such limitations from the
perspective of the navigators.
According to the bridge operations analysis results by STCW (The International Convention on
Standards of Training, Certification and Watchkeeping for Seafarers), the largest problem with
currently operating navigation equipment is the provision of excessive and unnecessary information
or inappropriate methods of data provision. In particular, the alert functionality with respect to
emergency situations is currently provided through RADAR and ECDIS, but such alert messages
are delivered through text messages or alarm sounds which are ineffective and not intuitive, so this
aspect requires further improvement. Additionally, an effective means of information delivery for
limited screen space needs to be further studied (Jeong J.S, 2012). In this respect, this paper
proposes an AR (Augmented Reality) technology-based navigational aid system that can be utilized
in bridge environments as an intuitive means of information delivery. Experimentation was
conducted to examine the application potential of the proposed system.
II. Analysis of Previous Study
As mentioned above, the complexity of the navigation equipment in the bridge can interfere
with safe operation; thus, navigational aid systems developed in the future need to support the
decision-making process of the navigator through effective information provision methods rather
than by simply increasing the amount of information delivered. Moreover, the intuitiveness of the
navigation data provided is a critical element that can resolve issues caused by excessive
information. This paper aims to improve upon this by utilizing augmented reality technology.
Augmented Reality is a field of VR (Virtual Reality) that fuses virtual objects or information
to the real environment through computer graphics technology so that virtual objects appear to
originate from the existing environment. This technology allows the display of information by
overlapping the information upon images of the real world. Considering that data obtained
OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality
23
through watch-keeping takes up a significant portion of the ship operation data provided, this
method of utilizing augmented reality as a method of information provision in a bridge may be
very effective.
Various studies are currently being carried out regarding such augmented reality technologies
along with the recent development of mobile devices, and applications to a diverse range of
industries including aerospace, automobile, and ships are increasing rapidly. The augmented
reality display technology, HUD (Heads-Up Display) system is being used in fighter jets and
commercial airliners to contain the time of the pilot and enhance the capabilities of the pilot (Kim
K.H. 2008). Currently, attempts to apply this HUD technology to the automobile industry are
being carried out in order to minimize visual interference to the driver while providing essential
information while driving. Generally, ships are relatively slower than aircraft or automobiles.
However, navigational equipment has been diversified with the development of new technologies.
Therefore, ships officers are requesting the efficient display of information on the bridge, and
various navigational aids systems are being researched and developed using augmented reality
technology as a means of displaying data.
In the European e-Navigation project, ACCSEAS, a wearable navigational aids system was
developed using HMD (Head-Mounted Display), as shown in Figure 1 (a), in order to enhance the
target recognition speed of the navigator. Though simulation experiments were conducted, further
research using the position data of the ship and navigator is still necessary for accurate registration
(ACCSEAS, 2015). Additionally, various navigation support applications have been developed
recently for mobile devices, as shown in Figure 1 (b), and such applications are being used in the
navigation of small ships, like yachts (B&G, 2015).
Figure 1: User Interfaces of AR Navigation System
(a) ACCSEAS Project, (b) Mobile application of SeaNav, B&G Source: ACCSEAS (2015), B&G (2015)
However, these systems all have different user interfaces, different methods of displaying
information, and different types of data expressed with no agreed upon standard, causing
inconveniences to the user and restrictions in their usage. Also, the indiscreet provision of
inaccurate data can result in serious problems such as marine accidents, further necessitating
research on what kind of data is to be shown and how such data will be delivered (Olivier Hugues,
2010). In this paper, a survey of navigators was conducted to determine key elements necessary
for navigation, and we then propose an augmented reality-based user interface which can
effectively express such data. Moreover, a navigational aid system to implement the proposed
OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality
24
interface is detailed, and we review experimental results from ship-handling simulator and an
open sea test to verify the efficiency of the proposed system.
III. Navigational Aids System based on AR
3.1. User Requirements As the first step in designing and implementing an augmented reality-based navigational aid
system, a survey was carried out to analyse user requirements. This survey was composed of
questions to determine necessary navigation data, effective methods of presenting such data, and
the arrangement or location of the presentation of the data. The survey was completed by 20
experienced ship officers. Analysis of the survey results revealed that most respondents preferred
navigation data to be presented and arranged in ways similar to how they are presented in current
navigation equipment and desired new functionality including the display of distance to a fairway.
Table 1: List of the Maritime Service Portfolios (MSPs) – taken from Annex 7 of NCSR 1/28)
Items Details Unit Type
GPS UTC/LAT/LON/SoG/CoG date/time/degrees/knots/degrees $xxRMC
heading heading degrees $xxHDT
STW speed through water knots $xxVHW
AIS AIS messages - $xxVDM
echo Sounder depth meter $xxDPT
tidal Current tidal current direction/speed degrees/knots $xxVHW
wind wind direction/speed degrees/knots $xxVWR
RoT rate of turn degrees/min $xxROT
rudder indicator rudder indicator degrees $xxRSA
RADAR target RADAR tracked target - $xxTTM
CCTV camera color image, pan/tilt degrees/degrees JPEG
Ownship attitude roll/pitch degrees/degrees -
Figure 2: User Interface of AR Navigation System
OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality
25
In addition, respondents showed positive responses toward the augmented reality system being
developed, stating that it could be used effectively in the entry and departure of ports. On the
other hand, responses showed that integration with the ECDIS system is necessary for the
provision of accurate navigation data and that data regarding fairways, anchorage, land area, and
planned routes are necessary for safe operation. Table 1 shows the necessary navigation data for
the augmented reality navigational aid system obtained from the survey results.
3.2. Design of User Interface
The information for the navigational aids system were categorized into 1) data regarding the
ownship in operation, 2) data regarding traffic ships, and 3) ECDIS data based on the survey
response analysis results. As shown in Figure 2, data regarding the ship in operation are mainly
displayed through the ODD (Overhead Data Display) in the bridge, composed of the location,
bearing, speed, and rate of turn (RoT), and displayed on the upper portion of the augmented
reality display using an intuitive interface, such as a graph. Data of traffic ships are displayed as
overlays on their corresponding positions, where detailed data was additionally displayed when
the user selected the data. The detailed data was generated based on the AIS data, and the targets
of RADAR and AIS were distinguished using symbols. The ECDIS data was displayed
selectively on the location of the ship in operation and followed the standard of ENC (Electronic
Navigational Charts) symbol. Additionally, the user was able to select whether to display each
data, and it was designed so that the user is able to adjust the data display color.
3.3. System Configuration
The hardware of the system is composed of a PTZ (Pan/Tilt/Zoom) camera, AHRS (Attitude
and Heading Reference System), NMEA Combiner, and user console with an additional joystick
device for camera control. The software system is composed of a data manager module, user
interface module, registration module, and augmented image rendering module. Figure 3 shows
the data flow between each module.
The first stage of the system collects navigational data through the NMEA combiner, and the
ENC data and planned route data are extracted through the ECDIS integration module. Following
this, the user interface module uses the collected navigation data to generate the display
component elements, processes user input such as through the joystick, and finally generates the
augmented image in real time through the registration and rendering modules for provision to the
navigator. In this experiment, a simulator system was also used to conduct tests in various
maritime conditions, and a simulator integration module was included for this. The detailed
specifications of each module are as follows.
OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality
26
Figure 3: Configuration of Navigation Aids System based on AR
(a) Data Manager Module
The data manager module processes navigation data collected from various navigation devices
of the bridge and performs integration and management functions. The collected navigation data
in our experiment followed the NMEA-0183 standard, was synchronized based on the GPS time
(UTC, Universal Time Coordinated), and was shared with other modules through Shared Memory.
And, multi-threading was used for performance enhancement to allow parallel processing of the
input navigation data.
(b) ECDIS Module
ENC are vector type charts that conform to International Hydrographic Organization (IHO)
specifications, as contained in Publication S-57. S-57 includes various feature object classes and
their attributes. The proposed system extracts and uses elements necessary for the augmented
reality navigational aids system from the ENC data of ECDIS. Data determined through analysis
of the requirements is extracted from ENC, and the extracted data is then stored in a separate
database system and displayed selectively depending on the location of the ownship in operation.
In addition, the distance to the next waypoint and the estimated arrival time data are calculated
from the route plan inputted through ECDIS.
(c) Unser Interface Module
The user interface module is the user interface (GUI) provided to the navigator and performs
the task of user input processing through mouse and joystick operation. Additionally, display
components are generated using the collected navigation data, and the option of whether to
display each component along with its colour is reflected.
(d) Augmented Image Rendering Module
In the augmented image rendering module, an augmented image is generated by integrating
the collected navigation data and ENC data. In order to generate the augmented image, the
augmented elements are arranged in a virtual 3-dimensional space and integrated with the real
world image in real time. In this study, a 3-dimensional graphics engine (Unity3D) that supports
GPU acceleration was used to enhance the performance of the image rendering. Additionally, the
OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality
27
registration process of matching the position of the virtual object with the position of the real
world image needs to be performed in advance for the virtual reality system. In this module,
registration is carried out by positioning the camera in the virtual space according to the ownship
position (latitude and longitude) and attitude (pitch and roll) data obtained in real time through the
GPS and AHRS sensors installed on the ship.
(e) Simulator Integration Module
A ship-handling simulator was used in this study in order to assess the applicability of the
implemented augmented reality-based navigation support service and system for various maritime
environments. The simulator integration module virtually generated a camera image and sensor
data, and the generation period and types of data used were identical to the actual ship
environment.
IV. Experiments
In this study, a prototype of the AR navigational aid system was manufactured based on the
implemented technology. Using the prototype, simulated and real ship application experiments
were performed. For each experiment, each function of the prototype was tested for normal
operations, application feasibility to the bridge environment was evaluated, and usability as
navigation equipment was assessed. Table 2 shows the specifications of the prototype used in the
experiment.
Table 2: Specification of Prototype system
Specification
PC system
CPU Intel i7 3.0GHz
RAM 16GB
GPU NVIDIA GTX750
Operating System Windows 7 pro 64bit
Monitor LCD monitor, 23 inch
PTZ Camera AXIS P5534E (image resolution : 1280×720 pixels)
4.1. Experiment Conditions The subjects of all the experiments were experienced ship officers. After being sufficiently
informed of the system instructions before the experiments were carried out, the navigator
subjects were instructed to use the system for port entry and departure situations. After the
experiment was finished, opinions on the usability of the system and additional requirements were
collected through surveys and interviews. For the simulator experiment, we utilized an FMB (Full
Mission Bridge) class ship-handling simulator system from the Korea Research Institute of Ships
and Ocean Engineering (KRISO). In order to make the experiments as realistic as possible, the
experiment scenario was designed using marine traffic analysis data from Busan North Port (Safe
Tech Research, 2013). For the experiment at open sea, the prototype system was installed on the
OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality
28
training ship, “Saenuri” of Mokpo National Maritime University, and the experiment was carried
out using entry and departure conditions at Mokpo Port. The detailed experiment conditions are
shown in Table 3, and the experiment environment is shown in Figure 4.
Figure 4: Experiments for usability test (a) AR navigation system with ship-handling simulator system, (b) Saenuri at Mokpo harbour,
(c) prototype of AR navigation system
Table 3: Experiments condition
Simulation test Sea test
target harbor Busan port entry Mokpo port entry
ownship Tanker SAENURI
traffic-ships 15 vessels > 30 vessels
weather dense fog light fog
# of subjects 12 5
4.2. Experiment Results The experiment included the process of identifying problems during the operation of the
prototype system along with each function. Experiment results showed that the hardware and
software systems operated in a stable manner in the simulator and real ship environment, and the
average augmented image refresh rate, which determines the performance of the entire system,
was measured to be above 30Hz. In addition, the data manager module processed all the received
data without data loss. Figure 5 shows the user screens of the simulator and real ship experiment.
The ownship information was displayed as a graph in the upper portion of the screen by
integrating the navigation data as shown in Figure 6 (a), and the azimuth circle data shown below
the graph operated according to the bearing of the ship in operation. The traffic ship information
was directly displayed on the location of the corresponding ship by integrating the RADAR and
AIS data as shown in Figure 6 (b). This feature is more intuitive than existing navigation
equipment in determining information of traffic ships; however, there were cases when the
overlay on the position of the other ships actually made recognition difficult. The ECDIS data was
displayed the ENC symbols as shown in Figure 6 (c). Since the same size symbols were used, it
was found that there were difficulties in identifying its distance. Meanwhile, the bearing and
distance information to the next waypoint calculated from the route plan data were displayed
according to the ownship information.
OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality
29
Figure 5: Screenshot of Experiments (a) simulation test, (b) sea test
Figure 6: Results of experiment (a) ownship information, (b) trafficship information, (c) ECDIS information
4.3. Analysis of User Survey After the simulator experiment, a survey was carried out on the officers with regard to the
satisfaction level for each function of the augmented reality system. The survey questions were
composed of 5 scale items and miscellaneous opinion items.
Analysis of the survey responses reveals that levels of satisfaction regarding the ownship
information was highest in items asking the satisfaction level for each function, as shown in
Figure 7, while the level of satisfaction of the traffic ship information was relatively low.
Additionally, the level of satisfaction for the entire system showed high points for efficiency and
effectiveness, while legibility and clarity obtained lower than average points. The relatively low
level of satisfaction for legibility and clarity was thought to be due to the difficulty in
distinguishing information, as the information overlapped in multitudes when other ships were
concentrated near the horizon during port entry and departure.
The officers who participated in the experiment were interviewed. They gave mostly positive
feedback with regard to the effectiveness of the augmented reality navigation equipment; however,
they also expressed concern with regard to the additional workload necessary to use the new
navigational equipment. Additionally, for the information display method in the augmented reality
image, responses varied depending on the inclination or preference of the respondent. It is evident
from the data that development of user interfaces optimized for each user is necessary.
OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality
30
(a) (b)
Figure 7: Survey results of AR navigation system (a) satisfaction of functions, (b) satisfaction of AR system
V. Conclusion
Although various types of navigation equipment with advanced technology are available to
help enhance operation safety for navigators, further research on the method of efficiently
displaying and servicing information rather than indiscriminately providing a bounty of
information is necessary. In this regard, this paper proposed a navigation aid system based on
augmented reality technology, and the proposed system provides various overlaid navigation
information on images from cameras to support swift and accurate decision-making by officers.
The ship-handling simulation and real vessel application experiments were carried out to verify
the functionality of the system. Through our experiments, stable operation of the prototype
hardware and software in a real ship environment was verified. In a survey conducted after the
experiment, most participants gave positive feedback with regard to the augmented reality
navigation aid system but expressed concern with the increase in workload from the new
navigation equipment. There were also cases when registration errors occurred during augmented
image registration due to location errors from the GPS, and elements that need improvement were
identified, including difficulty in distinguishing data due to excessive amount of information
displayed on the horizon.
As research and development continue, various navigation aid systems based on transparent
displays will be further developed and utilized in the bridge to maximize the effect of augmented
reality technologies. Such use of augmented reality technology is expected to prevent marine
accidents and increase ship operation efficiency in the future.
Submitted: March 8, 2016 Accepted: June 25, 2016
VI. Acknowledgements
The Contents of this paper are the results of the research project of Ministry of trade, industry
& energy (10041790, Development of Advanced Ship Navigation Supporting System based on
Oncoming International Marine Data Standard).
OH, PARK, KWON / Advanced Navigation Aids System based on Augmented Reality
31
References
B&G (2015), http://www.bandg.com Jeong, J. S. (2012), “A Study on the User Requirements for the Support of a Safe Navigation of the
Incoming Vessels to Port Waterways and the Outgoing Vessels”, Research report of Korea Research Institute of Ships and Ocean Engineering, pp. 73-83.
Jung M. (2015), “Analysis of User Requirement for the Improvement of ECDIS to Enhance Navigational Safety and Work Efficiency”, Journal of Navigation and Port Research, Vol. 39, No. 3, pp. 141-147.
Jung M. (2016), “User Requirement Analysis of ECDIS for the Development on S-Mode Guideline”, Journal of Navigation and Port Research, Vol. 40, No. 3, pp. 89-95.
Kim K. H. (2008), “Application of Head-Up-Display Technology to Telematics”, Electronics and Telecommunications Trends, Vol. 23, No. 1, pp. 153-162.
Mads Bentzen (2015), “ACCSEAS Final Report” Ministry of Oceans and Fisheries (2015), “Causal analysis of marine accident”, http://www.mof.go.kr Olivier Hugues (2010), “An Experimental Augmented Reality Platform for Assisted Maritime
Navigation”, ACM. Virtual Reality International Conference, pp. 87-92. SafeTechResearch (2013), “Maritime Safety Audit for New Marina Terminal of Busan North Port”
There is no conflict of interest for all authors.