helmet-mounted display technology for eva training in nasa

50th International Conference on Environmental Systems ICES-2021-276 12-15 July 2021

U.S. Government Funded Work

Helmet-Mounted Display Technology for EVA Training in

NASA’s Neutral Buoyancy Lab

Janine R. Moses1

KBR, 2400 NASA Parkway, Houston, TX 77058 and University of California-Davis, Davis, CA 95616

James R. Stoffel2

KBR, 2400 NASA Parkway, Houston, TX 77058 and

Ruby Z. Houchens3

KBR, 2400 NASA Parkway, Houston, TX 77058 and University of California-Davis, Davis, CA 95616

Jocelyn T. Dunn, Ph.D.4

KBR, 2400 NASA Parkway, Houston, TX 77058

Stephen K. Robinson, Ph.D.5

University of California-Davis, Davis, CA 95616

and

Andrew F.J. Abercromby, Ph.D.6

NASA Johnson Space Center, 2101 NASA Parkway, Houston, TX 77058

The Human/Robotic/Vehicle Integration and Performance (HRVIP) Lab at University of

California, Davis is collaborating with NASA’s Johnson Space Center (JSC) to design and test

an extravehicular activity (EVA) spacesuit helmet-mounted display (HMD) to enhance

astronaut situational awareness during underwater EVA training. An EVA HMD will enable

astronauts to monitor and react to real-time information including physiological biometrics,

spacesuit status, environmental factors, task procedures, and navigation aids. To meet

operational EVA challenges, HRVIP Lab is partnering with the JSC’s Human Physiology,

Performance, Protection, and Operations (H-3PO) Lab to create HMD prototypes and test

them during underwater EVA training in JSC’s Neutral Buoyancy Laboratory (NBL). Two

HMD mounting styles were designed and tested in the NBL. The swing arm HMD mount holds

the display a short distance in front of the helmet to allow focusing on text-based real-time

data. The surface HMD mount, positioned in the astronaut’s peripheral vision on the helmet

visor, displays flashing colors as a minimal distraction alert to the user to check system status.

NBL testing of the HMD prototype during 2020 has resulted in the following findings: (1)

users (astronauts) found the real-time biofeedback and EVA parameters useful and readable;

(2) there was minimal physical conflict between the HMD hardware on the spacesuit and EVA

training operations; (3) the peripheral visual cues from HMD’s visor surface mount were

effective only in certain scenarios; and (4) voice control enabled astronauts to use HMD

autonomously, but also requires system improvements for increased reliability. This HMD is

a test bed for evaluating data visualizations and interfaces for potential future flight

1 EVA Informatics Intern, KBR, 2400 NASA Parkway, Houston, TX 77058 and graduate student, UC Davis, Davis,

CA 95616 2 EVA Human Performance & Analog Engineer, KBR, 2400 NASA Parkway, Houston, TX 77058 3 EVA Informatics Intern, KBR, 2400 NASA Parkway, Houston, TX 77058 and undergraduate student, UC Davis,

Davis, CA 95616 4 Human Performance Engineer, KBR, 2400 NASA Parkway, Houston, TX 77058 5 Director, UC Davis Center for Spaceflight Research; Professor, UC Davis, Davis, CA 95616 6 H-3PO Laboratory Lead, NASA Johnson Space Center, 2101 NASA Parkway, Houston, TX 77058

International Conference on Environmental Systems

2

informatics. Feedback from these HMD evaluations will inform future heads-up displays, both

for EVA training and for the next-generation spacesuit.

Nomenclature

API = Application Programming Interface

ARGOS = Active Response Gravity Offload System

ASR = Automatic Speech Recognition

AWS = Amazon Web Services

BTU = British Thermal Unit

CAD = Computer Aided Design

COTS = Commercial Off-the-Shelf

CWS = Caution and Warning System

DAQ = Data Acquisition

ECS = Environmental Control System

EMU = Extravehicular Activity Mobility Unit

EVA = Extravehicular Activity

H-3PO = Human Physiology, Performance, Protection, and Operations

HMD = Helmet-Mounted Display

HRVIP = Human/Robotics/Vehicle Integration and Performance

HUD = Heads-Up Display

IRIS = Intelligent Response and Interaction System

ISS = International Space Station

JSC = Johnson Space Center

NASA = National Aeronautics and Space Administration

NBL = Neutral Buoyance Laboratory

OLED = Organic Light-Emitting Diode

PET = Phase-Elapsed Time

PGT = Pistol Grip Tool

PIL = Python Imaging Library

PLSS = Portable Life Support System

PoE = Power over Ethernet

TRL = Technology Readiness Level

xEMU = Exploration Extravehicular Activity Mobility Unit

I. Introduction

As human space exploration extends to deep space, communication delays increase. To supplement the limited

communication with Mission Control Center during exploration missions, extravehicular activities (EVAs) will

require new technology to enable crew autonomy. Heads-up displays (HUDs) aim to enhance astronaut situational

awareness, providing crew with real-time information about their EVA to help inform crucial decision-making1. As

an incremental step in beginning to utilize HUD technology for EVA training, the NASA Johnson Space Center’s

Human Physiology, Performance, Protection, and Operations (H-3PO) Laboratory has developed a sensor system to

enable real-time biofeedback for astronauts during EVA training in JSC’s Neutral Buoyancy Laboratory (NBL).

Specifically, this metabolic rate data collection system provides an approximation of crewmember physical workload

and spacesuit consumables usage. With this EVA informatics project, the H-3PO laboratory aims to enhance astronaut

training by tracking differences in EVA task designs and techniques to identify areas to improve and enable

efficiencies in EVA performance. In addition, H-3PO laboratory personnel plan to utilize this project as a test bed for

evaluating data visualizations and interfaces for potential future flight informatics systems.

This EVA Informatics project – Helmet-Mounted Display (HMD) – is a joint collaboration between University of

California, Davis’ Human/Robotics/Vehicle Integration and Performance (HRVIP) Laboratory and the H-3PO


3

Laboratory. HMD is a low-cost and low-tech prototype that was

developed and tested in 2020 for EVA training in the NBL (see Figure

1).

During this testing, astronauts had access to real-time biofeedback

data for the first time during NBL training. Another benefit of the project

is that user feedback from HMD is helping to inform the development of

more advanced HUDs for the Exploration Extravehicular Mobility Unit

(xEMU) and provides a test bed in a high-fidelity operational

environment for other informatics display concepts.

For this testing, HMD served two distinct applications. First, it

presented readable information, such as real-time physiological data.

Second, HMD presented a visual peripheral signal (e.g., flashing light)

on the display, as a precursor to a caution and warning system (CWS)

that would serve as a minimal-distraction alert to the EVA astronaut to

check system status. This paper describes the HMD hardware assembly

(Section II) and software systems (Section III) and their integration with

NBL testing (Section IV). The paper also outlines results and feedback

from subjects/crewmembers (Section V), with a focus on the

effectiveness and utility of HMD during underwater EVA training.

II. Hardware

The HMD hardware is low-tech, low-cost, and quick turn-around

as this project was developed and tested in less than a year. Extensive

subsystem testing was performed on each waterproofed piece of

hardware and each electronic component. The fully assembled system

then underwent testing, both before and after it was integrated with the

extravehicular mobility unit (EMU). This section outlines the major

components of HMD, with a focus on the elements with which the user

interacts: the display screen and mounting hardware.

Raspberry Pi & DAQ Box: A Raspberry Pi Model 3 B+ and a Power

over Ethernet (PoE) adapter are housed in an aluminum alloy

container, the data acquisition (DAQ) Box. The DAQ Box was

purchased commercial-off-the-shelf (COTS) and modified in-house to

be waterproof for more than 6 hours at 40-foot depth. It was secured

with Velcro in an empty cavity in the EMU’s portable life support

system (PLSS).

Cable Harness: The cable harness, designed and built in-house,

connects the display screen to the Raspberry Pi, and the Raspberry Pi

to its power and network source. Because Bluetooth and WiFi do not transmit underwater at long distances, HMD’s

Raspberry Pi is hardwired both for network connectivity (Ethernet) and power (via power over Ethernet). The

power/Ethernet cable exits the Raspberry Pi DAQ Box and is routed through the PLSS, through the umbilical in the

NBL (see Figure 2), and is connected to an Ethernet port on the

pool deck. The cable for the display screen is routed through the

PLSS to the helmet where the mounting hardware is attached.

Display Screen2: The display screen from Adafruit Industries

is a 1-inch × 1-inch (2.54-cm × 2.54-cm) OLED, 16-bit color

screen with 128 × 128 resolution. It was waterproofed in-house

with a transparent silicone elastomer material, see Figure 3. The

display screen is positioned 6-8 inches from the user’s eyes

(depending on where their head sits inside the EMU helmet).

This eye-to-display distance is slightly shorter than the distance

required for a naked human eye to resolve an image (9-10

inches). Thus, large text size is readable on HMD, but it is more difficult for the user to resolve detailed images or

small text size, especially if they use focal lenses in the EMU helmet. Typically, at water depth, colors are lost because

Figure 1. HMD on subject in NBL. The

HMD swing arm is mounted on the right

side of the subject’s EMU helmet visor.

Figure 3. Potted display screen in swing arm.

Figure 2. Umbilical from EMU to pool

deck power/Ethernet source. Courtesy

of NASA.


4

the light wavelengths from the surface are absorbed. However, because HMD’s OLED is its own light source, just

inches away from the viewer (compared to light traveling from the surface), the colors of the OLED screen are not

compromised. The EMU helmet visor bubble does not distort the display and the water columnates the light from the

screen, making HMD slightly more clear and crisp when viewed underwater.

Mounting Hardware – Swing Arm Mount: The Swing Arm Mount Assembly, in Figures 4 and 5, was designed

and built in house. It integrates the display screen into a modular and reconfigurable design, and is mounted to the

helmet visor with suction cups. The swing arm assembly can be positioned in four distinct positions on either side of

the EMU helmet, in the upper and lower quadrants of the visor (+/- 30 degrees). The overall design incorporates a

noninvasive, impact resistant, and visor-friendly configuration with no impact or change to the existing EMU system.

A hazard analysis was conducted with spacesuit engineers, and it was concluded that the scratch risk to the helmet

bubble is minimal. The swing arm assembly was created in a Computer Aided Design (CAD) program and

manufactured via Additive Manufacturing, also known as 3D printing. The swing arm assembly is made of tough

polylactic acid (PLA) filament material, ideal for printing lightweight and strong functional prototypes. The swing

arm mount was assembled with corrosion resistant fasteners to survive the underwater environment. To enable this

mount to conform to the curvature of the visor, the swing arm includes three lightweight nylon swivel joints (spherical

bearings) and strong suction cups. This provided a flexible platform for location, position, orientation, and safety for

early development.

Mounting Hardware – Surface Mount: The surface mount assembly as shown in Figure 6, also was designed

and built in house. It integrates the same display screen into a simple chassis, and contains suction cups with swivel

joints. The surface mount assembly was designed to the same overall requirements, manufacturing, and safety

standards. It can be integrated on either side of the EMU helmet. For this phase of the study, the surface mount

assembly was integrated approximately at the visor’s midline and as far aft as possible, see Figure 7.

Figure 4. HMD swing arm mount assembly.

Figure 5. HMD swing arm on

EMU visor.

Figure 7. HMD surface mount on

EMU visor. Partially obstructed

by the helmet light.

Figure 6. HMD surface mount assembly.


5

III. Software

A. Data Architecture and Metabolic Rate Data Collection at the NBL

At the NBL, an umbilical connects the Class III-W EMU spacesuit to the NBL’s environmental control system

(ECS) system that provides power, data, and breathing gas to the spacesuit. An earlier project in 2019 integrated a

metabolic rate data collection system with the ECS panel on the pool deck at the NBL. This project has provided the

foundation for the data architecture and data security for the HMD project. The metabolic rate data collection system

is a Raspberry Pi-based data acquisition system with a touchscreen application for starting and stopping data collection

(independent from HMD). This metabolic rate data collection system measures the flow rate of the breathing gas

supply to the suit and the concentration of carbon dioxide in the breathing gas returned to the ECS panel. These two

measures are combined to determine the volume of carbon dioxide that the suited crewmember is exhaling to estimate

metabolic rate or energy expenditure during the time course of the EVA training events in the NBL. This data is

processed and stored in a Cloud-based data architecture that enables remote access to these data via a custom-built

web application. With HMD, real-time metabolic rate data also can now be accessed by the crewmember as well.

The Raspberry Pi is the “brains” of HMD. It contains an SD card that hosts the Python code running HMD.

Commands are sent to the Amazon Web Services (AWS) GovCloud via an application programming interface (API),

either using voice control or remote manual control. The HMD scripts utilize open source software from Adafruit and

from the Python Imaging Library (PIL). Figure 8 shows a sequential concept of operations for controlling HMD during

NBL operations, from the crewmember verbalizing a voice command to the display changing modes accordingly:

1. Crewmember utters wake word.

2. Voice control software transcribes command and picks out key words or phrases.

a. Alternatively: the HMD team remotely and manually sends specific command words/phrases.

3. Command is sent via API to the GovCloud.

4. Raspberry Pi extracts the command from the GovCloud and interprets it.

5. If the command includes a request for metabolic rate data, then that data is extracted from the GovCloud

and processed (i.e., time-averaged) on the Raspberry Pi.

Figure 8. Data flow during NBL test. The green arrows represent data flow of commands to control HMD.

The blue arrows represent data flow for real-time metabolic rate.


6

6. The Raspberry Pi sends a signal to the display to change modes, turn on/off, etc.

B. Voice Control

Voice control enables subjects/crewmembers to interact with HMD autonomously. HMD utilizes a voice control

system called Intelligent Response and Interaction System (IRIS) which is currently being developed and tested for a

helmet display project for the xEMU. Audio feed was stripped from the NBL communications/video system and fed

to designated computers that host IRIS.

IRIS is made up of several key components:

1. Wake word: the wake word for the first subject, EV1, was “grapefruit” and the wake word for the second

subject, EV2, was “blueberry.” These were chosen because they are multisyllabic, distinct, and not likely to

be uttered during typical NBL operations. A beep indicates that IRIS recognized the wake word and is

beginning to transcribe speech.

2. Speech-to-text: Google’s Automatic Speech Recognition (ASR) transcribes the speech that is heard after

the wake word is uttered.

3. Natural language understanding: Google’s Dialogflow picks out phrases/words from a predetermined list

(i.e. “phased elapsed time”, “elapsed time”, or “PET”; “metabolic rate” or “met rate”).

4. Interfaces: external API through which to track IRIS in real-time.

IRIS handles both conversational and terse interactions, as long as the key words/phrases are uttered. For example,

a crewmember can say either “grapefruit, what is my phase-elapsed time?” or “grapefruit, PET.” Once a command is

sent, the HMD python script decodes the meaning of the command and directs the display screen to change display

modes accordingly. This was the first time that voice control was utilized with NBL communication loops. As

described in results (Section V), future work will be required to improve the audio integration and investigate why

Figure 9. HMD display modes. From left to right: welcome screen, metabolic rate (numerical),

metabolic rate (bar graph), phase-elapsed time (PET), timer #1, static image (EV locations), static

image (ISS translations), flashing blue square, flashing yellow circle.


7

some errors occurred during testing. During HMD testing, HMD personnel closely monitored the voice control system

and occasionally had to interfere if there was a missed transcription. The team would reset IRIS or manually send

command payloads to the display via the API, in place of voice commands.

C. Display Modes

Figure 9 shows the HMD display modes. Note that the metabolic rate and PET are real-time data. In the future,

HUDs may be used as navigational aids, so two of the display modes (the static images) mimic navigation settings.

IV. Underwater Testing

Prior to integrated underwater testing, extensive

subsystem testing was performed to ensure the

hardware/cable harness was waterproof and the

software/voice control system functioned as expected. The

NBL is a high-fidelity EVA training analog, mimicking

microgravity and housing a 1:1 scale mockup of the

International Space Station (ISS). HMD was integrated with

EVA training in this environment to better understand the

advantages of utilizing HMD as an EVA training aid. The

following section walks through a typical HMD test day in

sequential order, focusing on the choreography between

HMD and typical NBL operations.

Pre-Dive Brief: Before the day of the NBL run, the HMD

team briefs the subjects about the upcoming test and the

subjects choose their swing arm location (upper left/lower

left/upper right/lower right).

HMD Donning: The morning of the NBL test, the HMD

team sets up and tests the hardware and cable harness before

the subject dons the EMU. After suit donning, when the

helmet is secure, the HMD team secures the suction cups of

the swing arm onto the visor in the predetermined position. The subject verifies that there are no major conflicts

between HMD and visibility or movement.

Display Checks: Throughout the first half of the NBL run (over the course of 3-4 hours), the HMD team sits in the

test conductor room. During designated periods, the HMD team walks the subject through a HMD check by telling

them specific voice control commands to repeat. Meanwhile, the HMD team monitors the voice control system and

the Raspberry Pi (via remote access) to ensure that the commands are being sent and interpreted correctly. The subjects

also are encouraged to utilize HMD independently to check real-time data such as metabolic rate or phase-elapsed

time. Figure 10 shows a crewmember

checking the PET right after egressing the

airlock in the beginning of the NBL run.

Mounting Hardware Reconfiguration:

Approximately half-way through the NBL

run, two HMD team members dive in the NBL

to reconfigure the hardware. They detach the

swing arm suction cups from the visor,

remove the display from the swing arm, place

the display in the surface mount, and attach

the surface mount suction cups to the visor.

This process takes about 5 minutes. Figure 11

shows a HMD diver performing the

reconfiguration.

Peripheral Signal Tests: After

reconfiguration, the HMD test team (back in

the test conductor room) manually sends a

signal to HMD to display flashing

Figure 11. Mounting hardware reconfiguration. HMD diver,

Janine Moses, attaching the surface mount to the EMU visor.

Figure 10. HMD check during NBL testing. A

crewmember checks the PET during airlock egress at

the beginning of the NBL training run (reflection of

display through helmet is also visible).


8

colors/shapes. Without prior knowledge of when the signaling will take place, the subjects are asked to report when

they notice the flashing display. Each time a signal is sent to HMD, the response time is recorded: the time from when

the visual signal first appeared on the display to when the subject/crewmember reported seeing the flashing colors.

HMD Doffing: At the end of the NBL run when the subjects are out of the water, HMD is detached from the visor

so that the helmet can be removed. The rest of the hardware and cable harness is also removed from the EMU for post-

test checkouts and maintenance.

Post-Test Debrief & Survey: At the end of the day, the HMD team debriefs with the subjects/crewmembers and

administers an electronic survey (See Section V. Results).

V. Results

This section reports feedback from the users about their experiences with HMD during EVA training in the NBL

and about the HMD hardware itself. Feedback from HMD evaluations will inform future HUDs, both for EVA training

and for the next generation spacesuit.

During November and December 2020, HMD was tested during 5 NBL runs, each with 2 subjects/crewmembers

(10 users total). The first 2 tests included EVA training personnel as the suited subjects, and the following 3 tests had

current NASA astronauts as test subjects. HMD performance feedback was captured during the NBL tests by recording

users’ verbal comments and after the NBL test through an electronic survey3.

A. HMD Positioning, Visibility, and Physical Conflicts

Users did not report any visibility conflicts during NBL Ops; they had an

unobstructed view of their chest-mounted workstations, even with HMD. If

subjects used a Fresnel lens in the helmet bubble, they were advised to mount

HMD opposite the lens to avoid visibility conflict (although none of the

individuals in this subject pool used a lens). Tests showed minimal physical

conflicts with HMD in the NBL, except for during airlock ingress/egress.

During these operations, HMD was accidentally bumped or knocked off on

three occasions during airlock ops because both crewmembers were in a

confined space with one crewmember’s feet near the other’s helmet.

Before the underwater runs, each test subject was given the choice of

desired position for the swing arm mount (lower left/upper left/lower

right/upper right). Table 1 indicates the position chosen by the

subjects/crewmembers, and Figure 12 shows examples of positioning. When HMD was reconfigured, the surface

mount was placed on the same

side (left/right) of the visor

where the swing arm was

originally placed. Subjects

considered a variety of factors

when choosing the swing arm

mount’s position:

• Position in the suit: some

subjects tend to ride low

in the suit during NBL

training so they chose the

upper left/right positions

for better visibility of the

display and of their

workstation.

• Cuff checklist location:

Some subjects chose to

place the display on the

same side as the cuff checklist so that they could check both simultaneously, while others chose to place the

display on the opposite side of the cuff checklist to minimize any visibility interference.

• Access to PGT/other tools: Some crewmembers chose to place the display on the opposite side of their tools

(mounted to their waist). For example, if the pistol grip tool (PGT) was mounted on the right side of the EMU

Table 1. Chosen HMD Positions

Position # Subjects

Lower Left 5

Upper Right 3

Upper Left 1

Lower Right 1

a) Lower Left Position b) Upper Right Position

Figure 12. HMD swing arms mounted on EMU visors.


9

tool mount, the subject chose to mount HMD on the left side to reduce visibility interference when accessing

the PGT.

• Dominant eye/field of view: Some subjects chose to place the swing arm on the opposite side from their

dominant eye so as to not obstruct their primary view. Others chose to place it on the same side as their dominant

eye so it would be easier to check when sweeping their field of view.

• Other helmet equipment: Subjects ensured that, if HMD was positioned in one of the lower quadrants, the

Valsalva device (inside the helmet to block one’s nose and equalize pressure) did not interfere with visibility.

B. User Feedback

This section focuses on (1) the usefulness of HMD content specifically for EVA training in the NBL and (2) the

visibility/readability of the content itself. Table 24 shows the 2 scales subjects used to rate each display mode (listed

in Section III: Software).

Note that this part of the

survey was only

administered to the test

subjects for NBL Tests 2-5,

so there are 8 responses. The

highest rated display modes

were the real-time metabolic

rate and phase-elapsed time

(PET). Figure 14 shows 7 of

8 subjects/crewmembers rated the metabolic rate mode as “probably useful” or “certainly useful”. Figure 14

summarizes that all 8 subjects/crewmembers thought that access to phase-elapsed time was “probably useful” or

“certainly useful.” Subjects thought the metabolic rate display mode was useful because it provided a quantified

measure through which they could compare their relative exertion levels for various tasks during EVA training.

Subjects thought the PET display mode was useful because it provided high-level insight into timeline progress. Most

subjects thought that the metabolic rate and PET data were visible, but 4 subjects reported that the labels on the display

for “Met Rate” and “PET” were too small or lacked color contrast (see Figure 9: Display Modes). Users did not report

eye fatigue, but most did report that it was hard to focus on the smaller text/numbers, especially when switching from

far-field focus (workstation) to near-field focus (display screen). This is expected because HMD does not address the

optical challenges of a near-eye display, beyond offering a displaced swing-arm mount to position the display away

from the subjects’ eyes.

During their NBL underwater runs, half (5 of 10) of subjects independently checked HMD for their metabolic rate

or PET outside of the designated HMD tests. Those subjects appreciated HMD’s access to operationally useful data,

and utilized the displayed information to make decisions and increase their overall situational awareness. For example,

one crewmember who felt discomfort in their EMU was debating about continuing with the NBL training run or

pausing the run, returning to the pool deck to readjust the suit, and then resuming the run. This crewmember

Figure 13. Metabolic rate usefulness for EVA training and visibility ratings. Most subjects rated the metabolic

rate (numerical) display mode as “probably useful” or “certainly useful” and as “acceptable visibility” or

“excellent visibility”.

0

1

3

4

0

1

2

3

4

5

Certainly notuseful

probably notuseful

probablyuseful

certainlyuseful

NU

MB

ER O

F C

REW

USEFULNESS RATING

Met Rate (Numerical) Usefulness Scale

0

1

2

5

0

1

2

3

4

5

6

Very poorvisibility

Poor visibility Acceptablevisibility

Excellentvisibility

NU

MB

ER O

F C

RE

W

VISIBILITY RATING

Met Rate (Numerical) Visibility Scale

Table 2. Survey Scale. Subjects/crewmembers rated each display mode with scores

for (a) Usefulness (left) and (b) Visibility (right).


10

independently checked HMD for the phase-elapsed time, determined that it was still early in the NBL training run,

and chose to have their suit readjusted on the pool deck before proceeding.

C. Voice Control Feedback

The audio recordings of the subjects/crewmembers during the NBL runs were analyzed to determine the scenarios

under which conflict arose between the users’ HMD voice control and any verbal communications with the other EVs

or test conductor/personnel. The consensus was that voice control significantly enhances HMD use, interaction, and

autonomy. As such, the voice control system must continue to be improved for it to be integrated with nominal

operations, because it was the greatest source of frustration for users. There were a few times when a

subject/crewmember wanted to interact with HMD but needed to wait until a conversation between other test personnel

were complete before uttering voice commands. The following list summarizes the feedback received from the users

and from other test personnel:

Pros for the automatic voice interpretation software:

• Wake words (“blueberry” and “grapefruit”) were easy to memorize.

• The “beep” sound to acknowledge that the wake word was recognized by the software was helpful.

• Good at interpreting garbled speech, including English-as-a-second-language accents.

Cons for the automatic voice interpretation software:

• Required a lot of technical support in development stages (during earlier NBL tests).

• Not always consistent with picking up wake word (due to NBL audio quality).

• Sometimes “grapefruit” sounded like “break-break” which is used to completely halt an NBL test.

D. Peripheral Signaling

After HMD reconfiguration, the surface mount, positioned in the subject’s peripheral vision, displayed flashing

colors at the discretion of the HMD test team. Each time a signal of flashing colors was shown on HMD, the response

time was recorded from when the visual signal first appeared on the display to when the subject reported seeing the

flashing colors. Subjects most often noticed the visual signal (flashing colors) within 5 seconds under these scenarios:

• Their head was turned in a direction such that the HMD surface mount was in their visual field of view.

• They were translating along a truss section of the ISS mockup, also with the HMD surface mount in view.

• They were waiting for instructions from the test conductor, being repositioned by divers, or engaged in

other passive activities.

When the subjects were engaged in a more complex task or the display was not in their field of view, they often

did not notice the flashing signal. Any future caution and warning system based on peripheral signaling would

therefore require an audio alert to supplement (or replace) the visual cue. Also, during periods of significant chatter

over the NBL audio communications, a subject often had to wait for a lull in the conversation before reporting that

they had noticed the HMD signal, which resulted in inaccurately long recorded response times.

Figure 14. Phase-elapsed time (PET) usefulness for EVA training and visibility ratings. Most subjects rated

the PET display mode as “probably useful” or “certainly useful” and as “acceptable visibility” or “excellent

visibility.”

0 0

1

7

0

1

2

3

4

5

6

7

8

Certainly notuseful

probably notuseful

probablyuseful

certainlyuseful

NU

MB

ER O

F C

RE

W

USEFULNESS RATING

PET Usefulness Scale

0 0

3

5

0

1

2

3

4

5

6

Very poorvisibility

Poor visibility Acceptablevisibility

Excellentvisibility

NU

MB

ER O

F C

REW

VISIBILITY RATING

PET Visibility Scale


11

E. HMD Hardware Challenges

There were a number of challenges that arose during NBL operations related specifically to underwater testing.

Most of these challenges did not significantly affect the user’s experience, but they offer room for improvement during

future iterations of HMD technology. Examples include:

Potted display: The display and swing arm were jostled during airlock ingress and egress operations. On a few

occasions, the potting on the display tore when it was jostled, risking electronics exposure to water.

Suction cups: If the display was knocked by the subject/crewmember, a diver, or a mockup, the suction cups would

slide around, occasionally becoming detached from the visor.

Cable harness: If the display was mounted on the right side of the visor, there was 6-10 inches of excess cable that

needed to be secured in the PLSS. This was never a snag hazard, but it often required the HMD team to spend several

minutes managing the cables during HMD donning (on the pool deck) and reconfiguration (in the pool).

DAQ Box: The aluminum alloy container for housing the computing unit developed external chalky stains after

each 6-hour use in the chlorinated pool.

VI. Future Work

Work on this project over the course of the past year has elevated the technology readiness level (TRL) of this

device, from a concept to a fully operational HMD. However, this technology remains in its early phases of

development, and it is important that additional HMD capabilities are iteratively developed and tested. The results

from the initial HMD testing demonstrate that EVA informatics is a useful and important technology to study and

improve. In the next set of HMD evaluations, the team plans to inquire not just if each display mode was useful during

EVA training, but specifically why the visual content from a display mode was useful and how the information on the

display was utilized. The future work planned for HMD is outlined in the following subsections.

A. Upgrades for Additional NBL Testing

The next iteration of HMD will incorporate hardware upgrades specifically tailored to eliminate or mitigate the

hardware challenges outlined in Section V. In addition, in place of the visor-surface mount to test response to a

peripheral signal, HMD may utilize the swing arm mount for peripheral signaling. This will ensure that the flashing

light signal is visible to the

subjects/crewmembers regardless of the

task complexity with which they are

engaged. HMD software also will be

expanded to include more display modes,

such as custom text inputs for tool settings

and additional real-time biofeedback data.

The IRIS voice control system will be

improved to pick up the wake words with

greater consistency, a task that will require

additional integration with the NBL audio

communication system.

B. Upgrades for Use in Different

Spacesuits and Environments

Depending on the specific interests of

the EVA community, HMD may be

adapted to other spacesuits (see Figure 15),

such as the Mark III, Z-2, Z-2.5, and

xEMU, or to hard-hat surface-supplied

diving in the NBL. For example, the swing

arm mounting hardware may be redesigned

to accommodate the different shape of the

xEMU helmet bubble. The HMD technology and integration with NBL operations can be shifted to accommodate

these newer spacesuits, including during lunar (1/6-G) operations in the NBL.

HMD can be adapted to other training environments, such as the JSC Rock Yard. In the near-term, H-3PO

personnel also plan to utilize HMD for cognitive testing in the active response gravity offload system (ARGOS)

a) Hard-hat (Kirby Morgan 97) b) Z-2.5 Spacesuit

for Lunar EVA training

Figure 15. Lunar Ops in NBL. HMD can be modified to be used

during Lunar EVA training at the bottom of the NBL.


12

environment. Slight hardware or software upgrades may be necessary, but the overhead required to adapt HMD to

these environments will be significantly lower than at the NBL.

C. Upgrades for UC Davis Research

UC Davis’s HRVIP Lab has an elevated interested in pairing the use of this technology with motion-capture

software to gain a comprehensive understanding of pre-existing EVA training motion patterns and how the use of

HMD may enhance those patterns. HMD may be tested in a partial-G flight environment as we are considering HMD

implementation not only for EVA training in the NBL, but for EVAs on future space exploration missions.

VII. Conclusions

The HMD project provides an important first step in developing EVA Informatics to enhance astronaut autonomy.

Subjects/crewmembers provided high ratings for the usefulness and readability of real-time biofeedback and EVA-

related data, even utilizing HMD to make real-time decisions based off PET readings. HMD offers insight into the

benefits of real-time access to EVA Informatics with biofeedback, in addition to how EVA Informatics could be

integrated into NBL training on a daily basis. Even though the voice control system was the primary source of

frustration for users, when it worked successfully, it enabled greater autonomy. HMD is still a work in progress: it is

a low-tech and low-cost prototype that demonstrated the effectiveness and importance of EVA Informatics to enhance

training in a high fidelity analog environment. Results from HMD testing and lessons learned from the development

and test experience will help to:

1. Inform the next iteration of HMD for EVA training in the NBL.

2. Provide a test bed for low-cost and minimal impact testing of other HUDs in the NBL.

3. Inform the adaptation of HMD for use in other analog environments (JSC Rock Yard and ARGOS).

4. Inform the development of more advanced HUDs for the xEMU and other future spacesuits.

Even in its early stages, HMD proved to be effective and informative. HMD offers a glimpse into the benefits of using

biofeedback and other EVA parameters to enhance astronaut autonomy during EVA training.

Acknowledgments

This ongoing research is supported by the NASA JSC H-3PO Laboratory and by the NASA SSERVI Research

Institute: Radiation Effects on Volatiles and Exploration of Asteroids and Lunar Surfaces (REVEALS). Thank you to

Austin Alexander for software development and support and to the rest of the H-3PO team that implemented the

metabolic rate data collection system at the NBL. Thank you to the IRIS Voice Control Team, Aly Shehata and

William Baker, for providing the voice control software and support, and for the great teamwork and integration with

HMD. Thank you to Ryan Amick and Jerri Stephenson for the human factors-related input and guidance in creating

the surveys. Thank you to the NBL team: the test support personnel, test directors, divers, umbilical technicians,

spacesuit lab technicians, spacesuit engineers, flight leads, and NBL leadership for making HMD testing possible.

Thank you to the suited subjects and crewmembers for taking the time to learn about HMD and provide details about

their experience using HMD.

References 1. Carr, C. E., Schwartz, S. J., Rosenberg, I., “A Wearable Computer for Support of Astronaut Extravehicular

Activity,” Proceedings of the 6th Int’l Symposium on Wearable Computers, 0-7695-1816-8/02, IEEE, 2002.

2. Neighbors, H., Salazar, G., Steele, G., Sarma, K. R., Schmidt, J., Wiggs, J., Mejias-Rolon, Y., “OLED

Technology Evaluation for Space Applications,” AIAA Space, Pasadena, CA, 2015.

3. Mitra, P., “Human Systems Integration of an Extravehicular Activity Space Suit Augmented Reality Display

System,” Masters Thesis, Aerospace Engineering Dept., Mississippi State University, Starkville, MS, 2018.

4. Pauwels, R., Seynaeve, L., Bosmans, H., Bogaerts, R., & Jacobs, R., “Technical versus diagnostic image

quality in dental CBCT imaging,” European Congress of Radiology, C-2391, European Society of Radiology,

Vienna, Austria, 2013.

helmet-mounted display technology for eva training in nasa

Documents