augmented reality: telehealth demonstration application
TRANSCRIPT
Augmented Reality: Telehealth Demonstration ApplicationNicholas Boyd
Computer Science Department, University of NewHampshire, Durham, NH, [email protected]
Jacob HawkinsComputer Science Department, University of New
Hampshire, Durham, NH, [email protected]
Thomas YangComputer Science Department, University of New
Hampshire, Durham, NH, [email protected]
Scott A. ValcourtComputer Science Department, University of New
Hampshire, Durham, NH, [email protected]
ABSTRACTAugmented Reality (AR) as a technology will improve the waywe work and live in the future. The Microsoft HoloLens deviceallows for rendering of interactive virtual components into a realworld space. The HoloLens is an augmented reality headset and candisplay these virtual components in front of the user’s eyes, so thedata needed to complete a real-world task will always be available.The nature of a HoloLens device lends itself useful for applicationsin a healthcare setting. Potential benefits come from transitioningto a more hands-free environment such as allowing the loggingof data while in sterile environments without needing to sterilizerepeatedly from touching paper or tablet. This project developed anaugmented reality (AR) application to include a care plan trackerestablished by a patient’s doctor to allow the patient to do dailytasks without a health care worker’s supervision. The applicationdisplays the medications that the patient needs to ingest, daily tasksto complete, and health data to record. The application allows thephysician to retrieve useful patient information regularly withoutscheduled physicals. This project sets a baseline that will providefuture developers with documentation, research, and this sampleapplication to assist in the design and construction of more complexapplications in the future at the University of New Hampshire.
CCS CONCEPTS• General and reference; • Cross-computing tools and tech-niques; • Design; • Human-centered computing; • Visualiza-tion; • Visualization theory, concepts and paradigms; • Net-works; • Network types; • Cyber-physical networks;
KEYWORDSAugmented reality, Microsoft HoloLens, programming, network,cyberinfrastructure
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from [email protected] ’20, July 26–30, 2020, Portland, OR, USA© 2020 Copyright held by the owner/author(s). Publication rights licensed to ACM.ACM ISBN 978-1-4503-6689-2/20/07. . . $15.00https://doi.org/10.1145/3311790.3399629
ACM Reference Format:Nicholas Boyd, Jacob Hawkins, Thomas Yang, and Scott A. Valcourt. 2020.Augmented Reality: Telehealth Demonstration Application. In Practice andExperience in Advanced Research Computing (PEARC ’20), July 26–30, 2020,Portland, OR, USA. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3311790.3399629
1 INTRODUCTIONTelehealth and telemedicine have been growing in importance inserving the population with healthcare. [1] As these new areasemerge, they are prepared for advances in Augmented Reality (AR)for incorporation into the delivery of healthcare. To this end, ourproject team focused on the investigation of possible use cases forthe Microsoft HoloLens device in the healthcare field. The HoloLensis an AR computing platform that combines data presented in graph-ical form projected on a heads-up visual field in front of the user’seyes to augment the environment seen through the field glass.
The research involved in this project focuses on the developmentof the programming environment required to develop HoloLensapplications, along with identifying best practices for the use ofthe HoloLens in the telehealth field. [2] The project team consultedpublications and the Internet for development research, and usedthese strategies, along with a series of requirements gatheringsessions to create an application based on input from the Universityof New Hampshire (UNH) Nursing Department faculty.
The application developed demonstrates a care plan for a pa-tient at home, indicating for the patient what medication needsto be taken, exercises to be completed, and general daily tasks todocument.
2 RESEARCH APPROACHThe project team began with a requirements gathering exerciseinvolving two members of the UNH Nursing Department faculty.The discussion involved an open-ended set of questions with twomembers of the UNH Nursing Faculty: Ms. Marguerite Corviniand Dr. Gene Harkless. The group took the liberty to brainstormlong-term goals of an application involving the use of the MicrosoftHoloLens.
The two members of the UNH Nursing faculty are members ofthe UNH Telehealth Practice Center, [3] a research and teachingfacility focused on the effective connections between healthcare andtechnology. They explained the goal with their center is to movetowards putting care into the hands of the patient. This would bein the form of a care plan provided by the patient’s doctor. The
PEARC ’20, July 26–30, 2020, Portland, OR, USA Nicholas Boyd et al.
project team focused on the alignment of the application to be basedaround this concept, with the HoloLens serving as a tool for thepatient that would be provided by their doctor. The patient wouldbe given a set of instructions, the care plan, and those tasks wouldbe completed while wearing the HoloLens to assist the patient.
We solicited from the faculty members detailed information ondifferent types of patients that could benefit from a care plan withthe HoloLens. Since this application was designed to be a proofof concept and deployment details were out of scope, we chose tobuild the application to handle an initial set of data and tasks asdetermined by a simulated doctor providing the HoloLens withspecific patient actions and care plan as determined by the Nursingfaculty.
Generally, patient type input data would be a simple set of ques-tions that identified actions to create a care plan. These questionsmight be around the following questions:
• What are your goals and preferences for your health?• What is a long-term goal for your health?• What is a short-term goal for your health?• What do you feel is important for you to do each day?
From this line of questioning, we include two examples of careplans that could be installed into the application.
Patient 1: A cardiac patient – A 60 y/o woman who had open-heart surgery three months ago and is still in recovery. She liveswith her daughter (Ellie), son-in-law, and granddaughter. She de-veloped a care plan with her provider to meet her personal goalsand her health goals.
Have breakfast with Ellie at 7 amRemember to take medication Ticagrelor 90 mg andatorvastatin 80mg at 7 am (you can have the HoloLensproject a pill picture and prompt)Do exercise 8 am. – have HoloLens project the exer-cisesWeigh self daily; take Blood Pressure only once perweek (would be cool if this could somehow be sentvia the HoloLens)Take care of indoor plants 10 amCheck your email messages and pay bills 11:30 amTake medication with lunch at 12 pmHelp Ellie with her homework 2 pmHave dinner 5:30 pmWork on gardening plan and discuss with Ellie 6:30pmTake Ticagrelor 90 mg, metoprolol 100 mg , amlodip-ine 5 mg medication 7pm – have HoloLens projectpills picture and prompt
Example answers to the Person-centered planmight be as follows–
• What are your goals and preferences for your health? Tomanage my heart disease and go back to being active.
• What is a long-term goal for your health? To be able to ridemy bike for more than 4 miles.
• What is a short-term goal for your health? To be able to cookagain and spend time with my granddaughter.
• What do you feel is important for you to do each day? I enjoygardening. I would like to get to a point where I can gardeneach day.
Patient 2: Patient with Mild Cognitive Impairment –This is an80 y/o male who lives alone and has difficulty with memory issues.He has a dog, Minnow, and a neighbor who assists him. He has adaughter that lives in Vermont. He has worked with his providerto develop a plan to meet his every day personal goals as well ashis health goals. It would be helpful if there is a way to check theseoff so that when the patient does each one, he knows that he hascompleted the tasks and doesn’t get confused.
Walk Minnow at 7 amFeed Minnow at 8 amCall neighbor and check inWalk Minnow at 10 amCheck your email messagesTake medication with lunch at 12 pmMeet up with Bill to play cards 2 pmCall daughter after she gets home from work 5:30 pmHave dinnerWalk Minnow at 6:30 pmTake out the trash. Tomorrow is trash day!
Example answers to the Person-centered planmight be as follows–
• What are your goals and preferences for your health? To beable to stay in my home and live independently
• What is a long-term goal for your health? To be able toremember important things I need to do, like pay my billsand social outings.
• What is a short-term goal for your health? To be able to takecare of my dog.
• What do you feel is important for you to do each day? I enjoywalking my dog every day and visiting with friends.
3 DEVELOPMENT COMPONENTSAt the onset of the project, a collection of resources was necessary.These resources included both hardware components and softwarepackages. Any publicly-posted recipes of cookbooks on the use ofthe Microsoft HoloLens Gen 1 device have aged to a point wherethey are no longer correct in merging the hardware of the past withthe software of the present.
3.1 Hardware requirementsThe hardware required to develop a similar HoloLens project in-clude the components listed below.
• Microsoft HoloLens DK1,• An Android device,• A computer or laptop,• A DirectX 10 or greater-capable GPU, and• A CPU with SSE2 instruction set.
Augmented Reality: Telehealth Demonstration Application PEARC ’20, July 26–30, 2020, Portland, OR, USA
Figure 1: Microsoft HoloLens Demonstration ApplicationSample User Interface View
3.2 Software requirementsThe software packages and base code necessary to develop a similarHoloLens project are listed below.
• Android 9.0 or greater,• Windows 10 Enterprise, Professional, or Education,• Unity 3D,• Mixed Reality Toolkit (MRTK),• Visual Studio 2019,• HoloLens Emulator, and• Windows 10 SDK.
4 IMPLEMENTATIONThe current iteration of the project is implemented in the Unity 3Dengine using assets from the MRTK provided by Microsoft. VisualStudio is used to compile the final executable and deploy it to theHoloLens device. Once the application is compiled, it is deployedand used in tethered mode with the HoloLens directly commu-nicating with a developer’s computer. Once operation is verifiedand tested, the application can be moved to operate untetheredin a stand-alone fashion on the HoloLens device. An Android de-vice running Android 9.0 or greater is currently used to act as akeyboard during development as opposed to using the HoloLens’virtual keyboard primarily to aid in development ease of use.
5 EXAMPLE OPERATION OF APPLICATIONFollowing the sample use cases identified during the requirementsgathering phase, the project team built a sample application for usein the HoloLens. Since visualizing the application is difficult withouta contextual background, the project team built a sample room usingthe Unity 3-D package in which to display the application. Figure 1is an example of the application in operation in the sample roomas viewed from the perspective of a patient wearing the HoloLensdevice.
6 CONCLUSIONS AND FUTURE WORKThe requirements and ideas that the project team has gathered fromthe UNH Nursing Department allowed us to create an application
that is devoted to a care plan provided from a doctor for their specificpatient type. In addition to this application, we plan to use ourexperience with development to create documentation and researchfor general HoloLens development. This documentation will allowfuture team members to create more advanced applications withaugmented reality.
The baseline should allow future developers to more easily createmore complex applications based on the research and documen-tation provided. The documentation allows for easy setup of adevelopment environment and supports the deployment of codewithin the HoloLens ecosystem. Eventually, based on the demon-stration application and future documentation, there should beuseful methods to build towards future applications.
The telehealth field is one that can benefit significantly frommore HoloLens developers and the project team has provided re-search and a demonstration application that will make that feasiblein the future.
ACKNOWLEDGMENTSThe project team thanks the faculty and staff of the UNH Com-puter Science Department; Professor Matt Plumlee for his regularfeedback during the run of this project; The staff members andfaculty of the of the UNH Telehealth Practice Center, particularlyDr. Gene Harkless and Ms. Marguerite Corvini; The Northeast Cy-berteam Steering Committee; and our project sponsor, Dr. ScottValcourt. This work is partially sponsored by the National ScienceFoundation under Grant award: OAC-1659377.
REFERENCES[1] Baumann, Paula Kerler and Scales, Tim. “History of Information Communication
Technology and Telehealth.” Academy of Business Research Journal, vol. 3, 2016,pp. 48–52.
[2] Elgamal, Tarek, Chen, Bo, and Nahrstedt, Klara. “Teleconsultant: Communica-tion and Analysis of Wearable Videos in Emergency Medical Environments.” InProceedings of the 25th ACM International Conference on Multimedia (MM’17),October 2017, pp. 1241-1242. DOI: https://doi.org/10.1145/3123266.3127920
[3] UNH Telehealth Practice Center Website. chhs.unh.edu/telehealth-practice-center.Accessed 29 Apr. 2020.
A APPENDICESOn the following page is a graphic consisting of the project posteras presented at PEARC20.
PEARC ’20, July 26–30, 2020, Portland, OR, USA Nicholas Boyd et al.