![Page 1: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/1.jpg)
Researcher: Zi Siang See
Research Presentation: Zi Siang See
Technical developmentEmail (University of South Australia): [email protected] (Sunway University): [email protected] (Industrial collaboration): [email protected] (University Tunku Abdul Rahman): [email protected] (University Tunku Abdul Rahman): [email protected]
Zi Siang See, Mark Billinghurst, Vimalan Rengganaten, Shelly Soo (2016)
SIGGRAPH Asia 2016 Mobile Graphics And Interactive Applications
Medical Learning Murmurs Simulation
with Mobile Audible Augmented Reality
Research &
collaboration
Zi Siang See, Mark Billinghurst, Vimalan Rengganaten, Shelly Soo (2016) Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality. SIGGRAPH Asia 2016 Mobile Graphics And Interactive Applications.
![Page 2: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/2.jpg)
Researcher: Zi Siang See
Presentation Outline
1) Introduction, Background
2) Proposed Method & Apparatus
3) User Study
4) Discussion
Research &
collaboration
![Page 3: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/3.jpg)
Researcher: Zi Siang See
Figure: augmented reality (AR), examples /preview from search engine
Introduction: Augmented Reality (AR)
• Displays, trackers, and AR systems in general need to become more accurate,
lighter, cheaper, and less power consuming (Azuma, 2001).
![Page 4: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/4.jpg)
Researcher: Zi Siang See
Medical Student’s Life
• Educators and trainees work with expensive hardware
• Institutional-owned
• Hardware are not mobile
• Time and space constraints on fixed location-based environment
• Unable to include dynamic information on-demand
Figure: current medical learning laboratory, Sg Long, UTAR, MY.
![Page 5: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/5.jpg)
Researcher: Zi Siang See
Medical Student’s Life
• Educators and trainees work with expensive hardware
• Institutional-owned
• Hardware are not mobile
• Time and space constraints on fixed location-based environment
• Unable to include dynamic information on-demand
Figure: current medical learning laboratory, Sg Long, UTAR, MY.SAM
Harvey
Harvey, Gordon Center for Research in Medical
Education, University of Miami
Student Auscultation Manikin, Cardionics.
![Page 6: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/6.jpg)
Researcher: Zi Siang See
Medical Student’s Life
![Page 7: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/7.jpg)
Researcher: Zi Siang See
Medical Student’s Life
Simulator provides murmurs audio
Instructor /students use stethoscope to listen and practice
![Page 8: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/8.jpg)
Researcher: Zi Siang See
Conventional Mobile AR
• Almost everyone has a mobile device
• Mobile devices are internet-based + deliver 3D graphics
![Page 9: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/9.jpg)
Researcher: Zi Siang See
AR for Medical Education
• AR has been used for a variety of different medical applications
• VR diagnostic imaging and patient treatment visualization (Sielhorst et al., 2008;
King et al., 2016).
• AR has also been used for medical education
• Showing virtual CT data onto the body of a trainee (Zhu et al., 2014)
• Viewing human anatomy with AR (Kamphuis et al., 2014, Bacca et al., 2014).
![Page 10: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/10.jpg)
Researcher: Zi Siang See
AR for Clothes
• Augment alternative clothing /fashion experience (e.g. using Kinect)
• Additional AR content on markers imprinted on clothes for branding
• Previous research on using cloth for AR tracking
Trauman, 2015; Hilsmann and Eisert, 2009; Cushen and Nixon, 2012; Bradley et al., 2009.
![Page 11: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/11.jpg)
Researcher: Zi Siang See
• Works on everyone’s mobile devices + AL-60 digital stethoscope
• Clothing system with imprinted markers (low-cost)
• Wearable /portable everywhere anytime
Murmurs AR from Aortic /Tricuspid /Pulmonary
/Mitral
Process AR tracking,
markers captured from mobile
Provides
murmur audio
Cloth with
imprinted marker
Server,
processor
Mobile audible AR interface +AL-60(optional graphic visual)
Mobile AR + Clothes + Medical Learning
![Page 12: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/12.jpg)
Researcher: Zi Siang See
Mobile AR + Clothes + Medical Learning
Areas /position Possible murmurs detection
Aortic area Aortic valvular stenosisEarly diastolic aorticregurgitation
Pulmonary area Pulmonic valvular stenosis
Mitral area Acute mitral regurgitationMitral stenosis
Tricuspid area Tricuspid regurgitationTricuspid stenosisPericardial friction rubEarly diastolic aorticregurgitation
• Image-based tracking
• The prototype system is also transferable to other platforms
![Page 13: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/13.jpg)
Researcher: Zi Siang See
• Different murmur digital wave patterns
• Medical trainees need to identify different types of murmurs
• With the AR murmurs simulator, digitized audio content can be
assigned or customized to suit various medical simulations.
Mobile AR + Clothes + Medical Learning
![Page 14: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/14.jpg)
Researcher: Zi Siang See
Prototype Method and Apparatus
![Page 15: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/15.jpg)
Researcher: Zi Siang See
Prototype Method and Apparatus
Mobile audible AR with murmur simulation
AL-60 stethoscope for listening and practicing (portable, low-cost, ease-of-use)
![Page 16: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/16.jpg)
Researcher: Zi Siang See
User Study
• Measure usefulness of the prototype
• 15 medical students were involved in the test
• 9 female, and 6 male ranging in age between 20 to 23 years old (SD = 1.21).
![Page 17: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/17.jpg)
Researcher: Zi Siang See
User Study
Setup (S1, S2)
• Each participant was asked to configure the AL-60 stethoscope with
mobile device with the AR application (S1).
• Put on the T-shirt imprinted with AR markers and become the subject
model for the next participant (S2).
Condition (C1, C2)
Run the prototype AR application twice, listening for audible murmurs:
• Once with the visual annotation aid (C1).
• Once without visual aid (C2).
![Page 18: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/18.jpg)
Researcher: Zi Siang See
User Study
• Subject model was allowed to sit or lay down in a clinical
environment to mimic a mock patient visit session.
• Within-subjects experimental design, where each participant
experiences both of the AR visual display conditions.
![Page 19: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/19.jpg)
Researcher: Zi Siang See
User Study
No. Questions
Q1 I found it easy to use
Q2 I found it natural to use
Q3 I found it reliable
Q4 I found it physically challenging
Q5 I found it mentally challenging
Q6 I found it useful
![Page 20: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/20.jpg)
Researcher: Zi Siang See
User Study
No. Questions
Q1 I found it easy to use
Q2 I found it natural to use
Q3 I found it reliable
Q4 I found it physically challenging
Q5 I found it mentally challenging
Q6 I found it useful
* Perceived easiness and usefulness
![Page 21: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/21.jpg)
Researcher: Zi Siang See
* Perceived difficulties
User Study
No. Questions
Q1 I found it easy to use
Q2 I found it natural to use
Q3 I found it reliable
Q4 I found it physically challenging
Q5 I found it mentally challenging
Q6 I found it useful
![Page 22: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/22.jpg)
Researcher: Zi Siang See
: average results of C1 and C2 survey question.
Subjective Feedback
Figure: average results of C1 and C2 survey questions
ease of use
natural
reliable
Physically challenging
Mentally challenging
Usefullness
![Page 23: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/23.jpg)
Researcher: Zi Siang See
User Study: Results
• Wilcoxon Signed Rank test was used to analyze the results– check for significant difference between using visual (C1) and non-visual interfaces (C2).
• For Q1, using a one-tailed test we found that participants felt that the visual
aid made the interface significantly easier to use than the interface with non-
visual cues, Z = -1.82, p = 0.03.
• For Q2, finding the interface natural to use, there was a near significant
difference between C1 and C2, with Z = -1.60, p = 0.06.
• There was no difference between conditions in terms of how reliable
participants felt each condition was (Q3), Z = -0.17, p = 0.44.
• In terms of the physical challenge, participants felt that the non-visual
condition (C2) was significantly more physically challenging (Q4) than the
visual condition (C1), Z = -2.191, p = 0.014.
• Similarly, C2 was felt to be more mentally challenging (Q5) than C1, Z = -
2.31, p = 0.01.
• C1 was viewed as being more useful (Q6) than C2, Z = -1.87, p = 0.03.
• Overall, these results show that the interface with visual aids was better
than the not using any visual aids.
![Page 24: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/24.jpg)
Researcher: Zi Siang See
: average results of C1 and C2 survey question.
User Study: Setup
Figure: average results of S1 and S2 survey question.
ease of use
natural
reliable
Physically challenging
Mentally challenging
Usefullness
![Page 25: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/25.jpg)
Researcher: Zi Siang See
User Study: Setup
• Users found the configuration of the stethoscope (S1) relatively easy
(Q1) and useful (Q6), but it was also rated above average in terms
of physical (Q4) and mental challenge (Q5).
• Similarly from S2 users found it easy to wear the tracking T-shirt
(Q1) and that it was useful (Q6), but it was an above average
physical challenge to use (Q4).
![Page 26: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/26.jpg)
Researcher: Zi Siang See
User Study: Feedback
• Users said they “.. like the idea of it being portable.. ”,
• “The idea is pretty good, affordable and portable”,
• and “It is usually challenging to remember all murmurs and I hope
this new app can help me about it”.
• However users also felt that “.. it is quite difficult to use..”,
• “Markers are hard to detect under different lighting”,
• and users felt “.. quite exposed and uneasy wearing the shirt and
scanned by others”.
![Page 27: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/27.jpg)
Researcher: Zi Siang See
Discussion
• Most users felt that having the interface with visual cues was more
useful as it provides clear visual feedback for an AR murmur audio is
being tracked.
• It can be confusing when the system is used without visual
feedback, as users could not be sure if an AR audio cue will be
played.
• If there is no audio feedback users move on to the next marker to
search for other murmurs to be heard, as a result users were
moving the mobile application around without getting much murmurs
being experienced (slow cloud-based AR loading, crease, etc).
• The use of AR and cloth tracking simulates the physical experience
of holding a digital stethoscope to check on a heart murmur.
• Audible AR also allows the users to be able to hear uncommon
heart murmurs that rarely occur during a diagnosis of a real patient.
![Page 28: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/28.jpg)
Researcher: Zi Siang See
Conclusion
• This paper presents a medical education system teaching about
heart murmurs using mobile audible Augmented Reality.
• Existing mobile hardware using image-based tracking on clothes.
• Low cost and can be easily used in a wide variety of setting.
• Users felt that the system was natural and easy to use, and the use
of additional visual cues made the system much more useful.
• In the future, we would like to explore mobile audible AR for different
murmurs scenarios and the suitability of the proposed system being
tested in examination.
Murmurs AR from Aortic
/Tricuspid /Pulmonary /Mitral
![Page 29: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/29.jpg)
Researcher: Zi Siang See
Further Development
• Smaller devices
• Macro lens (smaller markers)
• T-shirt different sizes
• Markers on clip-on badges
• Instructors use it for exam
• Offline /cloud-based?
![Page 30: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/30.jpg)
Researcher: Zi Siang See
Demo
![Page 31: Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality](https://reader031.vdocuments.us/reader031/viewer/2022030221/588471d51a28ab5e248b4a29/html5/thumbnails/31.jpg)
Researcher: Zi Siang See
Questions
• Name/Email (feedback/ discussion)
Contact
Zi Siang See, Mark Billinghurst, Vimalan Rengganaten, Shelly Soo (2016) Medical Learning Murmurs Simulation with Mobile Audible Augmented Reality. SIGGRAPH Asia 2016 Mobile Graphics And Interactive Applications.
Paper