a guidance system for blind and visually impaired people via … · 2018-07-06 · walter seiffert...
TRANSCRIPT
Walter Seiffert Simões
A Guidance System for Blind and Visually Impaired People via Hybrid Data Fusion
Federal University of Amazonastechnology College
Graduate Program in Informatics
Vandermi J. da [email protected]
Luciano M. da [email protected]
Vicente de Lucena [email protected]
6th Workshop on Communications in Critical Embedded Systems
Script
IntroductionRelated workDesign ArchitectureImplementation of the Proposed ModelTests and ResultsFinal Considerations
Walter Seiffert Simões 2WoCCES2018
Introduction
IntroductionRelated workDesign ArchitectureImplementation of the Proposed ModelTests and ResultsFinal Considerations
Motivation
• “How to assist the visually impaired in their self-location and autonomous
navigation in indoor environments with confidence? ”
• Applying hybrid data fusion in a wearable device.
Walter Seiffert Simões 3WoCCES2018
Some Similar Research
• Lu presented a simultaneous mapping and navigation (SLAM)approach using heterogeneous visual information such as points,lines and planes, joined by a multilayer characteristics graph (MFG)algorithm.
• Ruiz introduced an indoor navigation system by reading RSSI valuesin RFID markers. The data were fused by a Kalman filter. Thealgorithms Pedestrian Dead Reckoning (PDR), ZARU (Zero-Angular-Rate Update) and ZUPT (zero-velocity-update-aided) were used toreport location, speed and angulation.
• Hattori proposed to construct a method of estimating IPS usingKalman fusion in a complementary way over wi-fi data with images.Thus, if one sensor does not work, it does not prevent the other fromassuming the task of providing the location.
But …
• Unlike all of the localization approaches discussed above, we believe thatthe hybrid fusion of wi-fi and visual markers may result in a betterlocational accuracy for indoor applications.
Related work
IntroductionRelated workDesign ArchitectureImplementation of the Proposed ModelTests and ResultsFinal Considerations
Walter Seiffert Simões 4WoCCES2018
Online
PreprocessingRoutes
Offline
Guide
Data processing
FiltersAnalysis
Processed data
Base of sensor values
Localdata
Audio Bank
Speech Recognition Algorithms
Decisions
Policies and rulesRoute offered to the user
Audio Guide
Algorithms and strategies
Location Algorithms
1
2
Marker data
Visual Classifier
RF Classifier
Location IdentifiersAuxiliary data
Speech Classifier
Auxiliary Data of Navigation
Construction of the classifiers
Radio Frequency Mapping Algorithm
Computational Vision Mapping Algorithm
Speech Pattern Mapping Algorithm
Location sensorsCamera RF
Mobile Mapping Device
User Interaction
Microphone
Input
Headphone
Output
Display
Wearable Navigation Device
Location sensorsCamera RF
User Interaction
Microphone
Entrada
Link between mapped data and navigation
Search data on base
Create
Initial route
Rebuild
Reset route
Adjust
Route Correction
Headphone
Saída
DisplayVibration-
tactile device
Keyboard Mouse
Reason
Search for data?No
Yes
Design Architecture
Walter Seiffert Simões 5
IntroductionRelated workDesign ArchitectureImplementation of the Proposed ModelTests and ResultsFinal Considerations
WoCCES2018
Design Architecture
Components of Mapping and Navigation Devices
Walter Seiffert Simões 6
IntroductionRelated workDesign ArchitectureImplementation of the Proposed ModelTests and ResultsFinal Considerations
WoCCES2018
Component DeviceCamera RGB night vision 8 Mpixels Mapping, Navigation
Network ESP8266 module Mapping, NavigationEasyVR Shield 3.0 – Voice Recognition Mapping, Navigation
Microphone Mapping, NavigationBone conduction headset Mapping, Navigation
Arduino nano Micro-controller Mapping, NavigationRaspberry pi 3, 1.2 GHz, 32 Gb Mapping, NavigationVibration Motor 1027 Arduino Navigation
Mapping Device Navigation Device
Wi-fi Device
Design Architecture
Mapping Scheme
Walter Seiffert Simões 7
IntroductionRelated workDesign ArchitectureImplementation of the Proposed ModelTests and ResultsFinal Considerations
WoCCES2018
Technique Technology Area (m2)Fingerprinting Wi-fi 5.0
FAB-MAP (Fast Appearance Based Mapping) Visual 1.67
Fingerprinting FAB-MAP
The mapping scheme is based on the fusion of radio frequency and visual information
Construction of Physical Devices – Visual Marker
Walter Seiffert Simões 8
Black border
Object of interest
Border code to indicate orientation
Exemple
Implementation of the Proposed Model
IntroductionRelated workDesign ArchitectureImplementation of the Proposed ModelTests and ResultsFinal Considerations
WoCCES2018
Design Architecture
Stimuli Location – Visual Marker
Walter Seiffert Simões 9
Data markers
Calibrateequipments
Capture picture
Set position of reference markers
ObjectMarker
CreateSamples
Traincascade
Positive images
Negative images
Classifier Haar-like
IntroductionRelated workDesign ArchitectureImplementation of the Proposed ModelTests and ResultsFinal Considerations
2000 images on each marker
10,000 negative images
false alarm = 2000/10.000 = 0,2
OpenCV algorithms
WoCCES2018
Design Architecture
Indoor Navigation – Wi-fi and Computer Vision
Walter Seiffert Simões 10
IntroductionRelated workDesign ArchitectureImplementation of the Proposed ModelTests and ResultsFinal Considerations
WoCCES2018
The HLVN (Heterogeneous Landmark-based Visual Navigation) reads the visual markers by recognizing the edge information and the central content of the labels.
Read visual label
Image processing
Visual Classifier
Has the object been found? HLVN Local datayes
No
Navigation System
Base image values
Read ID, RSSI, SNR
Process Data
RF Classifier
Was the Zone found? TBD Local dataYes
Navigation system
Base of zone values
KNN
No
The TBD (Track Before Detect) reads the ID, RSSI and SNR values of the radio frequency markers.
Indoor Navigation
Walter Seiffert Simões 11
Tests and Results
IntroductionRelated workDesign ArchitectureImplementation of the Proposed ModelTests and ResultsFinal Considerations
WoCCES2018
Auditorium
STAIRS
Real path EndBeginCorridor
Corri
dor
Classroom Laboratory
RSSI with SNR
Auditorium
STAIRS
Referential path EndBeginCorridor
Corri
dor
Classroom Laboratory
Visual
1
2
Indoor Navigation with Kalman Filter
Walter Seiffert Simões 12
Tests and Results
IntroductionRelated workDesign ArchitectureImplementation of the Proposed ModelTests and ResultsFinal Considerations
WoCCES2018
Moving Still
Tests and Results
Results
• Comparison with other published results.
Walter Seiffert Simões 13
IntroductionRelated workDesign ArchitectureImplementation of the Proposed ModelTests and ResultsFinal Considerations
WoCCES2018
Final Considerations
IntroductionRelated workDesign ArchitectureImplementation of the Proposed ModelTests and ResultsFinal Considerations
Results
• When observed in isolation, wi-fi and visual sensors showed an inefficiencyof their uses as a single value input;
• The measurement procedure based on sensor data fusion techniquesallowed the system to be more fault tolerant and capable of providing newinformation that it could not provide individually.
• The results showed that the approach adopted is promising. In fact, it can beapplied to different scenarios because of its scalable capability.
Walter Seiffert Simões 14WoCCES2018
A Guidance System for Blind and Visually Impaired People via Hybrid Data Fusion
Federal University of Amazonastechnology College
Graduate Program in Informatics
Thank you!Walter Seiffert Simões
Vandermi J. da [email protected]
Luciano M. da [email protected]
Vicente de Lucena [email protected]