a navigationsystem for blindpeople - laura giarré · a navigationsystem for blindpeople ......

A navigation system for Blind PeopleWhen the smartphone becomes a Guide DOG

Laura GiarréThe Blind Leading the Blind, P. Brugeul the elder. 1568, Napoli Capoimonte musuem

March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 1

Problem

March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia

Visually impaired are 300.000.000 (source WHO)• And increasing with aging of the population• Social inclusion enhanced with autonomous

mobility• Today: tactile pavements (very expensive!)

2

Can technology help a Visually Impaired Person• to live a more independent life, working and

socializing• to move autonomously and safely in an

unknown environment• to retrieve information about the places

nearby• to recognize the persons they are interacting

with and understand their feelings• to perform day life activities with increased

autonomy


Challenges for VIP

3


S

Machine LearningFiltering

Signal DetectionFeature ExtractionComputer vision

Haptic InterfaceVibrationBraille PadMusicAudio signalWearable vibratingbracelet/belts

Camera Sonar/laserAcceleremoterRFIDIMUiBeacon

CYBER SIGHT SENSE

Sensors Actuators

Computation

Solution

4

Stairway to Heaven

√ How Much Information√ Given Task√ Ergonomic/Functionality√ Cosmetic Acceptability√ Tests in Real World √ Embed Blind Experience


• Small in size• Hardware robust but light• Optimized Battery life• Appeal of exterior design• Cosmetic acceptability is more importantthan technological potential benefits• Functionality: TRANSMIT INFORMATION to theremaining sense without INTERFERING

Ergonomic


Laboratory vs real world• Early success with simplified tasks and conditions has bred

optimism about eventual success in performing complex tasks in real-world conditions:

• Walking through environment cluttered with small low contrastobstacles


• Use of other SPARED sensory modality to supply information normally gathered by the impairedsense• Aids employed regardless of the different clinicalfeatures of blindness•Basic Examples:

•Braille reading: fluently read through raised patterns

• Long cane: uses tactile information to explore the surroundings

Non invasive Solutions


• Miniaturization of actuators and electronics :new systems embedded into clothing,

devices worn on the finger, hands, wrist, forearm, tongue, head, chest, abdomen and feet

• Hearing and touch is the first and second major senses, and it gather enough information from the environment to perform daily tasks.

• Assistive devices provide acoustical and tactile feedback to compensate for visual information.


Wearable Solutions

9


• What is the easiest way to let blind people use Sensory Substitution Devices?– minimize the external infrastructure– use their own smartphone– develop cheap (computationally) algorithms

running on smartphones– make use of all the sensors already

embedded in the phones

A case study:

11

State of the art on indoor navigation

Laura Giarré University of Modena and Reggio Emilia

March 1st, 2018

• Indoor navigation:– Triangulation of RF signals (WiFi)– Direct sensing (RFIDs, ultrasound, Bluetooth, iBeacon,...)– Pattern matching– Dead‐reckoning (accelerometers, gyroscopes, …)

• Often dedicated hardware is required– Expensive devices, complex deployment

• Low accuracy (sometimes <5m)– Insufficient for navigation of the visually impaired

• Audio feedback is annoying!!!12

: what is not?


A system to avoid obstaclesCameras, ultrasounds, sonar, haptic belts, interfaces

A system to give back sightto directly stimulate the deficient sensorysystem: Invasive solutions to send to theoptical nerve acquired signals from avideocamera

13

: what is?


Arianna is a system for low vision and/or blind person,A smartphone-based solution for assisted navigation:• goal: improving social inclusion, accompanying the

users along the path• From mute paths to talking paths

– ICT for improving user interactions (robotics&vision)– Haptic interfaces

14

A smartphone-based solution for assisted navigation:• A set of paths and tags deployed in the environment • A smartphone (instrument of mediation between the

reality and the user to access the information • A vibration feedback signal to the users to follow pre-

defined path

system (autonomous mobility)


March 1st, 2018

Tape and QRcode laying

User arianna app installation

Map creator and server configuration o app configuration

System usage

Installation step:1) Tape laying

2) Map creator

3) Map selection

4) System usage

What is needed

Laura Giarré University of Modena and Reggio Emilia 16

1. The smartphone camera captures the image of the scene in front of the user;2. A computer vision algorithm identifies the path by recognizing two colored

lines on the floor;3. The user receives a vibration signal through the telephone to understand the

direction to follow;4. Some special references (tags) like QR code or ibeacon allow to identify

contents associated to the position

What the camera sees

Path identified by Arianna

MORE details..


Inside

Laura Giarré University of Modena and Reggio EmiliaMarch 1st, 2018

The algorithms

18

1. How to detect the path?– There are various functions in computer vision

that can be combined for our problem, taking into account some fundamental aspects:

– the identification must be fast, to allow the user who moves the phone in his hand not to perceive discontinuous signals;

– the battery life must be maximized – What can we use?

• the geometry of the strip (actually three parallel lines) • the colors of the strip


Classic Approach• Basic idea:

– Detect the Contours (Canny algorithm)– Find contour points along a line (Hough transformation)

Contour foundwith Canny

Lines for eachcontour point

rho

theta

Identification of aligned points(quantized rho, theta)


Example: Optimization to select only the path

Pre‐filteringto elimatedetails

Tresholds to define the maincontours

Lines with the majority of ‘scores’


A colour based approach• It is based on the representaion HSV (hue saturation

value) (more robust to ambient light changing with respect to RGB)

• Simple range filtering intervals– Es. filtering all the points in the intervals (H:0.4-0.6, S:0.23-0.53,

V:0.4-0.6)• Double filtering to increase robustness• No complex matrix operations: only “if”s• Low robustenss to light condition• Indoor and Outdoor


A mixed approach• At regular intervals or when variation in lighting is

detected..– Geometry analisys to identify path and sample/resample the

colours

• Regularly– Coulour based identification


2. If the path is lost?• Basic Idea:

– Indication if it has been lost from left or right (computer vision)

– Estimation of the relative movement of the user with respect to the path (inertial samrtphone sensors: (dead-reckoning)


Inertial sensors measurements• Gyroscope angular direction (at regular intervals T)

– alpha_k• Accelerometer filtered velocity (assumption: the user is

moving in 2D)– from [a_x, a_y, a_z] and [alpha, beta, gamma] to |v_k|

• Activity estimation (walking/standing) to stop the acquisition when the user is.


Sensor fusion: integration of Computer vision and dead‐reckoning

• When the path is visible again:– Increase the quality of the measurement of alpha (from the angle

theta detected with hough)– Reste the cumulative error (from the estimated rho with hough)

• To use the camera data it is important to know the angualr reference in the

• Conversion between the absoulte and relative orientation

Beta

Alpha=f(Beta,Theta)


Example of ‘fusion’Real Experiment

only dead‐reckoning dead‐reckoning + CVMarch 1st, 2018 Laura Giarré University of Modena and

Reggio Emilia 27

3. How to improve the system by modeling user reaction (HIL)?

• Assumptions: – User is walking at constant speed (noisy)– Corretions:

1) Proportional to the deviatuon from the path and to the direction to go back onto the path in a time T_X, whenthe path is visible to the camera2) Fixed correction (the sign depends if we lose the pathfrom right or left)

– The correction of the user is characterized by a proportional term (m, n)


User position estimation with EKF(Estimated Kalman Filter)

• Inertial sensors, camera, correction model u_k


User Simulator

Perception direction noise uniformely distributed in

[-π/6, π/6]


User simulator (wrong initialdirection)

Heading correction m=0.6 Distance correction n=0.4


Inside

Laura Giarré University of Modena and Reggio EmiliaMarch 1st, 2018

The System

32

The APP ( Android and iOS)

Drawer menuFollow the line

Places

Help Settings

Navigation

• Changeable colours (for low vision people)• Not only path recognition but also navigation from a to b.• Reproduction of contents (POI)• Voice-over (upon request)



I-beacon integrationThe point of interests (POI) or TAGS can be QR code (CV) or I-beacon (radio) • Ibeacon vs Qrcode:

• More precise recognition in a wider aerea• Not on the path (Hidden for cosmetic reasons)• More costly and they need battery

March 1st, 2018 34


Where to install ARIANNA

Airports

Offices/Mall

Sidewalks

Museums

35


How to evaluate the system?

• Run specific tests: decide a task (a pathpresenting curves, crossing points, beginningand end, Qr codes)

• Define some metrics: time, number of tagsmissed, etc (quantitative)

• Run follow up with questionnaires (qualitiative)• Importance of feedback from BlindCommunities!!!

36

Tests with Blind Communities• Istituto dei ciechi Milano, Bologna• Unione Ciechi Florence, Palermo• Demos & exhibitions:

– Project FIC3 App Launch Day, 2016– Taiwan Computex, 2016– EXPO Milan, 2015– National prize for Innovation, 2014– Smart City Exhibition, 2014– SZDIF14 China international industrial design fair, 2014– ABF MIT Challenge workshop, 2013 & 2015

• Collected subjective and objective feedback with questionnaires

Laura Giarré University of Modena and Reggio EmiliaMarch 1st, 2018 37

FIWARE accelerator

Recent installations (2017)

FARM CULTURAL PARKFavara, Agrigento

(Italy)


Blindinstitute of Palermo

March 1st, 2018 38

The team: University of Palermo, University of Modena and Reggio Emilia, In.sight, National Chiao Tung University,

University of Roma Tre


Web: in.sight.srlSocial: @insightsrl

References• Tzu-Kuan Chuang*, Ni-Ching Lin, Jih Shi Chen, Chen-Hao Hung, Yi-Wei

Huang, Chunchih Teng, Haikun Huang, Lap-Fai Yu, Laura Giarré, Hsueh-Cheng Wang Deep Trail Following Robotic Guide Dog in PedestrianEnvironments for People Who Are Blind and Visually Impaired - Learningfrom Virtual and Real Worlds 2018 IEEE International Conference onRobotics and Automation

• Giovanni Galioto, Ilenia Tinnirello, Daniele Croce, Federica Pascucci, Federica Inderst, Laura Giarré Sensor Fusion Localization and Navigation for Visually Impaired People 2018 European Control Conference

• Croce D., Giarré L., La Rosa F. G., Montana E., Tinnirello I.“Enhancingperformance in a smartphone-based navigation system for visually impairedpeople”. In Proceedings of the 24nd Mediterranean Conference on Controland Automation (MED’2016), 2016.

• Croce D., Gallo P., Garlisi D., Giarré L. Mangione S., Tinnirello I.“ARIANNA: a smartphone-based navigation system with human in theloop“, in Proceedings of the 22nd Mediterranean Conference on Controland Automation (MED’2014), 2014.

•March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 40

a navigationsystem for blindpeople - laura giarré · a navigationsystem for blindpeople ......

Documents