assisting visually impaired people with mobility through ... · pdf fileassisting visually...
TRANSCRIPT
Assisting Visually Impaired People
with Mobility through Technology in
the Age of ContextA SURVEY
Presented by Martin Goldberg
Committee:
• Prof. Zhigang Zhu (Advisor)
• Prof. Ioannis Stamos
• Prof. Zhanyang Zhang
• Dr. William H. Seiple
Assisting VIPs with Mobility in the Age of Context
Introduction
Mobile Devices
Sensors
Location Awareness
Social Media
Data
Integration
Conclusions
Assisting VIPs with Mobility in the Age of Context
Background and Motivation
Robert Scoble & Shel Israel
Age of Context: Mobile, Sensors, Data
and the Future of Privacy. Patrick Brewster Press, 2013
Introduces the concept of the 5 Forces of the
Age of Context, called a ‘technological
storm’ – their main concern is how this affects
our privacy.
We want to use this ‘technological storm’ as
a prism to see how technology has been,
and – perhaps – can be, used to aid Visually
Impaired Persons (VIPs) with mobility issues.
Assisting VIPs with Mobility in the Age of Context
Background and Motivation
https://nfb.org/blindness-statistics
Approximately
2.5%Blind People in US population.
Blind = visual acuity 20/200 or worse
VIP – can be 20/60 or worse
Most in senior years
Assisting VIPs with Mobility in the Age of Context
The Five Forces of the AoC
• Mobile Devices
• Sensors
• Location Awareness
• Social Media
• Data (Big Data & the miracle of Little Data)*
* S & I consider the immense amount of Data (Big Data) and Data
Mining techniques that can find particular Data in the vast data out
there – the core of their fear for our privacy.
Assisting VIPs with Mobility in the Age of Context
Definition: VIP
Visually Impaired Person
May have vision but does not always
see as clearly as required – sometimes
shapes, sometimes details; may not see
anything - although may still have light
perception.
Assisting VIPs with Mobility in the Age of Context
Definition: Mobility
The area dealing with a VIP’s ability to
‘get out and about’, preferably
without needing a helper’s assistance.
Assisting VIPs with Mobility in the Age of Context
Definitions: Wayfinding & Navigation
Wayfinding
Deciding a correct or optimal path to get from one
location to another.
Navigation
The process of managing the traversal decided by
wayfinding; being able to avoid hindrances or other
dangers - such as holes or edges.
Assisting VIPs with Mobility in the Age of Context
Definition: Orientation
1. Defining the correct direction to
follow
2. Defining the direction the VIP is
facing.
Assisting VIPs with Mobility in the Age of Context
Introduction
Mobile Devices
Sensors
Location Awareness
Social Media
Data
Integration
Conclusions
Assisting VIPs with Mobility in the Age of Context
Mobile Devices
Abascal & Civit (2000)Recognize the value in portable communication as a
way to “untether” a VIP from the house. An instance of
general applicability being applied to VIP assistance.
http://bitchinlifestyle.tv/1960s-vs-2010s/
woman-talking-on-rotary-phone/ http://www.123rf.com/photo_24039158_
girl-sitting-on-a-park-bench-and-talking-
by-a-cell-phone.html
Assisting VIPs with Mobility in the Age of Context
Mobile v. Portable DevicesKane (2009)
Freedom to Roam
a) Signature block
b) Water level Sensor
c) CCTV – Magnifier
d) D) Mobile Phone
S & I:
Mobile – has communication potential
Portable – lacking communications
Assisting VIPs with Mobility in the Age of Context
Smartphones
Company Product OS Revealed Marketed
Google Google Now Android 2011 2012
Apple Siri iOS 2012 2013
Microsoft Cortana Windows Phone 8.1 2014 2015
Cortana Siri Google Now
Totally Blind
VIP usage:
As the iPhone
has only one
button to use,
it is favored
among blind
users.
Assisting VIPs with Mobility in the Age of Context
Smartphone as Assistive TechnologyCoughlan and Manduchi (2008)
Search strategies of visually impaired
persons using a camera phone
wayfinding system.
Use of the smartphone camera has
been a major research topic, by these
authors and others.
Note the ‘awkwardness’ of how the
phone is worn on the neck.
Assisting VIPs with Mobility in the Age of Context
Mobile Assistive Technologies
http://dub.washington.edu/
http://faculty.washington.edu/
wobbrock/hci.html
Studies on VIP Needs and Uses of Assistive Technology
U. Washington, Seattle - Jacob O. Wobbrock
DUB Lab
2009 – Freedom to Roam – Kane (now at Colorado)
with J. Bigham (Rochester, now CMU)
2013 - Exploring the Use of Speech Input
- Azenkot (now at Cornell)
Rochester, NY – Jeffrey Bigham - VizWiz (now at CMU)
2013 – Blind Use of Social Networks - Brady
Assisting VIPs with Mobility in the Age of Context
Mobile Devices Commonly in Use
http://www.asus.com/us/Tablets/
ASUS_Transformer_Pad_TF103C/http://www.theweek.co.uk/iphone-6s/62138/
iphone-6s-and-6s-plus-smash-sales-records-
should-i-buy-the-new-iphone
Tablets
Stretch Screen TechnologySmartphones
Voice Over
Mobile devices with
cameras, stretch
screen technology
and text-to-speech
and speech
activation
technologies are very
programmable
Assisting VIPs with Mobility in the Age of Context
Wearable Devices – the Future?
https://en.wikipedia.org/wiki/Google_Glass
Google Glass
https://en.wikipedia.org/wiki/Smartwatch
Smart watch
https://learn.adafruit.com/
introducing-the-raspberry-pi-2-model-b?view=all
Single-board
computers such as
the Raspberry Pi
Possibilities …
Assisting VIPs with Mobility in the Age of Context
Introduction
Mobile Devices
Sensors
Location Awareness
Social Media
Data
Integration
Conclusions
Assisting VIPs with Mobility in the Age of Context
Sensors
A sensor is a transducer whose purpose is to
sense (that is, to detect) some characteristic
of its environs. It detects events or changes in
quantities and provides a corresponding
output, generally as an electrical or optical
signal.- Wikipedia (Sensors)
Assisting VIPs with Mobility in the Age of Context
Sensors - Sound
https://en.wikipedia.org/
wiki/Microphone
https://en.wikipedia.org/
wi`ki/Loudspeaker
https://en.wikipedia.org/
wiki/Headphones
Microphones – detect sound, transduce to electronic signals
1.Mid-range
2. Tweeter
3.Woofers
Transduce electronic
signals into sound
Loudspeakers Headphones
Assisting VIPs with Mobility in the Age of Context
Sensors – Images /
Depth and Direction
https://en.wikipedia.org/wiki/
Digital_camera
Digital Camera (images)
transduces light capture
into digital matrices
Depth and direction can
be measured by Laser or
Ultrasound, most
commonly using time-of-
flight measurements Sekiguchi (2014) ref 74 in paper
Assisting VIPs with Mobility in the Age of Context
Sensors – Force Sensors / Antennas
https://en.wikipedia.org/
wiki/Computer_keyboard
https://en.wikipedia.org/
wiki/Mouse_(computing)
https://en.wikipedia.org/
wiki/Touchscreen https://en.wikipedia.org/wiki/Antenna_(radio)
Force Sensors transduce
pressure readings into
electronic
signals; here
button-based
keyboard and
mouse are
shown, as well as
touch-screen
technology
Antennas
transduce
electromagnetic waves
into electronic pulses
Assisting VIPs with Mobility in the Age of Context
Sensors – Other Sensors
https://en.wikipedia.org/wiki/
Accelerometer
https://en.wikipedia.org/wiki/Gyroscope
https://en.wikipedia.org/wiki/Magnetometer
Accelerometers measure proper acceleration
(G-Force)
Gyroscopes are used to
measure and maintain
orientation
Magnetometers measure magnetic intensity
and can determine magnetic direction
Assisting VIPs with Mobility in the Age of Context
Sensors – as used on a Smartphone
http://www.boeboer.com/page/29/ (Samsung Galaxy S III)
Assisting VIPs with Mobility in the Age of Context
Body as Sensor
http://erc-assoc.org/about/erc_data/
spinoff-technology-argus-ii-retinal-prosthesis-system
http://www.news.com.au/technology/
blind-soldier-sees-with-his-tongue-using
-brainport-vision-technology/story-e6frfro0
-1225841742025
Argus II attempts to
bypass the retina,
passing signals derived
from a camera to the
optic nerve
Brainport attempts
to take a camera
input and, through
the tongue, teach
the brain to process
discernible images
Assisting VIPs with Mobility in the Age of Context
Tactile Graphics
http://people.bordeaux.inria.fr/abrock/
Anke Brock (2014)
User studies on interactive
tactile maps, with ‘talking
labels’
Earlier work:
Joe Cioffi – Click ‘n Go
Noted difficulty – must be
memorized, not useful
when in motion.
Assisting VIPs with Mobility in the Age of Context
Introduction
Mobile Devices
Sensors
Location Awareness
Social Media
Data
Integration
Conclusions
Assisting VIPs with Mobility in the Age of Context
Global Positioning System - GPS
“A visual example of a 24 satellite GPS constellation in motion with
the earth rotating. Notice how the number of satellites in view from a
given point on the earth's surface, in this example at 45°N, changes
with time.” – Wikipedia label
https://en.wikipedia.org/wiki/Global_Positioning_System
GPS uses 24 geo-orbiting Satellites each
sending signals to points on the Earth’s
surface at a latency of 38 ms. The signal
includes time stamps generated by atomic
clocks.
Assisting VIPs with Mobility in the Age of Context
GPS for VIPsProduct Software / OS Device Notes
Loadstone GPS Symbian OS Symbian (Nokia) Since 2006
LoroDux Java ME None specific Utilizes OpenStreetMaps,
under development
Mobile Geo Windows Mobile Windows enabled
devices
Seamless integration with Mobile Speak
BlindSquare iOS iPhones Based on FourSquare
combined with OpenStreetMaps
Trekker Breeze Firmware HumanWare Device not app for smartphone
previously Trekker (2003)
requires preset routes to work
BrailleNote GPS Firmware HumanWare PDA, allows note taking
and recording of points of interest
Assisting VIPs with Mobility in the Age of Context
GPS use by VIPs
2005 – GPS available on Cell Phones.
3 Satellites – give a position in 2-D space.
4 Satellites – give a position in 3-D space.
VIPs will typically rely on at least two devices as location
awareness is so critical to them – Kane (2009)
Assisting VIPs with Mobility in the Age of Context
When GPS is not Available
SmartPDR - Kang (2015)
Pedestrian Dead-ReckoningUses robotics location technology.
Using the IMU (Inertia Measurement
Unit) the application measures out
the pedestrian’s path, keeping track
of where the pedestrian is.
This interface may require
modification to be usable by VIPs.
Assisting VIPs with Mobility in the Age of Context
Augmenting the Environment
Many solutions suggest modifying the environment, typically with
RFIDs and beacons to interact with smartphone applications.
This is typical of indoor solutions, where GPS is not available.
An outside example is suggested by
Flores & Manduchi (2014),
where they place beacons at public
transport sites informing VIPs of their
location and guiding them to the
appropriate platform.
Assisting VIPs with Mobility in the Age of Context
Computer Vision Algorithms
Feng Hu (CCVCL – 2015)
Literature Review looks at various Computer Vision Algorithms
to aid with localization.
Main relevant points:
• 2D Solutions
• 3D Solutions
• 2D – 3D Solutions
• Emerging Technologies
Under consideration:
Using 2D camera pose in 3D model to orient VIP in Manhattan.
Assisting VIPs with Mobility in the Age of Context
Introduction
Mobile Devices
Sensors
Location Awareness
Social Media
Data
Integration
Conclusions
Assisting VIPs with Mobility in the Age of Context
Dictionary.com definition:
Websites and other online means of
communication that are used by large
groups of people to share information
and to develop social and professional
contacts.
Facebook Study (2014):
VIPs use Social Media just like everyone
else.
Social Media
Assisting VIPs with Mobility in the Age of Context
Social Media for VIPs
BlindSquare etc.
https://seshagiriprabhu.wordpress.com/2013/07/21/building-an-openstreetmap-tile-server-on-ubuntu-12-04-lts/http://www.foursquare.com
http://blindsquare.com/
A combination of two
existing applications for
iPhones
Software to
locate
friends
nearby
Crowd-sourced
electronic map
generation
Assisting VIPs with Mobility in the Age of Context
Social Media for VIPs
Crowdsourcing
https://itunes.apple.com/us/app/vizwiz/id439686043?mt=8
Provided by G. Olmschenk
Crowdsourcing is an interesting idea.
Ideally, volunteers log in
and help the VIP.
VizWiz – from J. Bigham’s
group had minimal
success.
Erin Brody ()2013)(same
group) found that VIPs as a rule meet this
technology with distrust. Time will tell …
Assisting VIPs with Mobility in the Age of Context
Introduction
Mobile Devices
Sensors
Location Awareness
Social Media
Data
Integration
Conclusions
Assisting VIPs with Mobility in the Age of Context
Big Data
The Miracle of Little Data
The Age of Context
S & I: Age of Context is a threat to our privacy.
Data today is stored in huge graph databases
which can then be mined to elicit specific data
Can we utilize this Age of Context
to aid VIPs with mobility?
on a specific subject, such as
one’s current location.
Assisting VIPs with Mobility in the Age of Context
http://www.aspire-consultancy.co.uk/0_html/products/
notetakers/braillesense/sync.htm
The SyncBraille
Braille Display http://www.antilogic.co.za/technology/
plan-b-tactile-map-translator-braille-maps/
Classic Data for VIPs
Tactile
Graphics, in
particular
Braille, are
now
electronically
available for
VIPs
Assisting VIPs with Mobility in the Age of Context
Image Data
https://en.wikipedia.org/wiki/Geotagging
Geotagging Li Fei-Fei (2005) – Textons for Bag-of-Words
Image understanding
Computer vision’s answers to aiding VIPs are
mostly through image data. Here we see some
image based algorithms.
Assisting VIPs with Mobility in the Age of Context
Introduction
Mobile Devices
Sensors
Location Awareness
Social Media
Data
Integration
Conclusions
Assisting VIPs with Mobility in the Age of Context
Integration of the Five Forces
Mobile Devices -
- Data
Social Media -
- Location Awareness
- Sensors
The Five Forces do not –
of course – exist on their
own.
Here we will consider
how some or all of the
forces have been
employed in
combination …
Assisting VIPs with Mobility in the Age of Context
Beyond Tactile Maps
Anke Brock (2015)
Tactile and audio sensors on
a mobile tablet provide
location awareness. The
data is haptic.
Talking tactile maps have
labels that give information
in audio format.
Assisting VIPs with Mobility in the Age of Context
Wearable Indoor Navigation System
Yicheng Bai (2014)
An example of a system
becoming more typical: the user
wears a camera and a GPS
enabled mobile device with
connectivity to the web. Data is
processed in the cloud. While
social media is not overtly
mentioned, it is suggested.
This is a concept paper.
Assisting VIPs with Mobility in the Age of Context
BlindNavi : A Navigation App for VIP
On SmartphoneHsuan-Eng Chen et al. (2015)
BlindNavi Project:
Similar but applies FourSquare
into the system. Uses a
smartphone to process the data
and location based on GPS.
Smartphone = Mobile device +
sensors, in particular camera.
Assisting VIPs with Mobility in the Age of Context
Talking Points 3
Rayoung Yang (2011)
We note that Talking Points 3
includes four of the five forces:
mobile devices, sensors, location
awareness, and data. Social
media could be added in such a
system for providing more relevant
and current information for VIPs.
Introduces the idea of interaction with the environment, a push-
pull approach.
Assisting VIPs with Mobility in the Age of Context
Introduction
Mobile Devices
Sensors
Location Awareness
Social Media
Data
Integration
Conclusions
Assisting VIPs with Mobility in the Age of Context
Summing Up the Age of Context
And VIPs
Research:
The possibilities are
there …
Evidence
suggests that
VIPs do not
readily accept
advances*
Assisting VIPs with Mobility in the Age of Context
Summing Up the Age of Context
And VIPsWhite Cane
&
Guide Dog
Traditional:
The iPhone has
been widely
adopted; but, few
VIPs readily utilize
applications*Brady (2013)
Assisting VIPs with Mobility in the Age of Context
Proposal:
An Interactive ApproachHere, we consider a
‘smart’ environment
that communicates
with the VIP who has a
mobile device capable
of processing image
and other data and of
communicating with
the environment.