James Coughlan, Ph.D.

The Crosswatch Traff ic Intersection Analyzer: a Roadmap for the Future

2

Traff ic intersections: diff icult for many visually impaired

travelersHow to align oneself

properly to crosswalk?What is the layout of the

intersection?How to remain inside

crosswalk while walking?

When is it safe to cross?How to avoid moving

vehicles?

Past work“Crosswatch” system (Ivanchenko et al., ICCHP 2010; etc.) – computer vision smartphone app that performs the following functions in real time for visually impaired user:•find crosswalk markings (e.g., zebra stripes), determine user’s orientation relative to crosswalk “corridor”•find Walk light and announce if it’s on

3

Crosswatch (past work)

4

Two kinds of misalignment relative to crosswalk corridor: (1) translated to the left; (2) direction error.

Crosswatch can calculate both kinds of misalignment.

Crosswatch (past work)

5

How Crosswatch finds crosswalk and calculates alignment

Crosswatch (past work)

Crosswalk scene Zoomed-inregion showingWalk light 6

Advantages of Crosswatch approach

Information provided to user in real time

System requires no new infrastructure since it analyzes the visual features that are already there

7

Related workAranda & Mares, ICCHP 2004: computer vision for Walk light detection

Zebralocalizer (Ahmetovic, Bernareggi & Mascetti, 2011): smartphone computer vision system

Bohonos et al., ICCHP 2008: Bluetooth beaconing at traffic intersections

8

Next stepsTravelers need a broad range of information about intersections:•What type of intersection, features in it•Where are the crosswalk and other important features•When is it time to cross?

9

“What” informationWhat is the intersection layout? Four-way, three-way, etc.?Are there stop signs? Signal lights? Walk light?Is there a pedestrian walk button?Is there a median strip somewhere in the crosswalk?

10

“Where” informationWhere are all the crosswalks? How am I oriented relative to the one I want?

Where is the pedestrian walk button relative to where I am standing?

How long is the crosswalk? How far is it to the median strip?

11

“When” informationWhen is it time to cross, according to Walk light (or traffic lights if there is no Walk light)?

Note: this is not necessarily the same as when it is safe to cross!

12

How to get What, Where, When information?

Realization: it is impractical to expect computer vision to figure out all this information on its own!

Intersections, and the features in them, are extremely variable, and not always easy to see (even for people with normal vision)

13

Walk buttons come in many shapes and sizes

14

Find the walk button!

15

Median strips

Without high-resolution depth information to show the elevated median strip, it is hard to find reliably

16

Complicated intersectionsWhere a visually impaired traveler may need the most guidance… computer vision may have the most difficulties!

17

18

Solution: GISUse computer vision where it is necessary: (b)find user’s location and orientation relative to intersection features(b) detect signal lights in real time (e.g., Walk light)

Augment with GIS (Geographic Information Systems)

19

How this worksGPS tells you which intersection you’re atSmartphone compass and accelerometer indicate direction you’re pointing inComputer vision analyzes visible features in scene and determines (x,y) GPS coordinates (to within about 1 meter)Use GIS to find information relevant to current location (look up online, or download all information for a city or region onto smartphone)

How this works (continued)

GIS has data for each intersection, including layout of all crosswalks, list of important features and landmarks and their locations

20

Much to doWhere to find this GIS information?

Some available through municipal/transit agencies, Google Maps

But much of it may require crowd-sourcing: volunteers will contribute information about intersections in their neighborhood to online GIS database 21

Much to do (continued)Develop Crosswatch algorithms for estimating high-resolution GPS coordinates

Appropriate Crosswatch user interface: for instance, how to help user pan left to right to capture entire intersection scene

User surveys (what should Crosswatch do and not do?) and testing

22

ConclusionCrosswatch needs to expand its functionality to provide extensive “what”, “where” and “when” information

Use computer vision where it is needed and rely on GIS to provide other information (data infrastructure)

Future idea: use GIS to help with route planning that is tailored to needs of traveler

23

24

Thanks to…Second author: Dr. Huiying Shen (Smith-

Kettlewell)

Collaborators: Dr. Megan Lawrence (Smith-Kettlewell), Dr. Volodymyr Ivanchenko (Amazon Lab 126)

Funding from NIH and NIDRR