1

SurroundSense: Mobile Phone Localization via Ambience Fingerprinting

Ionut Constandache

Co-authors: Martin Azizyan and Romit Roy Choudhury

2

Context

Pervasive wireless connectivity+

Localization technology=

Location-based applications

3

Location-Based Applications (LBAs)

For Example: GeoLife shows grocery list when near Walmart MicroBlog queries users at a museum Location-based ad: Phone gets coupon at Starbucks

iPhone AppStore: 3000 LBAs, Android: 500 LBAs

4

Location-Based Applications (LBAs)

For Example: GeoLife shows grocery list when near Walmart MicroBlog queries users at a museum Location-based ad: Phone gets coupon at Starbucks

iPhone AppStore: 3000 LBAs, Android: 500 LBAs

Location expresses context of user Facilitates content delivery

5

Location is an IP addressLocation is an IP addressAs if for content delivery

6

Thinking about Localizationfrom an application perspective…

7

Emerging location based apps need place of user, not physical location

Starbucks, RadioShack, Museum, Library

Latitude, Longitude

8

Emerging location based apps need place of user, not physical location

Starbucks, RadioShack, Museum, Library

Latitude, Longitude

We call this Logical Localization …We call this Logical Localization …

9

Can we convert from Physical to Logical Localization?

10

Can we convert from Physical to Logical Localization?

State of the Art in Physical Localization:1. GPS Accuracy: 10m 2. GSM Accuracy: 100m3. Skyhook (WiFi+GPS+GSM) Accuracy: 10m-

100m

State of the Art in Physical Localization:1. GPS Accuracy: 10m 2. GSM Accuracy: 100m3. Skyhook (WiFi+GPS+GSM) Accuracy: 10m-

100m

11

Widely-deployable localization technologies have errors in the range of several meters

Widely-deployable localization technologies have errors in the range of several meters

Can we convert from Physical to Logical Localization?

State of the Art in Physical Localization:1. GPS Accuracy: 10m 2. GSM Accuracy: 100m3. Skyhook (WiFi+GPS+GSM) Accuracy: 10m-

100m

State of the Art in Physical Localization:1. GPS Accuracy: 10m 2. GSM Accuracy: 100m3. Skyhook (WiFi+GPS+GSM) Accuracy: 10m-

100m

12

Several meters of error is inadequate to logically localize a phone

Physical LocationError

13

Several meters of error is inadequate to logically localize a phone

RadioShackStarbucks

Physical LocationError

The dividing-wall problem The dividing-wall problem

14

Contents

SurroundSense

Evaluation

Limitations and Future Work

Conclusion

15

Contents

SurroundSense

Evaluation

Limitations and Future Work

Conclusion

16

It is possible to localize phones by sensing the ambience

Hypothesis

such as sound, light, color, movement, WiFi …

17

Sensing over multiple dimensions extracts more information from the ambience

Each dimension may not be unique, but put together, they may provide a

unique fingerprint

18

SurroundSense

Multi-dimensional fingerprint Based on ambient sound/light/color/movement/WiFi

Starbucks

Wall

RadioShack

19

BB AACC DDEE

Should Ambiences be Unique Worldwide?

FFGG

HHJJ

II

LLMMNN

OO

PPQQ

QQ

RR

KK

20

Should Ambiences be Unique Worldwide?

BB AACC DDEE

FFGG

HHJJ

II

KKLL

MMNNOO

PPQQ

QQ

RR

GSM provides macro location (strip mall) SurroundSense refines to Starbucks

21

++

Ambience Fingerprinting

Test Fingerprint

Sound

Acc.

Color/Light

WiFi

LogicalLocation

Matching

FingerprintDatabase

==

Candidate Fingerprints

GSM Macro Location

SurroundSense Architecture

22

Fingerprints

Sound:(via phone microphone)

Color:(via phone camera)

Amplitude Values-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1

N

orm

aliz

ed

C

ount

0.14

0.12

0.1

0.08

0.06

0.04 0.02

0

Acoustic fingerprint (amplitude distribution)

Color and light fingerprints on HSL space

Lightn

ess

1

0.5

0

Hue

0

0.5

1 00.2

0.4

0.6

0.8

1

Saturation

23

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Moving

24

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Moving

Queuing

25

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Queuing Seated

Moving

26

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Pause for product browsing

Moving

27

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Pause for product browsing

Short walks between product browsing

Moving

28

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Walk more

Moving

29

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Walk more Quicker stops

Moving

30

Fingerprints

Movement: (via phone accelerometer)

WiFi: (via phone wireless card)

Cafeteria Clothes Store Grocery Store

Static

ƒ(overheard WiFi APs)

ƒ(overheard WiFi APs)

Moving

31

Discussion

Time varying ambience Collect ambience fingerprints over different time

windows

What if phones are in pockets? Use sound/WiFi/movement Opportunistically take pictures

Fingerprint Database War-sensing

32

Contents

SurroundSense

Evaluation

Limitations and Future Work

Conclusion

33

Evaluation Methodology

51 business locations 46 in Durham, NC 5 in India

Data collected by 4 people 12 tests per location

Mimicked customer behavior

34

Evaluation: Per-Cluster Accuracy

Cluster

No. of Shops

1 2 3 4 5 6 7 8 9 10

4 7 3 7 4 5 5 6 5 5

A

ccura

cy (

%)

Cluster

Localization accuracy per cluster

35

Evaluation: Per-Cluster Accuracy

Cluster

No. of Shops

1 2 3 4 5 6 7 8 9 10

4 7 3 7 4 5 5 6 5 5

A

ccura

cy (

%)

Cluster

Localization accuracy per cluster

Multidimensional sensing

36

Evaluation: Per-Cluster Accuracy

Cluster

No. of Shops

1 2 3 4 5 6 7 8 9 10

4 7 3 7 4 5 5 6 5 5

Fault tolerance

A

ccura

cy (

%)

Cluster

Localization accuracy per cluster

37

Evaluation: Per-Cluster Accuracy

Cluster

No. of Shops

1 2 3 4 5 6 7 8 9 10

4 7 3 7 4 5 5 6 5 5

A

ccura

cy (

%)

Cluster

Localization accuracy per cluster

Sparse WiFi APs

38

Evaluation: Per-Cluster Accuracy

Cluster

No. of Shops

1 2 3 4 5 6 7 8 9 10

4 7 3 7 4 5 5 6 5 5

No WiFi APs

Acc

ura

cy (

%)

Cluster

Localization accuracy per cluster

39

Evaluation: Per-Scheme Accuracy

Mode WiFi Snd-Acc-WiFi Snd-Acc-Lt-Clr SS

Accuracy 70%

74% 76% 87%

40

Evaluation: User Experience

Random Person Accuracy

Average Accuracy (%)0 10 20 30 40 50 60 70 80 90 100

1 0.9 0.8 0.7 0.6 0.5

CD

F

0.4 0.3 0.2 0.1 0

WiFISnd-Acc-WiFiSnd-Acc-Clr-LtSurroundSense

41

Economics forces nearby businesses to be different

Not profitable to have 3 coffee shopswith same lighting, music, color, layout, etc.

SurroundSense exploits this ambience diversity

Why does it work?

The Intuition:

42

Contents

SurroundSense

Evaluation

Limitations and Future Work

Conclusion

43

Limitations and Future Work

Energy-Efficiency

Localization in Real Time

Non-business locations

44

Limitations and Future Work

Energy-Efficiency Continuous sensing likely to have a large energy draw

Localization in Real Time

Non-business locations

45

Limitations and Future Work

Energy-Efficiency Continuous sensing likely to have a large energy draw

Localization in Real Time User’s movement requires time to converge

Non-business locations

46

Limitations and Future Work

Energy-Efficiency Continuous sensing likely to have a large energy draw

Localization in Real Time User’s movement requires time to converge

Non-business locations Ambiences may be less diverse

47

Contents

SurroundSense

Evaluation

Limitations and Future Work

Conclusion

48

SurroundSense

Today’s technologies cannot provide logical localization

Ambience contains information for logical localization

Mobile Phones can harness the ambience through sensors

Evaluation results: 51 business locations, 87% accuracy

SurroundSense can scale to any part of the world

49

Questions?

Thank You!

Visit the SyNRG research group @http://synrg.ee.duke.edu/