the limits of localization using signal strength: a comparative study eiman elnahrawy, xiaoyan li,...

The Limits of Localization Using Signal Strength: A Comparative

Study

Eiman Elnahrawy, Xiaoyan Li, and Richard Eiman Elnahrawy, Xiaoyan Li, and Richard MartinMartin

Dept. of Computer Science, Rutgers UniversityDept. of Computer Science, Rutgers University

SECON, October 7SECON, October 7thth 2004 2004

Dept. of Computer Science, Rutgers University

Sensors Everywhere

Tracking, monitoring, geometric-based routing, Tracking, monitoring, geometric-based routing, ……

““Localization in indoor environments”Localization in indoor environments”

Off-the-shelf radiosOff-the-shelf radios

Already used for communicationAlready used for communication

No additional hardwareNo additional hardware


Background: Radio-Based Localization

Two strategies:Two strategies:[1] Classifiers (matching or [1] Classifiers (matching or

learning): match RSS to learning): match RSS to existing fingerprints [(x,y),SS]existing fingerprints [(x,y),SS]

[2] Propagation model: distance as [2] Propagation model: distance as a function of signal strength: a function of signal strength:

Sj = kSj = k0j0j + k + k1j1j log Dj log Dj, ,

Dj = sqrt((x-xDj = sqrt((x-xjj))22-(y-y-(y-yjj))22))

[Bahl00, Ladd02, Roos02, Smailagic02, Youssef03, Krishnan04]

[-80,-67,-50]

RSS

(xj,yj)

(x?,y?)

[(x,y),s1,s2,s3]

[(x,y),s1,s2,s3][(x,y),s1,s2,s3]


Major Question

What is the best performance we can achieve in What is the best performance we can achieve in radio-basedradio-based localization localization without any additional without any additional hardwarehardware??

UltrasoundUltrasound

Very high frequency clocksVery high frequency clocks

Directional antennaDirectional antenna

Related Work: different technologies/algorithms Related Work: different technologies/algorithms [Want92, Priyantha00, Doherty01, Niculescue01, Savvides01, Shang03, He03, Hazas03, Lorincz04]


Strategy

Compared localization performance across a wide Compared localization performance across a wide spectrum of representative algorithms spectrum of representative algorithms

2 previous + 3 new “area-based” + variances 2 previous + 3 new “area-based” + variances = 11 algorithms= 11 algorithms

Used combination of traditional and new metricsUsed combination of traditional and new metrics

Ran extensive experiments across different Ran extensive experiments across different environments (2 different buildings) using 802.11environments (2 different buildings) using 802.11

Combined results with a prior small-scale evaluation Combined results with a prior small-scale evaluation on matching algorithms (classifiers)on matching algorithms (classifiers) [Battiti02][Battiti02]


Results in a Nutshell

All algorithms have similar performance All algorithms have similar performance

Fundamental limitations due to noise and systematic errorsFundamental limitations due to noise and systematic errors

Area-based algorithms allow Precision-Accuracy tradeoffsArea-based algorithms allow Precision-Accuracy tradeoffs

Within existing HW and using simple floor maps we can get Within existing HW and using simple floor maps we can get 95% room-accuracyMedian error of 10 feet and 97th percentile of 30 feet

To improve, use To improve, use additional hardwareadditional hardware (e.g., directional (e.g., directional antenna) or antenna) or a more complex environment modela more complex environment model (i.e., (i.e., model every shelf, desk!)model every shelf, desk!)


Outline

Introduction, Motivations, and Related WorkIntroduction, Motivations, and Related Work

Localization Algorithms: An OverviewLocalization Algorithms: An Overview

Experimental EvaluationExperimental Evaluation

Performance metricsPerformance metrics

Experimental setupExperimental setup

ResultsResults

Conclusion and Future WorkConclusion and Future Work


Point-based Versus Area-based

““Point-based” algorithmsPoint-based” algorithms

Returned answer is a single locationReturned answer is a single location

““Area-based” algorithmsArea-based” algorithms

Returned answer is an area (or volume) likely Returned answer is an area (or volume) likely to contain the localized sensor to contain the localized sensor

Ability to describe the uncertainty: possible Ability to describe the uncertainty: possible locloc

Accuracy Accuracy (likelihood of being in the area) vs. (likelihood of being in the area) vs. PrecisionPrecision (area size) (area size)


Radio-Based Algorithms at a Glance

(1) Simple Point Matching (1) Simple Point Matching (SPM)(SPM)

(2) Area Based Probability (2) Area Based Probability (ABP)(ABP)

(3) Bayesian Networks (3) Bayesian Networks (BN)(BN)

(6) RADAR(6) RADAR

(7) Averaged RADAR(7) Averaged RADAR

(8) Gridded RADAR(8) Gridded RADAR

(9) Highest probability(9) Highest probability

(10) Averaged highest (10) Averaged highest probabilityprobability

(11) Gridded highest (11) Gridded highest probabilityprobability

(4) Bayesian point(4) Bayesian point

(5) Averaged Bayesian(5) Averaged Bayesian

Matching SS vs. distance

Are

a-b

ase

dP

oin

t-b

as

ed


1. SPMFind Find ∩ tiles which fall tiles which fall within a within a “threshold”“threshold” of of RSS for each AP RSS for each AP

∩ ∩

=

2. ABP-cUsing Using “Bayes’ rule”“Bayes’ rule” compute compute likelihood likelihood of an RSS matching of an RSS matching a fingerprint for each tilea fingerprint for each tile

p(Ti|RSS) p(Ti|RSS) αα p(RSS|Ti) . p(Ti) p(RSS|Ti) . p(Ti)

Return a set of tiles bounded Return a set of tiles bounded by an overall probability that by an overall probability that the sensor lies in the area the sensor lies in the area (Confidence: user-defined)(Confidence: user-defined)

Build a regular grid of tiles, tile Build a regular grid of tiles, tile ↔↔ expected fingerprint expected fingerprint

Eager:Eager: start from low start from low threshold (= threshold (= δδ, 2 × , 2 × δδ, …), …)

Similar to MLESimilar to MLE

Confidence Confidence ↑↑ →→ Area size Area size ↑↑


Measurement At Each Tile Is Expensive!

Triangle-based linear interpolation using Triangle-based linear interpolation using “Delaunay Triangulation“Delaunay Triangulation” (Surface Fitting)(Surface Fitting)

Simple, fast, and efficientSimple, fast, and efficient

Insensitive to the tile sizeInsensitive to the tile size


3. Bayesian Networks

Graphical models that encode dependencies Graphical models that encode dependencies between variablesbetween variables

Propagation model: Dj is the distance to the jth Propagation model: Dj is the distance to the jth APAP

Sj = kSj = k0j0j + k + k1j1j log Dj log Dj Dj = sqrt((x-xDj = sqrt((x-xjj))22-(y-y-(y-yjj))22))

Network learns parameters kNetwork learns parameters k0j0j, k, k1j1j, x, y , x, y distribution (joint PDF) using training fingerprintsdistribution (joint PDF) using training fingerprints

S1 S2 S3 S4

D1 D2 D3 D4

X Y


Point Based Algorithms…

(4) Bayesian Point, (5) Averaged Bayesian(4) Bayesian Point, (5) Averaged Bayesian

(6) RADAR(6) RADARReturn the Return the “closest”“closest” fingerprint to the RSS in the fingerprint to the RSS in the training set using training set using “Euclidean Distance in signal space”“Euclidean Distance in signal space”

(7) Averaged RADAR, (8) Gridded RADAR(7) Averaged RADAR, (8) Gridded RADAR

(9) Highest Probability(9) Highest ProbabilitySimilar to ABP: a typical approach that uses Similar to ABP: a typical approach that uses “Bayes’ “Bayes’ rule”rule” but returns the but returns the “highest probability single “highest probability single location”location”

(10) Averaged Highest Probability, (11) Gridded Highest (10) Averaged Highest Probability, (11) Gridded Highest ProbabilityProbability


Performance Metrics

Traditional: Traditional: Distance error between returned and true Distance error between returned and true positionposition

Return avg, 95Return avg, 95thth percentile, or full CDF percentile, or full CDF

Does not apply to area-based algorithms!Does not apply to area-based algorithms!

Does not show accuracy-precision tradeoffs! Does not show accuracy-precision tradeoffs!


New Metrics: Accuracy Vs. Precision

Tile AccuracyTile Accuracy % true tile is returned % true tile is returned

Distance AccuracyDistance Accuracy distance between distance between true tile and returned tiles (sort and true tile and returned tiles (sort and use percentiles to capture use percentiles to capture distribution)distribution)

PrecisionPrecision size of returned area (e.g., size of returned area (e.g., sq.ft.) or % floor sizesq.ft.) or % floor size


Room-Level Metrics

Applications usually operate at the level of roomsApplications usually operate at the level of rooms

Mapping: divide floor into rooms and map tiles Mapping: divide floor into rooms and map tiles

(Point (Point ↔↔ Room): easy Room): easy

(Area (Area ↔↔ Room): tricky Room): tricky

Metrics: accuracy-precision Metrics: accuracy-precision Room AccuracyRoom Accuracy % true room is the returned room % true room is the returned room

Top-n rooms AccuracyTop-n rooms Accuracy % true room is among the % true room is among the returned rooms (for area-based algorithms)returned rooms (for area-based algorithms)

Room PrecisionRoom Precision avg number of returned rooms (for avg number of returned rooms (for area-based algorithms)area-based algorithms)


Experimental Setup

CoRECoRE

286 fingerprints (rooms + 286 fingerprints (rooms + hallways)hallways)

50 rooms50 rooms

200x80 feet200x80 feet

4 Access Points (somewhat linear)4 Access Points (somewhat linear)

Industrial research labIndustrial research lab

253 fingerprints (all hallways)253 fingerprints (all hallways)

20 rooms out of 115 rooms20 rooms out of 115 rooms

225x144 feet225x144 feet

5 Access Points (nicely 5 Access Points (nicely distributed)distributed)


[1] Impact of Training on Accuracy and Precision

Divide data into training and testing setsDivide data into training and testing sets

Both Both locationlocation and and numbernumber of training samples impact of training samples impact performanceperformance

Different strategies Different strategies [Fixed spacing vs. Average spacing][Fixed spacing vs. Average spacing]: as long : as long as samples are as samples are “uniformly distributed”“uniformly distributed” but not necessarily but not necessarily “uniformly spaced”“uniformly spaced” methodology has no measurable effect methodology has no measurable effect


Number of samples has an impact, Number of samples has an impact, but not strongbut not strong!!

Improving Improving AccuracyAccuracy worsens worsens PrecisionPrecision (tradeoff)(tradeoff)

0 50 100 150 200 25060

65

70

75

80

85

90

95

100

Training data size

% a

ccu

racy

Average Overall Room Accuracy

SPM

ABP-50

ABP-75

ABP-95

BN

0 50 100 150 200 2500

2

4

6

8

10

Training data size

% f

loo

r

Average Overall PrecisionSPM

ABP-50

ABP-75

ABP-95

BN


0 20 40 60 80 1000

0.2

0.4

0.6

0.8

1

pro

ba

bil

ity

ABP-75: Percentiles' CDF

Minimum25 PercentileMedian75 PercentileMaximum

0 20 40 60 80 1000

0.2

0.4

0.6

0.8

1

pro

ba

bil

ity



0 20 40 60 80 1000

0.2

0.4

0.6

0.8

1

distance in feet

pro

ba

bil

ity

SPM: Percentiles' CDF


0 20 40 60 80 1000

0.2

0.4

0.6

0.8

1

distance in feet

pro

ba

bil

ity



[2] A Deeper Look Into “Accuracy”


[3] Comparing All Algorithms: Accuracy

Traditional error along with median CDF for area-Traditional error along with median CDF for area-based algorithmsbased algorithms

Room-level accuracyRoom-level accuracy


0 20 40 60 80 1000

0.2

0.4

0.6

0.8

1

distance in feet

pro

bab

ilit

y

Error CDF Across Algorithms

BN-MedianABP-75 MedianR1R2GRP1P2GPB1B2

Striking similarity! Similar slope, percentilesStriking similarity! Similar slope, percentiles

Increasing sample size yields marginal improvementIncreasing sample size yields marginal improvement

Given [Battiti et al] compared a host of “learning Given [Battiti et al] compared a host of “learning approaches” including MLE and found similar performance approaches” including MLE and found similar performance » » ”all algo are qualitatively similar””all algo are qualitatively similar”


Algorithm

Pro

bab

ilit

y

Within Room Accuracy Stack

S A50 A75 A95 BN R1 R2 GR P1 P2 GP B1 B20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

FirstSecondThirdOthers

Similar top-room accuracySimilar top-room accuracy

Area-based algorithms are Area-based algorithms are superior at returning multiple superior at returning multiple rooms rooms

Again, increasing sample size yields marginal improvementAgain, increasing sample size yields marginal improvement


[4] Fundamental Uncertainty?Bayesian net used to generate spatial uncertainty PDF across Bayesian net used to generate spatial uncertainty PDF across x,yx,y

Wide distributions » Wide distributions » high degree of uncertaintyhigh degree of uncertainty

Attenuation along x in CoRE helps reducing uncertaintyAttenuation along x in CoRE helps reducing uncertainty

0 50 100 150 2000

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

x in feet

pro

ba

bil

ity

0 10 20 30 40 50 60 70 800

0.02

0.04

0.06

0.08

0.1

0.12

y in feet

pro

ba

bil

ity

0 50 100 150 2000

0.01

0.02

0.03

0.04

0.05

0.06

0.07

x in feetp

rob

ab

ilit

y0 20 40 60 80 100 120 140

0

0.01

0.02

0.03

0.04

0.05

0.06

y in feet

pro

ba

bil

ity

CoRE Research Lab


Take Home Lessons

All algorithms behave the same: there is a fundamental All algorithms behave the same: there is a fundamental uncertainty due to environmental effects!uncertainty due to environmental effects!

Point-based approaches find the peaks in the PDFsPoint-based approaches find the peaks in the PDFs

Area-based approaches explore more of the PDF but Area-based approaches explore more of the PDF but cannot narrow it (cannot eliminate the uncertainty)cannot narrow it (cannot eliminate the uncertainty)

Area-based approaches offer a precision-accuracy tradeoffArea-based approaches offer a precision-accuracy tradeoff

Uniform distribution with sufficient density: a rule of thumbUniform distribution with sufficient density: a rule of thumbSample density of 1/230ftSample density of 1/230ft22 (every 15 ft) (every 15 ft) Reasonable performance at 1/450ftReasonable performance at 1/450ft22 (every 21 ft) (every 21 ft)


Conclusion and Future Work

Characterized the limits of a wide variety of approaches to Characterized the limits of a wide variety of approaches to localization using 802.11localization using 802.11

Algorithms based on matching and signal-to-distance Algorithms based on matching and signal-to-distance functions are unable to capture effects on signal functions are unable to capture effects on signal propagationpropagation

Useful accuracyUseful accuracy, different ways to describe it to users, different ways to describe it to users

Uncertainty cannot be reduced!Uncertainty cannot be reduced!

Additional HW or complex models to improve accuracyAdditional HW or complex models to improve accuracy

Future work: hmm! Would it be worth the improvementFuture work: hmm! Would it be worth the improvement


Thank YouThank You

the limits of localization using signal strength: a comparative study eiman elnahrawy, xiaoyan li,...

Documents

new areabased variances

nutshellall algorithms

area confidence

geometricbased routing

k0j k1j log dj dj

limits of localization

threshold of rss

variablespropagation