one decade of sensorless sensing: wireless networks as
TRANSCRIPT
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
One Decade of Sensorless Sensing:Wireless Networks as Human Context Sensors
Neal Patwari
SPAWC 2015
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Outline
1 Introduction
2 RSS Device-Free Localization
3 Context Beyond Location
4 Conclusion
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Outline
1 Introduction
2 RSS Device-Free Localization
3 Context Beyond Location
4 Conclusion
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
My Talk: In Between Space
Wireless Communication Devices and Systems
Radar Devices and Systems
"Sen
sorle
ss"
Sens
ing
Radar research has produced amazing monitoring systems(Low cost) Comms devices perform channel estimationThe gap is wide
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Sensor Network Story (circa 2000)
Image Credit: The Sensor
Network Museum
Wireless sensors were predicted to:Cost .05 USDBe everywhere: walls, air, etc.Use ambient energy sources
We forgot about the cost/energy of thesensor
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Sensor Network Re-imagined
The radio itself, provided that it can measure the strength of theincoming signal, is the only sensor we use; with this sensorless
sensing approach, any wireless network becomes a sensornetwork.
— From Kristen Woyach, Daniele Puccinelli, Martin Haenggi,“Sensorless sensing in wireless networks: implementation andmeasurements”, IEEE WiOpt 2006
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Radio as a Sensor
According to Woyach et al., received signal strength (RSS) can:
Detect a person crossing a link lineClassify the environment as changed or unchangedDetect very small changes in position of TX or RXEstimate rotational velocity of the TX or RX
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Key Characteristics: Radio as a Sensor
Use wireless communication devices (low cost)Re-purpose existing channel estimatesWireless network becomes multistatic radarApplications in context awareness
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Radio Sensor Measurements
Wireless comms devices do estimate the channel, but mostdon’t allow access.
1 Received signal strength (RSS)2 MIMO channel state information (CSI)3 Phase measurement unit (PMU)
N transceivers→ O(N2) links
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Context Sensing
Image Credit: xkcd.com/138/
What/who is around us?What are people doing?Humans & computers should actappropriately for the context
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Application: Tracking People
Image: http://www.rfidjournal.com/
articles/view?11615
Not all people will wear tagsEfficiency: Smart buildingsSafety: Evacuations, factories,aging-in-placeSecurity: Monitoring,surveillance, health care
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Device-free Localization (DFL): Features of RF
Alternatives: Video, Audio, Thermal, InfraredRadio waves penetrate (non-metal) walls, furniture, smokeWorks in the dark, quietNot as privacy-invasive as audio or video surveillance
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Outline
1 Introduction
2 RSS Device-Free Localization
3 Context Beyond Location
4 Conclusion
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Zigbee RSS Measurement
CC2531 “USB dongle”2.4 GHz, IEEE 802.15.4,15 channelsRSS for each packet
30 40 50 60 70 80Time index
65
60
55
50
45
40
RSS
(dBm
)
Link 1Link 2
Person changes RSSTwo identical links:different changes
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Problem Statement: RSS Device-free Localization
5 7
18 19 20 2114 15 16 17
1
2
3
4
5 6 7 8 9 10 11 12 13
RSS changes most due to people in environment near linkOne person / object affects multiple linksMesh network of N nodes→ O
(N2) RSS measurements
Find: Count, locations of people
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
RSS-DFL: Survey of Current Capabilities
Experimental tests report 10 cm - 2 m avg.error using 5-35 nodes in 15-150 m2, and can
track 1-4 people.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Radio Tomographic Imaging (RTI)
1 Quantify “presence” on each link2 Presume it is linear combination of presence in pixels3 Pick regularization method4 Solve inverse problem
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
History: Shadowing as Linear Spatial Filter
xi
xj
xk
xl
link a link b
shadowing field ( )p x
Two nearby links’ shadowing iscorrelatedModel: shadow loss is a lineintegral of a spatially correlatedfield1
1N. Patwari and P. Agrawal, “Effects of correlated shadowing: connectivity, localization, and RF tomography,”
IPSN 2008.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Discrete-space Model
Consider simultaneously all M pair-wise links:
y = Wx + n
y = [y1, . . . yM ]T = measured “change” in RSSx = [x1, . . . xN ]T = discretized presence field (e.g.,dB/voxel)W = [[wi,j ]]i,j = weights; n = noise
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
∆ Shadowing Field Estimation Problems
Measure y, change in RSS from empty periodAssume known W . Estimate x.Ill-posed! Pixels� links, other issuesLinear model isn’t true physics; W is unknown.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Real-time Approaches to Image Estimation
Real-time requirement: linear estimator
x̂ = Πy
Projection Π needs only be calculated onceComplexity: Order of # Links × # pixelsRegularization: e.g., Tikanov, Least-squares
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Regularized Image Estimation Algorithms
1 Regularized inverse: minimize penalized squared error2
f (x) = ‖Wx− y‖2 + α‖Qx‖2
when Q is the derivative:
ΠTik =[W T W + α(DT
X DX + DTY DY )
]−1W T
2 Assume correlated image x and use regularized leastsquares.
ΠRLS =(
W T W + αC−1x
)−1W T
2J. Wilson and N. Patwari, “Radio tomographic imaging with wireless networks”, IEEE TMC, 2010.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Shadowing RTI
Experiment: Open deployment in atriumy is decrease in RSS compared to no person present
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Shadowing RTI with Passive Tags
Reader: 2 TX, 2 RX; 40 passive tags on floor of 16 m2
area3
30 cm average error
3B. Wagner, B. Striebing, D. Timmermann, “A system for live localization in smart environments”, IEEE ICNSC,
2013.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Variance RTI
Problem: Through-wallRSS changes don’t fitattenuation modelUse short-term RSSvariance for yAverage error: 45 - 63 cm,in 72 m2 area4
100 200 300 400 500 600 700−90
−80
−70
−60
RSS(dBm)
Vacant network area
100 200 300 400 500 600 700−90
−80
−70
−60
RSS(dBm)
Stationary human obstructing link
100 200 300 400 500 600 700−90
−80
−70
−60
RSS(dBm)
Moving human obstructing link
Time (samples)
Link (27,0) to (15.45,26.4)
Link (6,0) to (20,26.4)
4J. Wilson and N. Patwari, “ See through walls: motion tracking using variance-based radio tomography
networks”, IEEE TMC, 2011.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Multiple Channel RTI
Fading condition diversity.Anti-fade links are mostinformativeSpatial model (ellipse width)should be a function of fade leveland sign of RSS change5
Auto-update calibration forlong-term apartment (23 cmerror)6
5O. Kaltiokallio, M. Bocca, N. Patwari, “A fade level-based spatial model for radio tomographic imaging,” IEEE
TMC, 2013.6
M. Bocca, O. Kaltiokallio, and N. Patwari, “Radio tomographic imaging for ambient assisted living,” Evaluating
AAL Systems Through Competitive Benchmarking, 2013.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Multiple Person Tracking
Particle filtering, 0.7-1.0 m error 7
RTI-based, real-time, 1-4 people,< 55 cm error8
7F. Thouin, S. Nannuru and M. Coates, “Multi-target tracking for measurement models with additive
contributions,” ICIF 2011.8
M. Bocca et al., “ Multiple target tracking with RF sensor networks,” IEEE TMC, 2013.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
RTI in 3-D
Detect, classify vehicleson road 9
Classify person’s pose10
9C.R. Anderson, R.K. Martin, T.O. Walker, R.W. Thomas, “Radio tomography for roadside surveillance”, IEEE
JSTSP, 2014.10
B. Mager, N. Patwari, M. Bocca, “Fall detection using RF sensor networks”, PIMRC 2013.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
RSS Fingerprint
Attenuation/variance/histogram on each link forms highdimensional vectorTrain w/ person at each grid locationLearn map from RSS vector to coordinate2 m median error in hallways of 1500 m2 area11
1.7 m avg. error in 150 m2 area, tracking four people12
11M. Seifeldin et al., “Nuzzer: a large-scale device-free passive localization system for wireless environments”,
IEEE TMC 2013.12
C. Xu et al., “SCPL: indoor device-free multi-subject counting and localization using RSS”, IPSN 2013.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
RSS Fingerprint: Pros and Cons
Need training w/ person on each grid pointNo need for sensor coordsIncreased complexity in # peopleDatabase degrades as other things move
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
RSS Fingerprint: Degradation
Original state:8
7
6
5
4
3
2
1
01 2 3 4 5 6 7 8 9 10 11 12 130
Y c
oo
rdin
ate
(m
)
X coordinate (m)
Boxes of
Books
Houseplant
Dining set
TVconsole
Coat rack
Couch
Bags ofgroceries
Sink
Washingmachine
Bedroomdoor
Bathroomdoor
Node locations
Final state:8
7
6
5
4
3
2
1
01 2 3 4 5 6 7 8 9 10 11 12 130
Y c
oo
rdin
ate
(m
)
X coordinate (m)
Ironingboard
Filingcabinet
Node locations
Random change, retest, repeatError rate doubles each sixchanges, regardless of method
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Statistical Inversion Method I
Joint person tracking and sensorlocation13
Expectation Maximization(EM)-based algorithm30 cm error (open field, 49 m2)
13Xi Chen et al., “Sequential Monte Carlo for simultaneous passive device-free tracking and sensor localization
using received signal strength measurements”, IPSN 2011.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Statistical Inversion Method II
Learning of distribution of eachlink14
Gaussian mixture model15
13 cm error (open field)
14A. Edelstein, M. Rabbat, “Background subtraction for online calibration of baseline RSS in RF sensing
networks”, IEEE TMC 2013.15
Y. Zheng and A. Men, “Through-wall tracking with radio tomography networks using foreground detection”,
WCNC 2012.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Robust Line Crossing Location Estimation
(a)1 2 3 N
Short SegmentsBorder
Nodes
j=1 j=2
Link Lines (b)0 2 4 6 8 10
1
2
3
4
5
6
7
8
9
s1
s2
s3
NodesPerson Location
Person's TrackShort Segment
Time
0
1
2
3
4
Sta
te
Given: Link RSS measurementsProblem: Find between which nodes a person crossed.Each link is unreliable. Use redundant (longer links). How?Error correction coding16, hidden Markov model17
16P. Hillyard et al., “You’re crossing the line”, IEEE SPW 2015.
17P. Hillyard et al., “Demo: Detecting and Localizing Border Crossings Using RF Links” IPSN 2015.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Spatial Model for Variance
Need model for: What is the variance vs. person position?
Measurement at Bookstore, nodes on shelvesNormalize link, person position s.t. xr = (-1, 0), xt = (1,0)Find average variance by human position w.r.t. RX, TX
����
���
�
��
��
����
�
�
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Spatial Model: Setup
Human = tall cylinder diameter DReflectors in a plane. TX, RX, inplane ∆z above18
Propagation via single reflection,path loss ∝ d−n
18N. Patwari and J. Wilson, “Spatial models for human motion-induced signal strength variance on static links",
IEEE Trans. Info. Forensics & Security, 2011.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Spatial Model: Results
Mean variance ∝ spatial functions:
−2 −1 0 1 2−2
−1.5
−1
−0.5
0
0.5
1
1.5
2
X Coordinate
Y C
oord
inate
−3−6
−9
−12
−15
−18
−3−6
−9
−12
−15
−18
Matches with our, others’ results
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Outline
1 Introduction
2 RSS Device-Free Localization
3 Context Beyond Location
4 Conclusion
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Breathing Rate Estimation
0.2 0.3 0.4 0.5 0.630
40
50
60
70
80
90
100
110
Frequency (Hz)
No
rma
lize
d A
ve
rag
e P
SD
Norm. Avg. PSD
Actual Breathing Rate
Breathing causes periodic change in RSSMeasure many channels’ RSS over time (30 s)19
Peak of avg. PSD. Error: about 0.4 breaths/min
19O. Kaltiokallio et al., “Catch a breath: non-invasive respiration rate monitoring via wireless communication”,
IPSN 2014.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Channel State Information (CSI) Measurement
Hacked driver for Intel WiFi 802.11n 5300 NIC20
Gives channel gain (amplitude and phase)For 30 subcarriers from among OFDM subcarriersFor each antenna pair in 3x3 MIMO
20D. Halperin, “Tool Release: Gathering 802.11n Traces with Channel State Information”, ACM SIGCOMM
CCR, 2011.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Activity Recognition
Activities vary in temporal, frequency, dist’n characteristicsUsing training DB, can classify w/ machine learning 21 22
21S. Sigg, M. Scholz, et al., “RF-sensing of activities from non-cooperative subjects ...”, IEEE TMC 2014.
22B. Wei, W. Hu, M. Yang, C.T. Chou, “Radio-based Device-free Activity Recognition with Radio Frequency
Interference”, IPSN 2015.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Person Counting
Problem: Estimate the # ofpeople moving randomly inareaSolution: Match RSS hist toanalytical pdf 23
23S. Depatla, A. Muralidharan and Y. Mostofi, "Occupancy Estimation Using Only WiFi Power Measurements,"
IEEE JSAC 2015.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Gesture Recognition from Micro-Doppler
Gestures (left) vary in time-Doppler characteristicsSmall (<20 Hz) Doppler can be estimated from OFDMpackets (using an SDR RX) 24
Can be measured from AM at a passive RFID tag 25
24Q. Pu, S. Gupta, S. Gollakota, and S. Patel, “Whole-Home Gesture Recognition Using Wireless Signals”,
MobiCom 2013.25
B. Kellogg, V. Talla, and S. Gollakota “Bringing Gesture Recognition to All Devices”, NSDI 2014.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Keystroke Recognition
Typing on a keyboard near a TXchanges MIMO channel 26
Phase of changes can be measuredand used to estimate key typedTyping a few known words allowstrainingImplemented with SDRs, but could bedone with 802.11n CSI
26B. Chen, V. Yenamandra and K. Srinivasan, “Tracking Keystrokes Using Wireless Signals”, MobiSys 2015.
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Outline
1 Introduction
2 RSS Device-Free Localization
3 Context Beyond Location
4 Conclusion
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Open Research Area: Modeling
How to model human position effect on channelStatistical, temporalFunction of link length, environment, fade level
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Open Research: Context Awareness
Need for fundamental temporal, Doppler, statisticalfeatures from gestures and activities to reduce trainingreq’ts.Interface to channel estimates made on COTS RFICsCompared to radar devices, incomplete dataFind estimators, bounds for estimators, from such data
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Commercialization: Xandem
http://www.xandem.com
RSS-based motion detection system
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Security Pain Point
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Conclusion
Channel estimation in wireless comms enables contextawareness sensingSignificant commercial needsEg: localization, monitoring, activity, gesture recognitionMany estimation, detection, classification problems yet tobe solved
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors
Introduction RSS Device-Free Localization Context Beyond Location Conclusion
Questions and Comments
More info on http://span.ece.utah.edu/
Neal Patwari University of Utah
One Decade of Sensorless Sensing: Wireless Networks as Human Context Sensors