larson.defense
DESCRIPTION
TRANSCRIPT
disaggregated hot and cold water sensing with minimal calibration
semi-supervised training for infrastructure mediated sensing
Eric C. Larson
UbiComp Lab
electrical engineering
computerscience and engineering
University of Washington
how can indirect sensing and machine learning be used to reduce our
environmental footprint?
3
lake mead 1983
4
lake mead 2011
5
we are using water faster than it is being replenished
Pacific Institute for Studies in Development, Environment, and Security, 2011 6
we are using water faster than it is being replenished
Pacific Institute for Studies in Development, Environment, and Security, 2011 6
image: weiku.com
$2,994.83
7
8
water usage is vastly misunderstood
eco-feedback
10
eco-feedback
11 image: gardena, inc
eco-feedback
11 image: gardena, inc
image: showersmartimage: iSave
eco-feedback
Geographic Comparisons Dashboards
Metaphorical Unit Designs Recommendations 12
eco-feedback
13
eco-feedback
14
eco-feedback
15
eco-feedback
16
eco-feedback
17 video: courtesy Jon Froehlich
what are the potential water savings?
eco-feedback in electricity19
0%
5%
10%
15%
20%
1 2 3 4 5 Untitled 1
20%
12%9.2%8.4%
6.8%
3.8%
Enhanced Billing
Web Based
Daily Feedback
Realtime Feedback
Appliance Level + Personalized
Feedback
Ann
ual %
Sav
ings
Based on 36 studies between 1995-2010. Summarized by Ehrhardt-Matinez et al.>20% reduction: Gardner et al. (2008) and Laitner et al. (2009)
Appliance Level
eco-feedback in electricity
aggregate
disaggregated
Courtesy: Sidhant Gupta20
how can we sense water usage?
22
15
TurbineInsert
Thermistor
Flow
image: LBNL
22
15
TurbineInsert
Thermistor
Flow
image: LBNL
metersflow rate fixture flow
inline water
23
metersflow rate fixture flow
inline water
waterpressure
pressuresensor
machine learning
estimated
• central sensing point
• easy to install
• low cost
• can observe every fixture
HydroSense
25
• central sensing point
• easy to install
• low cost
• can observe every fixture
HydroSense
25
40#
50#
60#
70#
80#
Cold Line Pressure (Hose Spigot)
0 9 4.5
time (s)
psi
open close
HydroSense
26
kitchen sink
upstairs toilet
template matching
unknown event
27
downstairs toilet
kitchen sink
upstairs toilet
template matching
unknown event
27
downstairs toilet
feasibility study
• 10 homes
• staged calibration
• ~98% accuracy
Froehlich, J., Larson, E., Campbell, T., Haggerty, C., Fogarty, J., and Patel, S.N. HydroSense: infrastructure-mediated single-point sensing of whole-home water activity. Proceedings of the 11th ACM international conference on Ubiquitous computing, (2009), 235–244.
Larson, E., Froehlich, J., Campbell, T., et al. Disaggregated water sensing from a single, pressure- based sensor: An extended analysis of HydroSense using staged experiments. Pervasive and Mobile Computing, (2010).
28
70
50
30
pres
sure
(psi)
70
50
30
pres
sure
(psi)
initial study: staged events
kitchen sink kitchen sink
29
70
50
30
pres
sure
(psi)
70
50
30
pres
sure
(psi)
natural water use
30
how well does HydroSense work in a natural setting?
longitudinal evaluation
32
33
34
35
36
totals
days 33 33 30 27 33 156
events 2374 3075 4754 2499 2578 14,960
events/day 71.9 93.2 158.5 92.6 78.1 95.9
compound 22.2% 21.8% 16.6% 32% 21.3% 21.9%
data collection
Larson, E., Froehlich, J., Saba, E., et al. A Longitudinal Study of Pressure Sensing to Infer Real- World Water Usage Events in the Home. Pervasive Computing, Springer (2011), 50–69.
most comprehensive labeled dataset of hot and cold water ever collected
37
bathroom sink
natural water usage
70
50
30
pres
sure
(psi)
70
50
30
pres
sure
(psi)
8AM2 minutes
38
kitchen sink kitchen sink
toilet
bathroom sink
natural water usage
70
50
30
pres
sure
(psi)
70
50
30
pres
sure
(psi)
8AM2 minutes
38
kitchen sink kitchen sink
toilet
bathroom sink
natural water usage
70
50
30
pres
sure
(psi)
70
50
30
pres
sure
(psi)
template matching: 98% 74%10 fold cross validation
35%
8AM2 minutes
minimal
38
need a more realistic approach
templates feature vectors
matching parametric model
minimize training
39
need a more realistic approach
templates feature vectors
matching parametric model
minimize training
39
70
50
30
pres
sure
(psi)
10 psi7.32 psi
15 Hz
200 ms
feature vectors: dense features
43
70
50
30
pres
sure
(psi)
10 psi7.32 psi
15 Hz200 ms
feature vectors: dense features
x
d1
44
70
50
30
pres
sure
(psi)
feature vectors: sparse featuresun
labe
led
inst
ance
s
x
d1
45
70
50
30
pres
sure
(psi)
feature vectors: sparse features
codebook
x1
x56
x132
x240
0
0...
0
0...
0
0...
0
0...
x
d1
46
70
50
30
pres
sure
(psi)
feature vectors: sparse features
codebook
x
d1
x
s1
70
50
30
pres
sure
(psi)
70
50
30
pres
sure
(psi)
x
d1
x
s1 x
s2
x
d2 x
d3
x
s3
x
d4
x
s4
x
d5
x
s5
x
d6
x
s6 x
s7
x
d7
feature vectors: sequence
templates
matching parametric model
feature vectors
Traditional MethodsKNNSVMDecision TreesCRFDBN (i.e., HMM)
Ensemble MethodsKNN-subspaceBagged Trees
Stacking MethodsTB+CRFSVM+CRFTB+DBN
51
minimal training set
0
10
20
30
40
50
60
70
80
90
100
NN KNN TM HMM KNN-sub SVM HMM-TM CRF TB SVM+CRF TB+CRF
valv
e le
vel a
ccur
acy
(%)
error bars=std err.
1-2 labels per valve
52
dens
e fe
atur
es
spar
se fe
atur
es
55%valve
fixture
64%
category78%
supervised results summary
53
fixture level confusions
Kitc
hen
Sin
k
Mas
ter B
athr
oom
Sin
k
Sec
onda
ry B
athr
oom
Sin
k
Sec
onda
ry B
athr
oom
Toi
let
Mas
ter B
athr
oom
Toi
let
dishwasher laundry
Mas
ter B
athr
oom
Bat
h/S
how
er
Mas
ter B
athr
oom
Bat
h/S
how
er
Was
hing
Mac
hine
Dis
hwas
her
54
accu
racy trusted
not trusted
how accurate should the system be?
how can we be sure the user trusts the system?
highly criticalnoticeable
Lim, B. and Dey, A. Investigating intelligibility for uncertain context-aware applications. Proceedings of the 13th international conference on ubiquitous computing, (2011), 415.
~80%
~99%
55
accu
racy trusted
not trusted
category 78%
~80%
~99%
85%
90%
80%
goalsminimal
55%valve
64%fixture
laundry dishwasher
noticeable
10% 8% 0%
56
0 0.5 1 1.5 2 2.5 3 3.5 446
48
50
52
54
56
58
60
hours
psi
time
psi
fill
fill
fill
fill
cycl
e
cycl
e
cycl
e57
0 0.5 1 1.5 2 2.5 3 3.5 446
48
50
52
54
56
58
60
hours
psi
finding the dishwasher
0 0.5 1 1.5 2 2.5 3 3.5 446
48
50
52
54
56
58
60
hours
psi
template
59
dishwasher
0 0.5 1 1.5 2 2.5 3 3.5 446
48
50
52
54
56
58
60
hours
psi
finding the dishwasher
0 0.5 1 1.5 2 2.5 3 3.5 446
48
50
52
54
56
58
60
hours
psi
template
X
n
�tn
59
dishwasher
0 0.5 1 1.5 2 2.5 3 3.5 446
48
50
52
54
56
58
60
hours
psi
finding the dishwasher
0 0.5 1 1.5 2 2.5 3 3.5 446
48
50
52
54
56
58
60
hours
psi
template
X
n
�tn
X
n
�pn
59
dishwasher
0 0.5 1 1.5 2 2.5 3 3.5 446
48
50
52
54
56
58
60
hours
psi
finding the dishwasher
0 0.5 1 1.5 2 2.5 3 3.5 446
48
50
52
54
56
58
60
hours
psi
template
pressure difference
time difference
X
n
�tn
X
n
�pn
59
dishwasher
0 0.5 1 1.5 2 2.5 3 3.5 446
48
50
52
54
56
58
60
hours
psi
finding the dishwasher
0 0.5 1 1.5 2 2.5 3 3.5 446
48
50
52
54
56
58
60
hours
psi
template
pressure difference
time difference
X
n
�tn
X
n
�pn
59
dishwasher
0 0.5 1 1.5 2 2.5 3 3.5 446
48
50
52
54
56
58
60
hours
psi
finding the dishwasher
pressure difference
time difference
X
n
�tn
X
n
�pn
59
dishwasher
0 0.5 1 1.5 2 2.5 3 3.5 446
48
50
52
54
56
58
60
hours
psi
finding the dishwasher
pressure difference
time difference
X
n
�tn
X
n
�pn
X
n
�tcyclelaundry machine
59
laundry
dishwasher
laundry
dishwasher
truepositives false alarms precision
73% 15 90%
75% 16 89%
43% 58 85%
39% 119 82%
75% of all showering
cate
gory
60
accu
racy trusted
not trusted
category 78%
~80%
~99%
85%
90%
80%
goalsminimal
55%valve
64%fixture
laundry dishwasher
noticeable
10% 8% 0%
61
leveraging unlabeled data
labeled unlabeled
classifier classifierfeature set 2
high confidence high confidence
agree?
feature set 1
self labeled
multi-view classification
62
training 48%49%
labeled unlabeled
multi-view classification
self labeled
TB
SVM+CRF
55%53%
feature split: hot sensor vs. cold sensor
88%90%TB
SVM+CRF
99%
dense features
sparse features
63
training
training
training
0 5 10 15 2046
48
50
52
54
56
58
60
time of day (hours)
psi
kitchen sink, hot master bathroom toilet
multi-view classification
64
semi-supervised learning
rule based classifier
0 0.5 1 1.5 2 2.5 3 3.5 446
48
50
52
54
56
58
60
hourspsi
self labels
expert knowledge
virtual evidence
65
A B
C Dvirtual evidence
1
semi-supervised learning virtual evidence
kitchen sink kitchen sinkP=1
bath sink bath sinkP=1
toilet toiletP=1
P=0.01otherwise
66
argmax
A,BP (A)P (B|A)P (C = c|A)P (D = d|B)P (ve|A,B)
semi-supervised learning virtual evidence
67
semi-supervised learning virtual evidence
68
semi-supervised learning virtual evidence
69
semi-supervised learning virtual evidence
70
semi-supervised learning virtual evidence
71
semi-supervised learning results
0
10
20
30
40
50
60
70
80
90
100
TM HMM SVM TB SVM+CRF TB+CRF HMM-VE-Co
valv
e le
vel a
ccur
acy
(%)
10 fold cross validation
72
129
lumped fixture level, I can see that the previous problems in not detecting the dishwasher, washing
machine, or showers are starting to be mitigated, but are not completely solved. There are still a large
number of confusions for the kitchen sink and “within bathroom” confusions. There are also a number of
confusions between the dishwasher and kitchen sink, as well as the master bathroom shower and
secondary bathroom shower.
Figure 8-17. The lumped fixture level confusions for Co-DBN-VE using the minimal set of training instances
Implication: Despite the progress that virtual evidence has achieved in recognizing and labeling
sparse classes, it is still not in the 80-90% range I set out to achieve. There are still too many temperature
confusions and confusions within the rooms where fixtures reside. I now investigate ways to leverage the
behavior of the co-training classifier, together with virtual evidence to get into the 80% range. This
includes adding a dimension from the home owner—selective journaling.
8.6 The human component: cooperative, sparse labeling Up to this point, I have primarily relied on a few selected labels from the homeowner during one day of
water usage. This, as explained, was to reduce the overhead of calibrating the system. However, there is
no need for these labels to come from the same day—I have selectively chosen sequence learning
methods that learn their state transition probabilities from a global model, not a specific home. In this
-51957
6.6
0.6
1.1
-519
4.2
56
1.7
0.8
6.4
4.6
-11507
31
0.8
82
1.5
9.1
4.3
14
1.5
2.6
4.2
2.0
6.1
1.1
8.8
-11528
1.5
28
2.0
81
10
9.1
2.0
15
1.0
3.0
22
6.1
6.7
8.5
2.0
6.5
-1044
0.6
33
3.3
1.0
0.8
0.5
0.7
17
-2419
5.0
39
1.0
1.5
0.6
0.8
24
4.4
0.7
-7580
2.5
6.8
1.1
11
6.5
65
2.4
5.4
0.8
3.1
7.9
-7818
2.5
6.0
0.9
7.8
5.7
14
3.0
68
1.0
1.6
3.5
1.5
3.9
-3397
1.6
9.5
4.8
4.3
1.4
83
0.8
1.4
17
-3446
1.9
3.1
3.5
1.9
91
2.5
0.8
11
17
-265
6.4
42
-512
2.4
60
-3240
4.3
1.0
3.4
4.8
7.9
0.6
3.7
7.5
1.6
86
1.6
8.4
1.5
-3234
1.7
0.7
4.5
1.7
5.7
1.0
6.8
0.6
3.0
3.1
1.9
85
6.4
2.2
6.1
-543
0.7
66
3.2
-547
1.1
38
14 -539
9.1
43
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Dishwasher close,1
open,2
KitchenSink close,3
open,4
M.BathroomShower close,5
open,6
M.BathroomSink close,7
open,8
M.BathroomToilet close,9
open,10
S.BathroomShower close,11
open,12
S.BathroomSink close,13
open,14
S.BathroomToilet open,15
WashingMachine close,16
open,17
dishwasher laundry
dishwasher
laundry
73
129
lumped fixture level, I can see that the previous problems in not detecting the dishwasher, washing
machine, or showers are starting to be mitigated, but are not completely solved. There are still a large
number of confusions for the kitchen sink and “within bathroom” confusions. There are also a number of
confusions between the dishwasher and kitchen sink, as well as the master bathroom shower and
secondary bathroom shower.
Figure 8-17. The lumped fixture level confusions for Co-DBN-VE using the minimal set of training instances
Implication: Despite the progress that virtual evidence has achieved in recognizing and labeling
sparse classes, it is still not in the 80-90% range I set out to achieve. There are still too many temperature
confusions and confusions within the rooms where fixtures reside. I now investigate ways to leverage the
behavior of the co-training classifier, together with virtual evidence to get into the 80% range. This
includes adding a dimension from the home owner—selective journaling.
8.6 The human component: cooperative, sparse labeling Up to this point, I have primarily relied on a few selected labels from the homeowner during one day of
water usage. This, as explained, was to reduce the overhead of calibrating the system. However, there is
no need for these labels to come from the same day—I have selectively chosen sequence learning
methods that learn their state transition probabilities from a global model, not a specific home. In this
-51957
6.6
0.6
1.1
-519
4.2
56
1.7
0.8
6.4
4.6
-11507
31
0.8
82
1.5
9.1
4.3
14
1.5
2.6
4.2
2.0
6.1
1.1
8.8
-11528
1.5
28
2.0
81
10
9.1
2.0
15
1.0
3.0
22
6.1
6.7
8.5
2.0
6.5
-1044
0.6
33
3.3
1.0
0.8
0.5
0.7
17
-2419
5.0
39
1.0
1.5
0.6
0.8
24
4.4
0.7
-7580
2.5
6.8
1.1
11
6.5
65
2.4
5.4
0.8
3.1
7.9
-7818
2.5
6.0
0.9
7.8
5.7
14
3.0
68
1.0
1.6
3.5
1.5
3.9
-3397
1.6
9.5
4.8
4.3
1.4
83
0.8
1.4
17
-3446
1.9
3.1
3.5
1.9
91
2.5
0.8
11
17
-265
6.4
42
-512
2.4
60
-3240
4.3
1.0
3.4
4.8
7.9
0.6
3.7
7.5
1.6
86
1.6
8.4
1.5
-3234
1.7
0.7
4.5
1.7
5.7
1.0
6.8
0.6
3.0
3.1
1.9
85
6.4
2.2
6.1
-543
0.7
66
3.2
-547
1.1
38
14 -539
9.1
43
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Dishwasher close,1
open,2
KitchenSink close,3
open,4
M.BathroomShower close,5
open,6
M.BathroomSink close,7
open,8
M.BathroomToilet close,9
open,10
S.BathroomShower close,11
open,12
S.BathroomSink close,13
open,14
S.BathroomToilet open,15
WashingMachine close,16
open,17
DW
Shower
Shower
CW
dishwasher laundry
dishwasher
laundry
73
semi-supervised learning leveraging the homeowner
which labels are needed most?
can we leverage multi-view models?
74
semi-supervised learning leveraging the homeowner
can we leverage multi-view models?
• select low confidence examples
• ask homeowner for label
75
semi-supervised learning leveraging the homeowner
can we leverage multi-view models?
• select low confidence examples
• ask homeowner for label
AT&T LTEAT&T LTE 5:23 PM
did you just use water?
YesNo
75
semi-supervised learning leveraging the homeowner
can we leverage multi-view models?
• select low confidence examples
• ask homeowner for label
AT&T LTEAT&T LTE 5:23 PM
did you just use water?
YesNo
75
AT&T LTEAT&T LTE 5:23 PM
Fixture Selection
Do Not Disturb
Select Notification Times
OFF
Master Toilet
Recently Used:
Half Bath Toilet
Dishwasher
Master Sink Select Temp.
Kitchen Sink Select Temp.
Master Shower Select Temp.
More
Half Bath Sink Select Temp.
simulating labels from homeowner
AT&T LTEAT&T LTE 5:23 PM
did you just use water?
YesNo • ask for two labels every other day
• one morning and one evening
• only from 8AM-9PM
• randomly ask for previous event
76
AT&T LTEAT&T LTE 5:23 PM
Fixture Selection
Do Not Disturb
Select Notification Times
OFF
Master Toilet
Recently Used:
Half Bath Toilet
Dishwasher
Master Sink Select Temp.
Kitchen Sink Select Temp.
Master Shower Select Temp.
More
Half Bath Sink Select Temp.
10 15 20 25 30 35 40 45
0.65
0.7
0.75
0.8
0.85Co−Labeling in H1
Number of Labels
Valv
e Le
vel A
ccur
acy
of C
oLab
el−H
MM
Co−LabelingRandom Labelingco-labelingrandom labeling
iteration 1
iteration 3
iteration 5
iteration 10
simulating labels from homeowner
co-labeling for H1m
inim
al tr
aini
ng s
ettotals
days 33 33 30 27 33 156
events 2374 3075 4754 2499 2578 14,960
events/day 71.9 93.2 158.5 92.6 78.1 95.9
compound 22.2% 21.8% 16.6% 32% 21.3% 21.9%
77
totals
days 33 33 30 27 33 156
events 2374 3075 4754 2499 2578 14,960
events/day 71.9 93.2 158.5 92.6 78.1 95.9
compound 22.2% 21.8% 16.6% 32% 21.3% 21.9%
totals
days 33 33 30 27 33 156
events 2374 3075 4754 2499 2578 14,960
events/day 71.9 93.2 158.5 92.6 78.1 95.9
compound 22.2% 21.8% 16.6% 32% 21.3% 21.9%
0 1 2 3 4 5 6 7 8 9 10 11 12 1360%
70%
80%
90%
100%
valve fixture category
error bars=std err.
co-label iteration
accu
racy
78
133
followed by the toilet. This advocates the importance of using the DBN-VE as a baseline classifier,
because co-labeling only marginally increases the diversity of class examples. Even so, a shower example
is typically asked for in the first two iterations, but washing machines and dishwashers are not asked for
until typically the fifth or sixth iteration. This is not a problem, however, because the rule based classifier
and DBN-VE are able to leverage prior knowledge in classifying these fixtures and appliances.
A final investigation of the confusions reveals that the system, after 10 iterations of co-labeling, is
highly accurate among each fixture, although temperature confusions still exist (Figure 8-20). The most
common confusion is the secondary bathroom shower for the master bathroom shower.
Figure 8-20. The final confusion matrix for the CoLabel-DBN algorithm
For comparison to the other algorithms, I also show the improvement in the across fixture accuracy at
the valve, lumped fixture, and fixture category level, shown in Figure 8-21.
-32974
8.5
1.5
-328
6.1
72
1.1
0.8
5.0
0.7
1.5
0.6
-7180
18
92
0.6
5.4
2.6
4.9
3.4
2.3
3.5
0.9
5.1
-7211
18
0.9
92
1.4
4.9
0.7
4.4
1.4
1.4
7.3
1.0
0.7
3.2
-72053
2.1 -1651
4.2
67
0.6
2.4
0.8
31
2.5
5.2
-5253
1.2
5.0
0.6
18
4.2
88
1.8
7.5
1.5
1.6
5.7
1.0
-5430
1.2
0.5
5.3
4.6
10
1.8
89
1.7
1.7
1.4
0.7
3.2
-2398
2.9
0.6
81
12
-2433
0.8
94
0.6
8.1
-133
6.7
68
-259
2.8
42
-1628
1.9
2.9
0.5
25
0.8
95
0.7 -1625
2.2
3.1
10
1.3
96
6.6 -27290
-316
1.1
0.7
59
18 -308
0.9
0.9
15
60
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Dishwasher close,1
open,2
KitchenSink close,3
open,4
M.BathroomShower close,5
open,6
M.BathroomSink close,7
open,8
M.BathroomToilet close,9
open,10
S.BathroomShower close,11
open,12
S.BathroomSink close,13
open,14
S.BathroomToilet open,15
WashingMachine close,16
open,17
dishwasher laundry
dishwasher
laundry 79
implications for homeownerweek one• homeowner installs system• 1-2 examples per fixture
• show sparse classes: dishwasher, shower, laundry
80
implications for homeownerweek one• homeowner installs system• 1-2 examples per fixture
• show sparse classes: dishwasher, shower, laundry
week two• 2-4 labels, every 2 days• fixture category: 85%
80
implications for homeownerweek one• homeowner installs system• 1-2 examples per fixture
• show sparse classes: dishwasher, shower, laundry
week two• 2-4 labels, every 2 days• fixture category: 85%
week three• 9-12 more examples• fixture: 82%
80
implications for homeownerweek one• homeowner installs system• 1-2 examples per fixture
• show sparse classes: dishwasher, shower, laundry
week two• 2-4 labels, every 2 days• fixture category: 85%
week three• 9-12 more examples• fixture: 82%
end of week three• fixture: 87%• valve: 80%
80
summary contributions
• comprehensive disaggregated dataset• multi-view classification
• expert knowledge • compressed sensing
• framework for virtual evidence in IMS• co-labeling with multi-view• idea: inception to industry ready
81
how can indirect sensing and machine learning be used to reduce our
environmental footprint?
disaggregated hot and cold water sensing with minimal calibration
semi-supervised training for infrastructure mediated sensing
UbiComp Lab
electrical engineering
computerscience and engineering
University of Washington
eclarson.com [email protected]
@ec_larson
Eric C. Larson
disaggregated hot and cold water sensing with minimal calibration
semi-supervised training for infrastructure mediated sensing
UbiComp Lab
electrical engineering
computerscience and engineering
University of Washington
eclarson.com [email protected]
@ec_larson
Eric C. Larson