![Page 1: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/1.jpg)
Multimedia on the Mountaintop: Using Public Snow Images to Improve Water
Systems Operation
A. Castelletti, R. Fedorov, P. Fraternali, M. Giuliani Politecnico di Milano, Italy
ACM MM 2016, AmsterdamBNI session
![Page 2: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/2.jpg)
The (hopefully brave new) idea
• There is a lot of multimedia content out there, produced by– People– Ground sensors
• There are many environmental problems that lack affordable and accessible input data
• Question: is public web visual content good enough to help in such environmental problems?
![Page 3: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/3.jpg)
Observing the earth
• Not everything can be done from above• There is not a single satellite product good for all• (Useful) satellite products are costly• Clouds may be a problem
![Page 4: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/4.jpg)
The grand challenge: water scarcity• Climate change, urban concentration and agriculture
put water resources under stress• Predicting future availability is key• When you have mountains, water is stored as snow
UK_WATER SUPPLY UTILITY15 million customers2.6 Gl/day drinking water3 billion $ revenue (2013-14)
![Page 5: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/5.jpg)
The contentInput• User generated
– 700.000 Flickr images crawled so far within 300x160 km
• Sensor generated– 2000 webcams queried every
minute (10 – to 1500 images per web cam per day)
– More than 10M images crawled so far
Output• Virtual Snow Indexes:
numerical time series that are a proxy of the quantity of water stored in the snow pack (Snow Water Equivalent – SWE)
![Page 6: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/6.jpg)
The multimedia pipelines
• Differences– Web cam images have high temporal density, UG images
have broader spatial coverage– UG photos searched by keywords may be irrelevant,
webcam images always portrait mountains– UG photo mountain classifier already discards bad
weather images
![Page 7: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/7.jpg)
UG Image relevance
• 7000 images randomly sampled and used for a crowdsourcing experiment: “Do you see a mountain in this picture?”
• Classifier trained (94% precision, 96.3% recall)
![Page 8: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/8.jpg)
Webcam image enhancement
Remove/attenuate:• Variability of illumination• Shadows• People & irrelevant objects
Daily median image
![Page 9: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/9.jpg)
Mountain peak identification
orginal image edge maps
skyline estimationDEM generated virtual panoram
VCC best matching
![Page 10: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/10.jpg)
Snow mask extractionSnow classification at the pixel level
Snow mask extraction
![Page 11: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/11.jpg)
Snow Virtual Indexes
![Page 12: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/12.jpg)
The case study• Regulation of mountain inflow dependent lakes
Lake Como Catchment area Lake Como 4500 km2
Reservoirs Lake Como 247 Mm3 Alpine HP 545 Mm3
StakeholdersFarmers:
irrigated area 1400 km2
Floods:lake and downstream
….
![Page 13: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/13.jpg)
Local folklore
![Page 14: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/14.jpg)
Formalization: 2 objectives optimization
• Decide the daily lake outflow ( lake level)
• So to– Maximize water for
downstream irrigation– Minimize # of flood days
• Respecting– Minimum outflow
requirement for ecological preservation of effluents
• Based on– Policy input (X)
• Regulator's policies– Baseline: regulator only considers lake
level and day of year– Upper bound: regulator knows the
water that will be available (lake inflow) in the future
– P_x: regulator knows partial information (x) on the water that will be available (lake inflow) in the future
• What is X?– P1: Official snow water equivalent
data estimated from Region Lombardy– P2: virtual snow indexes from nearby
mountain images– P3: official SWE data + virtual snow
indexes
PS: Upper bound policy can be calculated retrospectively for the past, where you know how much water you actually got day by day
![Page 15: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/15.jpg)
Assessment method
Select information based on its
expected value(Iterative
Input Selection)
Design control policy based on selected input
information
Quantify performance of policy + selected
information
Quantify value of perfect
information Expected Value of Perfect Information (EVPI)
Inflow data series Outflow data series
Baseline policy Upper
boundpolicy
Input data
series(exogenous
variables)
Most Valuable Information
(X)
X_informed control policy(P_x)
J(P_x)Performance of
P_x
Performance metricsHyper Volume Indicator
(HV)
Performance improvement over baseline(ΔHV)
![Page 16: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/16.jpg)
Assessment results
![Page 17: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/17.jpg)
Thank you & … see you soon in the PlayStore
![Page 18: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/18.jpg)
Content processing pipeline• Photo contains/does not contain mountain landscape
binary classifier– SVM with Dense SIFT, Spatial Histograms. 7k annotated
images (majority of 3 votes). 95.1% Accuracy on balanced dataset.
• Peak identification / Photo orientation estimation– Ad-hoc algorithm with edge extraction and vector cross-
correlation. 160 images manually aligned w.r.t. Digital Elevation Model. 75-81% of images correctly aligned (depending on weather conditions).
• Pixel-wise snow/non snow classifierRandom Forest, trained/evaluated on 60 manually segmented images (single annotator) for a total of 7M of labeled pixels. 91% accuracy.
![Page 19: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/19.jpg)
Iterative input selection
Select information based on its
expected value(Iterative
Input Selection)
Design control policy based on selected input
information
Quantify performance of policy + selected
information
Quantify value of perfect
information Expected Value of Perfect Information (EVPI)
Inflow data series Outflow data series
Baseline policy Upper
boundpolicy
Input data
series(exogenous
variables)
Most Valuable Information
(X)
X_informed control policy(P_x)
J(P_x)Performance of
P_x
Performance metricsHyper Volume Indicator
(HV)
Performance improvement over baseline(ΔHV)
D=distance metric
![Page 20: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/20.jpg)
Policy search
Select information based on its
expected value(Iterative
Input Selection)
Design control policy based on selected input
information
Quantify performance of policy + selected
information
Quantify value of perfect
information Expected Value of Perfect Information (EVPI)
Inflow data series Outflow data series
Baseline policy Upper
boundpolicy
Input data
series(exogenous
variables)
Most Valuable Information
(X)
X_informed control policy(P_x)
J(P_x)Performance of
P_x
Performance metricsHyper Volume Indicator
(HV)
Performance improvement over baseline(ΔHV)
![Page 21: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/21.jpg)
![Page 22: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/22.jpg)
Good decisions matter
WATER DEFICIT
FLOOD THRESHOLD
EFFECT OF REGULATION
![Page 23: Multimedia on the mountaintop: presentation at ACM MM2016](https://reader035.vdocuments.us/reader035/viewer/2022062823/587301a31a28ab99088b46ef/html5/thumbnails/23.jpg)
For more info• A. Castelletti, R. Fedorov, P. Fraternali, M. Giuliani:
[email protected]• http://snowwatch.polimi.it/