GlimpseData: Towards Continuous Vision-Based Personal Analytics
GlimpseData: Towards Continuous Vision-Based Personal AnalyticsSeungyeop Han withRajalakshmi Nandakumar, Matthai Philipose, Arvind Krishnamurthy, and David Wetherall
University of WashingtonMicrosoft1
Personal Analytics: Measure, analyze, and share data about life2
2What?What you have eatenCalorie counterRemind medication
Where?Vision-based LocalizationRemind visitsIndoor-location based ads
SeungyeopWho?Whom you have met todayRemind peoples nameFoster social interactionCheck peoples mood Visual Data adds New Dimensions
Hamburger354 cal33Challenges of Vision-based AnalysisResourcesCycles, bandwidth, power are limitedVision Algorithms Privacy and securityUser interaction with applications
44Challenges of Vision-based AnalysisResourcesCycles, bandwidth, power are limitedVision Algorithms Privacy and securityUser interaction with applications
55Resource is the Problem670 mW avg
Phone700 mW avg150 gCloud.01 serverWWAN700mW10GB/moWiFi500mW5MbpsResource consumption vs budget for mobile imager/cloud-based classifier:100-300mW imager(10mW)700mW WWAN(70mW)675GB/mo WWAN data(5GB/mo)1 server/wearercompute(0.01 server/wearer)Not All Frames are InterestingInput Visual StreamIs the frame interesting?Low-Power Sensors
YESCan we use lower power sensors to filter out uninteresting frames?77GlimpseDataData collection and analysis framework to study continuous sensor-augmented visual data
Case study on predicting whether a frame contains faces88Requirements for Data Collection System
Inclusive wrt. sensorsWidely available platformUnobtrusive
99Smartphone as a Data-Collecting Front-end
1010Smartphone as a Data-Collecting Front-end
SensorsAudioCamera
Location
Thermal CameraAndroid applicationRunning as a service~5 video frame per secondSync with timestampsCollect all possible sensorsCustom-built 16x4 temp array40x15 FOV
1111Collected DatasetTotal 116 minutes over 7 days~1M Sensor readings~100k Thermal camera frames>30k RGB frames (~5% are face frames)
1212 [demo] VisualizationJS visualizer running on browsers.
1313GlimpseDataData collection and analysis framework to study continuous sensor-augmented visual data
Case study on predicting whether a frame contains faces1414Case Study: Filtering Non-Face Frames with Low Data-rate SensorsBuild classifier determining non-face frames.Test with OpenCV face detector, and manually inspect frames.
Goal: filter out as many frames as possible while not missing frames with faces1515Not All Frames Have FacesInput Visual StreamDoes the frame have faces?Low-power sensor data
YESAccelerometer,Gyroscope, Light,Sound, Location,Thermal SensorFilterCan we determine if a frame is unlikely to have a face before running face detector?less than 5% frames contain faces in the data1616Thresholding on Single Sensor17Applying a threshold on each sensor value is promising
17Thresholding on Single Sensor18Applying a threshold on each sensor value is promising
Note: Face detector is resilient to noise18
Joint Classifier19With a logistic regressor using all sensors, it can filter out 60% of frames while missing only 10% frames with faces.19DiscussionHow to contribute data for the research communityNeed data with better quality (e.g., synchronization is an issue)Privacy concerns with sharing data
Beyond filteringDesign APIs for multiple applicationsNeed better classification methods20SummaryVisual data is extremely rich and could be useful and continuous analysis may be feasible.But, collecting large, rich dataset is challenging.
We presented GlimpseData, a data collection and analysis framework. A case study showed promising results for filtering uninteresting frames and feasibility of such study.
2121Q&Ahttps://github.com/UWNetworksLab/GlimpseData
2222