ubiquitous and mobile computing cs 528: tagsenseemmanuel/courses/cs528/s16/...tagsenseleverages this...
TRANSCRIPT
![Page 1: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/1.jpg)
Ubiquitous and Mobile Computing CS 528: TagSense
John Breen and Neha Mahajan
Computer Science Dept.Worcester Polytechnic Institute (WPI)
![Page 2: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/2.jpg)
What is TagSense?
TagSense is a first step towards automatic Image Tagging in order to speed up future
image retrieval.
![Page 3: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/3.jpg)
The Problem
Let’s Say You Have a Few Pictures…
![Page 4: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/4.jpg)
The Problem: Reality
Reality is slightly different…
![Page 5: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/5.jpg)
Why TagSense?
![Page 6: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/6.jpg)
How it Works
![Page 7: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/7.jpg)
How it Works
● Wifi Ad‐hoc Network Created
● All Phones Record Sensing Data
● Send Data to Camera Man to Process
![Page 8: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/8.jpg)
How it Works
??
![Page 9: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/9.jpg)
How to get “Who”
We only want information on people in the photo…
![Page 10: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/10.jpg)
How to get “Who”
● Accelerometer Based Motion
● Complementary Compass
● Correlating Picture and Accelerometers
![Page 11: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/11.jpg)
Accelerometer Based Motion
![Page 12: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/12.jpg)
Complementary Compass
![Page 13: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/13.jpg)
Correlating Picture and Accelerometers
![Page 14: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/14.jpg)
How to get “Who”
![Page 15: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/15.jpg)
Performance Evaluation
To evaluate TagSense, a real life experiment is conducted with 8 Google Nexus One phones, in which one phone is used as a camera while other phones are carried in pocket. When a picture is clicked, the camera triggers and gathers sensor data over the WiFi ad‐hoc mode.
The evaluation is done to answer the following questions:
● How well does TagSense tag people compared to approaches based on face recognition.
● How does human behaviour in different scenarios affect the individual tagging methods(posing, compass, motion) employed by TagSense.
● How well can TagSense recognize activities and context.
![Page 16: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/16.jpg)
Tagging People
![Page 17: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/17.jpg)
Tagging People
![Page 18: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/18.jpg)
Tagging People ‐Metrics
The goal of a tagging scheme is to achieve high precision, high recall, and low fall‐out.
![Page 19: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/19.jpg)
Performance of different TagSense methods under different scenarios.
![Page 20: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/20.jpg)
Performance comparison of TagSense with iPhoto and Picasa for name based image search.
![Page 21: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/21.jpg)
Tagging Activities and Context
![Page 22: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/22.jpg)
Tag Based Image Search
![Page 23: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/23.jpg)
Limitations of TagSense
● TagSense vocabulary of tags is quite limited.
● TagSense does not generate captions.
● TagSense cannot tag pictures taken in the past.
● TagSense requires users to input a group password at the beginning of a photo session.
● TagSense methods for tagging people are complex.
![Page 24: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/24.jpg)
Future of TagSense
● Smartphones are becoming context‐aware with personal sensing.
● Smartphones may have directional antennas.
● The granularity of localization will approach a foot.
● Smartphones are replacing point and shoot cameras.
![Page 25: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/25.jpg)
Conclusion●Mobile phones are becoming inseparable from humans and replacing traditional cameras. TagSense leverages this trend to automatically tag pictures with people and their activities.
● TagSense uses three different methods based on posing, compass and movement to identify people in a picture.
● TagSense has lower precision and comparable fall‐out but higher recall as compared iPhoto/Picasa.
![Page 26: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/26.jpg)
ANY QUESTIONS?
![Page 27: Ubiquitous and Mobile Computing CS 528: TagSenseemmanuel/courses/cs528/S16/...TagSenseleverages this trend to automatically tag pictures with people and their activities. TagSenseuses](https://reader034.vdocuments.us/reader034/viewer/2022042300/5ecb2e07135db918ed05c18f/html5/thumbnails/27.jpg)
References
● http://www.memorybook.com/images/landingPgs/design‐programs/convPileOfPics.png
● https://tctechcrunch2011.files.wordpress.com/2013/02/photo‐pile‐erik‐kessels2.jpg
● http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.224.9848&rep=rep1&type=pdf
● http://www.sigmobile.org/mobisys/2011/slides/tagsense.pdf● Qin, Chuan, et al. "Tagsense: a smartphone‐based approach
to automatic image tagging." Proceedings of the 9th international conference on Mobile systems, applications, and services. ACM, 2011.