Download - Systems to Capture Everything: Beyond cameras and desktops Gordon Bell, Jim Gemmell, Roger Lueder
Systems to Capture Everything:
Beyond cameras and desktopswww.MyLifeBits.com
Gordon Bell, Jim Gemmell, Roger LuederGordon Bell, Jim Gemmell, Roger Lueder
Outline
MyLifeBits aka MemexMyLifeBits aka MemexHow has the project evolved? How has the project evolved? How do we use MyLifeBits? How do we use MyLifeBits? How is it built?How is it built?Shape of the database? Shape of the database?
CARPE- Continuous archiving and CARPE- Continuous archiving and recording of personal experiencerecording of personal experienceWhat is the vision?What is the vision?Relevance for devices and software?Relevance for devices and software?
I am data
History: Telepresence
Tele-presentations
Tele-meetings
Ambience and Presence:Being there while being here
Dining at home on the “Orient Express”
History: The remote worker re-discovers the PERSONAL
computer
Oct 1998
Can we scan your books and put them online?
Raj Reddy
Sure! Don’t worry about
copyright stuff. Microsoft has lots of lawyers
1999 – Scanning starts in earnest“we” start to scan,
put content intofolders & files
My docs and archive
Self....Biographical
X- Employer
EmployerEmployer
X-EmployerProjectProject
ProjectProject
Employer
Library/file cab
Library/file cab
Library/file cab
Library/file cab
Library/file cab
Library/file cab
Active Employer
Library/file cab
Library/file cabLibrary/file cab
<1980s Library/file cabLibrary/file cab
Project
BusinessInvests, family $s, & Legal
Personal, including Medical
Library/file cab
Now that it’s in CyberspaceHow do you remember the 20,000+ file names?
Or in which of 1500 folders they live?
What’s about a tool for finding stuff?
Jan 2001 CACM “A Personal Digital Store”
16 GB; +2/yr16 GB; +2/yr A good place to A good place to
stopstop Began search for Began search for
search engines, search engines, especially for email.especially for email.
Jim suggests that Jim suggests that we build a system we build a system that would be that would be easier to use and easier to use and have many more have many more capabilities.capabilities.
2001 Capture goes beyond paper
Jim, I don’t need no
stinkin’ database!
Gordon, You should be using a database.
Re-discovery of MemexAs We May Think, Vannevar Bush, 1945
““A memex is a device in which an individual stores all A memex is a device in which an individual stores all his books, records, and communications, and which his books, records, and communications, and which is mechanized so that it may be consulted with is mechanized so that it may be consulted with exceeding speed and flexibility”exceeding speed and flexibility”
Full-text search, text & audio annotations, and Full-text search, text & audio annotations, and hyperlinkshyperlinks
Even more capture
Telephone calls, more video, all web pages Telephone calls, more video, all web pages visited, keyboard and mouse usage logging, visited, keyboard and mouse usage logging, radio, TV…radio, TV…
2003 - SenseCam
Feb 2005
Epiphany!Epiphany!
Memex is a database Memex is a database & &
personal TP systempersonal TP system
Demo Clips & Screens
747 Screen…
Vue de jour
Timeline
Pivoting: contact> call> t> web page
GPS Photo location
Reports
The Stew family tree
Copyright Mark Stewart, 2004
Vibe report
Quindi Meeting Capture
SenseCam
SenseCam around Cambridge
MyLifeBits Software
Everything goes in a database
MyLIfeBits need all the features of a databaseMyLIfeBits need all the features of a database(Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, (Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, Replication)Replication)
If we didn’t use one, we’ll eventually create one!If we didn’t use one, we’ll eventually create one! Files as blobs; sync with file system for legacy appsFiles as blobs; sync with file system for legacy apps We are part of Jim Gray’s Bay Area Research LabWe are part of Jim Gray’s Bay Area Research Lab
SQLSQL
MyLifeBits Software
MyLifeBits store
database
Voice Voice annotation annotation tooltool
Telephone Telephone capture toolcapture tool
TV capture TV capture tooltool
TV EPG TV EPG download tooldownload tool
Radio Radio capture capture & EPG& EPG
PocketPC PocketPC transfer transfer tooltool
PocketRadio PocketRadio playerplayer
Import filesImport files
MyLifeBits MyLifeBits ShellShell
Browser Browser tooltool
InternetInternet
IM captureIM capture
GPS import & GPS import & Map displayMap display
SenseCamSenseCam
Screen saverScreen saver
Text Text annotation annotation tooltool
MAPI MAPI interfaceinterface
Legacy Legacy email clientemail client
OutlookOutlookinterfaceinterface
files
Legacy Legacy applicationsapplications
VIBEVIBElogginglogging
RoomRoomCaptureCapture
IMG_Images2
PK,FK1,I1,I2,I3 item_id
Width HeightI1 Date Taken Camera Make Camera ModelI2 LatitudeI3 Longitude Elevation
Outlook_CalendarItems2
PK,FK1 item_id
Subject Start End Description Location Creation Time Modified
Common ground with WinFS:Items, Links & Meta-data
AnnotatesAnnotates
Caller in Phone CallCaller in Phone Call
Photo of EventPhoto of Event TAPI_PhoneCalls2
PK,FK1,I1 item_id
Phone Call Type CID CID Name CID #I1 Begin End Seconds Connected Ended Roaming Trimmed Recorded Transcript
PhotoFinder - Shneiderman and Kang
The Shape & Size of Gordon’s LifeBits
MyLifeBits10/31/2005
242K items 110 GB
by number of Items.
MyLifeBits10/31/05
110 GB242 K items
By Size (GB)
Bell Growth: 1GB/month
=1.1 TB/lifetime
Size (MB) by TypeSize (MB) by Type
1
10
100
1000
10000
1895 1905 1915 1925 1935 1945 1955 1965 1975 1985 1995 2005
Year Mpix Manufacturer1997 .25 Ricoh1999 1 Kodak2001 2 Canon2002 3 Sony2003 4 Sony2005 5 Panasonic
Year Mpix Manufacturer1997 .25 Ricoh1999 1 Kodak2001 2 Canon2002 3 Sony2003 4 Sony2005 5 Panasonic
15,000 photos
Monthly & Lifetime Storage Use
Item Daily number
Total* MB|GB
Month|Life
1 MB Books|reports 0.1 3
5KB Emails 100 13
100 KB Image scans 5 13
0.4 MB Photos 10 100
75 KB Web pages|docs 100 188
100 MB Music 0.1 250
1 KB/s Listened audio, speech 40,000 1,000
50 KB Daily photos 1,000 1,250
2 GB/hr TV 4 200,000
Observations about use(rs)
1.1. Cell phone sized device (CPSD) will be the platform!Cell phone sized device (CPSD) will be the platform!2.2. On Applications… think about CPSD as the platform and contextOn Applications… think about CPSD as the platform and context
Search is the “killer app” pretty much as Bush described. Search is the “killer app” pretty much as Bush described. Screen savers “memory refreshers” also provide ambienceScreen savers “memory refreshers” also provide ambience Where did my day to?Where did my day to?
3.3. Users are unwilling to spend time managing their computers or data. Users are unwilling to spend time managing their computers or data. Meta-data, classification, etc. must be automaticMeta-data, classification, etc. must be automatic User-input meta-data e.g. Dublin Core – naïve’ Librarian’s dream.User-input meta-data e.g. Dublin Core – naïve’ Librarian’s dream. We have nice scheme for classification using facets. It requires work.We have nice scheme for classification using facets. It requires work.
4.4. Time is the most important meta-data. Photos: place (GPS), subject.Time is the most important meta-data. Photos: place (GPS), subject.5.5. Folders are a good and bad idea. Folders are a good and bad idea.
Most users don’t know what they are or how they workMost users don’t know what they are or how they work If used, over time, they become useless: too many, miss-file, etc.If used, over time, they become useless: too many, miss-file, etc.
6.6. User User shouldshould put “every” information fragment into the system. e.g., to put “every” information fragment into the system. e.g., to dos, call backs, business cards numbers, attention events. It pays. dos, call backs, business cards numbers, attention events. It pays.
7.7. Same information in multiple places always becomes obsolete.Same information in multiple places always becomes obsolete.
Capturing Everything:
Phone calls in context of cell phone as a Phone calls in context of cell phone as a platform for communication and captureplatform for communication and capture
Formal Meetings Formal Meetings RoomsRoomsEverything in daily lifeEverything in daily lifePersonal health and medical monitoringPersonal health and medical monitoringMemex for scientists and engineersMemex for scientists and engineers
BodyMedia Output
Real time health monitoringPolysomnogram for sleep apnea.
Microsoft Research SensCam II
Sensors:Sensors:VGA camera w/ wide-angle lensVGA camera w/ wide-angle lens light level in R,G,B and whitelight level in R,G,B and whiteambient temperatureambient temperaturepassive infrared for person detectionpassive infrared for person detectionaccelerometersaccelerometers three, programmable buttons, LEDs, sounderthree, programmable buttons, LEDs, sounderaudio level & audio recordingaudio level & audio recordingUSB 2 and SD memory. 1-2 K photos/dayUSB 2 and SD memory. 1-2 K photos/dayNot GPSNot GPS
SenseCam University Grant ProgramMSFT supplies money, software, SenseCams
Memex vision: Notebook for engineers & scientistsMemex vision: Notebook for engineers & scientists Medical & health: observations & memory recall, Medical & health: observations & memory recall,
including diet and exerciseincluding diet and exercise Education: How do people learn? Education: How do people learn?
Help me learn/remember!Help me learn/remember! Tourist e.g. museum experienceTourist e.g. museum experience PlumbingPlumbing
SecuritySecurity Filtering many images, voice & location annotationFiltering many images, voice & location annotation
More real time experience capture
Real time medical & health monitoringReal time medical & health monitoringMIT. Deb Roy home capture to understan MIT. Deb Roy home capture to understan
how his children learnhow his children learnU. of Tokyo. Ubiquitous homeU. of Tokyo. Ubiquitous homeColumbia U. Voice & sound record & profileColumbia U. Voice & sound record & profileMIT. iDat. Electronic lab that records MIT. iDat. Electronic lab that records
everything into your notebookeverything into your notebook
Experience Retrieval in a Ubiquitous Home(chamds, byon, yamasaki, aizawa)@hal.k.u-tokyo.ac.jp
Experience Retrieval in a Ubiquitious Home
MIT iDAT Project aka notebook
Samsung challenge Going beyond plain old photography and videographyGoing beyond plain old photography and videography
Print, view, and file in scrapbook or shoeboxPrint, view, and file in scrapbook or shoebox Digitized bits offers worldwide sharing and easy sharingDigitized bits offers worldwide sharing and easy sharing Screensaver is useful, but is it a killer app?Screensaver is useful, but is it a killer app?
The cell phone sized device (CPSD)… one deviceThe cell phone sized device (CPSD)… one device Next generation platformNext generation platform Phones and messaging e.g. sms, mail, web, iM, bloggingPhones and messaging e.g. sms, mail, web, iM, blogging Audio, photo, video record and viewing (incl. broadcast)Audio, photo, video record and viewing (incl. broadcast) Within 5 years and with supplemental devices, will take on Within 5 years and with supplemental devices, will take on
the PCthe PC
Capture, storage, retrieval, and displayChallenge putting them together
Capture ….Capture …. Cell phone sized devices (CPSD). The “killer app”!!Cell phone sized devices (CPSD). The “killer app”!! Consumer… photo, video, audio… experienceConsumer… photo, video, audio… experience ProfessionalProfessional
StorageStorage CaptureCapture ArchivalArchival
Retrieval = f(use). Archive… ambienceRetrieval = f(use). Archive… ambience DisplayDisplay
Personal: Cell phonePersonal: Cell phone PCPC WallWall
www.MyLifeBits.com
BONUS SLIDES
Challenges Data-typesData-types
Quantity expanding i.e. info explosionQuantity expanding i.e. info explosion New capabilities e.g. real time create new data-typesNew capabilities e.g. real time create new data-types Meta-data to increase value & provide pivots Meta-data to increase value & provide pivots
Going beyond a PC to a distributed environmentGoing beyond a PC to a distributed environment
Network environment, including media centerNetwork environment, including media center Into the cloud. Especially important for social aspectsInto the cloud. Especially important for social aspects Periphery… smart buildings, objects, Periphery… smart buildings, objects, Backup, migration, and caching for beyond a TerabyteBackup, migration, and caching for beyond a Terabyte Expanding network: PC > LANs > web > p2p(eer)Expanding network: PC > LANs > web > p2p(eer)
Schema sharing among disparate systemsSchema sharing among disparate systems CARPE (real time data capture)CARPE (real time data capture)
Rooms, phone calls, SenseCam, Health transducers, etc.Rooms, phone calls, SenseCam, Health transducers, etc. Security, privacy, forgetfulness, deniability, etc.Security, privacy, forgetfulness, deniability, etc.
More challenges
Dear Appy: Monitoring and automatic migration of files that Dear Appy: Monitoring and automatic migration of files that are unlikely to be understood on future platforms as well as are unlikely to be understood on future platforms as well as platform migration.platform migration.
Get What I Need: GWIN…Endless, but evolutionary Get What I Need: GWIN…Endless, but evolutionary improvements in search: misspellings, stemming synonyms improvements in search: misspellings, stemming synonyms
Endless frontier of schema and extensions to them for new Endless frontier of schema and extensions to them for new applications e.g. making org charts, family relationships.applications e.g. making org charts, family relationships.
CARPE… a whole new game!CARPE… a whole new game! Versioning is essentialVersioning is essential Scaling.. We don’t know what happens at a TerabyteScaling.. We don’t know what happens at a Terabyte What can, should be, or will be in the cloud? Books… videosWhat can, should be, or will be in the cloud? Books… videos Will we be allowed to use such systems? Copyright laws vary: Will we be allowed to use such systems? Copyright laws vary:
E.g. ripping CDs, copy of anything, photos, conversationsE.g. ripping CDs, copy of anything, photos, conversations
The “dear appy” problemDear Appy, Dear Appy,
How committed are you?How committed are you?Please come back to me.Please come back to me.Forever yours truly, Forever yours truly, Lost and forgotten dataLost and forgotten data
Who’s responsible?Who’s responsible?Media or 8 track cassette, 8” floppyMedia or 8 track cassette, 8” floppyEvolving platform, file, and databaseEvolving platform, file, and databaseEvolving, incompatible standards & formats for Evolving, incompatible standards & formats for legacy data that disregard ancestorslegacy data that disregard ancestorsEvolving and/or disappearing appsEvolving and/or disappearing apps
Is Cyberspace a safe store?
Don’t your physical records e.g. paper last forever?
What about information on your CDs, tapes, hard drives, solid state devices?
Automatic classification problem XML on bills and imported content… transactionsXML on bills and imported content… transactions We need to download classifications rather than We need to download classifications rather than
build thembuild them Definitions & synonyms should help find what I wantDefinitions & synonyms should help find what I want
Today it is too expensive to manually classify Today it is too expensive to manually classify scanned paper. E.g. “right time” meta-data is critical!scanned paper. E.g. “right time” meta-data is critical!
We hope “the system” can classify papers and other We hope “the system” can classify papers and other documents e.g. bills. Ideally, build Dublin Coredocuments e.g. bills. Ideally, build Dublin Core
In 10 years we need all documents to appear In 10 years we need all documents to appear electronically & classified electronically & classified with a little help from mewith a little help from me