11

MyLifeBits: Attempting to realize the Memex Vision

Jim Gemmell & Roger Lueder Jim Gemmell & Roger Lueder

Gordon BellGordon Bell

http://research.microsoft.com/barc/MediaPresence/MyLifeBits.aspxhttp://research.microsoft.com/barc/MediaPresence/MyLifeBits.aspx

22

Outline … MyLifeBits

Background…fulfilling the Memex visionBackground…fulfilling the Memex visionCyberizing everythingCyberizing everythingFile to database transitionFile to database transitionUse…beyond searchUse…beyond searchLong-term agenda and outlookLong-term agenda and outlook

33

MemexPosited by Vannevar Bush in “As We May Think”

The Atlantic Monthly, July 1945

““A memex is a device in which an individual stores A memex is a device in which an individual stores all his books, records, and communications, and all his books, records, and communications, and which is mechanized so that it may be consulted which is mechanized so that it may be consulted with exceeding speed and flexibility”with exceeding speed and flexibility”

Supports: Annotations, links between documents, Supports: Annotations, links between documents, and “trails” through the documentsand “trails” through the documents

““yet if the user inserted 5000 pages of material a day yet if the user inserted 5000 pages of material a day it would take him hundreds of years to fill the it would take him hundreds of years to fill the repository, so that he can be profligate and enter repository, so that he can be profligate and enter material freely”material freely”

44

Sketch of memex

Bush’s camera on the head

66

Capturing what you see

88

"The PC is going to be the place where you store the information and really the center of control“ Billg

1/7/2001MyLifeBits is a project to “cyberize” everything!MyLifeBits is a project to “cyberize” everything!

What? Recall of all articles, books, CDs, photos, What? Recall of all articles, books, CDs, photos, video, communication (e.g. mail, phone), video, communication (e.g. mail, phone), meetings,and web meetings,and web Why? …Why? …“because we can”“because we can”

Office: communicate, store, & workOffice: communicate, store, & work Home & Media Center: ambiance &entertainmentHome & Media Center: ambiance &entertainment Immortality for progeny. Memory aidsImmortality for progeny. Memory aids

Goal: understand the 1 TByte PC for Lonfor Longhorn Goal: understand the 1 TByte PC for Lonfor Longhorn need, utility, cost, feasibility and tools.need, utility, cost, feasibility and tools.

9

LifeLog: A potential research program

LifeLog:

A (sub)system that captures, stores, and makes accessible the flow of one person’s experience in and interactions with the world

LifeLog Thrust:

Capture the “story” of a human Living Content Ontology (format)

The End of the Line…

Biographies

Sagas Family Bibles

Home Movies

Photo Albums Videos

Cave Paintings

BlogsLifeLog

1111

The guinea pig Gordon Bell is digitizing his lifeGordon Bell is digitizing his life Has now scanned virtually all:Has now scanned virtually all:

Books written (and read when possible)Books written (and read when possible) Personal documents (correspondence including memos and email, Personal documents (correspondence including memos and email,

bills, legal documents, papers written, …)bills, legal documents, papers written, …) PhotosPhotos Posters, paintings, photo of things (artifacts, …medals, plaques)Posters, paintings, photo of things (artifacts, …medals, plaques) Home movies and videosHome movies and videos CD collectionCD collection And, of course, all PC filesAnd, of course, all PC files

Now recording: phone, radio, TV (movies), web pages… Now recording: phone, radio, TV (movies), web pages… conversations and meetings to comeconversations and meetings to come

Paperless throughout 2002. 12” scanned, 12’ discardedPaperless throughout 2002. 12” scanned, 12’ discarded.. Only 30 GB!!!Only 30 GB!!!

1212

I am data

1313

Capture and encoding

1414

Quindi conference capture

1515

I mean everything

1919

gbell wag: 67 yr, 25Kday life

1

1 0

1 0 0

1 , 0 0 0

1 0 , 0 0 0

1 0 0 , 0 0 0

1 , 0 0 0 , 0 0 0

1 0 0 -5 K B

M s g s

1 0 0 -5 0 K B

p a g e s

5 -1 0 0 K B

T i f s

0 . 1 -1 M B

B o o k s

1 0 -4 0 0 K Bj p e g s

4 0 K s1 K B p ss o u n d

0 . 1 -1 0 0 M Bs o n g s

1 -1 0 G B

V i d e o s

L i f e t i m e s t o r a g e ( G B )

2020

MyLifeBits organization: time and space

Timeline/Context(space)

Personal(some $s)

GB Co.(angel, etc.)

ProfessionalACM, etc., …@Microsoft.com,

New co’s.

Archival (time) Working

2121

MyLifeBits: Some Lives(t) PersonalPersonal

Parents, children, grandkidsParents, children, grandkids CGB himselfCGB himself GKBGKB Close friendsClose friends

GB $sGB $s Personal incl. several legal Personal incl. several legal

structuresstructures Properties: autos, real estate,Properties: autos, real estate, Investments & contractsInvestments & contracts

Past prof. companies/organiz’nsPast prof. companies/organiz’ns DECDEC Carnegie-Mellon U.Carnegie-Mellon U. DEC, NSF, Encore, Ardent, DEC, NSF, Encore, Ardent,

Me Inc., Me Inc.,

CGB@ MicrosoftCGB@ Microsoft MLBMLB ClustersClusters TelepresenceTelepresence WWW presenceWWW presence

Computer History MuseumComputer History Museum BOD memberBOD member Fund-raisingFund-raising CyberMuseumCyberMuseum

Startups & boardsStartups & boards Bell-Mason DirectorBell-Mason Director Diamond & Vanguard Brds.Diamond & Vanguard Brds.

23

Personal LifeLog Applications

Conservator

Baby Book

Companion Caretaker

Babysitter

AdvisorMentorTutor

Autobiography

Photo Album

PersonalAssistant

Diary/Journal

Biography

Financial Manager

Medical Manager

ExecutorObituary

Others Self

Assistant for Elderly

Application controlled by:

Oth

ers

Sel

fA

pp

lica

tio

n u

sed

by:

Personal Proxy

Parole Officer

Pers Flight Recorder

Meeting Prep

Captain’s Log

Trustee

2525

MyLifeBits is:

Memex and more (audio and video)Memex and more (audio and video) Universal store for all personal stuffUniversal store for all personal stuff Guiding principles for the system:Guiding principles for the system:

1.1. Full text search & Full text search & collectionscollections (> than hierarchy) (> than hierarchy)

2.2. Visualizations for search, display, insightVisualizations for search, display, insight

3.3. Annotations and links add value and essentialAnnotations and links add value and essential Increase search ability and value of information.Increase search ability and value of information. So make many kinds and them easy to create!So make many kinds and them easy to create! Stories are the ultimate annotationStories are the ultimate annotation

4.4. Keep the links when you author: “transclusion”Keep the links when you author: “transclusion”

2626

MLB database: size and content?

Database features are essential: Database features are essential: Consistency, Indexing, Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, replication.Pivoting, Queries, Speed/scalability, Backup, replication.

Folders &Files were the starting point >> database into sets Folders &Files were the starting point >> database into sets aka “collections” that are identical to the folder structureaka “collections” that are identical to the folder structure

Outlook (msgs, attachments, calendar, contacts)Outlook (msgs, attachments, calendar, contacts) Web trails including voice message annotation Web trails including voice message annotation Journal (Outlook), trails: every document use & transactionJournal (Outlook), trails: every document use & transaction What about? What about?

Money (transactions, payees, etc.)…is their lifelog/trailMoney (transactions, payees, etc.)…is their lifelog/trail Streets and trips to cross-index to all docsStreets and trips to cross-index to all docs Attributes for photos for retrieval? Location, time, settingsAttributes for photos for retrieval? Location, time, settings Presentations as a report or trail. Each slide an object!Presentations as a report or trail. Each slide an object!

InternetMyLifeBits

store

database

files

Voice annotation tool

Text annotation tool

Legacy applications

MAPI interface

Legacy email client

Radio EPG tool

PocketPC transfer tool

Telephone capture tool

Radio capture tool

TV capture tool

TV EPG download tool

Browser tool

PocketRadio player

MyLifeBits Shell

2929

Annotation like this…

VoiceAnnotation

3030

Pivot to look at all of MLB(t)Call, contact, pivot by time to find web page

3131

Find brig, image, and look for 80

3232

Here are the photos

3333

Timeline view tells a story

3434

Finding scatological works

3535

Statistics of use

4545

Value of media depends on annotations

““Its just bits until it is annotated”Its just bits until it is annotated”

4646

System annotations provide base level of value

Date 7/7/2000Date 7/7/2000

4747

Tracking usage – even better

Date 7/7/2000. Opened 30 times, emailed to 10 Date 7/7/2000. Opened 30 times, emailed to 10 people (its valued by the user!)people (its valued by the user!)

4848

Get the user to say a little something is a big jump

Date 7/7/2000. Opened 30 times, emailed to 10 people. “BARC Date 7/7/2000. Opened 30 times, emailed to 10 people. “BARC dim sum intern farewell Lunch”dim sum intern farewell Lunch”

4949

Getting the user to tell a story is the ultimate in media value

A story is a “layout” in time and spaceA story is a “layout” in time and space Most valuable content (by selection, and by being well annotated)Most valuable content (by selection, and by being well annotated) Stories must include links to any media they use (for future navigation/search – Stories must include links to any media they use (for future navigation/search –

“transclusion”).“transclusion”). Cf: MovieMaker; Creative Memories PhotoAlbumsCf: MovieMaker; Creative Memories PhotoAlbums

Dapeng was an Dapeng was an intern at BARC intern at BARC for the summer for the summer of 2000of 2000

We took him to We took him to lunch at our lunch at our favorite Dim Sum favorite Dim Sum place to say place to say farewellfarewell

At table L-R: Dapeng, Gordon, Tom, Jim, Don, At table L-R: Dapeng, Gordon, Tom, Jim, Don, Vicky, Patrick, JimVicky, Patrick, Jim

5050

Value of media depends on annotations

Auto-annotate whenever Auto-annotate whenever possible e.g. GPS cameraspossible e.g. GPS cameras

Make manual annotation Make manual annotation as easy as possible. XP as easy as possible. XP photo capture, voice, photo capture, voice, photos with voice, etcphotos with voice, etc

Support gang annotationSupport gang annotation Make stories easyMake stories easy

no

ne

au

to

au

to-u

sag

e

use

r-ba

sic

use

r-story

Annotations

“Its just bits until it is annotated”

5151

5353

CD

VCR

Cassette

Plasma Panel

DVD

MediaCenter

Computer

Set top

Set top

Kbd Mse

Wfr

Spkr

SpkrIR

Cable/Satellite

Ethernet SVHS-wide

5.1 digital

5 speakers

stereo

stereo

stereoVideo*

5.1 digitalcomp.

stereoVideo*

Video*

Cables/linksSpeaker 5+1Plasma 2 or 3Cable/Enet 2IR 8Stereo 45.1 digital 2Comp./S-video 3Plasma panel 1Power 10Kbd/mse 2Monitor II (opt.) 4Camera 2Total 42 – 46 Things 18+remotes

*Video = composite or S-videoCamera

Mic

Receiver

Legacy

Legacy

Legacy

Redundant

5454

5555

Media center 2

5656

Photos

5757

Caneel Bay Vacation Jan. 1998

Gordon, Gwen, Brig, Pam, Gordon, Gwen, Brig, Pam, Fiona, Bob, Laura and KolbeFiona, Bob, Laura and Kolbe

5959

The Agenda for the Tbyte(s), Lifetime, PC:The killer app after office and mail.

1.1. Guarantee that data will live forever! “dear appy” problemGuarantee that data will live forever! “dear appy” problem2.2. Cheap, easy, and data-rich (e.g. time, place) capture:Cheap, easy, and data-rich (e.g. time, place) capture:

GPS and time everywhereGPS and time everywherePaper capture has to be as easy as discard (scanner/shredder)Paper capture has to be as easy as discard (scanner/shredder)Personal meeting capture...Personal meeting capture...E-book…e-magazines & journals need to have critical mass! E-book…e-magazines & journals need to have critical mass! Telephony and audio capture with indexingTelephony and audio capture with indexingMedia Center compatible for entertainment (photos, video, TV, radio)Media Center compatible for entertainment (photos, video, TV, radio)

3.3. Content analysis (critical for photo & video!)Content analysis (critical for photo & video!)4.4. Information control: privacy, security, expunge/deniability,… Information control: privacy, security, expunge/deniability,… 5.5. One One dbase for everything (articles, books, conversations, ... dbase for everything (articles, books, conversations, ...

financial transactions) …vs. long-term use of hierarchical files. financial transactions) …vs. long-term use of hierarchical files. Is dbase intuitive?Is dbase intuitive?

6.6. Annotations/meta-information add every-increasing valueAnnotations/meta-information add every-increasing valueEasy annotation for aiding search and Easy annotation for aiding search and it becomes the contentit becomes the content

7.7. The “killer apps”: Alzheimer, immortality, surrogate memory?The “killer apps”: Alzheimer, immortality, surrogate memory?8.8. GUI’s to improve use (e.g. time to learn, use, retention)GUI’s to improve use (e.g. time to learn, use, retention)

6060

The “dear appy” problemDear Appy, Dear Appy, How committed are you? How committed are you?

Please come back to me, Please come back to me, Lost and forgotten dataLost and forgotten data

Who’s responsible?Who’s responsible?mediamediaplatform, file, and databasesplatform, file, and databasesevolving standards and formatsevolving standards and formatsevolving and/or disappearing appsevolving and/or disappearing apps

  

6161

The Amnesia Control Problem

Full sharing of bits that are mineFull sharing of bits that are mine I created them, OK to copy and distributeI created them, OK to copy and distributeDRM: purchased for my own useDRM: purchased for my own use

““OK to look at, but I only own half the bits”OK to look at, but I only own half the bits”Controlling forgetfulnessControlling forgetfulness

Private, do not “demo”Private, do not “demo”Expunge forever... “this never happened”Expunge forever... “this never happened”

6262

The Content Analysis Problem

1.1. ““Cliplets”: Automatic segmentation of a Cliplets”: Automatic segmentation of a pile of documents and video into pile of documents and video into individual documents and scenes.individual documents and scenes.

2.2. Item typing: Would like a minimal Dublin Item typing: Would like a minimal Dublin Core for each item: date, creator, title, Core for each item: date, creator, title, source, abstract, and typesource, abstract, and type

3.3. ““Type” classification: articles, letters, Type” classification: articles, letters, memos, etc.memos, etc.

4.4. Ontology creation for collectionsOntology creation for collections

6363

The End