computer vision technologies for remote collaboration using physical whiteboards, projectors and...
TRANSCRIPT
Computer Vision Technologies for Remote Collaboration Using Physical Whiteboards, Projectors and Cameras
Zhengyou ZhangZhengyou Zhang
Microsoft ResearchMicrosoft Researchmailto: mailto: [email protected]@microsoft.com
http://http://research.microsoft.comresearch.microsoft.com/~zhang//~zhang/
Motivations
WB provides a large shared space WB provides a large shared space for the participants to focus their attention and for the participants to focus their attention and
express their ideas spontaneouslyexpress their ideas spontaneously Many meetings use whiteboard heavily Many meetings use whiteboard heavily
brainstorming sessions, lectures, project planning brainstorming sessions, lectures, project planning meetings, patent disclosures, etc meetings, patent disclosures, etc
WB is ubiquitous in knowledge workers’ officesWB is ubiquitous in knowledge workers’ offices
→ → Whiteboard will still exist for long timeWhiteboard will still exist for long time
Motivations (cont’d)
Difficulties: Difficulties: – Content is hard to archive or shareContent is hard to archive or share– Busy in note taking, instead of spending time Busy in note taking, instead of spending time
sharing and absorbing ideas sharing and absorbing ideas Need tools to increase productivitiesNeed tools to increase productivities
– Don’t lose any sparksDon’t lose any sparks– Share with othersShare with others– Keep for later reviewKeep for later review– Integrate with your office documentIntegrate with your office document
Motivations (cont’d)
Projector Projector – A great tool for presentationA great tool for presentation– Not so convenient for discussionsNot so convenient for discussions
Whiteboard-Whiteboard-Camera SystemWhiteboard-Whiteboard-Camera System– Whiteboard: Projecting surface (Output) and Whiteboard: Projecting surface (Output) and
Writing surface (Input)Writing surface (Input)– Seamless integration of computer Seamless integration of computer
presentation with whiteboard discussionspresentation with whiteboard discussions– Enable remote collaboration on a shared Enable remote collaboration on a shared
workspaceworkspace
Whiteboard Scenarios
Note taking with a high-res digital cameraNote taking with a high-res digital camera– Whiteboard It!Whiteboard It!
Note taking with a low-res PC cameraNote taking with a low-res PC camera– Whiteboard ScanningWhiteboard Scanning
Real-time collaborationReal-time collaboration– Live WhiteboardLive Whiteboard
Meeting (WB + Audio / Video) ArchivingMeeting (WB + Audio / Video) Archiving– Whiteboard ArchivingWhiteboard Archiving
♥ Whiteboard It! ♥
Convert WB Content into an Electronic DocumentConvert WB Content into an Electronic Document
Overview of the System
♥ Whiteboard Scanning ♥
Get a High-res Scan With an Inexpensive Get a High-res Scan With an Inexpensive PC CameraPC Camera
How to do it?How to do it?– Take multiple pictures with overlapTake multiple pictures with overlap– Stitch them Stitch them automaticallyautomatically into a high-res image into a high-res image– Image enhancement (Image enhancement (Whiteboard It!Whiteboard It!))
System Overview
Example
+
+
Document Scanning w/ Low-Res camera
♥ Live Whiteboard ♥
Real-time distributed collaborationReal-time distributed collaboration
Whiteboard
DisplayMic (array)
Table Projector
Speaker
PC
Meeting Room/Office
Windows Communications Desktop
TabletPC
TabletPC Wire or wireless
Camera
Offices
A typical image sequence
Segmentation of the person and WB background is neededSegmentation of the person and WB background is needed
Demo
Integration with Windows MessengerIntegration with Windows Messenger
Whiteboard update
Annotations
Windows Messenger
Windows MessengerT120
Local WB client Remote WB client
USB 2
Video
Real-Time WB Processing
Video
Whiteboard changes
♥ Whiteboard Archiving ♥
Capture whiteboard content + audio/Video Capture whiteboard content + audio/Video – high-resolution digital still camera high-resolution digital still camera
Produce key framesProduce key frames– A KF usually corresponds to a major topicA KF usually corresponds to a major topic– Print as notes, or cut & paste into documents Print as notes, or cut & paste into documents
Record time stamps of each strokeRecord time stamps of each stroke Efficient meeting browsingEfficient meeting browsing
– Key frames to navigate between sectionsKey frames to navigate between sections– Strokes to bring up the audio at the moment Strokes to bring up the audio at the moment
when they were writtenwhen they were written
A Typical System Installation
Example: Sample Images
Key Frame Extraction
Number of strokesNumber of strokes
0
100
200
300
400
500
600
700
800
900
1000
1 21 41 61 81 101 121 141 161 181
Chapter 1 Chapter 2
Key Frame 1
Key Frame 2
Browsing Interface: Demo
Key Frame Thumbnails
Future Strokes
Current Strokes
Raw Image
VCR & Timeline Control
♥ Projector-Whiteboard-Camera System ♥
Communi-cation
Network
Video Mixer
Presentation
Video Mixer
P
Visual Echo Cancellation
Wri t i
ngs
A
A AnnotationA
P P
P
W
W
Local Meeting Room
Remote Room
Whiteboard
Camera
ProjectorDisplay
Remote Room
P+A
P+A+W
P+W
Visual Echo Cancellation
PrerequisitesPrerequisites– Geometric calibrationGeometric calibration– Color calibrationColor calibration
recover writings W
albedo estimation
& color
clustering
Color Lookup Table(given by color calibration)
Color Mapping & Geometric Warping
estimated visual echo E
captured image I
Homography H (given by geometric calibration)
VEC Results
Conclusions
Tools to use whiteboard and projectorTools to use whiteboard and projector Whiteboard It!Whiteboard It! for anyone who has a for anyone who has a
digital camera and it is easy to usedigital camera and it is easy to use Whiteboard ScanningWhiteboard Scanning for anyone who for anyone who
has a PC camera and it is easy to usehas a PC camera and it is easy to use Whiteboard ArchivingWhiteboard Archiving and and StreamingStreaming
for meeting rooms and collaboration for meeting rooms and collaboration Integration of Integration of ProjectorProjector and and WhiteboardWhiteboard
for seamless remote collaborationfor seamless remote collaboration
Acknowledgements
Thanks to Li-wei He, Zicheng Liu & Thanks to Li-wei He, Zicheng Liu & Hanning Zhou for contributionsHanning Zhou for contributions