Visual Screen: Transforming an OrdinaryScreen into a Touch Screen

Zhengyou Zhang & Ying Shan

Vision Technology GroupMicrosoft Research

zhang@microsoft.com

Motivation

Transform an ordinary screen into a touch

screen using an ordinary camera

Configuration

Position a camera so it can see the whole screen

Fingertip Tracker

Action

Detector

&

Event

Generator

Mouse EventsVideo

Input

Visual Screen

FingertipDetector

HomographyMapping

Screen Detection

Virtual Touch Screen

Non-flatness correctionCalibration

The System

System Diagram

Calibration

• Mapping from image coordinate to the screen coordinate

• Homography if the screen is flat• More accurate method required when the

screen is curved

Plane Plus Residue Flow

H

The actual screen coordinate can be found by the homography corrected by an interpolated residue vector

Mapping

Plane Plus Residual Flow

Before correction After correctionOriginal calibration points Reprojected image points

Segmentation

• Images of screen pixels have some degrees of invariance in the color space

• Compute a color model for the screen without the indicator

• Compute a color model for the indicator • Use standard Bayes classifier to segment the

indicator from the screen background

Segmentation

Before segmentation After segmentation

Locate the Finger Tip

•Initial location from horizontal histogram•Fit the centerline of the finger•Finger tip is the intersection of the centerline and the segment boundary

Experiments: Draw Bubbles for Fish

Experiments: Barney Under the Sea

Thank You