Interactive Space – An application of OpenCV

Sam Siciliano

Overview

• Who I am and the project background.• The System

– Components– Communication

• The Software– APIs– Custom Code– User Interface

• Ideas, Expansions and Surprises• Demonstration and Videos

Who I am

• Graduated summer before last from PSU with a degree in computer engineering

• Embedded Systems Engineer for Logic Product Development

• OMSI Employee at the time of the project

The Project

• Portland State University Senior Capstone Project done in conjunction with OMSI

• Team of 6

• Was in OMSI 3-4 months, currently lives on my laptop.

The System – Block Diagram

The System - Components• “Technobeam” intelligent light fixture

– DMX-512 Lighting Protocol– 180 degrees pan, 90 degrees tilt– Adjustable colors, patterns via “litho wheel” and “gobos”– Iris and adjustable focus

The System - Components

• The DMX-512 transceiver– Shifts from RS-232 to RS-485– Increases baud rate, from 1152 to 2500

The System – Components

• The Server– Dell, p4 2GHz Machine.– Windows XP Professional– Generic Video Capture Card

Software Components

Input Video Stream

via DirectX 9 and

Direct Show

ComputerVision

CalibrationBased

Mapping

TB/PTSerial

ControlClasses

Frame Xvid, Yvid XTB, YTB

Video Capture Via DirectX

• Direct Show and Filter Graphs

Example of Simple Video Capture:

Graph Used for the Project:

Video Capture Via DirectX

• Install DirectX 9.0 SDK (note – DirectShow is not included with latest version!!)

• Use the DirectShow API to Create a Filter Graph– Insert ProxyTrans Filter that comes with

OpenCV

• Define a callback function and start playback

Computer Vision

• Initial Conditions– “cvgoodfeaturestotrack”– Region and Trigger

Selection

• Tracking– Optical Flow– Point Removal Rules– Centroid Calculation

AlgorithimGet Trackable

Features in ROI(points P[] – 1)

Save Frame (Frame N – 1)

Get Next Frame(Frame N)

Pass in all to OpticalFlow

FunctionTo get updated

points (P[])

> 0 points in P[]?

Calculate Centroid ofP[]

Send centroid to remapping

functions

Save Current Frame(now N – 1)

Save Current Points(now P – 1)

Object LostNo

Yes

Throw out invalid points in P[]

Calibration and Mapping

• UI setup to gather calibration data points a set of 2 inputs, 2 outputs

• Points are collected once per given setup

• Accounts for “fish-eye” lens distortion

Calibration and Mapping - 2

• Our problem context:

• First Solution: get to calibration points (lower left, upper right) to obtain thresholds, linearly scale and shift:

???

Xvid(from 0 to 320)

Yvid(from 0 to 240)

Pan (from 0 to 216)

Tilt (from 0 to 216)

aba

ba PanXPanPan

XXPan input

aba

ba TiltYTiltTilt

YYTilt input

Calibration and Mapping - 3

• Worked fine with initial camera we were using.

• When switching though to a camera with a wide angle lens, system wasn’t adequate due to “fisheye” distortion.

• New Scheme: 9 point calibration and piece-wise linear interpolation.

Calibration and Mapping - 4

2113

243

34

34

13

24

34

34

Panyyy

PanPanx

xx

PanPanc

cyyy

PanPanx

xx

PanPanPan vidvid

x1,y1, pan1 x2,y2, pan2

x3,y3, pan3

Ideas and Surprises

• Feedback enabled “throwing” of the light

• Hypersonics and Music

• Use fuzzy logic with camera on pan/tilt unit

Questions?