development of image processing based feedback systems for interactive gaming using non-traditional...

Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers


Adam HedjiMantas PulinasPhilip SanIII. Viktorious

SSIP 2009, Debrecen, Hungary


Project NatalX-Box 360Microsoft Corporation

Full Body Motion Detection

New project that promises to use two

cameras for full-body motion detection


CHECKERS!


Overview of Code

Board Plane Geometry Detection



Overview of Code

Relaying of Data with Checkers A.I.

Relaying of Data with Checkers A.I.

FeedbackFeedback

Identification of Square Status by

Colour Classification

Identification of Square Status by


Segmentation of Individual Game

Squares

Segmentation of Individual Game

Squares



This is blue!This is blue!


Geometry Detection

• Find the corners

• 2 approaches were used:• Edge

detection and region growing

• Hough transforms


Geometry Detection – Edge Detection and Region Growing


Geometry Detection – Edge Detection and Region Growing• Edge detection with Sobel

operator• Dilate the image to fill the

gaps in the border




gaps in the border• Thin the image to get the

true border




gaps in the border• Thin the image to get the

true border• Dilate the image several

times to remove useless edges


Geometry Detection – Edge Detection and Region Growing• Edge detection with Sobel operator• Dilate the image to fill the gaps in

the border• Thin the image to get the true

border• Dilate the image several times to

remove useless edges• Region growing• Determine the corner tiles• Track corners in real-time


Geometry Detection – Projective Correction• Edge detection with Sobel operator• Dilate the image to fill the gaps in

the border• Thin the image to get the true

border• Dilate the image several times to

remove useless edges• Region growing• Determine the corner tiles• Track corners in real-time


Geometry Detection – Projective Correction

Calculate homography using corner coordinates


Geometry Detection – Hough Transform

• Hough transform– We had time so we developed a better

solution– Based on Hough transformations– Better real-time line detection

Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional ControllersDevelopment of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers


Colour ClassificationHow can the computer classify the colours?

(171,154,158)(171,154,158)

(32,61,105)(32,61,105)


Colour ClassificationUse image processing algorithms to make the RGB values only 0 or 255

(255,255,255)(255,255,255)

(0,0,255)(0,0,255)



(255,255,255)

(0,0,0)

(255,255,0)

(0,0,255)

•Select individual tile

•Analyse the predominant colour inside to classify the square state (white, black, blue, yellow)

•Sample of pixels used as opposed to whole square



•How to determine if each R, G and B values are 0 or 255?

-Need to choose threshold value

e.g.A given pixel of value (15, 19, 250)A threshold of 126Output is (0,0,255)


Colour Classification - But how to choose the threshold?• Webcams can be of relatively poor quality and

provide poor contrast.– For example, blue pieces used were relatively

hard to distinguish from the black tiles.

Development of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional ControllersDevelopment of Image Processing Based Feedback Systems for Interactive Gaming Using Non-Traditional Controllers


But how to choose the threshold?

• Can instead normalise the image and use a threshold of 127, given by (256/2)-1.– Select a white tile and take

an average of the colour values.

– Do the same for a black tile.– Use these averages to

normalise the image

p1=(241,209,210)

p2=(232,204,214)

p3=(240,211,205)

BlackWhite

BlackInputBGROutput

),,(


Colour Classification - Normalisation



• Then reduce the image to absolute values of 0 and 255

• Use a threshold of 126 (half of full intensity value)


Summary

• Main results– Successfully used 2 approaches to chess

board detection• Edge detection and region growing• Hough transform

– Removing perspective distortion– Identification of individual tiles and pieces,

including classification– Connection to engine interface with feedback

system


Improvements

• Aim towards real-time 60fps processing• Use a more efficient programming language

such as C++• Use of GPU using CUDA or Open CL

programming language• More complex algorithms

– Motion detection of hand• Use of overlay of 3D structures onto camera

image.• Virtual humans...


Unused Virtual Human


The Team

Adam HedjiUniversity of Zagreb, [email protected]

Mantas PaulinasVilnius Gediminas Technical University, [email protected]

Viktor BlaskovicsUniversity of Szeged, [email protected]

Philip SanUniversity College London, [email protected]

development of image processing based feedback systems for interactive gaming using non-traditional...

Documents

feedback systems

interactive gaming

body motion detection

true border slide

realtime slide

hungary slide

useless edges slide

sobel operator