video object tracking and replacement for post tv production lyu0303 final year project spring 2004
TRANSCRIPT
![Page 1: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/1.jpg)
Video Object Tracking and Replacement for Post TV Production
LYU0303 Final Year Project
Spring 2004
![Page 2: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/2.jpg)
Outline
Brief project overview and targets achieved New components added to the proposed
system Working principles of individual parts Future work Q & A
![Page 3: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/3.jpg)
Overview
Post-TV production processes and changes the content of a video clip
Difficult for computers to process video in a global view
Step-by-step programming enables the computer to “think” more thoroughly
![Page 4: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/4.jpg)
Overview
We have introduced the following parts in the last semester: Bitmap I/O RGB <-> HSV converter Edge detector Edge equation finder Translation detector Texture mapper
New parts have been added in order to increase the functionality of the system
![Page 5: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/5.jpg)
Additional Basic Components
The following parts have been added Graphical User Interface (GUI) Corner point finder Video file reader and writer Video player processor (frame-based) Improved texture mapper
The following part has been modified Bitmap I/O Edge finder Texture mapper Removed Equation processor
![Page 6: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/6.jpg)
Graphical User Interface
Although UI is not a main project part, a good GUI can significantly decrease the time needed for processing and program maintenance
Since C++ language is being used, the Microsoft Foundation Classes are adopted to develop the user interface
Provide most of the basic functions needed for information input and output
![Page 7: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/7.jpg)
Graphical User Interface
![Page 8: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/8.jpg)
Corner Point Finder
As we know that processing dots are much faster than processing lines, we may wish to try to find the corner points directly before finding the edge equations
Stick some brightly coloured labels at the corners of the rectangular surface
Can apply on cylindrical objects as well to indicated the curve control points
A simple K-Means method is used to group the points together.
![Page 9: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/9.jpg)
Corner Point Finder In order to know the orientation of the mapped
texture, we need to know which is the “first” point For simplicity we just make the first point larger so
that it contains more corner points than the others
First point
![Page 10: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/10.jpg)
Corner Point Finder
We repeat the process until all the corner points has been processed
Points which are too far away from the main groups are discarded
Find out the centroids of the groups and move them out to give an estimation of corner positions
Group No. of Member
1 1
2 1
3 1
1 2
4 1
1 5
2 5
3 5
4 5
![Page 11: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/11.jpg)
Corner Point Finder
![Page 12: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/12.jpg)
Video File I/O
As we are talking about video processing, it is inevitable to handle video reading and writing
DirectShow interfaces are being used in the project
Able to process any type of AVI video as long as the Window Media Player can play them
![Page 13: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/13.jpg)
Video Player
We need to choose a video renderer that is suitable for our application
To reserve the function of real-time processing and video texture in the future, the Video Mixing Renderer 9 (VMR9) is used
Allows the rendering of more than one video or bitmap on the screen at the same time
![Page 14: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/14.jpg)
Video Processor
Video are sequence of bitmap frames We capture each video frame and process t
hem in the same way as processing the bitmaps
After a video frame is being processed, we add them to an AVI file
Default video compressor is DivX Mpeg4 encoder (FourCC code “divx”)
![Page 15: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/15.jpg)
Video RendererVideo System Structure
![Page 16: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/16.jpg)
Texture Mapper
A graphics design technique used to wrap a surface of a 3-D object with a texture map
The 3-D object acquires a surface texture similar to the texture map.
Colors, brightness values or altitudes
![Page 17: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/17.jpg)
Texture Mapper
Mapping
function
(u,v)(r,c)
Texture coordinates
Image coordinates
![Page 18: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/18.jpg)
line by line process each pixel on every line last semester, we mapped cubes. How about this
semester?
Scan-line conversion
Scanline yk
Scanline yk+1
scanning order
for every line
![Page 19: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/19.jpg)
Cylindrical mapping
How to map this cylinder with different textures?
![Page 20: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/20.jpg)
Cylindrical mapping
Mapping cylinders
3 parts: Surface detection Line scanning Texture lookup
+ Orientation determination
![Page 21: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/21.jpg)
Surface detection
1. Hough algorithm
to detect the two straight edges
four points are detected
![Page 22: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/22.jpg)
Surface detection
2. Scanning from top to bottom
the scanning slope is orthogonal
yellow points are detected
by color transition
![Page 23: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/23.jpg)
Surface detection
3. Model the curved edges by ellipses
x2 / a2 + y2 / b2= 1 2 ellipses are
needed rotational factor
x
y
z
![Page 24: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/24.jpg)
Scan-line mapping4. Scan-line from left t
o right
Problem:
How to determine the corresponding texture for each pixel, say p=(x,y) ?
x
y
z
(x,y)
![Page 25: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/25.jpg)
Texture lookup5. Rotate the cylinder t
o the upright position along z-axis
6. Translate the cylinder to (0,0) and determine the z coordinate p=(x’,y’,z’)
7. Rotate the along x-axis upright from viewpoint
x
y
z
(0,0)h
![Page 26: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/26.jpg)
Texture lookup8. Detect the marker w
hich indicates where the texture starts
9. Look up the texture coordinate (u,v) by
(y’+h/2)/h = v
tan-1(x’/z’) / 2 = u
x
y
z
(u,v)
![Page 27: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/27.jpg)
Cylindrical mappingAfter mapping the top we have :
TEXTURES
MAPPING
MAPPED CYLINDER
![Page 28: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/28.jpg)
Spherical mapping
How to map this sphere with different textures?
![Page 29: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/29.jpg)
Spherical mapping
Mapping spheres
Also 3 parts: Surface detection Line scanning Texture lookup
+ Orientation determination
![Page 30: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/30.jpg)
Surface detection
1. Scan from 4 directions and get the boundary points
2. Determine center
![Page 31: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/31.jpg)
Line scanning
3. Scan from left to right.
4. For each pixel p=(x,y), look up the texture by transforming it to the upright position
x
y
z
(0,0)
Upright
![Page 32: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/32.jpg)
Texture lookup
4. Translate to origin
5. Look up z-coordinate
6. Rotate the ball along z-axis
7. Rotate the ball along x-axis (x’,y’,z’)
x
y
z
(0,0)
(5)(6)
![Page 33: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/33.jpg)
Texture lookup8. Lookup the texture by
v = cos-1(y’/R) / u = (cos-1(x’/(R sin(v))) ) / (2)
![Page 34: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/34.jpg)
Spherical mapping After mapping:
Mapped Sphere
Texture
![Page 35: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/35.jpg)
Shadow mapping
Mapping of surface brightness Method:
Covert the image from RGB HSV Modify the V (intensity) value
5 types of shadow mapping
![Page 36: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/36.jpg)
Shadow mapping
1. Mapping with texture brightness
often unrealistic
2. Mapping with image brightness
![Page 37: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/37.jpg)
Shadow mapping
3. Mapping with average brightness of the image
more realistic
4. Mapping with other shadow maps (e.g. wood)
![Page 38: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/38.jpg)
Shadow mapping
5. Transparency mapping
(background information has to be obtained)
![Page 39: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/39.jpg)
Problem with sphere mapping
The texture is “squeezed” at the pole
![Page 40: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/40.jpg)
Solution
Modify the texture to this:
Mapped sphere becomes:
No distortion at
pole
![Page 41: Video Object Tracking and Replacement for Post TV Production LYU0303 Final Year Project Spring 2004](https://reader035.vdocuments.us/reader035/viewer/2022062511/5519c887550346047c8b48f4/html5/thumbnails/41.jpg)
Q & A