video object tracking and replacement for post tv production
DESCRIPTION
Video Object Tracking and Replacement for Post TV Production. LYU0303 Final Year Project Fall 2003. Outline. Project Introduction Basic parts of the purposed system Working principles of individual parts Future Work Q&A. Introduction. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/1.jpg)
Video Object Tracking Video Object Tracking and Replacement for and Replacement for
Post TV ProductionPost TV Production
LYU0303 Final Year ProjectLYU0303 Final Year Project
Fall 2003Fall 2003
![Page 2: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/2.jpg)
OutlineOutline
Project IntroductionProject Introduction Basic parts of the purposed systemBasic parts of the purposed system Working principles of individual partsWorking principles of individual parts Future WorkFuture Work Q&AQ&A
![Page 3: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/3.jpg)
IntroductionIntroduction
Post-TV production software changes the Post-TV production software changes the content of the original video clips.content of the original video clips.
Extensively used in video-making Extensively used in video-making industries.industries.
Why changing the content of a video?Why changing the content of a video? Reducing video production costReducing video production cost Performing dangerous actionsPerforming dangerous actions Producing effects those are impossible in realityProducing effects those are impossible in reality
![Page 4: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/4.jpg)
Difficulties to be overcomeDifficulties to be overcome Things in video can be treated individually Things in video can be treated individually
called “video objects”.called “video objects”. Computers cannot perform object detection Computers cannot perform object detection
directly because…directly because… Image is processed byte-by-byteImage is processed byte-by-byte Without pre-knowledge about the video objects to be detectedWithout pre-knowledge about the video objects to be detected Result is definite, no fuzzy logic.Result is definite, no fuzzy logic.
Though computers cannot perform object Though computers cannot perform object detection directly, it can be programmed to detection directly, it can be programmed to work indirectly.work indirectly.
![Page 5: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/5.jpg)
Basic parts of the purposed systemBasic parts of the purposed system
Simple bitmap reader/writerSimple bitmap reader/writer RGB/HSV converterRGB/HSV converter Edge detectorEdge detector Edge equation finderEdge equation finder Equation processorEquation processor Translation detectorTranslation detector Texture mapperTexture mapper
![Page 6: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/6.jpg)
RGB/HSV converterRGB/HSV converter
Human eyes are more sensitive to the Human eyes are more sensitive to the brightness rather than the true color brightness rather than the true color components of an object.components of an object.
More reasonable to convert the More reasonable to convert the representation of colors into HSV (Hue, representation of colors into HSV (Hue, Saturation and Value (brightness)) model.Saturation and Value (brightness)) model.
After processing, convert back to RGB and After processing, convert back to RGB and save to disk.save to disk.
![Page 7: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/7.jpg)
RGB/HSV converterRGB/HSV converter
RGB to HSVRGB to HSV HSV to RGBHSV to RGB
![Page 8: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/8.jpg)
Edge detectorEdge detector
Usually, a sharp change in hue, saturation or Usually, a sharp change in hue, saturation or brightness means that there exist a boundary brightness means that there exist a boundary line.line.
HSV: (0,0,0)
HSV: (0,255,255)
![Page 9: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/9.jpg)
Edge equation finderEdge equation finder
Derives mathematical facts out of the edge Derives mathematical facts out of the edge points.points.
Works with voting algorithm of Hough Works with voting algorithm of Hough Transform.Transform.
Automatically adjusts tolerance value to Automatically adjusts tolerance value to minimize the effect of noise points.minimize the effect of noise points.
This helps when the edge is not completely This helps when the edge is not completely straight or blurred.straight or blurred.
![Page 10: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/10.jpg)
Edge equation finderEdge equation finder
0
4590
(x1,y1)
Angle in degree Frequency
0 1
45 3
90 1
135
135 1
Desired linear equation in point-slope form:
)1(*45tan1 xxyy
![Page 11: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/11.jpg)
Equation processorEquation processor
After finding out the equation, constraints After finding out the equation, constraints can be applied in order to remove can be applied in order to remove redundant equations, get shadows or redundant equations, get shadows or detect occultation.detect occultation.
Find out the corner points for translation Find out the corner points for translation detector and texture mapper.detector and texture mapper.
![Page 12: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/12.jpg)
Translation detectorTranslation detector
A simple object motion tracker.A simple object motion tracker. Collects the data of the first key frame to Collects the data of the first key frame to
accelerate the process of the remaining accelerate the process of the remaining video frames.video frames.
Can be beneficial if the video segment is Can be beneficial if the video segment is long and the scene seldom changes.long and the scene seldom changes.
![Page 13: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/13.jpg)
Texture MapperTexture Mapper
A graphics design process in which a 2-D A graphics design process in which a 2-D surface, called a surface, called a texture maptexture map, is "wrapped , is "wrapped around" a 3-D object. around" a 3-D object.
The 3-D object acquires a surface texture The 3-D object acquires a surface texture similar to the texture map.similar to the texture map.
![Page 14: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/14.jpg)
Texture MapperTexture Mapper
Mapping
Original position of pixel
New position of pixel
![Page 15: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/15.jpg)
Texture MapperTexture Mapper
Every polygon is assigned 2 sets of Every polygon is assigned 2 sets of coordinatescoordinates
Image coordinates Image coordinates (r, c)(r, c) : location of pixel : location of pixel in the imagein the image
Texture coordinates Texture coordinates (u, v)(u, v) : location in : location in texture image which contains color texture image which contains color information for image coordinatesinformation for image coordinates
![Page 16: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/16.jpg)
Texture MapperTexture Mapper
Mapping functionsMapping functions map texture map texture coordinates to image coordinates or vice coordinates to image coordinates or vice versa.versa.
They are usually determined by image They are usually determined by image points whose texture coordinates are points whose texture coordinates are given explicitly.given explicitly.
![Page 17: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/17.jpg)
Texture MapperTexture Mapper
(r1, c1)
(u1, v1)
(r2, c2)
(u2, v2)
(r4, c4)
(u4, v4)
(r3, c3)
(u3, v3)
(u1, v1) (u2, v2)
(u3, v3)(u4, v4)
![Page 18: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/18.jpg)
Texture MapperTexture Mapper
Scan conversionScan conversion: the process of scanning : the process of scanning all the pixels and perform the necessary all the pixels and perform the necessary calculation.calculation.
Forward mappingForward mapping maps from the texture maps from the texture space to image spacespace to image space
Inverse mapping Inverse mapping maps from the image maps from the image space to texture spacespace to texture space
![Page 19: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/19.jpg)
Scan conversion with forward Scan conversion with forward mappingmapping
Algorithm: Algorithm: for u = umin to umaxfor u = umin to umax for v = vmin to vmixfor v = vmin to vmix r = R(u,v)r = R(u,v) c = C(u,v)c = C(u,v) copy pixel at source (u,v) copy pixel at source (u,v) to destination (r,c)to destination (r,c)
![Page 20: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/20.jpg)
Scan conversion with forward Scan conversion with forward mappingmapping
Advantage:Advantage:Easy to compute as long as the forward Easy to compute as long as the forward mapping function is known.mapping function is known.
DisadvantageDisadvantagePixel-to-pixel mapping is not 1-1.Pixel-to-pixel mapping is not 1-1.Holes may appear.Holes may appear.Can result in aliasing.Can result in aliasing.
![Page 21: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/21.jpg)
Scan conversion with forward Scan conversion with forward mappingmapping
![Page 22: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/22.jpg)
Scan conversion with inverse Scan conversion with inverse mappingmapping
Algorithm: Algorithm: for (r,c) = polygon pixelfor (r,c) = polygon pixel u = TEXR(r,c)u = TEXR(r,c) v = TEXC(r,c)v = TEXC(r,c) copy pixel at source (u,v) copy pixel at source (u,v) to destination (r,c)to destination (r,c)
![Page 23: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/23.jpg)
Scan conversion with inverse Scan conversion with inverse mappingmapping
Advantage: Advantage:
Every destination pixel is filled (no holes).Every destination pixel is filled (no holes).
Allow easy incorporation of pre-filtering & resampling Allow easy incorporation of pre-filtering & resampling operations to prevent aliasingoperations to prevent aliasing
![Page 24: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/24.jpg)
Scan conversion with inverse Scan conversion with inverse mappingmapping
Take advantage of Take advantage of Scanline Polygon Fill AlgorScanline Polygon Fill Algorithmithm
For a row scan, maintain a list of scanline / polFor a row scan, maintain a list of scanline / polygon intersections.ygon intersections.
Intersection at scanline r+1 efficiently computeIntersection at scanline r+1 efficiently computed from row r.d from row r.
Scanline yk+1
Scanline yk{xk, yk}
{xk+1, yk+1}
![Page 25: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/25.jpg)
Scan conversion with inverse Scan conversion with inverse mappingmapping
Coordinates at a non-boundary level are compCoordinates at a non-boundary level are computed by linearly interpolating (u,v) coordinates uted by linearly interpolating (u,v) coordinates of bounding pixels on the scanline.of bounding pixels on the scanline.
Scanline yk+1
Scanline yk{xk, yk}
{xk+1, yk+1}
![Page 26: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/26.jpg)
Scan conversion with inverse Scan conversion with inverse mappingmapping
Suppose (ri,ci) maps to (ui,vi), i = 1,…, 5Suppose (ri,ci) maps to (ui,vi), i = 1,…, 5 (r4,c4) = s (r1,c1) + (1-s) (r3,c3) { s is known }(r4,c4) = s (r1,c1) + (1-s) (r3,c3) { s is known } (u4,v4) = s(u1,v1) + (1-s)(u3,v3) {u4,v4 are known}(u4,v4) = s(u1,v1) + (1-s)(u3,v3) {u4,v4 are known} Similarly, (u5, v5) can be found.Similarly, (u5, v5) can be found. t = (c-c4)/(c5-c4)t = (c-c4)/(c5-c4) (r,c) = t*(u5,v5) + (1-t)*(u4,v4)(r,c) = t*(u5,v5) + (1-t)*(u4,v4)
Scanline yk(r4, c4)
(r1, c1)
(r3, c3) (r2, c2)
(r5, c5)
(r, c)
image
![Page 27: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/27.jpg)
Basic 2D linear mappingBasic 2D linear mapping
Scaling & TranslationScaling & Translationu = ar + du = ar + dv = bc + ev = bc + eupright rectangle upright rectangle uprig upright squareht square
Euclidean mappingEuclidean mappingu = (cosu = (cos)r – (sin)r – (sin)c + d)c + dv = (sinv = (sin)r + (cos)r + (cos)c + e)c + erotated unit square rotated unit square upr upright squareight square
![Page 28: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/28.jpg)
Basic 2D linear mappingBasic 2D linear mapping
Similarity mappingSimilarity mappingu = s(cosu = s(cos)r – s(sin)r – s(sin)c + d)c + dv = s(sinv = s(sin))r + s(cosr + s(cos)c + )c + eerotated square rotated square upright u upright unit squarenit square
Affine mappingAffine mappingu = f(cosu = f(cos)r – g(sin)r – g(sin)c + d)c + dv = h(sinv = h(sin)r + i(cos)r + i(cos)c + e)c + erotated rectangle rotated rectangle uprigh upright unit squaret unit square
DEMO !
![Page 29: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/29.jpg)
Basic 2D linear mappingBasic 2D linear mapping
Projective mappingProjective mappingThe most general 2D linear mapThe most general 2D linear mapSquare Square arbitrary quadrangle ! arbitrary quadrangle !
u = (a11r+a12c+a13) / (a31r+a32c+1)u = (a11r+a12c+a13) / (a31r+a32c+1) v = (a21r+a22c+a23) / (a31r+a32c+1)v = (a21r+a22c+a23) / (a31r+a32c+1)
The 8 variables a11,a12, … , a32 have to be The 8 variables a11,a12, … , a32 have to be found out.found out.
![Page 30: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/30.jpg)
Basic 2D linear mappingBasic 2D linear mapping
We have a system of 8 equations solving 8 We have a system of 8 equations solving 8 unknowns.unknowns.
(x1,y1)
![Page 31: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/31.jpg)
Future WorkFuture Work
Mapping cansMapping cans Speed optimizationSpeed optimization Movie manipulationMovie manipulation Use of 3D markersUse of 3D markers
![Page 32: Video Object Tracking and Replacement for Post TV Production](https://reader035.vdocuments.us/reader035/viewer/2022070419/56815a95550346895dc810cd/html5/thumbnails/32.jpg)
Q & AQ & A
See the foot notes.