challenges and solutions for real-time immersive video
TRANSCRIPT
![Page 1: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/1.jpg)
Dr. Oliver Schreer 1Short Course, SP&T, 14th of April 2005
Challenges and solutions for real-time immersive video communication
Part II - 14th of April 2005
Dr. Oliver Schreer Fraunhofer Institute for Telecommunications
Heinrich-Hertz-Institut, Berlin, Germany
![Page 2: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/2.jpg)
Dr. Oliver Schreer 2Short Course, SP&T, 14th of April 2005
Structure of the Short CoursePart I (13th of April 2005)
Introduction
Scope of immersive video communication
Quick tour into projective geometry
Part II (14th of April 2005)Quick tour into camera model and epipolar geometry
Analysis of real-time video streams
Part III (15th of April 2005)The concept of real-time immersive video conferencing
A quick tour into stereo image processing
Hybrid-recursive disparity estimation
![Page 3: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/3.jpg)
Dr. Oliver Schreer 3Short Course, SP&T, 14th of April 2005
Examples of Projective Transformations
![Page 4: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/4.jpg)
Dr. Oliver Schreer 4Short Course, SP&T, 14th of April 2005
The Projective Plane PP22
projective line withcorrespondence to affine planeprojective point with
correspondence to affine plane
ideal line or line at infinity
W=0ideal point or
point at infinity
affine plane PP22
W=1
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
WYX
m
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
0YX
m
projectiveplane PP22
X
YW
C
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛==
1YX
Wmmaffine
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛=
W00
l
![Page 5: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/5.jpg)
Dr. Oliver Schreer 5Short Course, SP&T, 14th of April 2005
OutlineCamera model
Epipolar geometry
Analysis of real-time video streams
Camera model
Epipolar geometry
Analysis of real-time video streams
![Page 6: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/6.jpg)
Dr. Oliver Schreer 6Short Course, SP&T, 14th of April 2005
The Object of Interest
![Page 7: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/7.jpg)
Dr. Oliver Schreer 7Short Course, SP&T, 14th of April 2005
The Pinhole Camera
Gemma Frisius, 1545De Radio Astronomica et GeometricaAlbrecht Dürer, 1525
Underweysung der Messung
![Page 8: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/8.jpg)
Dr. Oliver Schreer 8Short Course, SP&T, 14th of April 2005
The Pinhole Camera
( )vu,=m
( )www Z,Y,X=M
![Page 9: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/9.jpg)
Dr. Oliver Schreer 9Short Course, SP&T, 14th of April 2005
Camera Coordinate System
C
YC
ZC
XC
M
m
![Page 10: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/10.jpg)
Dr. Oliver Schreer 10Short Course, SP&T, 14th of April 2005
World Coordinate System
C
YC
ZC
XC
Yw
Zw
Xw
(R,t)
M
m
![Page 11: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/11.jpg)
Dr. Oliver Schreer 11Short Course, SP&T, 14th of April 2005
Sensor Coordinate System
C
YC
ZC
XC
M
m
c
y
xf
Yw
Zw
Xw
(R,t)
![Page 12: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/12.jpg)
Dr. Oliver Schreer 12Short Course, SP&T, 14th of April 2005
Image Coordinate System
C
YC
ZC
XC
M
m
v
u
v0
u0
c
y
xf
Yw
Zw
Xw
(R,t)
![Page 13: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/13.jpg)
Dr. Oliver Schreer 13Short Course, SP&T, 14th of April 2005
The perspective projection
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
====
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
1
~~~
11vu
~W
W
W
WWNCNC
C
C
N ZYX
MMMZYX
ss PPDAPAPAPm
Embedding of euklidian3D point in the
projective space PP33
Mapping of the point in theprojective space PP33 to the
projective plane PP22
Division by 3rd componentleads to the point on the affine
plane, the image plane
![Page 14: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/14.jpg)
Dr. Oliver Schreer 14Short Course, SP&T, 14th of April 2005
Non-linear Lens Distortion
radial distortion
distorted undistorted
![Page 15: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/15.jpg)
Dr. Oliver Schreer 15Short Course, SP&T, 14th of April 2005
OutlineCamera model
Epipolar geometry
Analysis of real-time video streams
Camera model
Epipolar geometry
Analysis of real-time video streams
![Page 16: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/16.jpg)
Dr. Oliver Schreer 16Short Course, SP&T, 14th of April 2005
Parallel Stereo Camera System
m1 m2
f f
M
ρ
Bu1
I1
C1
I2
u2 C2
![Page 17: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/17.jpg)
Dr. Oliver Schreer 17Short Course, SP&T, 14th of April 2005
Convergent Stereo Geometry
C2
m2
Π
C1
I1 I2
M
m1
(R,t)
![Page 18: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/18.jpg)
Dr. Oliver Schreer 18Short Course, SP&T, 14th of April 2005
The Epipoles
M
m1
C1 C2
m2
Π
I1 I2
e2e1
(R,t)
![Page 19: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/19.jpg)
Dr. Oliver Schreer 19Short Course, SP&T, 14th of April 2005
The Epipolar Lines
M
m1
C1 C2
m2
l1 l2
Π
I1 I2
e2e1
(R,t)
![Page 20: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/20.jpg)
Dr. Oliver Schreer 20Short Course, SP&T, 14th of April 2005
Convergent Stereo View and Epipolar Lines
![Page 21: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/21.jpg)
Dr. Oliver Schreer 21Short Course, SP&T, 14th of April 2005
Thank you for your attention in the first part!
Coffee break
![Page 22: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/22.jpg)
Dr. Oliver Schreer 22Short Course, SP&T, 14th of April 2005
OutlineCamera model
Epipolar geometry
Analysis of real-time video streams
![Page 23: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/23.jpg)
Dr. Oliver Schreer 23Short Course, SP&T, 14th of April 2005
The 3D Video Processing Chain
AnalysisAnalysis SynthesisSynthesis
TransmissionDecoding
Analysis•Segmentation•Tracking•Rectification•Depth analysis (stereo)
Encoding(MPEG-4, H.264)
Synthesis
Composition of thescene
Analysis•Segmentation•Tracking
![Page 24: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/24.jpg)
Dr. Oliver Schreer 24Short Course, SP&T, 14th of April 2005
Segmentation and Shadow Detection
change detection shadow detection in the YUV-space
background adaptation
![Page 25: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/25.jpg)
Dr. Oliver Schreer 25Short Course, SP&T, 14th of April 2005
Shadow in Colour SpacesColour in Human Perception:
huesaturationintensity
B
R
G
RGB
V
S H
HSV
YUV
U
V
Y
What is a shadow?Change of intensityChange of saturationConstant hue
Problems:YUV is common video formatColour space transf.Real-time constraints
![Page 26: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/26.jpg)
Dr. Oliver Schreer 26Short Course, SP&T, 14th of April 2005
Shadowed Areas in the YUV Space
Y/V-plane (blue)Y/U-plane (blue)
Y/V-plane (green)Y/U-plane (green)
![Page 27: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/27.jpg)
Dr. Oliver Schreer 27Short Course, SP&T, 14th of April 2005
Solution: Shadow Detection in YUV
U
V
Y
P
α
Approximation of Colour:
Shadow detection:Change of saturation and intensityConstant Hue
Influence of shadow to YUV-value:
Saturation: Distance to originHue: vector angle in UV-plane
Linear shift towards originConstant angle in the U/V-plane
![Page 28: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/28.jpg)
Dr. Oliver Schreer 28Short Course, SP&T, 14th of April 2005
Fast Decision SchemeInvestigate segmented object
Y smaller than BG ? Angle difference withinboundary ?
U
V
Y
Pshadowed
PBG
U
V
Pshadowed
PBG
α2
α1
![Page 29: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/29.jpg)
Dr. Oliver Schreer 29Short Course, SP&T, 14th of April 2005
Experimental Results
Pure Segmentation Shadow Detection
![Page 30: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/30.jpg)
Dr. Oliver Schreer 30Short Course, SP&T, 14th of April 2005
Texture Adaptive FeatheringProblems: corona, flickering
![Page 31: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/31.jpg)
Dr. Oliver Schreer 31Short Course, SP&T, 14th of April 2005
Transparency Masks
Composition into new scene
I = α⋅F + (1-α)⋅B
Segmentation
binary transparent
![Page 32: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/32.jpg)
Dr. Oliver Schreer 32Short Course, SP&T, 14th of April 2005
Usage of Delta Band
delta bandbinary maskoriginal image withcontour
![Page 33: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/33.jpg)
Dr. Oliver Schreer 33Short Course, SP&T, 14th of April 2005
Comparative Results
![Page 34: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/34.jpg)
Dr. Oliver Schreer 34Short Course, SP&T, 14th of April 2005
Avatar Animation - A Tele-Consulting System
Handsegmentation and Tracking
Filter
Feature Point Tracking Player
Video In
Character
2D Feature Conversion
Viseme ConversionViseme Generator
Audio In
Net
wor
k
Operator
Display
Customer Avatar Animation Parameter
(FAP’s, BAP’s)
Speaker Audio Grabber N
etw
ork
Inte
rface
Net
wor
k In
terfa
ce
Video Analysis
![Page 35: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/35.jpg)
Dr. Oliver Schreer 35Short Course, SP&T, 14th of April 2005
Segmentation and Tracking of HandsSegmentation based on skin colour
Specific colour range in UV-spaceindependent of human race
Bounding Boxes
Tracking of bounding boxesReduction of processing speed
Stabilisation of segmentation
![Page 36: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/36.jpg)
Dr. Oliver Schreer 36Short Course, SP&T, 14th of April 2005
Overview of the Approach
Region GrowingApproach
on high resolutionOriginal image
Subsampling + Tracking
on low resolution
![Page 37: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/37.jpg)
Dr. Oliver Schreer 37Short Course, SP&T, 14th of April 2005
Tracking of Hands in Critical Situations
![Page 38: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/38.jpg)
Dr. Oliver Schreer 38Short Course, SP&T, 14th of April 2005
Facial Feature Tracking
tracking of a few facial features
approximation of head rotation by relative motion of featurescompared to center of gravity of skin pixels
![Page 39: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/39.jpg)
Dr. Oliver Schreer 39Short Course, SP&T, 14th of April 2005
Experimental Result: Reconstructed Nick/Turn Angles
head nick and turn angle
-0.6
-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
1 frame
FAP
valu
e
nick angleturn angle
![Page 40: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/40.jpg)
Dr. Oliver Schreer 40Short Course, SP&T, 14th of April 2005
Animation of an Avatar
![Page 41: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/41.jpg)
Dr. Oliver Schreer 41Short Course, SP&T, 14th of April 2005
ConclusionCamera model provides a sufficient frame work for the imagingprocess of one camera
Epipolar geometry describes the relation between two cameras – theimaging process of two cameras
Advanced segmentation for real-time mixed reality applicationsrequires
Speed and robustness
Shadow detection
Alpha blending
![Page 42: Challenges and solutions for real-time immersive video](https://reader031.vdocuments.us/reader031/viewer/2022020706/61fc840e8d33c02b785e141d/html5/thumbnails/42.jpg)
Dr. Oliver Schreer 42Short Course, SP&T, 14th of April 2005
..... thank you for your attention
END