![Page 1: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/1.jpg)
HURDLES AND OPPORTUNITIES
TOWARDS ENABLING TRULY
IMMERSIVE HOLOGRAPHIC-TYPE
COMMUNICATIONMaria Torres Vega
DEPARTMENT OF INFORMATION TECHNOLOGY
IDLAB-IMEC
![Page 2: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/2.jpg)
WE HAVE MOVED FROM STATIC TO DYNAMIC CONTENT…
2
![Page 3: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/3.jpg)
… AND ARE MOVING FAST TOWARDS IMMERSIVE MEDIA
3
![Page 4: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/4.jpg)
VIRTUAL REALITY (VR) COMES IN MANY FLAVORS
4
![Page 5: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/5.jpg)
MOST VR USE CASES CONSIDER THREE DEGREES OF FREEDOM (3DOF)
5
![Page 6: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/6.jpg)
SOME CONSIDER FIVE DEGREES OF FREEDOM (BUT NEVER CATCH ON)
6
![Page 7: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/7.jpg)
OTHERS ENVISION AN ENTIRELY NEW WAY OF DOING THINGS
7
• 6 Degrees of Freedom
• True Immersiveness:
• Real-time interactivity
• Ultra-high quality to avoid
cybersickness
![Page 8: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/8.jpg)
THE TRULY IMMERSIVE
HOLOGRAPHIC TYPE COMMUNICATION
END-TO-END SYSTEM
8
![Page 9: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/9.jpg)
Holographic Edge Unit 1 Holographic Edge Unit 2 Network
Router
Packet
gatewayService
gateway
Edge
computing
Engine
Edge
computing
Engine
Service
gateway
Packet
gatewayHTC Server HTC Client
HTC Client HTC Server
THE HOLOGRAPHIC TRANSMISSION CHAIN
![Page 10: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/10.jpg)
HOLOGRAPHIC CAMERAS AND RECEIVERS
10
Head Mounted Devices
(HMD)
Holographic Displays
Holographic Projectors and Receivers Light-field Cameras
RGB-Depth Cameras 360-Degree Cameras
Holographic Transmitters Holographic Receivers
![Page 11: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/11.jpg)
HOLOGRAPHIC RENDERING:LIGHT-FIELD VS POINT CLOUDS
11
Image-based solutions: Light-field Videos
Wijnants et al., 2018
• Large set of cameras: Images from different
angles and views
• Massive amount of data:
30° viewing, 10° tilt = 3300 separate images
Volumetric-based solutions: Point clouds
• Less cameras, more optimized
• Sets of 3D volume pixels, or voxels
![Page 12: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/12.jpg)
DYNAMIC POINT CLOUD OBJECTS
12
![Page 13: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/13.jpg)
WHAT DOES TRUE IMMERSIVENESS
REQUIRE FROM THE NETWORK
INFRASTRUCTURES?
13
![Page 14: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/14.jpg)
1. ULTRA-HIGH BANDWIDTHS REQUIREMENT
14
HD
(1080p)
360°(4K)
360° (16K)
Hologram
(Point cloud)
Hologram
(Light field)
1-5Mbps 15-25Mbps 100Gbps-2Tbps100-500Mbps 0.5-2Gbps
![Page 15: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/15.jpg)
Holographic Edge Unit 1 Holographic Edge Unit 2 Network
Router
Packet
gatewayService
gateway
Edge
computing
Engine
Edge
computing
Engine
Service
gateway
Packet
gatewayHTC Server HTC Client
HTC Client HTC Server
2. ULTRA-LOW LATENCY VS ULTRA-HIGH RELIABILITY
Visual feed (max 20ms)
Audio feed (max 100ms)
Tactile feed (max 1ms)
Server Processing
delay
Client Processing
delay
Network Transmission
delay
![Page 16: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/16.jpg)
3. SYNCHRONIZATION OF STREAMS
16
Synchronized Desynchronized
![Page 17: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/17.jpg)
HOW TO ACHIEVE TRULY IMMERSIVE
HOLOGRAPHIC TYPE COMM.?
A CROSS-LAYER APPROACH
17
![Page 18: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/18.jpg)
TRULY IMMERSIVE HTC: A CROSS-LAYER APPROACH
18
End-user Optimizations
Over-The-Top & Transport Optimizations
Novel Network Architectures
HEU 1 Client HEU 2 Server
![Page 19: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/19.jpg)
A CROSS-LAYER APPROACH: END-USER
19
HEU 1 Client HEU 2 Server
End-user Optimizations
![Page 20: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/20.jpg)
END-USER OPTIMIZATONS
20
Client-side Optimizations
Visual feed
Tactile feed • Continuous
• Real-time
• accurate
Audio feed
1.Viewport Prediction
2.Synch. of
feeds
3.QoS/QoE
assessment
Server-side Optimizations
Viewpoint info
& Quality
Representations
Tile
s
3D tiling and Multiple
representations
![Page 21: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/21.jpg)
A CROSS-LAYER APPROACH: TRANSPORT
21
Viewport prediction, sensor
synchronization and QoS/QoE3D Spatial Segmentation &
Multi-dimensional compression
HEU 1 Client HEU 2 Server
Over-The-Top & Transport Optimizations
![Page 22: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/22.jpg)
22
CURRENT VIDEO TRANSMISSION: QUALITY VS DELAYHTTP ADAPTIVE STREAMING: QUALITY OPTIMIZATION
RTP/UDP STREAMING: LATENCY OPTIMIZATION
Time
Bitrate
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
Tile
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
Time
Bitrate
11111111
111
111
2
2
2
11
1
3
33
2
2
111
11
2 2
2
111
11
2
Bandwidth
2
INTERNET
Video Web
Server
Synch.
ClientsWebRTC & QUIC:
Browse-based
real time
streaming
RTP/UDP
streaming
INTERNET
☺ Quality and Bandwidth
optimization
Segmentation: no life
Processing, buffering
and protocol overhead: no
real-time
☺ Latency optimization
Very limited quality
control -> problem for
HTC
Very low resilience
(packet loss prone)
Is it possible to get the best of both worlds?
![Page 23: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/23.jpg)
Holographic Edge Unit 1 Holographic Edge Unit 2 Network
Router
Packet
gatewayService
gateway
Edge
computing
Engine
Edge
computing
Engine
Service
gateway
Packet
gatewayHTC Server HTC Client
HTC Client HTC Server
OVER THE TOP & TRANSPORT OPTIMIZATIONS
Low latency transport (combining advantages of TCP and UDP) (QUIC over HTTP, for example)
More intelligent buffering
techniques
(window based)
Distributed
rendering and
encoding
Smarter retransmission
techniques
(HTTP push)
![Page 24: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/24.jpg)
TRULY IMMERSIVE HTC: A CROSS-LAYER APPROACH
24
Low-Latency quality aware OTT protocols
Distributed rendering
Smart
Re-transmissionsNovel Buffering
Techniques
HEU 1 Client HEU 2 Server
Viewport prediction, sensor
synchronization and QoS/QoE3D Spatial Segmentation &
Multi-dimensional compression
Novel Network Architectures
![Page 25: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/25.jpg)
TOWARDS FULLY DECENTRALIZED NETWORKS…
25
Current network
infrastructure
Software Defined
networksHierarchically
decentralized SDN
SDN with logically
(&completely)
decentralized controller
![Page 26: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/26.jpg)
FLOW SET-UP LATENCY ANALYSIS FOR DIFFERENT SDN ARCHITECTURES
26
0
2
4
6
8
10
12
14
16
18
20
Centralized Flat Distributed HierarchicallyDistributed
Completely DistributedAvg.
Flo
w S
etu
p L
ate
ncy (
ms)
SDN Control Plane Architectures
Average Flow Setup Latency Completely distributed architectures
have the lowest flow setup delay!
But there is still room for
improvement
![Page 27: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/27.jpg)
CONCLUSIONS
27
Low-Latency quality aware OTT protocols
Distributed rendering
Smart
Re-transmissionsNovel Buffering
Techniques
HEU 1 Client HEU 2 Server
Viewport prediction, sensor
synchronization and QoS/QoE3D Spatial Segmentation &
Multi-dimensional compression
Distributed Virtualized Novel Network Architectures
What are the network challenges to enable truly immersive HTC?
1. Very High Throughput (> 100Gbps)
2. Very low end-to-end Latency:
1. Visual < 20ms
2. Audio < 100ms
3. Tactile < 1ms
3. Perfect synchronization of flows
How can truly immersive HTC be achieved?
A cross-layer approach:
1. End-user: synchronization &prediction + 3D tiling
2. Transport protocol optimizations
3. Novel virtualized distributed architectures
![Page 28: DEPARTMENT OF INFORMATION TECHNOLOGY IDLAB-IMEC …€¦ · Video Web Server Synch. WebRTC & QUIC: Clients Browse-based real time streaming RTP/UDP streaming INTERNET ☺Quality and](https://reader034.vdocuments.us/reader034/viewer/2022050122/5f52856456837d5f795baff2/html5/thumbnails/28.jpg)
28
Thank you for your attention!
Any questions or comments?