i minds2009 future media prof rik van de walle (ibbt mm lab u gent)

Future MEDIA

2

Overview

What is this about?

Future media - evolutions and trends

Some key trends

Resulting research challenges / plans

Achievements so far

a lot of projects: please see IBBT website...

spin-off companies & patents

other

3

Future media – evolutions and trends

Approaching the ‘Zettabyte Era’

by 2012, annual global IP traffic will exceed half a

zettabyte, of which 90% will be video traffic

mobile data traffic will double each year

Super Hi-Vision or UDTV (Ultra High Definition Video)

7680 4320 pixels

Japan plans to deploy

UDTV broadcasting in 2015

requires ca. 600 Mbit/s

with current technologies

4


Emerging high-definition user-generated content

multiple versions of online content (e.g., YouTube)

soon to be consumed on high-resolution displays

need for on-the-fly adaptation

Managing huge content collections

automatic detection of concepts

interactive semantic search

multi-modal analysis and

retrieval algorithms

multimedia exploration systems

5


Recognition of many object classes,

scalable in the number of classes

(both for training and at run-time)

Limit the need for supervision

6


Exploit context information

(scene, other objects, geometry, …)

Couple textual descriptions to image content (nouns-

objects, adjectives-attributes, verbs-actions) and exploit text to provide contextual information


Distributed smart camera systems

multi-camera analysis, embedded and distributed

processing

applications connect to general purpose video analysis

services

applications concentrate on high-level aspects; camera

system handles low-level and access control

privacy issues

8


In the brain regions always connect bi-directionally,

in computer vision recognition all too often is

the endpoint of a one-way bottom-up pipeline.

Recognition has a pivotal role to play in terms of motion

analysis, 3D reconstruction, inpainting, ...

We need recognition to talk to all kinds of

complementary visual processes.

We even contend that a lot of progress

in these other areas could actually come from

recognition, rather than from further refinements along

the current lines of thinking

9


Applications based on recognition

automatic resizing

3D city reconstruction

traffic safety

content-based image retrieval

10


Applications based on recognition

video summarization

virtual editors

surveillance

…

11


HCI - interactive 3D environments

natural and intuitive interaction

techniques for emerging environments

‘serious games’

(education, therapy, ...)

Video surveillance

cameras are being installed virtually everywhere

video analysis as security sensor

intuitive control room applications

combining all event information

12


Mobile and context-sensitive interactive systems

13


Multi-modal interaction in 3D and virtual environments

14


Interactive workspaces

15


User-centered software engineering

16


Computer graphics and computer animation

modeling, rendering and animation

video-based CG and animation

stylized drawn animation

volume rendering and information visualisation

Image based modeling and rendering

for real world objects/subjects/scenes

Physically based illumination simulation for virtual

prototyping Real-time photo-realistic hair

and skin for games and movie special effects

17

In conclusion: some KEY TRENDS

Zillions of data

need for efficient indexing mechanisms

Different platforms

content reformatting

User-generated content, interactivity

semantic-level interaction with video content

multimodal data and multimodal interaction

Uncontrolled environments

increased robustness

better generalization

18

In conclusion: some KEY TRENDS

Towards cognitive image understanding

increase number of categories

reduce level of supervision

integrate cognitive information

in other image processing tasks

global reasoning: relations, context, ...

modular systems

"In 10 years we will have the first systems that can

interpret a never seen photo or video and that can be put into an unknown environment and react intelligently"

19

Resulting research challenges

Advanced video coding

coding techniques for very high resolutions

e.g., characteristics of 16x16 pixels

increasing coding efficiency without increasing

computational complexity (too much)

360p 720p 2360p

20


New coding paradigm:

Distributed Video Coding

mobile video applications

multi-view video entertainment

visual sensor networks

wireless video cameras

wireless low-power surveillance

21


Scalable graphics

MESHGRID

22


Scalable graphics

wavelet-based approach

23


Scalable graphics

move towards real-world applications

design, architecture

education, entertainment

cartography

nedicine, modeling

virtual reality

24


Long-term goal

adaptation and delivery framework for personalized

and immersive interactive multimedia experiences

super HD / 3D

sensory effects

Immersive Multimedia Experience

(“rollercoaster experience”)

Annotation

Retrieval

Universal Multimedia

Experience

Adaptation

Delivery

Universal Multimedia

Access


Content adaptation and transcoding

HDTV

originally acquired

high definition material

SDTV

legacy applications

common pan/scan

editing

(AVID)

Mobile (iPod)

semantically repurposed

original (4x) SR (4x)


Transform low-resolution rubbish into high-resolution

content (first steps…)

27


Multimedia data analysis and feature extraction

texture detection and classification

content-based multimedia

information retrieval

multi-camera video surveillance

object detection and tracking

immersive control rooms


Camera selection:

Making sense of the

information overload


Stitching: making sense of it

captured data

30


Metadata technology and exploitation

semantic web technology

W3C activities on media

annotation

Open Linked Data

linking external

domain knowledge

enable reasoning

31


Metadata technology for media production

metadata is much more than ‘data about media objects’

pursue model-driven product development

better communication: between staff and machines

during acquisition (associating audiovisual material)

for editing and repurposing (automate production tasks)

Computer Graphics

+ Scene Editing and Interaction

Add / remove / clone objects

Change shape / appearance / lighting

Animation

Interaction / Navigation

- Synthetic 3D models

realistic modelling is a painstaking task

Video/film

+ Ease of capture and visualisation

Film with camera

Project on screen

+ Film “language” well understood

- Scene editing difficult, no navigation/interaction as in

CG.


Video-Based Computer Graphics bridges the gap (multi-view video as shape & appearance primitive in CG)


Architectures supporting massive multi-user, multi-platform scalable

networked multimedia (virtual communities, gaming, networked virtual

environments, …)

NVE server(s) IP

34


Game technology

efficient

production tools

(e.g., digital sculpting)

real-time interaction with environment (physics)


Application domains: virtual communities, gaming,

networked virtual environments, …

Research topics:

scalability issues

scalability assessment


Network intelligence

Error resilience

Error concealment


Media-aware

security and authentication

watermarking

fingerprinting

content authentication

Applications

control of quality, legitimacy and authenticity of content

disclosure of digital archives

content distribution control

art authentication and dating

37

38

Achievements - a lot of projects...

HI-MASQUERADE

MUTABLE

IFIP

POKUMON

TELEON

GEISHA

PECMAN

COCOMEDIA

VIDEO Q-SAC

ISYSS

VICATS...

39

Achievements - spin-off companies

Androme

Voice and video over IP solutions

iDTV

computer graphics&animation

networked virtual environments

aQuartic

media streaming environment

metadata handling and exploitation

semantic web technologies (RDF, OWL)

content adaptation software platform (‘NinSuna’)

40


Eyetronics

systems & services for 3D scanning

high quality

flexible & efficient scanning

main applications: gaming & film production

GeoAutomation

3D acquisition of structural information in cities

based on mobile mapping (vehicle & cameras)

main application: digital surveying

41


QuESD

sports and video analysis systems

access control systems, including

embedded devices and video streaming solutions

eSaturnus

digital image acquisition & rendering

main applications: supporting surgery & endoscopy

42

Achievements - patents

Patents

active patent policy

while NOT excluding open-source approach!

some numbers:

5 granted

3 pending

43

Achievements - other

Strong participation in standardization

still image coding: JPEG

video coding: MPEG / VCEG / VQEG

semantic web: W3C

A lot of contract research

bilateral

various projects (EU, IWT, ...)

Strong collaboration with a lot of

recognized research partners

Contact info

IBBT Offices

Zuiderpoort Office Park

Gaston Crommenlaan 8 (bus 102)

B-9050 Gent-Ledeberg

Belgium

t: +32 9 331 48 00

f: +32 9 331 48 05

e: [email protected]

Or even better: just talk to one of us during iMinds

44

i minds2009 future media prof rik van de walle (ibbt mm lab u gent)

Documents