digital interactive entertainment dr. yangsheng wang professor of institute of automation chinese...
TRANSCRIPT
Digital Interactive Entertainment
Dr. Yangsheng Wang
Professor of Institute of Automation
Chinese Academy of Sciences
http://hci.ia.ac.cn/JointLab/onlinegame-En.htm
Overview ofDigital Entertainment Industry
Digital Entertainment Industry
Year 2008 the market size will be about $100B and major products are TV games、PC games and mobile games.
Microsoft, Sony and Nintendo provide most digital entertainment products.
China has a very big potential market for digital entertainment products.
Internet and Game in China
The total amount of Chinese Internet users is up to 87 millions in 2004, about 25% are online game(OG) users.
OGs are one of major parts of digital entertainment contents in China. Almost half of the OGs on the domestic market mainly come from Korea. Some Chinese game companiers are beginning to develop their own OGs.
PC Stand-alone games are losing their market. Mobile games are developed rapidly, but the income is on
the low level at the present stage. Overseas game companies enter Chinese market via Join
t-venture.
source： CGPA&IDC 2004
Internet Users in China
Online Game Market in China
source： CGPA&IDC 2004
Stand-alone Game Market in China
source： CGPA&IDC 2004
New Research & Applications of Natural Interactive Technologies
Traditional games mainly depend on mouse, keyboard, joystick and special input device.
Game players desire more natural and intelligent interaction methods, including voice, gesture, expression, gaze, and intelligent sensor, etc.
User-oriented multi-model HCI (Human Computer Interaction) technologies has become the hotspot of research and application in many domains.
Natural Interactive Technologies (NIT)
Computer Vision for Computer Games and Graphics Camera is used as a powerful interface device for games by
sensing user’s body position, head orientation, direction of gaze, pointing commands, and gestures.
Developer: Mitsubishi Electric Research Laboratories. Applications: Sega games
NIT Research and Application-I
Vision device
Vision-based Human-Computer Interface for Games By a camera, the player's body movements are projected rig
ht into the game as an unencumbered input and control device.
Developer: Sony Computer Entertainment America Applications: Sony PlayStation2 EyeToy games
NIT Research and Application-II
Camera
Voice Control for Games and Simulations Many of today's games have a lot of keyboard commands th
at can be hard to remember. Speech recognition technologies are used in this area for game control.
Product: Game Commander (Mindmaker) Product: Game Voice (Microsoft SideWinder)
NIT Research and Application-III
Interactive Entertainment Research Group in CASIA
Auto-Construction of a Game Object; Player-Oriented Game Multi-model Human & Computer Intera
ction; Natural Interactive Entertainment Platf
orm.
The Outline of Our Research
Photo-stereo Based Model Reconstruction
Images capture
Normal calculation
Surface reconstruction
flashlight
flashlight flashlight
camera
bracket
control card
Images capture
Images of the surface under different lighting conditions is captured in a very short period, that the register is avoid.
Normal calculation
A photo-stereo method is used to retrieve the surface normal based on the reflect property of the surface.
Surface reconstruction
The relevant height between the points can be integrated from the derivative field.
A pyramid integration method is used to optimize the results
L
qdypdxpzpz )()( 0
2|),(ˆ),(|)ˆ,ˆ( yxpyxpqpF dxdyyxqyxq 2|),(ˆ),(|
Orthogonal images Based Model Reconstruction
Generic Model and its feature points selection
Two images feature points editing
Generic model deformation
Texture merging and mapping
Generic model and its feature points selection
79 feature points according MPEG4
Images feature points editing andtexture merging
frontal face side face
texture merging result
Generic model deformation andtexture mapping
texturegeneric model
model reconstruction result
deforming
Individual game characters
Individual faces give players more fun and immersion.
player A
player B
Face detection
Deformation & Texture mapping
Expression recognition
Expression retargeting & Animation
Video-Based Model Reconstruction and Animation
Capture user’s frontal face with a camera, then deform an existent face model and map the face image to create an individual face model.
Five kind of user’s expressions are recognized real-time to drive the relevant animations.
Auto-Extraction of key points of a face
An RealBoost-based statistical learning is adopted to build a model for each facial point. During extraction, the models are used to locate facial points for each face image.
Deformation & Texture mapping
Origin image filling image
generic model
merging texture
reconstructed individual models
deforming
Expression recognition and retargeting
Expression recognition
Source expression
models
Individual expression
results
surprise anger sad happy fear
A novel approach to real time facial expression recognition using an embedded HMM which uses observation vectors that are composed of DCT results.
Getting into a Game - Change face
Player’s frontal face is captured by a camera.
Player’s face feature points are detected to deform character’s face model.
Player’s face image is used change character’s face texture.
Head detection and pose estimation
Fist detection and motion analysis
Other motion detection and analysis
Video-Based Motion Detection and Analysis
Head detection and pose estimation
front left right up down close far
locating the position of a head
recognizing the pose of the head
The novel scheme based on Markov model fusing appearance, color, and motion information.
Fist detection and motion analysis
prototype DEMO
training samples
Driving a car using your hands and voice
Player’s hands control the car’s direction.
Player’s voice controls the car’s speed and status.
Motion Detection and Analysis
Project video into game
Motion detection
Parameters calculation
Game objects control
Playing a game with your body
Player’s body movement is projected right into a game
The motion detection and analysis results are used to activate objects in a game.
Online-Game Strategic Research Group
Online-Game Strategic Research Group
A meeting of the group
The group includes major digital entertainment companies, universities, research institutes, and digital entertainment association in China.
Write strategic report of digital entertainment for Chinese government.
Establish “China National Center for Digital Entertainment”.
The goal of the Strategic Research Group
Thank you!