03 about imiimi.ntu.edu.sg/about_imi/documents/aar/imi_annual... · imi is an incubator of...
TRANSCRIPT
1
2
03 About IMI
03 IMI Missions
07 Research @ IMI
17 PhD Programme in New Media
33 International Events organized @ IMI
41 Industrial Collaboration
45 BeingThere Centre (BTC)
61 Publications
3
About IMI
IMI Missions
To develop cutting-edge new media research within
IMI.
To lead the international collaboration BeingThere
Centre on 3D Telepresence at NTU.
To create synergy and interactions with NTU schools
to empower New Media interdisciplinary research
through integrated projects and joint PhD
supervisions.
To promote and develop industrial applications in
Singapore and elsewhere in the world.
Dissemination& IndustrialApplications
InterdisciplinaryResearchProjects
BeingThereCentre
IMI Core Research
IMI is an incubator of multidisciplinary, cutting-edge and media-related research ideas. It is directed by Professor Nadia Magnenat
Thalmann who is a world pioneer in Interactive Digital Media (IDM) research, particularly in all aspects of Virtual Humans simulation. IMI
strives to build a strong reputation as a global leading IDM hub. In response to Singapore’s national priority to develop IDM as a strategic
R&D area, IMI aspires to facilitate and promote cross-disciplinary collaboration in media research at NTU. At the same time, it also seeks
to provide a one-stop interface to the outside community by forging partnerships with the Singapore government, industry players, and
other international entities. Another important role of IMI is to enrich the educational experience of students by exposing them to the
multifaceted field of IDM.
The Institute is dedicated to creating an environment where technology and creativity can coexist and develop. The general objective of
IMI core research is to create true synergies between the real and the virtual worlds and to develop interactions between real people and
virtual or artificial creatures like virtual humans, virtual animals and robots in a way of creating a real social relationship.
IMI attracts talent researchers from over 15 countries working in this organization.
4
“...This research institute is dedicated to creating an environment where technology and creativity
can coexist and develop.”
NTU President and IMI Founding Chairman
Professor Bertil ANDERSSON
5
“IMI strives to be an incubator of multidisciplinary cutting edge media related research ideas and
establish Singapore as a key player at the forefront of the global interactive digital media
innovation.”
Director, Institute for Media Innovation
Professor Nadia MAGNENAT THALMANN
6
IMI Core Group
1st row from the left: Mr RIVERA Xerxes Marlon Jr. N, Mr Qui C.T. TRAN, Mr Derek HO, Mr KOH Kian Hua, Dr ZHANG Zhijun, Dr Aryel
BECK, Dr XIAO Yang, Ms KHIN Yaminn
2nd row from the left: Prof Daniel THALMANN, Dr Frank Yunqing GUAN, Ms Jess LIM, Mr Roman VLASOV, Prof Nadia MAGNENAT
THALMANN, Mr William LAI, Dr Samuel LEMERCIER, Dr Zerrin YUMAK, Dr Jun LEE
7
8
Core Research Areas
The general objective of IMI Core Research framework is to create true synergies between the real and the virtual worlds. This means the
development of natural interaction between real people and virtual or artificial creatures like virtual humans, virtual animals and robots in
a way of creating a real social relationship. IMI core research areas include 3D Telepresence, Virtual Humans and Social Robots, 3D
Serious Games, 3D Crowd Simulation, 3D Immersion, 3D Medical Simulation and 3D Fashion.
9
Virtual humans are social agents in the virtual world, while social robots belong to the real, physical world. They are meant to interact with
each other as well as with real humans. In order to create believable and long-lasting relationships, they must be empowered with
personality and memory, and be able to express artificial emotions and moods. To achieve autonomous behaviour, they must be aware of
their social environment, understand what is going on around them, who is doing what, what are the relationships between people, and
how humans feel.
Virtual Humans and Social Robots
Nadine is one of the most realistic female humanoid robots around the
world and is the first to be modelled after a western female, Prof Nadia
Magnenat Thalmann. It has been constructed by Kokoro’s Actroid
technology. This robot has a realistic artificial skin and is able to display
very realistic facial and body expressions. It also has a controllable
upper-body capable of realistic human movements. Nadine has a total of
26 DoF for facial expressions and upper body movements. Nadine can
speak, display emotions, understand some situations and express natural
gestures. This makes it ideal for the study of Human-Robot Interaction.
10
IMI is working together with MIRALab at the University of Geneva on a Virtual Try On application which combines virtual models, virtual
fashion and garment simulation technologies into a single interactive application. Research has been led on fast physics based
simulation models and interactive intuitive interface. In the application, the user has the possibility to dress a virtual mannequin by
choosing a garment from a selection of clothes. The garment will automatically adapt to the 3D mannequin shape by means of a
physical simulation. Through an easy to use touch-screen manipulation, the user can interactively change the fabric colour, fabric types,
modify the design and select other options which will be immediately visualized. Once the mannequin has been dressed, the user can
see her walk around in the garments he/she selected. Knowing that the creation of a real dress needs less than one week, this rapid
prototyping technology will dramatically shorten the time to design clothes according to the customers’ taste.
3D Fashion
11
Crowd simulation is the process of simulating the movement of a large number of virtual humans or characters. Virtual cities have
become common in video games and other virtual applications. However, most of them are uninhabited or with only a few people.
Benefiting from crowd simulation technology, it becomes possible that large-scale virtual environments are populated with up to tens of
thousands of virtual humans. Research in crowd simulation has been widely explored in a number of fields such as computer graphics,
civil engineering, sociology and robotics. Crowds have wide applications for both real-time and non-real-time simulations. Virtual crowds
are used to simulate epic battles in the movie industry. Crowds are exploited to populate virtual worlds in computer games. In safety
training, trainees can participate in the evacuation scenarios to practice controlling crowds in emergency situations like earthquake, fire,
flood, etc.
The research of crowd simulation in IMI focuses on the interaction and immersion of the virtual crowds. The immersive 3D display
system has been successfully built, which has great potential for the training and evaluation of emergency evacuation and other real-
time applications of crowd simulation with interaction.
3D Crowd Simulation
12
3D Serious Games
We develop educational games for learning and training purpose with the aid of virtual
tutors or virtual trainers. This research is cross-school and interdisciplinary with inputs
from School of Mechanical & Aerospace Engineering, School of Computer
Engineering, National Institute of Education, School of Art, Design and Media. The
research program also partners with local companies including Underwater World
Singapore, PEC Ltd and AWWA Special School. Part of the projects under the
program received funding support from industry.
IMI is working on 3D medical simulation to develop novel 3D interactions and
immersive techniques for medical simulations. The project will allow medical students
in their early training to access realistic information (anatomic and medical) in a
manner of low cost, fidelity and easy to repeat. They also can learn how to solve
problems in medical emergencies e.g. recurrent Obstetric Management
Mistakes(OMM). With the collaboration of the Lee Kong Chian School of Medicine,
we apply various surgical simulations such as a baby birth simulation. Students can
have surgical training using virtual patients with different parameters to perform
surgical operations using interactive devices (e.g., cybergloves) and predict optimal
surgical outcome. This study is very interdisplinary as it implies medicine, biology,
visualisation and interaction techniques. New Media is essential to this research as it
allows visualizing realistic medical simulation with proper user interactions.
3D Medical Simulation
13
Professors Associated to IMI through Interdisciplinary Seed Projects
Assoc Prof HE
Ying, SCE
Asst Prof Ina
Conradi CHAVEZ,
ADM
Asst Prof Justin
DAUWELS, EEE
Asst Prof YUAN
Junsong, EEE
Prof Daniel
THALMANN, IMI/SCE
Assoc Prof CAI
Yiyu, MAE
14
Interdisciplinary Seed Projects
Integrating a User within a Group or a Crowd using Computer Vision and
Synthetic Vision
Period of Funding: 1 Dec 2012 - 30 Jun 2014
PI: Asst Prof YUAN Junsong (EEE)
Co-PI: Prof Daniel THALMANN (SCE)
The project consists in integrating smoothly a user (avatar) inside a moving Virtual
Group in front of a large 3D screen (or if possible in the IMI Immersive room). The
Virtual Group will be composed of autonomous Virtual Humans (agents). Gestures
and actions of the user will be analysed using Computer Vision, e.g. regular and depth
cameras. One original aspect of the project is to combine the classical Computer
Vision with Synthetic Vision as first introduced by Renault et al [[1]]. Agents will be
able to navigate and avoid collisions using the information coming from a camera
(Computer Vision) [2,3,4] and using a synthetic vision rendered from their point-of-
view.
3D Interactive Visualization of the Human Body in an Immersive Situation
Period of Funding: 1 Mar 2013 - 28 Feb 2014
PI: Assoc Prof CAI Yiyu (MAE)
Co-PI: Prof Nadia MAGNENAT THALMANN (IMI)
Clinical Collaborator: Dr TEO Yee Hong (TTSH)
International Collaborator: Prof Franz-Erich WOLTER (Leibniz University Hannover,
Germany)
This project aims to develop novel 3D interactive and immersive technique for medical
education application. The project if developed will allow medical students in their
early learning journey to access realistic information (anatomic and medical) in a
manner of low cost, fidelity and easy to repeat. Based on this project, we will apply to
various surgical simulations such as obstetric management and knee recovery after
orthopaedic surgery. Students can have surgical training using virtual patients with
different parameters to perform surgical operation using interactive devices (e.g.,
cybergloves) and predict optimal surgical outcome.
15
Robust and Accurate Face Segmentation and Expression Transfer for
Autonomous Avatars
Period of Funding: 1 Jul 2012 - 28 Feb 2014
PI: Assoc Prof HE Ying (SCE)
We will be using a completely new paradigm for 3D facial expression acquisition and
retargeting. It envisions a suite of new algorithms and tools that allow users to acquire
analyze process and visualize 3D facial expressions more easily and effectively than
conventional methods. It will provide high resolution facial motion data without
requiring the subject to wear cumbersome suits or tracking devices and markers and
different expressions of the same subject can be mapped to the same canonical 2D
domains. As a result, the captured data are stored in Geometry Video, a highly
compact representation that can be easily visualized and steamed. Therefore, the
captured data can be widely applied, with the potential to revolutionize a variety of
real-world applications in the entertainment industry, biomechanics, mass
communication, medical applications, etc.
Interdisciplinary Seed Projects
Virtual Pink Dolphins – Phase II
Period of Funding: 1 Oct 2010 - 31 Jan 2014
PI: Assoc Prof CAI Yiyu (MAE)
Co-PIs: Asst Prof CHIA Kok Hwee (NIE)
Prof Daniel THALMANN (IMI)
Assoc Prof ZHENG Jianmin (SCE)
Prof Nadia MAGNENAT THALMANN (IMI)
Mr Norman Kiak Nam KEE (NIE)
Asst Prof Kenneth FEINSTEIN (ADM)
This project is interested to investigate the modeling, simulation and visualization of
virtual pink dolphins for the purpose to design serious games to replace the real ones
for applications in special education, therapy, and entertainment. Virtual Reality
technology will be studied aiming to equip the virtual pink dolphins with multi-sensorial
functions (visual, audio, and tactile) for human-dolphin communication. Special efforts
are being made to study the virtual pink dolphin assisted therapy (VPDAT) and
human-mediated learning for Autism Spectrum Disorder (ASD) children.
16
Interdisciplinary Seed Projects
Monitoring of Group Discussions: A Socio-engineering Approach
Period of Funding: 1 Dec 2011 - 31 Dec 2013
PI: Asst Prof Justin DAUWELS (EEE)
Co-PIs: Prof Nadia MAGNENAT THALMANN (IMI)
Prof Daniel THALMANN (IMI)
Asst Prof Martin CONSTABLE (ADM)
Human group interaction is very complex and rich. Monitoring any group can be done
using non-verbal (facial expressions, body languages, gestures, etc ) or/and verbal
(voice tone, voice pitch,...etc) signals. Despite the impressive advancements in video-
audio signal processing, analyzing different aspects of human interaction as empathy,
hostility, (dis-)agreement, flirting, dominance, superiority, inferiority, etc. remains very
challenging. Investigating these aspects is crucial for building intelligent robots that
can participate in conversations in a natural way, or to fulfill certain social
responsibilities in our life. As our ultimate goal is to improve the current computer
systems and social robots that suffer from a lack of social skills, we will explore the
social signals and social behaviours, including social interactions (like turn taking and
backchannelling), social attitude (like alliance), and social relations/ roles to build more
socially intelligent model for robots.
Unframed
Period of Funding: 15 Aug 2011 - 31 Dec 2013
PI: Asst Prof Ina Conradi CHAVEZ (ADM)
Co-PI: Asst Prof Mark Joseph CHAVEZ (ADM)
The project goal is to use IMI extraordinary creative platform (The Immersive Room)
as a space to author new media art experimentation. The works will explore surreal,
abstract and realistic way of painting using medium of 3D stereo animation.
17
EEE29%
MAE13%
HSS3%
SCE47%
WKWSCI8%
18
PhD Programme in New Media
“IMI offers doctoral students a vibrant multidisciplinary research environment. In particular, they have
an opportunity to work in the area of cutting-edge interactive 3D simulation with top equipment as the
IMI Immersive Room and real-time motion capture in a multidisciplinary team. Through their personal
work, seminars, courses, and interaction with research experts, PhD candidates are prepared for
research positions in leading academic institutions as well as private and public organizations.”
The PhD Program, along with the Interdisciplinary Graduate
School (IGS) is at the core of IMI's effort to promote New Media.
Playing a crucial part in ensuring the future of New Media in
Singapore, IMI aims to educate the next generation of scientists
and technologists and encourage them to share and benefit from
their knowledge of engineering, art, design, business,
educational and behavioral research. The 38 PhD students
pursuing their PhD study in IMI come from different NTU schools
and are specialized in a wide spectrum of research disciplines.
Professor Daniel THALMANN
PhD Coordinator
Distribution of IMI PhD Students jointly supervised with NTU schools
19
Congratulations to Dr Pravin KAKAR, 1st PhD Graduate from IMI
“I am really feeling privileged to be a part of this institute which has really enabled my research and I hope that IMI scales greater high
along with times and all the best for the future.”
Dr Pravin KAKAR
20
PhD Interdisciplinary Seminars
11 April 2013
Presented by
Dr Jingwen DAI
Dr ZHANG Zhijun
Subbarao Nikhil NARAYAN
26 September 2013
Presented by
HOU Junhui
FANG Hui
LIANG Hui
26 November 2013
Presented by
Claudia KUSTER
CHENG Peng
ZHANG Yuzhe
Shakeel AHMAD
29 October 2013
Presented by
DONG Lu
GUO Guibing
TAN Jiat Chow
LI Qiaohong
21
PhD Interdisciplinary Seminars
12 March 2013
Presented by
LAI Danbo
LI Bingbing
Jaroslaw Slawomir
KOCHANOWICZ
25 February 2013
Presented by
EE Changshun Andrew
Yasir TAHIR
CHEN Yong
15 January 2013
Presented by
Dr LIU Gengdai
DONG Lu
Athirai Aravazhi IRRISSAPPANE
GU Yuanlong William
22
Assoc Prof ANG
Pei-Hui, Rebecca,
HSS
Assoc Prof CAI
Jianfei, SCE
Assoc Prof CAI
Yiyu, MAE
Assoc Prof CHAN
Kap Luk, EEE
Assoc Prof CHAU
Lap Pui, EEE
Assoc Prof CHEN
I-Ming, MAE
Asst Prof Mark
Joseph CHAVEZ,
ADM
Assoc Prof CHEN
Shen-Hsing,
Annabel, HSS
Asst Prof CONG
Gao, SCE
Asst Prof Martin
CONSTABLE,
ADM
Asst Prof Justin
DAUWELS, EEE
Assoc Prof HE
Ying, SCE
Asst Prof HU
Guoqiang, EEE
Asst Prof JUNG
Younbo, WKWSCI
Asst Prof Andy
KHONG Wai
Hoong, EEE
Prof KOH Soo
Ngee, EEE
Assoc Prof Vladimir
Vladimirovich
KULISH, MAE
IMI PhD Supervisors
Asst Prof FU
Chi Wing, SCE
Asst Prof CHEW
Lock Yue, SPMS
Asst Prof Kenneth
FEINSTEIN, ADM
23
Prof LUKE Kang
Kwong Kapathy,
HSS
Assoc Prof May
Oo LWIN,
WKWSCI
Assoc Prof Pina
MARZILIANO,
EEE
Assoc Prof MIAO
Chun Yan, SCE
Assoc Prof SEET
Gim Lee, Gerald,
MAE
Assoc Prof Alexei
SOURIN, SCE
Dr OLGA
Sourina, EEE
Assoc Prof TAN
Ah Hwee, SCE
Assoc Prof THENG
Yin Leng, WKWSCI
Assoc Prof WANG
Libo, EEE
Asst Prof YUAN
Junsong, EEE
Asst Prof
ZHANG Jie, SCE
Asst Prof ZHANG
Wei, SCE
Assoc Prof ZHENG
Jianmin, SCE
Assoc Prof SOON
Ing Yann, EEE
IMI PhD Supervisors
Prof Daniel
THALMANN,
IMI/SCE
Assoc Prof James
Patrick WILLIAMS,
HSS
Assoc Prof ZHU
Ce, EEE
Assoc Prof LIN
Feng, SCE
Assoc Prof LIN
Weisi, SCE
24
Current PhD Projects
Vision Based Optimal Path Formation Control and Obstacle Avoidance for Multiple Mobile
Robots
Student: Shakeel AHMAD
Supervisor: Asst Prof HU Guoqiang (EEE)
Co-supervisor: Prof Nadia MAGNENAT THALMANN (IMI)
The optimal path control and obstacle avoidance problem of vision-guided mobile robot with dynamical
and sensing constraints will first be investigated.
Secure Multi-modal Interface with Augmented Reality
Student: DINH Quang Huy
Supervisor: Assoc Prof SEET Gim Lee, Gerald (MAE)
Co-supervisor: Prof Nadia MAGNENAT THALMANN (IMI)
Mentor: Assoc Prof LIN Weisi (SCE)
Memetic Stigmergy: A Multi-method Approach in Studying Decentralized Loci across Social
Networks
Student: ENG Weiwen Herbert
Supervisor: Assoc Prof Patrick Williams (HSS)
Co-supervisor: Asst Prof Chew Lock Yue (SPMS)
Mentor: Asst Prof Kenneth Feinstein (ADM)
Simulation and Automatic Navigation of an Intravascular System
Student: CHEN Yong
Supervisor: Assoc Prof CAI Yiyu (MAE)
Co-supervisor: Prof Daniel THALMANN (SCE)
To make the vascular surgery easier and more efficient, the project aims to design an automatic
navigation system for catheter manipulation. The system consists of catheter control, navigation and
simulation. The coordinates of the catheter tip is obtained by an embedded sensor, and the whole
system works in real time.
25
Current PhD Projects
The Role of Embodiment and Presence on the Development of Leadership Capabilities in
Massively Multiplayer Online Role-playing Games
Student: EE Changshun Andrew
Supervisor: Asst Prof JUNG Younbo (WKWSCI)
Co-supervisor: Prof Nadia MAGNENAT THALMANN (IMI)
Given the advent and prevalent use of virtual environments in New Media, such as in social media (e.g.
Second Life), users – especially the young and adolescent – may acquire and master skills essential for
proficient navigation of these environments. Specifically, I am interested in the role of virtual
embodiment and presence on leadership development among users of these virtual environments.
Study of Nonverbal Communication in Telepresence Robot
Student: GU Yuanlong William
Supervisor: Assoc Prof SEET Gim Lee, Gerald (MAE)
Co-supervisor: Prof Nadia MAGNENAT THALMANN (IMI)
The research topic focuses on the use of nonverbal communication to improve the interactive
experience in human-robot interaction (HRI). The objective is to improve the interpersonal relation
between the system and interactant, by showing emotion, degree of involvement and etc, through the
mode of nonverbal communication.
Three Dimensional Content Processing and Delivery over Communication Network
Student: HOU Junhui
Supervisor: Assoc Prof CHAU Lap-Pui (EEE)
Co-supervisor: Prof Nadia MAGNENAT THALMANN (IMI)
This project aims to improve the quality and speed of real-time 3D processing and transmission. The
results obtained from this project would be useful for areas requiring interactive 3D visual output, e.g.
computer games, and virtual reality.
Developing a Testbed for Evaluating Trust and Reputation Systems in E-commerce
Environment
Student: Athirai Aravazhi IRRISSAPPANE
Supervisor: Asst Prof ZHANG Jie (SCE)
Co-supervisor: Prof Nadia MAGNENAT THALMANN (IMI)
My research focus is to develop a comprehensive testbed to evaluate the robustness of the Trust and
Reputation systems, so that it would be beneficial for the researchers in the field to analyse and
compare their trust models with the purpose of improving their performance.
26
Robotic Enhanced Tele-communication System
Student: LI Bingbing
Supervisor: Assoc Prof CHEN I-Ming (MAE)
Co-supervisor: Prof Nadia MAGNENAT THALMANN (IMI)
This project is aim at developing a robot system which enhance the current telecommunication system
in terms of delivering non-verbal information such as body movement and emotional information.
Quality Assessment in Computer Graphics
Student: LI Qiaohong
Supervisor: Assoc Prof LIN Weisi (SCE)
Co-supervisor: Prof Daniel THALMANN (SCE)
The research aims to combine human vision system and 3D scene information into the objective quality
assessment metrics enable automatic detection and elimination of rendered images of unacceptable
quality.
Current PhD Projects
Automated Thyroid Nodule Detection using 2D Ultrasound Images
Student: Subbarao Nikhil NARAYAN
Supervisor: Assoc Prof Pina MARZILIANO (EEE)
Co-supervisor: Prof Nadia MAGNENAT THALMANN (IMI)
The main aim of this project is to eliminate the subjectivity involved when measuring the volume of a
thyroid gland/nodule and hence reduce the high inter-observer variation during identifying a thyroid
nodule. The prognosis for thyroid carcinoma is based on the maximum dimension of the nodule under
observation hence eliminating subjectivity will significantly improve the chances for accurate diagnosis.
Monitoring of Group Discussions: a Socio-engineering Approach
Student: Yasir TAHIR
Supervisor: Asst Prof Justin DAUWELS (EEE)
Co-supervisor: Prof Daniel THALMANN (SCE)
The research is focused on developing a real time automated system that can monitor group discussion
using audio and video signals, which can provide suggestion in real time to improve group interaction.
The ultimate goal is to improve the current computer systems and social robots that suffer from a lack
of social skills, we will explore the social signals and social behaviours, including social interactions,
social attitude, and social relations/ roles to build more socially intelligent model for robots.
27
Acoustic Source Localization in Room Environment
Student: WU Kai
Supervisor: Asst Prof Andy KHONG W. H. (EEE)
The knowledge of the acoustic source location is of much importance in several applications including
hands-free communication systems, teleconferencing, automatic camera steering, etc. We are currently
working on speech source localization in this reverberant and noisy environment.
Neuroimaging as a Surrogate Marker to Assess Cognitive Substrates in Cognitive Training
Software
Student: TAN Jiat Chow
Supervisor: Assoc Prof THENG Yin Leng (WKWSCI)
Co-supervisor: Assoc Prof Annabel CHEN (HSS)
Brain training software has experienced rapid growth in recent years. These brain training software
make claims of improving cognitive or brain functions and the goal of the proposed work is to evaluate
these claims using neuroscience methods e.g. near infra-red optical imaging (NIRS),
electroencephalography (EEG).
Current PhD Projects
Learning Based Scene Understanding
Student: WANG Anran
Supervisor: Assoc Prof CAI Jianfei (SCE)
Co-supervisor: Asst Prof WU Jianxin (SCE)
The objective of this research is to investigate the existing interactive image segmentation algorithms
and then design a better solution. We aim at designing and developing more intuitive and intelligent
image segmentation algorithms using learning technology. We are particularly interested in how to
combine the interactive image segmentation and learning technology to fast segment a large set of
images.
Distributed and Constrained Conformal Mapping
Student: ZHANG Minqi
Supervisor: Assoc Prof HE Ying (SCE)
The research is focused on creating efficient conformal mapping algorithm by proposing a distributed
and constrained conformal mapping. The distributed feature makes it favorable for the GPU and
become very efficient.
28
Markerless Motion Capture of Human Based on Monocular Camera
Student: BIAN Zhenpeng
Supervisor: Assoc Prof CHAU Lap Pui (EEE)
Co-supervisor: Prof Nadia MAGNENAT THALMANN (IMI)
Motion capture of human is a highly active research area due both to the number of potential
applications and its inherent complexity. The research focuses on real-time markerless motion capture
of human, especially based on monocular camera.
Current PhD Projects
Resolving Data Sparsity and Cold Start in Recommender Systems
Student: GUO Guibing
Supervisor: Asst Prof ZHANG Jie (SCE)
Co-supervisor: Prof Daniel THALMANN (IMI)
The main research problems we desire to address are the two severe issues that original
Collaborative filtering inherently suffers from: data sparsity and cold start. Data sparsity arises from the
phenomenon that users in general rate only a limited number of items; Cold start refers to the difficulty
in bootstrapping the Recommender systems for new users. We aims to solve these issues in both 2D
and 3D environments.
Group Structure and Social Behaviour Simulation Based on Implementation of Psychological
Models of Emotionality and Motivation
Student: Jaroslaw Slawomir KOCHANOWICZ
Supervisor: Assoc Prof TAN Ah Hwee (SCE)
Co-supervisor: Prof Daniel THALMANN (IMI)
This specific research aims to identify the certain significant concepts within psychology and sociology,
to enable implementation of a generic model of autonomous agents with individualised needs, goals,
expression styles, capabilities, group identities, culture, coping strategies, history, world model, private
social map and other characteristics.
Depth Camera Based Hand Gesture Recognition
Student: LIANG Hui
Supervisor: Asst Prof YUAN Junsong (EEE)
Co-Supervisor: Prof Daniel THALMANN (IMI)
The goal of this research is to design and implement algorithms for the recognition of postures and
gestures of a user immersed in a Virtual Environment with Virtual Humans. This research will develop
novel real-time recognition methods with trade-off between accuracy and speed in order to solve the
problem efficiently.
29
Design of a Reputation Mechanism for 3D E-commerce
Student: FANG Hui
Supervisor: Asst Prof ZHANG Jie (SCE)
Co-supervisor: Prof Nadia MAGNENAT THALMANN (IMI)
The research attempts to design a feedback-based reputation mechanism to solve the inherited trust
problems in 3D e-commerce which is getting more feasible and popular. With 3D technology, the
constructing processes such as feedback provision, feedback computation, feedback presentation and
user decision making will be changed.
Modeling and Animating 3D Chinese Water Color Paintings
Student: CHENG Peng
Supervisor: Assoc Prof MIAO Chun Yan (SCE)
Co-supervisor: Prof Nadia MAGNENAT THALMANN (IMI)
The research is focused on simulating the behavior of Chinese ink and analysing various artistic effects
of colour ink diffusion. The study also looks how a scheme presents can be simulated automatically for
their computer generated simulation.
Current PhD Projects
Modeling Virtual Human with Episodic Memory
Student: ZHANG Juzheng
Supervisor: Assoc Prof ZHENG Jianmin (SCE)
Co-supervisor: Prof Nadia MAGNENAT THALMANN (IMI)
The research goal of this project is to refine the existing methods; episode representation, indexing
scheme, matching methods and storage policy , if any, or develop new ones to find a more effective
way to encode, store and retrieve episodic memory.
On Developing High-quality Immersive Rendering for Real-time 3D Teleconference
Student: ZHAO Mengyao
Supervisor: Asst Prof FU Chi-Wing (SCE)
Co-supervisor: Assoc Prof CAI Jianfei (SCE)
This project aims to investigate time-critical methods to analyze and model the input image-based
information on real-time 3D object reconstruction and rendering from real scenes.
30
Automatic 3D Clothing: from 2D Patterns to Garment
Student: ZHANG Yuzhe
Supervisor: Assoc Prof ZHENG Jianmin (SCE)
Co-supervisor: Prof Nadia MAGNENAT THALMANN (IMI)
This project aims to develop algorithms and solutions for automatic modeling and prepositioning of
garment from 2D pattern. And it mainly focuses on the process of converting 2D pattern into 3D
garment and its application in virtual try-on.
3D Simulation of Articulation in Physiological Human
Student: ZHANG Wenjing
Supervisor: Assoc Prof ZHENG Jianmin (SCE)
Co-supervisor: Prof Nadia MAGNENAT THALMANN (IMI)
The research is to model the articulation of human to help with the diagnosis and treatment of certain
disease. The research is mainly divided into two parts to build an accurate geometry model of human
articulation from MRI.
Interactive Image Segmentation
Student: ZHU Hongyuan
Supervisor: Assoc Prof CAI Jianfei (SCE)
Co-supervisor: Prof Nadia MAGNENAT THALMANN (IMI)
Image segmentation plays an important role in many fields of Computer Vision and Computer Graphics.
Interactive image segmentation is an ad-hoc topic in recent years. The main objective of this research
is to improve the segmentation with higher-level semantics provided by user input.
Current PhD Projects
Research on Multimedia Community-based Question Answering
Student: YUAN Quan
Supervisor: Asst Prof CONG Gao (SCE)
Co-supervisor: Prof Nadia MAGNENAT THALMANN (IMI)
This research focuses on Multimedia Community-based Question Answering (MCQA), a new type of
Community-based question-answering (CQA) that enables users to post questions with photos and
video. This research involves with question organization, multimedia question retrieval, expert user
discovery and recommendation, and QA in 3D virtual space.
31
Cyber-learning in Cyberworlds
Student: LAI Danbo
Supervisor: Assoc Prof Alexei SOURIN (SCE)
Co-supervisor: Assoc Prof ZHAO Dongsheng (NIE)
This project proposes to immerse the learners in 3D AI-assisted shared virtual collaborative
environments where mathematical formulas are used for immediate interactive definition and multi-
sensory rendering of geometry, appearance and physical properties of the shapes.
Adolescents' Information Disclosure on Facebook: Impact of Their Parents and Personality
Student: LIU Cong
Supervisor: Assoc Prof ANG Pei-Hui, Rebecca (HSS)
Co-supervisor: Assoc Prof May Oo LWIN (WKWSCI)
Personality traits (i.e., narcissism and social anxiety) and parental mediation on adolescents’
information disclosure on Facebook was studied. Parental mediation influences risky online behaviors
of their children in both direct and indirect ways. Implications on promoting adolescents’ online safety
will be discussed.
Current PhD Projects
Perceptually Based Selective Rendering
Student: DONG Lu
Supervisor: Assoc Prof LIN Weisi (SCE)
Co-supervisor: Assoc Prof ZHU Ce (EEE)
In this project, a perceptual model is to be exploited for image rendering (especially in illumination and
shadow determination). Visual attention, contrast sensitivity and masking will be investigated.
Experiments will be conducted to validate the effectiveness of the proposed method.
CBAN: An Embedded Patient Monitoring Body Area Network
Student: Saurabh RASTOGI
Supervisor: Assoc Prof LIN Feng (SCE)
Co-supervisor: Asst Prof Mark CHAVEZ (ADM)
After examining various approaches for this purpose we will describe the design methodology for the
proposed Body Area Network (BAN) and explore the configuration space looking for the most effective
solution, performance, cost and energy-wise. This BAN solution provides higher levels of user
satisfaction as expected from novel healthcare delivery scenarios (e.g., mobility and customization)
while the architecture in its heart achieves these targets by complex hardware methods (such as
multitasking, parallel processing) and performs beyond the expectations of accurate real time
diagnosis, analysis and monitoring of diseases.
32
Development of Aesthetic Algorithms for Acquired and Generated Digital Image
Student: ZHANG Xiaoyan
Supervisor: Assoc Prof Kap Luk CHAN (EEE)
Co-supervisor: Asst Prof Martin CONSTABLE (ADM)
This project explores the structural models behind visual appeal and the development of algorithms to
apply them to acquired and/or generated digital images. Operations such as post-exposure refocusing,
re-rendering of the scene from an alternate viewpoint, single-image based on matting and composition
and automatic scene segmentation will be explored.
Computer-aided Evaluation of Prosody Evaluation for Language Learning
Student: ZHAO Sixuan
Supervisor: Prof KOH Soo Ngee (EEE)
Co-supervisor: Prof K. K. LUKE (HSS)
In our project, a reference-independent prosody evaluation method using Support Vector Machine
(SVM) is proposed to evaluate the learner’s prosody without reference utterances.
Current PhD Projects
Statistical Machine Learning Techniques with Applications to Image Retrieval
Student: ZHANG Lining
Supervisor: Assoc Prof WANG Libo (EEE)
Co-supervisor: Assoc Prof LIN Weisi (SCE)
In this work, we study several statistical machine learning techniques, e.g., supervised learning, semi-
supervised learning, subspace learning and active learning in a synergistic way to maximize the
effectiveness of a learning task, and thus enhance the performance of the image retrieval systems.
Quantification of Brain Responses for Neurofeedback Game in 3D Virtual Environments
Student: WANG Qiang
Supervisor: Asst Prof OLGA Sourina (EEE)
Co-supervisor: Assoc Prof Vladimir Vladimirovich KULISH (MAE)
The main objective of our project is exploiting new algorithms and new features for the brain state
recognition for real-time applications with as fewer electrodes as possible.
33
34
Real-time Interaction with Nadine Humanoid Robot at Swissnex Singapore
End of 2013 Party │ 5 December 2013
The Ambassador of Switzerland in Singapore, Thomas
Kupfer (2nd from the left)
Presentation of Nadine Humanoid Robot online
Nadine sings a song of Adele Dr. Suzanne Hraba-Renevey, Executive Director of
Swissnex Singapore (2nd from the right)
35
19th ACM Symposium on Virtual Reality Software and Technology
(VRST 2013) │ 6-8 October 2013
The ACM Symposium on Virtual Reality Software and Technology (VRST) is an international forum for the exchange of experience and
knowledge among researchers and developers concerned with virtual reality software and technology. This year, VRST 2013 was
organized by the Institute for Media Innovation (IMI) from 6-8 October 2013 on the Nanyang Technological University (NTU) campus.
VRST 2013 provided an opportunity for VR researchers to interact, share new results, show live demonstrations of their work, and
discuss emerging directions for the field. The event was sponsored by ACM SIGCHI and SIGGRAPH.
Papers have been selected among 89 submissions by an International Program Committee following a strict peer review. 20 full papers
and 18 short papers have been selected for the Conference.
The technical program was composed of 8 sessions: medical VR and Virtual Rehabilitation, Human Factors, 3D Interaction, Navigation,
Avatars and Robots in TelePresence, Simulation, Rendering, and VR Display Technologies.
36
Institute for Media Innovation (IMI)
organized Public Lecture by Prof Petteri
Nisunen and Tommi Grönlund at the NTU
Reality Theatre, Research Techno Plaza on
1 October 2013
The NTU President Professor Bertil Andersson (1st row, 2nd from the right at the public
lecture)
Public Lecture by Prof Petteri Nisunen and Tommi Grönlund │ 1 October 2013
37
Perception and Behaviour of Virtual Humans Workshop │ 10 May 2013
Distinguished Lecture: The e-Documentation of the Past: Risks and Challenges │ 26 September 2013
Distinguished Lecturer: Dr. Marinos Ioannides, Cyprus University of Technology, Digital Heritage Lab, Cyprus
Assoc Prof Miriam Reiner, Technion Touch Lab,
Technion University, Israel
38
In the year 2013, IMI accommodated 34 visits (441 visitors from more than 13 countries).
Ms Anne-Catherine Lyon, State Councillor and Head
of Department of Education and Youth Development,
Canton of Vaud, Switzerland
Delegates from global offices of Temasek Holdings
Dr Lykke Friis and Dr Esben Lunde Larsen, members of
the Danish Parliament, accompanied by Prof Er Meng
Hwa, NTU Vice President (International Affairs)
Visitors to IMI
Mr Lim Chin Siang, Director of IDMPO from MDA
39
Journalists from Nature, The Economist and Le Monde
Delegates from Beijing Institute of Fashion Technology,
China
Final Year Medical students from Imperial College
London
The finalists for Nanyang Assistant Professorship Scheme
Visitors to IMI
40
Prof Franz-Erich WOLTER
Welfenlab
Leibniz University of Hanover
Germany
Assoc Prof Miriam REINER
Technion Touch Lab
Technion University
Israel
Assoc Prof ZHANG Hong
Wuhan University of Science and Technology
China
Prof Hector OROZCO
Autonomous University of Mexico
State
Mexico
Asst Prof Arjan EGGES
Utrecht University
The Netherlands
Sybren A. Stuvel
PhD Student
Utrecht University
The Netherlands
Visiting Professors, Researchers and PhD Students
41
42
PEC Ltd is a specialist engineering group servicing the oil and gas, petrochemical, oil and chemical terminal, and pharmaceutical
industries. Their two core business activities are project works and maintenance services in the region.
IMI and PEC Ltd tied up to develop simulation technology for engineering applications.
NTU-PEC Research Collaboration Project Kickoff Meeting │ 31 January 2013
43
Signing of NTU-NVIDIA Collaboration MOU │ 28 January 2013
The MOU signing ceremony between the Institute for Media Innovation, NTU and NVIDIA was held on 28 January 2013.
Desirous of promoting and developing cooperation in the field of GPU technologies, the MOU aims to foster closer ties between the two
parties and to promote cooperative activities including exchange of information, organization of joint workshops, seminars and
conferences.
44
Industrial Collaboration
Collaboration between IMI and Asian Women’s Welfare Association (AWWA)
Special School
“The collaboration with IMI has benefited both my ASD and Multiple Disabilities students.
It is amazing to see them interacting with the 3D dolphins through the use of hand
signals. Best of all, they were able to transfer that learning, out of the Immersive Room
into their classroom.”
Mrs Ruby SEAH
Principal, AWWA School
Collaboration between IMI and Lee Kong Chian School of Medicine
To explore the use of a highly immersive augmented reality environment (HIARE) to
provide a technological frame-work for uniformity of Phase 3 medical student safe and
controlled exposure to acute medical emergencies.
Virtual Pink Dolphin-assisted Therapy for Children with Autism
Funding Supported by Temasek Trust-funded Singapore Millennium Foundation
Research on autism and development of therapy for autism are of worldwide interest
today. Several studies show dolphin-assisted-therapy (DAT) has positive effect to children-
with-autism in terms of anxiety/stress relief, communication/learning improvement, and
cognitive-performance/self-confidence increase.
45
46
About BeingThere Centre
BeingThere Centre is an International Research Centre for Telepresence and
Telecollaboration mainly based at the Institute for Media Innovation (IMI) in Nanyang
Technological University (NTU), Singapore.
It is funded by Media Development Authority of Singapore with joint effort from ETH
Zurich (Switzerland), UNC Chapel Hill (USA) and NTU (Singapore). The objective of
the BeingThere Centre is to make major technological and systems-level advances
leading to a 3D credible experience of Telepresence in a fitted room using a mobile
platform. The result will be a breakthrough in the quality of interpersonal
communication at a distance allowing for eye contact and proper motion parallax
among a group of users.
BeingThere Centre
(IMI)
Est 15 Dec 2010
NTU
UNCETH
47
“BeingThere Centre based at the Institute for Media Innovation in Nanyang Technological University,
Singapore is a centre of breakthrough in Virtual Reality.”
Chair, BTC Steering Committee
NTU Deputy President and Provost
Professor BOEY Yin Chiang, Freddy
48
BeingThere Centre Steering Committee
Prof Freddy BOEY
Chair of Steering Committee
Deputy President and Provost
NTU, Singapore
Prof Anselmo LASTRA
Chair, Department of Computer Science
UNC Chapel Hill, USA
Mr Chun Cheng YEO
Assistant Chief Executive (Industry)
Executive Director, IDMPO
Media Development Authority,
Singapore
Dr. Silke MEYNS
Head of Team Patents and Licensing
ETH Zurich, Switzerland
49
BeingThere Centre International Advisory Board
Prof Susanto RAHARDJA
Chair of International
Advisory Board
Institute for Infocomm
Research, AStar, Singapore
Dr Mod MARATHE
Cisco Research and
University Relations,
USA
Dr.-Ing Aljoscha SMOLIC
Disney Research, Zurich
Prof Craig GOTSMAN
Technion, Israel
Prof Mel SLATER
University of Barcelona,
Spain
BeingThere Centre Co-Directors
Prof Nadia MAGNENAT THALMANN
Co-director
NTU, Singapore
Prof Markus GROSS
Co-director
ETH Zurich, Switzerland
Prof Henry FUCHS
Co-director
UNC Chapel Hill, USA
50
Midterm Review Panel Members
“The review panel believes the Centre is a great return on investment for MDA … The BeingThere Centre has encouraged
multidisciplinary collaboration at NTU. It brings faculty members and students from Computer Science, Electronic Engineering,
Mechanical Engineering, Social Science, and others, to work together within the same framework. The Centre has made worldwide
Impact of Talent. 6 former research fellows were offered faculty/top company positions in US, Canada, France and China. Clearly,
the Centre has substantially increased the international visibility of NTU. We congratulate the co-directors and members of the
BeingThere Centre for their impressive work.“
- From BeingThere Centre Midterm Review Panel
Dr ZHANG Zhengyou
Chair of Midterm Review Panel
Microsoft Research, Redmond, USA
Prof Michitaka HIROSE
The University of Tokyo, JAPAN
Prof Werner PURGATHOFER
Vienna University of Technology, AUSTRIA
BeingThere Centre Midterm Review │ 11 - 12 November 2013
51
KPI Fulfilment of BeingThere Centre as of 30 September 2013
BeingThere Centre Midterm Review │ 11-12 November 2013
BTC Midterm Review Meeting Chair of Midterm Review Panel trying out
the BTC Project
Midterm Review Panel Meeting with
Steering Committee Members
52
BeingThere Centre Associated Professors
Assoc Prof CAI
Jianfei, SCE
Assoc Prof CAI
Yiyu, MAE
Assoc Prof CHAU
Lap Pui, EEE
Asst Prof FU
Chi Wing, SCE
Asst Prof YUAN
Junsong, EEE
Assoc Prof ZHENG
Jianmin, SCE
Assoc Prof CHAM
Tat Jen, SCE
Assoc Prof CHEN
I-Ming, MAE
Asst Prof Andy
W. H. KHONG,
EEE
Assoc Prof SEET
Gim Lee, Gerald,
MAE
Prof Daniel
THALMANN,
IMI/SCE
Prof SUN
Chengzheng,
SCE
Assoc Prof HE
Ying, SCE
Asst Prof Justin
DAUWELS, EEE
Asst Prof Michael
Donald PATTERSON,
HSS
53
BeingThere Centre Engaged Researchers and PhD Students
NTU Researchers
Dr Zerrin YUMAK
Dr Aryel BECK
Dr Xiao YANG
Dr ZHANG Zhijun
Dr WONG Choon Yue
Dr Jun LEE
Mr BURHAN
Mr DENG Teng
Mr ISKANDARSYAH
Ms PANG Wee Ching
Mr REN Jianfeng
Mr TAN Fuwen
Mr XU Di
Mr Guo Yu
NTU PhD Students
Ms ZHAO Mengyao
Ms WANG Anran
Mr LI Bingbing
Mr GU Yuanlong, William
Mr HOU Junhui
Mr LIANG Hui
Mr WU Kai
Mr ZHANG Minqi
Mr ZHANG Juzheng
UNC Researchers
Prof Miriam REINER
Mr Andrei STATE
Dr Renjie CHEN
Dr Jingwen DAI
UNC PhD Students
Mr Andrew MAIMONE
Mr Peter LINCOLN
Mr Ryan SCHUBERT
Mr DOU Mingsong
ETH Researchers
Dr Luca BALLAN
Dr Jean-Charles BAZIN
Dr Tobias MARTIN
ETH PhD Students
Ms Claudia CLUSTER
Mr Nicola RANIERI
54
PIs: Assoc Prof CHAM Tat Jen (NTU)
Prof Henry FUCHS (UNC)
Team Members: Assoc Prof CAI Jianfei (NTU)
Assoc Prof Philip FU (NTU)
Mr Andrei STATE (UNC)
Mr TAN Fuwen (NTU)
Mr DENG Teng (NTU)
Mr XU Di (NTU)
Dr Renjie CHEN (UNC)
Prof Miriam REINER (UNC)
This project is aimed at exploring and developing technical and research methodologies for scene
acquisition and display to enable the room-based Telepresence system. The grand vision for the
Telepresence system is to simulate a highly realistic experience for users whereby distant remote
rooms appear to be separated only by transparent glass walls. This entails that as a user walks
about within the local room, her view of the remote room not only appears 3D, but is also
geometrically consistent with her positional point of view, and rendered with sufficient perceptual
fidelity to make the user believe she is physically there – right next to the remote room.
Novel Scene Acquisition and Display Technologies for a Next-generation Room-based Telepresence System
BeingThere Centre Project 1
• A. Maimone, X. Yang, N. Dierk, A. State, M. Dou and H. Fuchs, General-purpose Telepresence with Head-worn Optical See-
through Displays and Projector-based Lighting, IEEE Virtual Reality 2013, Orlando, USA, March 16 - 23, 2013
• M. Zhao, F. W. Tan, C. W. Fu, C. K. Tang, J. F. Cai and T. J. Cham, High-quality Kinect Depth Filtering for Real-time 3D
Telepresence, International Conference on Medical Education (ICME 2013), San Jose, California, USA, July 15 - 19, 2013
55
BeingThere Centre Project 2
PIs: Prof Markus GROSS (ETH)
Prof Marc POLLEFEYS (ETH)
Team Members: Dr Luca BALLAN (ETH)
Dr Tobias MARTIN (ETH)
Dr Jean-Charles BAZIN (ETH)
The project aims to design and build and entirely novel, mobile display platform enabling
autostereoscopic display of collaborators at remote locations. The core novelty is a humanized,
transparent OLED display combined with an LCD layer as an adaptive parallax barrier. A set of
cameras and time-of-flight depth sensors captures the three dimensional appearance, head
position, and distance of the remote communication partner in real-time. This information will be
utilized to control the parallax barrier and to steer the holonomic motion platform.
Mobile, Autostereoscopic 3D Display and Telepresence Platform
• C. Kuster, N. Ranieri, Agustina, H. Zimmer, J. C. Bazin, T. Popa and M. Gross, Towards Next Generation 3D Teleconferencing
Systems, 3DTV Conference 2012 Special Session on “Telepresence”, Zurich, October 15 - 17, 2012
• J. C. Bazin and M. Pollefeys, 3-line RANSAC for Orthogonal Vanishing Point Detection, IEEE/RSJ International Conference on
Intelligent Robots and Systems 2012, Algarve, Portugal , October 7 - 12, 2012
56
PIs: Assoc Prof CHEN I-Ming (NTU)
Assoc Prof SEET Gim Lee, Gerald (NTU)
Research Prof Greg WELCH (UNC)
Team Members: Dr WONG Choon Yue (NTU)
Mr BURHAN (NTU)
Ms PANG Wee Ching (NTU)
Dr Jingwen DAI (UNC)
The objective of the project is to improve on the state of art technology of deploying Telepresence
robots for telecommunication. The plan is to address the need for Telepresence robot technology
which provides rich interactivity without requiring the robot’s user to devote significant cognitive
activity to the operation of the robot.
Specifically, the intention is to develop a mobile Telepresence robot capable of navigating
autonomously or semi-autonomously (directed but not controlled) in a typical office environment
while allowing safe and relaxed interaction without the need to be consciously concerned about
preventing collisions, and maintaining a comfortable distance and pose with respect to the other
humans involved in the interactions.
Animatronic Avatar: Development of a Robotic Avatar for Telepresence Interaction and Communication
BeingThere Centre Project 3
• G. Seet, P. W. Ching, Burhan, V. Lastrebov, W. Gu and W. C. Yue, A Design For A Mobile Robotic Avatar – Modular Framework,
3DTV Conference 2012 Special Session on “Telepresence”, Zurich, October 15 - 17, 2012
• W. C. Pang, G. Seet and X. Yao, A Multimodal Person-following System for Telepresence Applications, ACM Sym. on Virtual
Reality Software and Technology (VRST2013), Singapore, October 6 – 8, 2013
57
BeingThere Centre Project 4
PIs: Prof Nadia MAGNENAT THALMANN (NTU)
Asst Prof YUAN Junsong (NTU)
Prof Daniel THALMANN (NTU)
Team Members: Assoc Prof HE Ying (NTU)
Asst Prof Andy W. H. KHONG (NTU)
Assoc Prof Lap-Pui CHAU (NTU)
Assoc Prof Jianmin ZHENG (NTU)
Assoc Prof CAI Yiyu (MAE)
Dr Aryel BECK (NTU)
Dr Zerrin YUMAK (NTU)
Dr ZHANG Zhijun (NTU)
Dr Xiao YANG (NTU)
Dr Jun LEE (NTU)
Mr Jianfeng REN (NTU)
The general objective of this project is to partially replace a real participant by its virtual
counterpart. The virtual participant is supposed to give a partial illusion that the real person is
present. This implies that the Virtual Human (VH) should look the same as the real human, speaks
with the same intonation, and be somehow aware of the real situation, the real participants, and the
task currently performed. The VH should react at the right time based on the perception he/she has
from the real participants. It implies to evaluate what each real participant is doing. Perception will
be obtained by visual and audio input and recognition. The virtual participant reacts according to
the input and its current knowledge. Its reactions encompass animation (body and facial gestures)
and speech synthesis.
Autonomous Virtual Humans and Social Robots for Telepresence
• J. Hou, L.P. Chau, Y. He, M. Zhang and N. Magnenat Thalmann, Rate-distortion Model Based Bit Allocation for 3-D Facial
Compression Using Geometry Video, IEEE Transactions on Circuits and Systems for Video Technology, No. 99, February 25, 2013
• Z. Yumak, N. Magnenat Thalmann, Multi-party Interaction with a Virtual Character and Human-like Robot, The 19th ACM
Symposium on Virtual Reality Software and Technology (VRST2013), Singapore, October 6-8, 2013
58
PIs: Assoc Prof CHAM Tat Jen (NTU)
Prof Henry FUCHS (UNC)
Prof Markus GROSS (ETH)
Team Members: Prof Miriam REINER(UNC)
Dr Jean-Charles BAZIN (ETH)
Dr Tobias MARTIN (ETH)
Mr ISKANDARSYAH (NTU)
Mr GUO Yu (NTU)
In this research project, broad experimental investigations are conducted into advanced hardware
and software technologies to support future systems for immersive 3D Telepresence with multiple
users per site, unburdened by any head gear, supporting full bidirectional eye contact. 3D
capture/acquisition, 3D display, as well as the related domain of 3D data encoding and
representation in the context of Telepresence are investigated.
This project particularly focus on the feasibility of approaches for the ambitious and challenging
tasks. With a focus on advanced research, rather than complete prototypes, the next generation of
3D acquisition and display, far beyond the capabilities of existing systems is believed to be
reached.
Advanced Technologies for 3D Capture, Communication, and Display
BeingThere Centre Project 5
• J. C. Bazin, H. Li, I. S. Kweon, C. Demonceaux, P. Vasseur and K. Ikeuchi, A Branch-and-Bound Approach to Correspondence
and Grouping Problems, The IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), pp. 1565 – 1576, July, 2013
• C. Kuster, T. Popa, J. C. Bazin, C. Gotsman and M. Gross, Gaze Correction for Home Video Conferencing, SIGGRAPH Asia 2012,
Singapore, November 29 – December 1, 2012
59
BeingThere Centre Achievements Highlights in 2013
CGI Achievement Award, Germany, 2013
Professor Nadia Magnenat Thalmann received CGI Achievement Award in Germany in recognition of her
outstanding accomplishment, academic excellence and service to the Computer Graphics Community in CGI 2013
Conference held in Germany, 2013.
2013 VR Career Award, USA, 2013
Prof. Henry Fuchs received the "2013 VR Career Award in recognition of his lifetime contributions to research and
practice in virtual environments, telepresence and medical applications." in IEEE Virtual Reality 2013 Conference
held in Orlando, FL, USA, March 16-23, 2013.
IEEE Computer Society Award in IEEE VR 2013 Conference, USA, 2013
Prof Greg Welch received IEEE Computer Society award for “Outstanding Performance as Co-Chairman of the
IEEE VR 2013 Conference,” in IEEE Virtual Reality 2013 Conference held in Orlando, FL, USA, March 16-23,
2013.
Best Short Paper Award in IEEE VR 2013 Conference
The paper titled "General-Purpose Telepresence with Head-Worn Optical See-Through Displays and
Projector-Based Lighting“ by Andrew Maimone, Xubo Yang, Nate Dierk, Andrei State, Mingsong Dou and
Henry Fuchs was given the "Best Short Paper Award" in IEEE Virtual Reality 2013 Conference held in
Orlando, FL, USA, March 16-23, 2013.
Business Development
A spin-off company was registered from the research work by the BeingThere Centre’s researcher Rajan
Sobhana Rashobh and professor Andy Khong from NTU.
Best Paper Award (second place) in SIAM Conference on Geometric and Physical Modeling (GD/SPM’13)
The paper titled "A Simple and Local Method for Computing Quasi-conformal Map on 3D Surfaces“ by Zhang
Minqi and He Ying was given the "Best Paper Award (second place)" in Society for Industrial and Applied
Mathematics (SIAM) conference on Geometric & Physical Modeling (GD/SPM 13) held in Denver, Colorado,
USA, November 11 - 14, 2013.
60
BeingThere Centre Co-organized Workshop on Virtual Humans and Cognitive
Modeling │ 26 April 2013
Prof Matthias Scheutz, Tufts University, USAProf Andrew Ortony, Northwestern University, USA
Director, Programme in Computational Social
Cognition IHPC/A*STAR, Singapore
61
62
Selected Keynotes & Invited Talks
N. Magnenat Thalmann, Cloth Simulation, Virtual Try On and Fashion Show, Xiamen University, China, October 14, 2013
N. Magnenat Thalmann, Interacting with Virtual Characters and Social Robots, Virtual Humans and Cognitive Modeling Workshop,
Institute for Media Innovation, Singapore, April 26, 2013
N. Magnenat Thalmann, Interacting with Social Robots and Virtual Humans, Workshop on 3D Displays and Related Technologies,
Nanyang Executive Centre Auditorium, NTU, Singapore, February 4, 2013
Books & Book Chapters
S. Sarda , M. Constable, J. Dauwels, S. Dauwels, M. Elgendi, M. Zhou, U. Rasheed, Y. Tahir, D. Thalmann and N. Magnenat Thalmann,
Real-time Feedback System for Monitoring and Facilitating Discussions, Natural Interaction with Robots, Knowbots and
Smartphones - Putting Spoken Dialog Systems into Practice, Lecture Notes in Computer Science 2013, Springer, 2013
Journal Papers
H. Zhu, J. Zheng, J. Cai, and N. Magnenat Thalmann, Object-level Image Segmentation Using Low Level Cues, IEEE Transactions
on Image Processing (TIP), October 2013
S. Zhao, S. N. Koh, and K.K. Luke, Reference-independent Prosody Evaluation Based on Prosodic Unit Segmentation, IEICE
Transactions on Information and Systems, Vol. E96-D, No. 9, September 2013
C. Liu, R.P. Ang and M.O. Lwin, Cognitive, Personality, and Social Factors Associated with Adolescents’ Online Personal
Information Disclosure, Journal of Adolescence, Elsevier, No. 1467, August 2013
L. Zhang, L. Wang, and W. Lin, Geometric Optimum Experimental Design for Collaborative Image Retrieval, IEEE Transactions on
Circuits and System for Video Technology, August 2013
J.C. Bazin, H. Li, I.S. Kweon, C. Demonceaux, P. Vasseur and K. Ikeuchi, A Branch-and-Bound Approach to Correspondence and
Grouping Problems, The IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), pp. 1565 – 1576, July 2013
63
Journal Papers
X.Q. Wu, J.M. Zheng, C.L. Wu and Y.Y Cai, Variational Structure–texture Image Decomposition on Manifolds, Signal Processing,
Elsevier, Vol. 93, pp. 1773–1784, July 2013
H. Liang, J. Yuan, D. Thalmann and Z. Zhang, Model-based Hand Pose Estimation via Spatial-temporal Hand Parsing and 3D
Fingertip Localization, The Visual Computer, Vol.29, No. 6-8, pp 837-848, June 2013
J. Ren, X. Jiang and J. Yuan, Noise-Resistant Local Binary Pattern with an Embedded Error-correction Mechanism, IEEE
Transactions on Image Processing, vol. 22, issue 10, pp. 4049-4060, June 2013
H. Zhou, J. Zheng and L. Wei, Texture Aware Image Segmentation using Graph Cuts and Active Contours, Journal of Pattern
Recognition, Vol. 46, Issue 6, pp. 1719-1733, June 2013
P. Chiang, Y.Y. Cai, K.H. Mak and J.M. Zheng, A B-spline Approach to Phase Unwrapping in Tagged Cardiac MRI for Motion
Tracking, Magnetic Resonance in Medicine, Vol. 69, pp. 1297–1309, May 2013
G.D. Liu, S. Choudhary, J.Z. Zhang and N. Magnenat Thalmann, Let's Keep in Touch Online: A Facebook Aware Virtual Human
Interface, The Visual Computer, May 2013
Y.B. Wang, R. Dubey, N. Magnenat Thalmann and D. Thalmann, An Immersive Multi-agent System for Interactive Applications, The
Visual Computer, Volume 29, Issue 5, pp. 323-332, May 2013
S. R. Carvalho, R. Boulic, C.A. Vidal and D. Thalmann, Latent Motion Spaces for Full-body Motion Editing, The Visual Computer,
Vol.29, No.3, pp.171-188, March 2013
G. Guo and M. Elgendi, A New Recommender System for 3D E-commerce: An EEG Based Approach, 23rd International Joint
Conference on Artificial Intelligence (IJCAI2913), Journal of Advanced Management Science, Vol. 1, No. 1, pp. 61-65, March 2013
Q, Wang and O. Sourina, Real-time Mental Arithmetic Task Recognition From EEG Signals, IEEE Transactions on Neural Systems
and Rehabilitation Engineering, Vol. 21, No. 2, pp. 225-232, March 2013
J. Hou, L.P. Chau, Y. He, M. Zhang and N. Magnenat Thalmann, Rate-distortion Model Based Bit Allocation for 3-D Facial
Compression Using Geometry Video, IEEE Transactions on Circuits and Systems for Video Technology, No. 99, February 2013
S.Q. Xin, X. Wang, J. Xia, W. Mueller-Wittig, G.J. Wang and Y. He, Parallel Computing 2D Voronoi Diagrams Using Untransformed
Sweepcircles, Computer-Aided Design Journal, Vol. 45, Issue 2, pp. 483-493, February 2013
Y. Cai, N.K.H. Chia, D. Thalmann, N.K.N. Kee, D. Lu, J. Zheng and N. Magnenat Thalmann, Design and Development of A Virtual
Dolphinarium To Increase Functional Developmental Learning in Children with Autism Spectrum Disorder, IEEE Transactions on
Neural Systems & Rehabilitation Engineering, Vol.21, No. 2, pp. 208-217, 2013
64
A. Wang, H. Zhu, J. Cai and J. Wu, Multi-class Cosegmentation with Pairwise Active Learning, 2013 Pacific-Rim Conference on
MultiMedia (PCM 2013), Nanjing, China, December 13-16, 2013
T. Deng, H. Li, J. Cai, T.J. Cham and H. Fuchs, Kinect Shadow Detection and Classification, ICCV 2013 Workshop: Big Data in 3D
Computer Vision, Sydney, Australia, December 8, 2013
J. Hou, L.P. Chau, Y. He and N. Magnenat Thalmann, Expression-invariant and Sparse Representation for Mesh-based
Compression for 3D Face Models, IEEE Conference on Visual Communication and Image Processing (VCIP 2013), Sarawak,
Malaysia, November 17-20, 2013
X. Ying, Zhenhua and Y. He, A Parallel Algorithm for Improving the Maximal Property of Poisson Disk Sampling, Proceedings of
the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, Society for Industrial and Applied Mathematics (SIAM)
conference on Geometric & Physical Modeling (GD/SPM 13), Denver, Colorado, USA, pp. 179-179, November 11-14, 2013
M. Zhang and Y. He, A Simple and Local Method for Computing Quasi-conformal Map on 3D Surfaces, Society for Industrial and
Applied Mathematics (SIAM) conference on Geometric & Physical Modeling (GD/SPM 13), Denver, Colorado, USA, November 11-14,
2013
Y. Tahir, U. Rasheed, K. Hui, S. Dauwels, J. Dauwels, D. Thalmann, N. Magnenat Thalmann, NAO Robot as a Social Mediator: A User
Study, International Conference on Social Robotics (ICSR2013), Bristol, UK, October 27-29, 2013
U. Rasheed, Y. Tahir, S. Dauwels, J. Dauwels, D. Thalmann and N. Magnenat Thalmann, Real-Time Comprehensive Sociometrics for
Two-Person Dialogs, 4th International Workshop on Human Behavior Understanding (HBU2013), Barcelona, Spain, October 22, 2013
G. Guo, J. Zhang, D. Thalmann and N. Yorke-Smith, Prior Ratings: A New Information Source for Recommender Systems in E-
Commerce, ACM Sym. on Virtual Reality Software and Technology (VRST2013), Singapore, October 6–8, 2013
W.C. Pang, G. Seet and X. Yao, A Multimodal Person-following System for Telepresence Applications, ACM Sym. on Virtual
Reality Software and Technology (VRST2013), Singapore, October 6–8, 2013
A. Wang, H. Zhu, J. Cai and J. Wu, Multi-class Cosegmentation with pairwise active learning, ACM Sym. on Virtual Reality Software
and Technology (VRST2013), Singapore, October 6–8, 2013
Y. Xiao, J. Yuan and D. Thalmann, Human-Virtual Human Interaction by Upper Body Gesture Understanding, ACM Sym. on Virtual
Reality Software and Technology (VRST2013), Singapore, October 6–8, 2013
Conference Papers
65
Z. Yumak and N. Magnenat Thalmann, Multi-party Interaction with a Virtual Character and Human-like Robot, The 19th ACM
Symposium on Virtual Reality Software and Technology (VRST2013), Singapore, October 6-8, 2013
Z. Yuzhe, Z. Jianmin and N. Magnenat Thalmann, Virtual Try-on with Kinect, ACM Sym. on Virtual Reality Software and Technology
(VRST2013), Singapore, October 6–8, 2013
J. Zhang, J. Zheng, N. Magnenat Thalmann, Design Mood Dynamics for Virtual Human, ACM Sym. on Virtual Reality Software and
Technology (VRST2013), Singapore, October 6–8, 2013
L. Liu, B. Li, C. Sun, I. Chen, M. Sung and T. Goh, Motivated Learning Embodied Implementation in Autonomous Systems - A Case
Study in NAO Robot, International Symposium on Robotics and Mechatronics (ISRM 2013), Singapore, October 2-4, 2013
M.S. Dou, H. Fuchs and J.M. Frahm, Scanning and Tracking Dynamic Objects with Commodity Depth Cameras, International
Symposium on Mixed and Augmented Reality (ISMAR 2013), Adelaide, Australia, October 1-4, 2013
A. Maimone and H. Fuchs, Computational Augmented Reality Eyeglasses, International Symposium on Mixed and Augmented Reality
(ISMAR 2013), Adelaide, Australia, October 1-4, 2013
C. Chen, J. Cai, J. Zheng, T.J. Cham and G. Shi, A Color-guided, Region-adaptive and Depth-selective Unified Framework for
Kinect Depth Recovery, IEEE Multimedia Signal Processing (MMSP) Workshop 2013, Italy, September 30 - October 2, 2013
J. Hou, L.P. Chau, Y. He and N. Magnenat Thalmann, Human Motion Capture Data Recovery via Trajectory-based Sparse
Representation, Proceedings of IEEE International Conference on Image Processing (ICIP '13), Melbourne, Australia, September 15-18,
2013
J. Ren, X.D. Jiang and J. Yuan, Learning Binarized Pixel-difference Pattern for Scene Recognition, IEEE International Conference
on Image Processing (ICIP), Melbourne Australia, September 15-18, 2013
J. Ren, X. D. Jiang and J. Yuan, Relaxed Local Ternary Pattern for Face Recognition, IEEE International Conference on Image
Processing (ICIP), Melbourne Australia, September 15-18, 2013
Q. Yuan, G. Cong, Z. Ma, A. Sun and N. Magnenat Thalmann, Who, Where, When and What: Discover Spatio-temporal Topics for
Twitter Users, Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2013),
pp 605-613, Chicago, United States, August 11-14, 2013
Conference Papers
66
H. Fang, Trust Modeling for Opinion Evaluation by Coping with Subjectivity and Dishonesty, in Proceedings of the 23rd
International Joint Conference on Artificial Intelligence (IJCAI 2013), Beijing, China, August 3-9, 2013
H. Fang, Y. Bao and J. Zhang, Misleading Opinions Provided by Advisors: Dishonesty or Subjectivity, in Proceedings of the 23rd
International Joint Conference on Artificial Intelligence (IJCAI 2013), Beijing, China, August 3-9, 2013
G. Guo, Improving the Performance of Recommender Systems by Alleviating the Data Sparsity and Cold Start Problems, 23rd
International Joint Conference on Artificial Intelligence (IJCAI2013), Beijing, China, August 3-9, 2013
G. Guo, J. Zhang and N. Yorke-Smith, A Novel Bayesian Similarity Measure for Recommender Systems, 23rd International Joint
Conference on Artificial Intelligence (IJCAI2013), Beijing, China, August 3-9, 2013
H. Wang, B. Zou and G. Guo, Qualitative and Quantitative Preferences Based Web Service Composition by Integrating with Trust,
23rd International Joint Conference on Artificial Intelligence (IJCAI2013), Beijing, China, August 3-9, 2013
W. Gu, G. Seet and N. Magnenat Thalmann, Improved Human Interaction in Telepresence Robot using Real-time Face
Segmentation, 32nd Chinese Control Conference (CCC2013), Xi'an, China, July 26-28, 2013
M. Zhao, F.W. Tan, C.W. Fu, C.K. Tang, J.F. Cai and T.J. Cham, High-quality Kinect Depth Filtering for Real-time 3D Telepresence,
International Conference on Multimedia and Expo (ICME 2013), San Jose, California, USA, July 15-19, 2013
N.S. Narayan, P. Marziliano, and C.G.L. Hobbs, Automatic Removal of Manually Induced Artefacts in Ultrasound images of
Thyroid Gland, in 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE EMBC 2013),
Osaka, Japan, July 3-7, 2013
Rajan S. Rashobh and Andy W.H. Khong, A Multichannel Time-domain Subspace Approach Exploiting Multiple Time-delays for
Acoustic Channel Equalization, The 38th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), May
26–31, 2013
J. Ren, X. D. Jiang and J. Yuan, Dynamic Texture Recognition Using Enhanced LBP Features, The 38th IEEE International
Conference on Acoustics, Speech, and Signal Processing (ICASSP), Vancouver, Canada, May 26–31, 2013
K. Wu, S. T. Goh and A.W.H. Khong, Speaker Localization and Tracking in the Presence of Sound Interference by Exploiting
Speech Harmonicity, in Proc. IEEE Int. Conf. Acoust. Speech Signal Process (ICASSP), Vancouver, Canada, May 26-31, 2013
Conference Papers
67
Conference Papers
S. Zhao, S.N. Koh, I. Y. Soon and K.K. Luke, Feedback Utterances for Computer-aided Language Learning Using Accent
Reduction and Voice Conversion Method, in Proc. 38th International Conference on Acoustics, Speech, and Signal Processing
(ICASSP), Vancouver, Canada, 2013 (Oral), pp. 8208-8212, May 26–31, 2013
S. Zhao, S.N. Koh, I. Y. Soon and K.K. Luke, Phonetic Segmentation Using Statistical Correction and Multi-resolution Fusion, in
Proc. 38th International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vancouver, Canada, 2013 (Oral), pp. 6694-
6698, May 26–31, 2013
L. Dong, W. Lin, C.W. Deng, C. Zhu and H.S. Seah, To Exploit Uncertainty Masking for Adaptive Image Rendering, IEEE
International Symposium on Circuits and Systems, Beijing, China, May 19-23, 2013
H. Zhu, J. Cai, J. Zheng, J. Wu and N. Magnenat Thalmann, Salient Object Cutout Using Google Images, IEEE International
Symposium on Circuits and Systems (ISCAS 2013), Beijing, China, May 19-23, 2013
H. Fang, J. Zhang and N. Magnenat Thalmann, A Trust Model Stemmed from the Diffusion Theory for Opinion Evaluation, in
Proceedings of the 12th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 2013), May 6-10, 2013
J. Kochanowicz, A.H Tan and D. Thalmann, Enabling Generative, Emergent Artificial Culture, International Conference on
Autonomous Agents and Multiagent Systems (AAMAS 2013), Saint Paul, Minnesota, USA, May 6-10, 2013
L. Zhang, L. Wang and W. Lin, A Semantic Subspace Learning Method to Exploit Relevance Feedback Log Data for Image
Retrieval, IEEE Symposium Series on Computational Intelligence (SSCI2013), Singapore, April 16-19, 2013
A. Maimone, X. Yang, N. Dierk, A. State, M. Dou and H. Fuchs, General-purpose Telepresence with Head-worn Optical See-through
Displays and Projector-based Lighting, IEEE Virtual Reality 2013, Orlando, USA, March 16-23, 2013
Agustina and C.Z. Sun, Dependency-conflict Detection in Real-time Collaborative 3D Design Systems, The ACM Conference on
Computer Supported Cooperative Work and Social Computing (ACM CSCW 2013), Texa, USA, February 23-27, 2013
Agustina and C.Z. Sun, XPointer: an X-ray Telepointer for Relaxed-space-time WYSIWIS and Unconstrained Collaborative 3D
Design Systems, The ACM Conference on Computer Supported Cooperative Work and Social Computing (ACM CSCW 2013), Texa,
USA, February 23-27, 2013
S. Zhao, I.Y. Soon, S.N. Koh and K.K. Luke, Automatic Phonetic Segmentation, Audio Day workshop (ADSC), Singapore, January,
2013