iti-certh augmented and virtual reality laboratory (avrlab)
DESCRIPTION
ITI-CERTH Augmented and Virtual Reality Laboratory (AVRLab) - PowerPoint PPT PresentationTRANSCRIPT
ITI-CERTH Augmented and Virtual Reality Laboratory (AVRLab)
The AVRL of the Informatics and Telematics Institute exhibits substantial research activity, both basic and industry-oriented, as
well as technology transfer actions, in the area of virtual, augmented and mixed reality. This area includes modeling and
simulation of complex virtual environments, haptic force feedback applied research, 3D modeling and compression, deformable objects and cloth modeling, collaborative virtual environments, new sensor
development for head/hands/body tracking, multimedia analysis/synthesis, gesture recognition, head motion modelling and tracking, advanced applications of virtual/augmented/mixed reality in education and training, e-health, assistive technologies, culture,
industry and e-business.
The ITI-CERTH AVRLab operates a fully operational and well-
equipped Virtual Reality Laboratory that serves as a
reference for augmented and virtual reality applications and
demonstrations in Greece. It employs a high quality scientific
group of personnel for the design, development and demonstration
of advanced augmented and virtual reality applications and
services. It participates in research networks with assorted institutes and industrial partners
in Greece and Europe and has established a spin-off company, to commercially exploit the research
results of the Laboratory.
Some significant applied research fields where ITI is activated are:
Haptics Computer Graphics 3D Semantics Mixed and Augmented
Reality Virtual, Augmented and
Mixed Reality Applications
HapticsExtensive study and development of very
efficient approaches for haptic rendering, real time force feedback and real time collision detection for
complex virtual environments. Applications include virtual
environments for the visually impaired, geometry education in 3D
environments, virtual assembly planning, real time interaction with
complex cultural objects, etc
Computer GraphicsResearch and development of efficient
algorithms for modeling, representation animation and
monitoring of deformable and complex virtual objects. It includes new approaches for cloth modeling,
collision and self-collision detection, new interaction paradigms, real time
avatar simulations, smart objects and complex behavioral modeling
3D Semantics Research and development of advanced
algorithms for search and retrieval of 3D content from large databases. Research includes new modeling approaches, descriptor extraction using rotation/translation/scaling
invariant features and identification of appropriate matching and data fusion
strategies for optimal descriptor selection
Mixed and Augmented RealityExtensive study and development of new
approaches in real time head/hands/body tracking for mixed
reality applications. Research directions include real time 3D
acquisition, stereo and multiview image sequence processing, feature detection and tracking, coding and
compression, advanced prediction and detection approaches, etc
Virtual, Augmented and Mixed Reality Applications
Application oriented research in areas such as education and training, e-
health, assistive technologies, culture, industry, e-business, phobia
treatment, etc. Includes various approaches utilizing and integrating
appropriate virtual/augmented/mixed reality strategies for enhanced
application realism
The goals of the SIMILAR European network
• Integrating a task force on multimodal interfaces that respond efficiently to speech, gestures, vision, haptics and direct brain connections
• Merging 32 excellent European laboratories in Human-Computer Interaction (HCI) and in Signal Processing in a single research network.
• Address challenges in medical, edutainment and disabled areas
The SIMILAR partners
SIMILAR institutions are spread all over Europe (and one in Africa)
Red : Research Blue : Industry
Secondary Objectives
– SIMILAR will develop a common theoretical framework for fusion and fission of multimodal information using Signal Processing tools constrained by Human Computer Interaction rules.
– SIMILAR will develop a network of usability test facilities.
– SIMILAR will manage an International Journal, Special Sessions in conferences, summer schools, interact with industrial partners and promote new research activities.
– SIMILAR will address a series of grand challenges in edutainment, disabled people and medical applications and will develop new interfaces for environments where the user is unable to use his hands, like in surgical operation rooms, or in cars.
The 7 SIGs (Special Interest
Groups )
– Information Fusion and Fission– Multimodal Analysis and Synthesis– Context Aware Adaptation– Usability– Medical Applications – Edutainment Applications– Disabled People Applications
car
Multimodal Signal Proc.
EdutainmentDisabledAnd Cars
Medical
Platforms(OpenInterface) Usability
Fusion &Fission (IT)
Contextaware.
SIMILAR Synopsis
A general specification for multimodal interfaces geared to solve interconnection problems between hardware and software from heterogeneous origins.
OpenInterface is mainly driven by : FPMs with Multitel (Belgium), ITI/CERTH (Greece), UCL-Tele (Belgium), CLIPs-IMAG(France).
OpenInterface will be accessible on the web : http://www.openinterface.org .
What is OpenInterface ?
Collaborative research between a large amount of partners leads to interconnection
problems…
Interconnection problems (1)
• John is working on a voice recognition algorithm and would like to test it into a real application…
• Brenda has a 3D visualisation system that would be interesting to drive with voice…
Interconnection problems (2)
… but programming interfaces are not the same !
?
Interconnection problems (3)
• Brenda’s 3D visualisation system looks very promising and Wiliam would like to use it to inspect his 3D segmented medical structures…
… but programming interfaces are not the same !
?
•Solution :
SIMILAR Platforms (OpenInterface)
Creation of a common language, using XML, that will be understood by anyone and used to describe module’s interfaces.
Interconnection problems (4)
SIMILARPlatforms
Interconnection problems (5)
• Once a common language as been created, everybody will be able to drive partner’s plugin easily and in a way that could be re-used to drive other partner’s module of the same type.
Zoom 1,5Zoom 1,5
• Easy development of multimodal applications • Avoid redundances of common tasks
– data formats support, usual audio/image/video processing,…
• Easy integration between partners’ modules using SIMILARPlatforms specifications
• Once a module is finished, he could then be integrated in the OpenInterface library to facilitate his deployment.
OpenInterface Main Goals
OpenInterface Architecture
OpenInterface : Component based architecture (1)
• Each functionality is a component
• Components are grouped into plugins (shared libraries)
• The content of each plugin is described using the SimilarPlatforms language
KERNEL
speech
visualization
Video plugins
Image plugins
Fusion/fission
gesturerecognition
Conception: Component based architecture (2)
OpenInterface Editor
•The OpenInterface Editor will– Facilitate the generation of new multimodal applications by
simple drag and drop of partner’s modules in a intuitive graphical user interface,
– Use the OpenInterface library and the SIMILARPlatforms language to automaticaly add new modules,
– Allow rapid creation of protopytes to test new multimodal interfaces.
Similar eNTERFACE workshops
• Similar organizes annual workshops on multimodal signal processing and interaction– Graduate, master, PhD, postdoctoral students as well as
Seniors meet for four weeks and work together in common or complementary research areas and develop novel applications!
• eNTERFACE 2005 in Mons, Belgium was a complete success
• eNTERFACE 2006 in Dubrovnik was also successful• Hopefully eNTERFACE projects will continue far
beyond SIMILAR
eNTERFACE 2005 projects
• Project 1: Combined Gesture-Speech Analysis and Synthesis
• Project 2: Multimodal Caricatural Mirror • Project 3: Biologically-driven musical instrument • Project 4: Multimodal Focus Attention Detection in an
Augmented Driver Simulator • Project 5: Multilingual Multimodal Biometric
Identification/Verification • Project 6: Speech Conductor • Project 7: A Multimodal (Gesture+Speech) Interface
for 3D Model Search and Retrieval Integrated in a Virtual Assembly Application
eNTERFACE 2006 projects
• Project 1: An Agent Based Multicultural User Interface in a Customer Service Application
• Project 2: Multimodal tools and interfaces for the intercommunication between visually impaired and “deaf and mute” people
• Project 3: Sign Language Tutoring Tool • Project 4: Multimodal Character Morphing • Project 5: Introducing Network-Awareness for Networked
Multimedia and Multi-modal Applications • Project 6: An instrument of sound and visual creation driven by
biological signals • Project 7: Emotion Detection in the Loop from Brain Signals and
Facial Images • Project 8: Realtime and Accurate Control of Expression in
Singing Synthesis • Project 9: Multimodal Driving Simulator
ITI-CERTH in eNTERFACE 2005
• ITI-CERTH was the leader of Project 7.
• The work has been published in IEEE Multimedia. “Masterpiece: Physical Interaction and 3D content-based search in VR Applications”, vol.13, no.3, pp. 92-100, July, 2006.
ITI-CERTH in eNTERFACE 2006• ITI-CERTH was the
leader of Project 2.– Intercommunication
between blind and deaf-mute users
– Utilizing cross-modal transformation
– Tools are integrated in treasure hunting game that serves for both entertainment and education to the new technologies for the disabled
– Modalities: Haptics, Speech, Sign language, Gesture
CSCW Tool(Game)
I
nt
er
ne
t
SpeechRecognition
text Text to SignLanguage
Sign LanguageSynthesis
AvatarAnimation
Haptic Device
SpeechSynthesis
Sign LanguageAnalysis
text
SynchronisedEnvironmentParameters
SynchronisedEnvironmentParameters
Haptic ForceFeedbackInterface
CSCW Tool(Game)
Voice Data
text
text
Net
wo
rk
THANK YOU!
INFORMATICS & TELEMATICS INSTITUTE
1st km. Thermi-Panorama Road
PO BOX 361, 57001 THERMI THESSALONIKI, GREECE
TEL: +30 2310 464160 FAX: +30 2310 464164
http://www.iti.gr
Konstantinos Moustakas
Email: [email protected]