research themes coordinated multi modal generation for virtual humans

Research themes

Coordinated Multi Modal Generation for Virtual HumansCoordination of multi modal generation with perception input. Real-time coordinated interaction. Dynamic animation, synchronized with extrapolations of a stream of incoming events/observations.

Conversational Agents: Theoretical WorkThe Virtual Conductor will serve as a context for theoretical investigation of dialog as a continuous perception/production feedback loop in which both interaction parties are at all times 'sending' as well as 'receiving'.

Conducting ApplicationsThe Virtual Conductor (VC) may be used for several applications. The most relevant at this point are the use of the VC for rehearsals and for tutoring of conducting students.

Music Perception TechnologyThe VC presents a novel context to work on applications of music analysis technology.

Activities

Analyzing Conducting by HumansTo implement the right behavior in a VC we need to know how different intentions are expressed by human conductors, and when.

Music Perception TechnologyTo conduct effectively, the VC needs to analyze the music in real time using algorithms implemented and/or adapted from literature.

Multi Modal Generation Animation technology, adaptive (re)planning, synchronization, etc…

Evaluation with Human MusiciansConduct tests with human musicians to evaluate the interaction models and the perception technology.

Human Media InteractionP. BosA. NijholtD. ReidsmaZs. Ruttkayd.reidsma@ewi.utwente.nl

Current State•Animated Virtual Conductor•Conducts a group of human musicians •Using 1-, 2-, 3- and 4-beat patterns.•Conducts volume and tempo (changes). •Tempo interaction: prepares, and leads the musicians through, tempo changes or corrections.•Can rehearse problematic passages with the ensemble

The conductor has been evaluated several times with groups of human musicians. The musicians could follow the tempo and dynamic changes of the conductor reasonably well. The conductor could interact successfully with the musicians, correcting their tempo if they played too fast or to slow. •Interacting with a Virtual Conductor, Bos, P., Reidsma, D., Ruttkay, Z.M., and Nijholt, A. (2006) in: Proc. of 5th International Conference on Entertainment Computing, Cambridge, UK. pp 25-30, LNCS 4161 Springer Verlag, DOI 1007/11872320_3

•http://hmi.ewi.utwente.nl/showcases/Human Music Interaction/

The Virtual Conductor