1

Jacoban instruction agent in VR

Marc Evers (evers@cs.utwente.nl)6 March 2000

2

Contents

context & project descriptionapproach & technologycurrent state & next stepsarchitecture of the systemdesign detailsconclusions

3

Context

pilot project of VR Valley Twente objective: establishing a regional

knowledge centre on VR organisations: UT (INF, WB), HE, HSA,

Carp, MST, ... 6 pilot projects http://www.vr-valley.com/

4

Jacob Project

Jacob as assistant tasks: instruction, navigation, etc.

Jacob shows intelligent behaviourmultimodal interactioninteraction between multiple agentsmanipulation of objects in the virtual

world

5

Approach

use existing knowledge and theories Intelligent Tutoring Systems, Computer

Graphics, AI, … software engineering focus

how to build such a system in a maintainable, adaptable, …, way

use OO techniques & software architecture knowledge

6

Technology

runs on any PC or Workstation with web browser+VRML browser+3D graphics card

programming languages Java 1.1; VRML 97

standards H-Anim for Jacob’s body

under consideration: Java 3D, JavaSpeech

7

Current state

Jacob teaches construction task Towers of Hanoi

manipulation of objects in virtual world by Jacob and by user

Jacob has explicit model of the tasksimulation of physical processes

gravity, collisionssimple animation

8

Next steps

natural language dialogue‘intelligence’ for Jacobmulti user Jacob (DeepMatrix)integration in VMC, in 2 steps:

first, Jacob will play isolated role in VMC second, Jacob interacts with VMC

9

Basic architecture

3D platform: VRML/EAI; Java 3D; ...

abstraction of 3D platform

abstract 3D World: agents, cubes, tables, ...

‘task’ layer (?)

10

World View

Cube Avatar

VWorld

avoidCollision()enforceWorldConstraints()agentMovesObject()update()

VEObject

position : Vec3DboundinBox : BoundingBox

canCollide()avoidCollision()getBoundingBox()startGravity()isOnTopOf()lock()isLocked()setPosition()

0..*

+agents

0..*

+objects

0..*0..*

Jacob

lookAt()moveObject()walkTo()shakeHead()

11

Class VWorld

simulation of physical aspects gravity, collisions other physical constraints

management of VE objects agents & objects

keeps track of the state of VE objects

observations

12

Class VEObject

defines general behaviour of VE objects bounding box, position, size gravity, (b)locking relations like isOnTopOf

13

Subclasses of VEObject

Cube detects picking, dropping, dragging by

userAvatar

knows position and orientation of userJacob

Jacob’s manifestation in the world lookAt, moveObject, walktTo, shakeHead

14

‘Physics’ protocolc : Cube : VWorld : GravityThread : World

Listener : DraggingSensor

NodeProxy

2: agentMovesObject(c)

1: drop performed

3: enforceWorldConstraints( )

4: startGravity( )

5: start( )

7: update( )

6: setPosition( )

8: worldStateChanged()

15

‘Physics’ protocol

c : Cube : VWorld

: GravityThread

: DraggingSensorNodeProxy

: WorldListener

3: enforceWorldConstraints( )7: update( )

2: agentMovesObject(c)

4: startGravity( )

5: start( )

6: setPosition( )

8: worldStateChanged()1: drop performed

16

Collision avoidance protocolc : Cube : VWorld : VEObject b1 : Bounding

Boxb2 : Bounding

Box

1: setPosition(v)2: avoidCollision(c,v)

3: avoidCollision(c,v)

4: canCollide( )

6: getBoundingBox( )

5: getBoundingBox( )

7: avoidCollision(b2, v)8: get...

17

Collision avoidance protocol

c : Cube

: VWorld

: VEObject

b1 : BoundingBox

b2 : BoundingBox

4: canCollide( )5: getBoundingBox( )

1: setPosition(v)

2: avoidCollision(c,v)

3: avoidCollision(c,v)

6: getBoundingBox( )

7: avoidCollision(b2, v)

8: get...

18

Instruction View

InstructionMethod

start()

InstructionRule

trigger()

0..*

0..*

0..*

0..*

InstructionModel

taskStateChanged()

Jacob

lookAt()moveObject()walkTo()shakeHead()

manipulates

VWorld

avoidCollision()enforceWorldConstraints()agentMovesObject()update()

Schema<<Interface>>

SchemaIterator

updateCurrent()

Task

worldStateChanged()currentSchema()

observes

observes

19

Class Task

knows how to create the ‘scene’has knowledge on performing the task

Schema: hierarchical structure Composite design pattern

action, explanation, pre/postcondition, invariant

schema interface is generic SchemaIterator determines current task

state

20

Class InstructionModel

InstructionRule rules that fire other rules or start

instruction methods based on conditionsInstructionMethod

executed until goal or state is reached example:

while user does nothing: wait, give hint, wait, show move

21

Task update protocol : VWorld t : Task : Schema

Iterator : Schema : Instruction

Model

1: worldStateChanged( )

2: updateCurrent( ) 3: precondition(t)

4: postcondition(t)

5: invariant(t)

6: getSubSchemas( )

7: taskStateChanged( )

8: currentSchema( )

22

Task update protocol

: VWorld

t : Task : SchemaIterator

: Schema

: InstructionModel

1: worldStateChanged( ) 2: updateCurrent( )

7: taskStateChanged( )

8: currentSchema( )3: precondition(t)4: postcondition(t)

5: invariant(t)6: getSubSchemas( )

23

Concluding remarks

current results can be generalised Jacob based on existing knowledge and

theories instruction agent concept is very

generalinstruction, assistance, design guidance…

demo & documentation http://www.cs.utwente.nl/~evers/jacob/

24

Jacob Screenshot