virtual environments: system architectures anthony steed simon julier department of computer...
DESCRIPTION
Reminder - VE is an Immersive, Mediated Communication Medium User Interface Devices Environment User Synthetic Environment Real Environment Mediated MediumTRANSCRIPT
![Page 1: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/1.jpg)
Virtual Environments: System Architectures
Anthony SteedSimon JulierDepartment of Computer ScienceUniversity College London
http://www.cs.ucl.ac.uk/teaching/VE
![Page 2: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/2.jpg)
Overview
• Problem Statement & Requirements
• Data Representations (Contents)
• Execution Models (Dynamics)
• Latency
![Page 3: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/3.jpg)
Reminder - VE is an Immersive,Mediated Communication Medium
User
Interface Devices
Environment
User
Synthetic Environment
Real Environment
Mediated Medium
![Page 4: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/4.jpg)
Ellis’s Content, Geometry, Dynamics Model
• Contents is data• Dynamics is code or
rules to change content• Today we will look at
common data and code representations
Contents:Actors and
Objects
Geometry:Dimensions, Metrics and
Extent
Dynamics:Interaction Rules
“Virtual Environments and Environmental Instruments”, S. Ellis, 1996
Environment
![Page 5: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/5.jpg)
Sources and Sinks of Data
• Input “Data”– Data from trackers (e.g. 3D position plus 3D rotation for head
and hand = 12D data)– Data from input devices (e.g. 2D joystick, buttons)– Possibly audio, haptic, physiological input
• Output “Data”– Displays systems, audio, video
![Page 6: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/6.jpg)
Displays Have Different Requirements
• Video (N copies – for stereo and multiple screens)– Maintain copy of visual state– Render as fast as possible (~60Hz)– Synchronise with other renders
• Audio– Maintain copy of audio state– Render without glitches (requires fast interrupt)
• Haptics– Maintain copy of haptic data
Render as fast as possible (~1000Hz)
![Page 7: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/7.jpg)
“Under the Hood” of the Code
• We will find code modules for many different tasks:– Managing data and assets– Reading devices – Audio rendering– Video rendering– User input– Networking– Complex calculations such as physics (see seminar
tomorrow!)
![Page 8: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/8.jpg)
Overview
• Problem Statement & Requirements
• Data Representations (Contents)
• Execution Models (Dynamics)
• Latency
![Page 9: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/9.jpg)
Graphs
• A graph consists of vertices and edges
• Vertices define the “state” information
• Edges define “relationships”
• Scene-graphs are directed and acyclic
Arbitrary graphDirected graphDirected acyclic graph
![Page 10: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/10.jpg)
Scene-graphs
• In a scene-graph, vertices are often called nodes– Store state information– Can include arbitrary property
information• All graphs have a root node
which defines the base of the tree
• All other nodes divided into two types:– Group nodes– Leaf Nodes
Root node
Group nodes
Leaf nodes
![Page 11: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/11.jpg)
Group Nodes
• Group nodes have multiple nodes as children– Child nodes can be other group nodes or leaf nodes
• Applies common state information to multiple objects– State information propagates down the graph
• Examples include:– Transformations– Switch nodes– Effects
• Bump mapping, scribing, specular highlights
– (In recent times) Shader programs
![Page 12: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/12.jpg)
Examples (OpenSceneGraph)
Anisotropic Lighting
ScribingCartoon
Bumpmapping
![Page 13: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/13.jpg)
Leaf Nodes
• Leaf nodes cannot have children• State information relates to the appearance of specific
objects• Examples include:
– Geometry– Image based rendering
• Billboards• Impostors
![Page 14: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/14.jpg)
Examples (OpenSceneGraph)
Impostors Billboards
![Page 15: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/15.jpg)
Overview
• Problem Statement & Requirements
• Data Representations (Contents)
• Execution Models (Dynamics)
• Latency
![Page 16: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/16.jpg)
Dynamics
• These are the rules of interaction between the contents
• These can be:– Differential equations of Newtonian dynamics to describe
kinematic and dynamic relationships– Grammatical rules for pattern-matched triggered actions
• Many different ways of doing this from imposing numerical approximations to Newtonian physics, through to plain old C++ / Java / XVR coding
![Page 17: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/17.jpg)
Dynamics Representation Model
• Separate from the contents, how do we want to represent our dynamics?
• Its leads to quite critical computer science questions about the separation between code and data
• From a scene-authors point of view two basic models– Standalone process model– Within-scene-graph model
![Page 18: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/18.jpg)
Implementing Dynamics as Standalone Processes• Dynamics implemented
as separate processes / threads
• Can change state of the graph in arbitrary ways– Change values of nodes– Add / remove nodes
Dynamics
![Page 19: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/19.jpg)
Problems of Standalone
• Treats scene-graph as a black-box, makes initial scene-creation very easy (load nodes in to scene-graph, changes a few fields on nodes)
• Composing scenes together, or instantiating existing objects is very hard
• End up with quite complicated code• The code probably has variables in local scope which
“should” be in the scene-graph• Couples together rendering and animation to the same
frame rate
![Page 20: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/20.jpg)
Implementing Dynamics Within the Scene-Graph
• Fairly “autonomous” dynamics can be achieved by embedding dynamics within the scenegraph
• Animations are group nodes which apply state changes to their children
• Examples include:– Animation paths– Particle systems
Animation Node
Animated nodes
![Page 21: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/21.jpg)
Example Animation and Particle System
![Page 22: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/22.jpg)
Problems with Within-Scene-Graph Model
• Difficult to get all the code in there– e.g Difficult to write code that interfaces to other applications– Some valiant attempts such as VRML97/X3D
• Does provide for easy composition of scenes• Different parts of the scene do need an explicit way of
talking to each other– In VRML/X3D there are routes– In many other systems there is an event system
• Many systems provide a scripting language support– Python and Lua are very common
![Page 23: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/23.jpg)
Execution Model Ties Everything Together
• So far we we’ve talked about a disparate set of systems:– A master environment– Separate representations for different output modes– User interfaces for controlling the environment
• The execution model “glues” all these parts together– Closely related to distributed systems as well
![Page 24: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/24.jpg)
Execution Models Tying Things Together
• Example:– The position on an object is changed– The update needs to be reflected in:
• The master database• The different scenegraphs• Over the network (if connected)
– How can all of this be coordinated?• Two main models:
– Kernel model– Actor model
![Page 25: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/25.jpg)
Simple Kernel Model
• Treats a VR application like a traditional graphical application:
while(true){
read_trackers();set_body_position();//Changes scene graphdo_animation(); //Changes scene graphrender_left_eye();render_right_eye();render_sound();poll_trackers();
}
![Page 26: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/26.jpg)
do_animation
• After the user’s position is set by the read_trackers and set_body_position (see later in course) …
• … do_animation manipulates the scene• As we have noted, is responsible for:
– Animations, physics, interaction• This is where the standalone or all the within-scene-
graph dynamics are updated
![Page 27: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/27.jpg)
Problems with Simple Kernel Model
• Everything happens in serial order• Rendering only happens at a fixed rate, so if part of
the animation slows down, the rendering slows down (very noticeable in many video games!)
• What if there are different output requirements such as haptics (1000Hz)?
• What if the input rates are much higher (e.g. 200Hz)
![Page 28: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/28.jpg)
Modified Kernel Model (1)
• Has a fast loop which calls different functions at different frame rates:
while(true){
fast_function();if (elapsed>30ms) slow_function();
}fast_function(){read_trackers(), do_haptics();}
slow_function(){render_left_eye();render_right_eye(); render_sound();}
![Page 29: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/29.jpg)
Modified Kernel Model (2)
• Uses a simple main loop, for the video rendering, but “background” threads for reading devices
• Main thread is very much like the Simple Kernel Model, but the function read_trackers and poll_trackers are now in a separate thread, and the main thread just copies information
• This is a good match to how operating systems actually schedule non-blocking input and output
![Page 30: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/30.jpg)
Pros / Cons of the Kernel Model
• Advantages:– Simple to understand– Application programmer keeps their own data structures within the loop
• Disadvantages:– Implementation needs care because of different update rates– Usually requires some awareness of parallel programming issues– Lots of complexity ends up in the do_animation() method
![Page 31: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/31.jpg)
Actor Model• Virtual environment is
realised by a set of collaborating asynchronous processes (actors)
• Actors send messages to one another
• Processes share a common database
• Database is a central repository of the scene graph
DatabaseAudio
Video1
Video2
TrackingSpeech
Collision
Application
![Page 32: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/32.jpg)
Setting Object State Using in the Actor Model
• Setting the object state is often achieved using the subject-observer design pattern
• The object in the database is the subject• Different renderers / networking systems are the
observers• When the subject’s state is updated, the observers are
automatically notified
![Page 33: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/33.jpg)
Pros / Cons of the Actor Model
• Advantages:– Application program does not care about distribution / what
rendering systems used– Update rates are locally very good– Scales well to multiple cores
• Disadvantages:– Difficult to code efficiently– Difficult to query between code modules in different actors,
needs very clear separation of responsibilities– End to end latency difficult to control (see next section)
![Page 34: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/34.jpg)
Overview
• Problem Statement & Requirements
• Data Representations
• Execution Models
• Latency
![Page 35: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/35.jpg)
End-to-End Latency
• Total time from head movement to scene movement
Mine, Mark (1993). Characterization of End-to-End Delays in Head-Mounted Display Systems, UNC Chapel Hill Computer Science Technical Report TR93-001.
![Page 36: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/36.jpg)
End-to-end latency
![Page 37: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/37.jpg)
Problems
• Single process• For the main CPU, tracker processing is actually
mostly just “waiting”, could schedule something else IF and only IF (IFF) we know how long it will take– Previously we did poll and read separately
• If there are multiple video streams, we could do them in parallel IFF rendering changes no state in the scene graph (depending on your scene graph this isn’t a good bet!)
![Page 38: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/38.jpg)
Example Timing Problem
Animate Render Animate Render
T0 = tracker poll
T1 = video output
Animate Render Animate Render
T0 = tracker poll
T1 = video output
Ideal case
Worst case
![Page 39: Virtual Environments: System Architectures Anthony Steed Simon Julier Department of Computer Science…](https://reader035.vdocuments.us/reader035/viewer/2022062505/5a4d1bf17f8b9ab0599e6453/html5/thumbnails/39.jpg)
Summary
• Representing the environment is difficult• The representation has to be rich enough to capture the contents,
geometry and dynamics• Each display mode requires its own form of the environment to
optimise the display• Want to make content as rich as possible to support dynamic
models• Otherwise behaviour is expressed only in code.• At run-time there are logically concurrent processes (rendering,
collision, audio etc…)• Execution models need to reflect this concurrency