INFORMATIKINFORMATIK

An An EEfficient Spatio-Temporal fficient Spatio-Temporal Architecture for Animation RenderingArchitecture for Animation Rendering

Vlastimil HavranVlastimil Havran, Cyrille Damez, , Cyrille Damez, Karol Myszkowski, and Hans-Peter SeidelKarol Myszkowski, and Hans-Peter Seidel

Max-Planck-Institut fMax-Planck-Institut für Informatik, ür Informatik, Saarbrücken, GermanySaarbrücken, Germany

Vlastimil HavranVlastimil Havran, Cyrille Damez, , Cyrille Damez, Karol Myszkowski, and Hans-Peter SeidelKarol Myszkowski, and Hans-Peter Seidel

Max-Planck-Institut fMax-Planck-Institut für Informatik, ür Informatik, Saarbrücken, GermanySaarbrücken, Germany

INFORMATIKINFORMATIKMotivationMotivation

In the traditional approach to rendering of high In the traditional approach to rendering of high quality animation sequences every frame is quality animation sequences every frame is considered separately.considered separately.

• The temporal coherence is poorly exploited:The temporal coherence is poorly exploited:

redundant computations.redundant computations.

• The visual sensitivity to temporal detail cannot be properly The visual sensitivity to temporal detail cannot be properly accounted for:accounted for:

too conservative stopping conditions,too conservative stopping conditions,

temporal aliasing.temporal aliasing.

In the traditional approach to rendering of high In the traditional approach to rendering of high quality animation sequences every frame is quality animation sequences every frame is considered separately.considered separately.

• The temporal coherence is poorly exploited:The temporal coherence is poorly exploited:

redundant computations.redundant computations.

• The visual sensitivity to temporal detail cannot be properly The visual sensitivity to temporal detail cannot be properly accounted for:accounted for:

too conservative stopping conditions,too conservative stopping conditions,

temporal aliasing.temporal aliasing.

INFORMATIKINFORMATIKGoalGoal

Developing an architecture for efficient rendering Developing an architecture for efficient rendering of high-quality animations which better exploits the of high-quality animations which better exploits the spatio-temporal coherence between frames:spatio-temporal coherence between frames:• Visibility: Multi-frame ray tracingVisibility: Multi-frame ray tracing

• Global illumination: Bi-directional Path Tracing extended to Global illumination: Bi-directional Path Tracing extended to re-using samples between framesre-using samples between frames

• Texturing and shading: sharing information between Texturing and shading: sharing information between framesframes

• Motion blur: conservative computation in 2D image spaceMotion blur: conservative computation in 2D image space

• Cache use: coherent patterns of access to data structures Cache use: coherent patterns of access to data structures in memory (along motion compensation trajectories)in memory (along motion compensation trajectories)

• Memory requirements: unlimited image resolutionMemory requirements: unlimited image resolution

Developing an architecture for efficient rendering Developing an architecture for efficient rendering of high-quality animations which better exploits the of high-quality animations which better exploits the spatio-temporal coherence between frames:spatio-temporal coherence between frames:• Visibility: Multi-frame ray tracingVisibility: Multi-frame ray tracing

• Global illumination: Bi-directional Path Tracing extended to Global illumination: Bi-directional Path Tracing extended to re-using samples between framesre-using samples between frames

• Texturing and shading: sharing information between Texturing and shading: sharing information between framesframes

• Motion blur: conservative computation in 2D image spaceMotion blur: conservative computation in 2D image space

• Cache use: coherent patterns of access to data structures Cache use: coherent patterns of access to data structures in memory (along motion compensation trajectories)in memory (along motion compensation trajectories)

• Memory requirements: unlimited image resolutionMemory requirements: unlimited image resolution

INFORMATIKINFORMATIKPrevious WorkPrevious Work

• Industrial solutions such as PhotorealisticRenderman, Industrial solutions such as PhotorealisticRenderman, Maya Rendering System, or SoftimageMaya Rendering System, or Softimage

– frame–by–frame computationframe–by–frame computation

– postprocessing of animation to decrease temporal postprocessing of animation to decrease temporal aliasing aliasing

• Industrial solutions such as PhotorealisticRenderman, Industrial solutions such as PhotorealisticRenderman, Maya Rendering System, or SoftimageMaya Rendering System, or Softimage

– frame–by–frame computationframe–by–frame computation

– postprocessing of animation to decrease temporal postprocessing of animation to decrease temporal aliasing aliasing

INFORMATIKINFORMATIKPrevious WorkPrevious Work

Temporal Coherence in Ray TracingTemporal Coherence in Ray Tracing• Coherence of shaded pixelsCoherence of shaded pixels

– Interpolation between frames [Maisel and Hegron 92]Interpolation between frames [Maisel and Hegron 92]

– Reprojection [Adelson and Hughes 95]Reprojection [Adelson and Hughes 95]

– 4D radiance interpolants [Bala et al. 99]4D radiance interpolants [Bala et al. 99]

• Coherence in acceleration data structuresCoherence in acceleration data structures

– Space-time solutions [Glassner 88, Groeller 91]Space-time solutions [Glassner 88, Groeller 91]

– Static vs. dynamic objects [Besuievsky and Pueyo 01, Static vs. dynamic objects [Besuievsky and Pueyo 01, Lext and Moller 01, and Wald et al. 02]Lext and Moller 01, and Wald et al. 02]

Temporal Coherence in Ray TracingTemporal Coherence in Ray Tracing• Coherence of shaded pixelsCoherence of shaded pixels

– Interpolation between frames [Maisel and Hegron 92]Interpolation between frames [Maisel and Hegron 92]

– Reprojection [Adelson and Hughes 95]Reprojection [Adelson and Hughes 95]

– 4D radiance interpolants [Bala et al. 99]4D radiance interpolants [Bala et al. 99]

• Coherence in acceleration data structuresCoherence in acceleration data structures

– Space-time solutions [Glassner 88, Groeller 91]Space-time solutions [Glassner 88, Groeller 91]

– Static vs. dynamic objects [Besuievsky and Pueyo 01, Static vs. dynamic objects [Besuievsky and Pueyo 01, Lext and Moller 01, and Wald et al. 02]Lext and Moller 01, and Wald et al. 02]

INFORMATIKINFORMATIKPrevious WorkPrevious Work

Temporal Coherence in Global IlluminationTemporal Coherence in Global Illumination• Coherence of illumination samplesCoherence of illumination samples

– Render cache [Walter et al. 99]Render cache [Walter et al. 99]

– Shading cache [Tole et al. 02]Shading cache [Tole et al. 02]

– Reusing photon hit points [Myszkowski et al. 01]Reusing photon hit points [Myszkowski et al. 01]

– Selective Photon Tracing [Dmitriev 02]Selective Photon Tracing [Dmitriev 02]

• Coherence in visibility computationCoherence in visibility computation

– Global lines [Besuievsky and Pueyo 01]Global lines [Besuievsky and Pueyo 01]

• Coherence in generation of random numbersCoherence in generation of random numbers

– Random number sequence associated with each light Random number sequence associated with each light transport path [Lafortune 96, Jensen 01, Wald et al. 02]transport path [Lafortune 96, Jensen 01, Wald et al. 02]

Temporal Coherence in Global IlluminationTemporal Coherence in Global Illumination• Coherence of illumination samplesCoherence of illumination samples

– Render cache [Walter et al. 99]Render cache [Walter et al. 99]

– Shading cache [Tole et al. 02]Shading cache [Tole et al. 02]

– Reusing photon hit points [Myszkowski et al. 01]Reusing photon hit points [Myszkowski et al. 01]

– Selective Photon Tracing [Dmitriev 02]Selective Photon Tracing [Dmitriev 02]

• Coherence in visibility computationCoherence in visibility computation

– Global lines [Besuievsky and Pueyo 01]Global lines [Besuievsky and Pueyo 01]

• Coherence in generation of random numbersCoherence in generation of random numbers

– Random number sequence associated with each light Random number sequence associated with each light transport path [Lafortune 96, Jensen 01, Wald et al. 02]transport path [Lafortune 96, Jensen 01, Wald et al. 02]

INFORMATIKINFORMATIK

Space-Time Architecture: Space-Time Architecture: PrinciplePrinciple

• Compute samples using a variant of Path-TracingCompute samples using a variant of Path-Tracing

• Pixels color = mean of sample valuesPixels color = mean of sample values

• 2 types of samples:2 types of samples:

– Native samples:Native samples:

ExpensiveExpensive

Computed from scratchComputed from scratch

– Recycled samples: Recycled samples:

CheapCheap

Based on previous computations of native Based on previous computations of native samples (using reprojections)samples (using reprojections)

• Compute samples using a variant of Path-TracingCompute samples using a variant of Path-Tracing

• Pixels color = mean of sample valuesPixels color = mean of sample values

• 2 types of samples:2 types of samples:

– Native samples:Native samples:

ExpensiveExpensive

Computed from scratchComputed from scratch

– Recycled samples: Recycled samples:

CheapCheap

Based on previous computations of native Based on previous computations of native samples (using reprojections)samples (using reprojections)

INFORMATIKINFORMATIKMotion CompensationMotion Compensation

• Camera and object motion compensationCamera and object motion compensation

• Memory access coherence …..Memory access coherence …..

• Camera and object motion compensationCamera and object motion compensation

• Memory access coherence …..Memory access coherence …..

INFORMATIKINFORMATIKBi-directional Path TracingBi-directional Path Tracing

INFORMATIKINFORMATIKCamera MotionCamera Motion

INFORMATIKINFORMATIKOccluded ConnectionOccluded Connection

INFORMATIKINFORMATIKPath ChangePath Change

INFORMATIKINFORMATIKMulti-Frame Ray TracingMulti-Frame Ray Tracing• Aggregate queries Aggregate queries

– Single ray geometrySingle ray geometry

– Results possible for all frames in Results possible for all frames in [f[f(i-R-S)’(i-R-S)’ff(i+R)(i+R)]]

• Two types of visibility queriesTwo types of visibility queries

– Ray shootingRay shooting

– Visibility between two pointsVisibility between two points

• Three settings of visibility queriesThree settings of visibility queries

– Exact result for single frame Exact result for single frame ff(i)(i)

– Exact for single frame Exact for single frame ff(i) (i) + validity for all other frames+ validity for all other frames

– Exact results for all frames in Exact results for all frames in [f[f(i-R)’(i-R)’ff(i+R)(i+R)]]

• Aggregate queries Aggregate queries

– Single ray geometrySingle ray geometry

– Results possible for all frames in Results possible for all frames in [f[f(i-R-S)’(i-R-S)’ff(i+R)(i+R)]]

• Two types of visibility queriesTwo types of visibility queries

– Ray shootingRay shooting

– Visibility between two pointsVisibility between two points

• Three settings of visibility queriesThree settings of visibility queries

– Exact result for single frame Exact result for single frame ff(i)(i)

– Exact for single frame Exact for single frame ff(i) (i) + validity for all other frames+ validity for all other frames

– Exact results for all frames in Exact results for all frames in [f[f(i-R)’(i-R)’ff(i+R)(i+R)]]

INFORMATIKINFORMATIKMulti-Frame Ray TracingMulti-Frame Ray Tracing

Additional techniques usedAdditional techniques used• Mail boxes (cache) for ray transforms, objects, and kd-treesMail boxes (cache) for ray transforms, objects, and kd-trees

• Frame masks for inserted kd-trees and shadow cacheFrame masks for inserted kd-trees and shadow cache

Additional techniques usedAdditional techniques used• Mail boxes (cache) for ray transforms, objects, and kd-treesMail boxes (cache) for ray transforms, objects, and kd-trees

• Frame masks for inserted kd-trees and shadow cacheFrame masks for inserted kd-trees and shadow cache

Construction of Spatial Data StructureConstruction of Spatial Data Structure• Instantiation of dynamic objects for range of framesInstantiation of dynamic objects for range of frames

• Static Static versus versus dynamicdynamic objects separation objects separation

• Construction of global kd-tree over static objectsConstruction of global kd-tree over static objects

• Hierarchical clustering over dynamic objectsHierarchical clustering over dynamic objects

• Construction of kd-trees for clusters of objectsConstruction of kd-trees for clusters of objects

• Insertion of kd-trees for clusters into global kd-tree leavesInsertion of kd-trees for clusters into global kd-tree leaves

• Refinement of global kd-tree leaves + empty space cutting Refinement of global kd-tree leaves + empty space cutting off (efficiency improvement methods)off (efficiency improvement methods)

Construction of Spatial Data StructureConstruction of Spatial Data Structure• Instantiation of dynamic objects for range of framesInstantiation of dynamic objects for range of frames

• Static Static versus versus dynamicdynamic objects separation objects separation

• Construction of global kd-tree over static objectsConstruction of global kd-tree over static objects

• Hierarchical clustering over dynamic objectsHierarchical clustering over dynamic objects

• Construction of kd-trees for clusters of objectsConstruction of kd-trees for clusters of objects

• Insertion of kd-trees for clusters into global kd-tree leavesInsertion of kd-trees for clusters into global kd-tree leaves

• Refinement of global kd-tree leaves + empty space cutting Refinement of global kd-tree leaves + empty space cutting off (efficiency improvement methods)off (efficiency improvement methods)

INFORMATIKINFORMATIKThe Animation BufferThe Animation Buffer

• Iterates over all pixels in S consecutive framesIterates over all pixels in S consecutive frames

• If more samples are requiredIf more samples are required

– Compute a native sample for frame Compute a native sample for frame ffii

– Reproject it and recycle it for all frames in Reproject it and recycle it for all frames in [f[f(i-R)’ (i-R)’ ff(i+R)(i+R)]]

• S+2R frames are kept in the bufferS+2R frames are kept in the buffer

• Iterates over all pixels in S consecutive framesIterates over all pixels in S consecutive frames

• If more samples are requiredIf more samples are required

– Compute a native sample for frame Compute a native sample for frame ffii

– Reproject it and recycle it for all frames in Reproject it and recycle it for all frames in [f[f(i-R)’ (i-R)’ ff(i+R)(i+R)]]

• S+2R frames are kept in the bufferS+2R frames are kept in the buffer

SSSSRRRR RRRR

INFORMATIKINFORMATIKThe Animation BufferThe Animation Buffer

• Iterates over all pixels in S consecutive framesIterates over all pixels in S consecutive frames

• If more samples are requiredIf more samples are required

– Compute a native sample for frame Compute a native sample for frame ffii

– Reproject it and recycle it for all frames in Reproject it and recycle it for all frames in [f[f(i-R)’ (i-R)’ ff(i+R)(i+R)]]

• S+2R frames are kept in the bufferS+2R frames are kept in the buffer

• Iterates over all pixels in S consecutive framesIterates over all pixels in S consecutive frames

• If more samples are requiredIf more samples are required

– Compute a native sample for frame Compute a native sample for frame ffii

– Reproject it and recycle it for all frames in Reproject it and recycle it for all frames in [f[f(i-R)’ (i-R)’ ff(i+R)(i+R)]]

• S+2R frames are kept in the bufferS+2R frames are kept in the buffer

SSSSRRRR RRRRSSSSSaving to a diskSaving to a disk

INFORMATIKINFORMATIKShading ComputationShading Computation

• A simplified version of RenderMan Shading LanguageA simplified version of RenderMan Shading Language

• Each shader decomposed into Each shader decomposed into

– View-independent componentView-independent component

re-usable, shared between framesre-usable, shared between frames

– View-dependent componentView-dependent component

recomputed for each framerecomputed for each frame

• A simplified version of RenderMan Shading LanguageA simplified version of RenderMan Shading Language

• Each shader decomposed into Each shader decomposed into

– View-independent componentView-independent component

re-usable, shared between framesre-usable, shared between frames

– View-dependent componentView-dependent component

recomputed for each framerecomputed for each frame

INFORMATIKINFORMATIKMotion BlurMotion Blur• Accuracy & Quality Accuracy & Quality

– The same sample point is considered for multiple framesThe same sample point is considered for multiple frames

In other frame-by-frame architectures the motion of In other frame-by-frame architectures the motion of objects must be computed explicitely by additional objects must be computed explicitely by additional samples.samples.

– Temporal changes in shading are properly accounted forTemporal changes in shading are properly accounted for

Difficult in other architecturesDifficult in other architectures

• EfficiencyEfficiency

– 2D computation2D computation

• Accuracy & Quality Accuracy & Quality

– The same sample point is considered for multiple framesThe same sample point is considered for multiple frames

In other frame-by-frame architectures the motion of In other frame-by-frame architectures the motion of objects must be computed explicitely by additional objects must be computed explicitely by additional samples.samples.

– Temporal changes in shading are properly accounted forTemporal changes in shading are properly accounted for

Difficult in other architecturesDifficult in other architectures

• EfficiencyEfficiency

– 2D computation2D computation

INFORMATIKINFORMATIKResultsResults

Speedup for bi-directional path Speedup for bi-directional path tracingtracing• Moving camera, moving objects: 7.7Moving camera, moving objects: 7.7

• Moving objects only:Moving objects only: 13.3 13.3

• Moving camera only:Moving camera only: 8.8 8.8

Speedup for bi-directional path Speedup for bi-directional path tracingtracing• Moving camera, moving objects: 7.7Moving camera, moving objects: 7.7

• Moving objects only:Moving objects only: 13.3 13.3

• Moving camera only:Moving camera only: 8.8 8.8

Proportion of native samples 2.4 - 4.7 %Proportion of native samples 2.4 - 4.7 %Proportion of native samples 2.4 - 4.7 %Proportion of native samples 2.4 - 4.7 %

Cost of native samples (profiler) 44 - 64 % Cost of native samples (profiler) 44 - 64 % of the whole computation time. of the whole computation time.Cost of native samples (profiler) 44 - 64 % Cost of native samples (profiler) 44 - 64 % of the whole computation time. of the whole computation time.

Disc caching overhead 10 % slowdown Disc caching overhead 10 % slowdown (for 1% of main memory used)(for 1% of main memory used)Disc caching overhead 10 % slowdown Disc caching overhead 10 % slowdown (for 1% of main memory used)(for 1% of main memory used)

Time per frame 240 – 415 sec.Time per frame 240 – 415 sec.Time per frame 240 – 415 sec.Time per frame 240 – 415 sec.

INFORMATIKINFORMATIKMotion blur renderingsMotion blur renderings

Motion Blur - computational overhead 25%Motion Blur - computational overhead 25%Motion Blur - computational overhead 25%Motion Blur - computational overhead 25%

Time per frame 120 sec. Time per frame 150 sec.Time per frame 120 sec. Time per frame 150 sec.Time per frame 120 sec. Time per frame 150 sec.Time per frame 120 sec. Time per frame 150 sec.

INFORMATIKINFORMATIKVideosVideos

INFORMATIKINFORMATIKConclusionsConclusions• We presented an efficient architecture for rendering of high-We presented an efficient architecture for rendering of high-

quality animationsquality animations

• In our architecture path tracing, texturing and shading, motion In our architecture path tracing, texturing and shading, motion blur can be efficiently computed.blur can be efficiently computed.

• Our architecture is efficient in particular for scenes in which a Our architecture is efficient in particular for scenes in which a limited number of objects moves locally in the scene and limited number of objects moves locally in the scene and camera motion is slow. camera motion is slow.

– For more complex motion scenario the animation segment For more complex motion scenario the animation segment length can be reduced which in the limit may boil down to length can be reduced which in the limit may boil down to the traditional frame-by-frame computation.the traditional frame-by-frame computation.

• Data structures handling dynamic objects require additional Data structures handling dynamic objects require additional memory, which is acceptable on modern computers. memory, which is acceptable on modern computers.

• The memory overhead involved in storing multiple images is The memory overhead involved in storing multiple images is negligible due to efficient buffering.negligible due to efficient buffering.

• We presented an efficient architecture for rendering of high-We presented an efficient architecture for rendering of high-quality animationsquality animations

• In our architecture path tracing, texturing and shading, motion In our architecture path tracing, texturing and shading, motion blur can be efficiently computed.blur can be efficiently computed.

• Our architecture is efficient in particular for scenes in which a Our architecture is efficient in particular for scenes in which a limited number of objects moves locally in the scene and limited number of objects moves locally in the scene and camera motion is slow. camera motion is slow.

– For more complex motion scenario the animation segment For more complex motion scenario the animation segment length can be reduced which in the limit may boil down to length can be reduced which in the limit may boil down to the traditional frame-by-frame computation.the traditional frame-by-frame computation.

• Data structures handling dynamic objects require additional Data structures handling dynamic objects require additional memory, which is acceptable on modern computers. memory, which is acceptable on modern computers.

• The memory overhead involved in storing multiple images is The memory overhead involved in storing multiple images is negligible due to efficient buffering.negligible due to efficient buffering.

INFORMATIKINFORMATIKFuture WorkFuture Work

• Reusing samples for many pixels in the same frame Reusing samples for many pixels in the same frame as suggested in [Bekaert et al. 2002, Szirmay-Kalos as suggested in [Bekaert et al. 2002, Szirmay-Kalos 2002]2002]

• Improving the efficiency of visibility tests for Improving the efficiency of visibility tests for reprojected samples using a shaft culling approach.reprojected samples using a shaft culling approach.

• Skipping some visibility tests based on the spatio-Skipping some visibility tests based on the spatio-temporal coherence of neighboring samples as temporal coherence of neighboring samples as implemented in the Maya Rendering System [Sung et implemented in the Maya Rendering System [Sung et al. 2002]al. 2002]

• Reusing samples for many pixels in the same frame Reusing samples for many pixels in the same frame as suggested in [Bekaert et al. 2002, Szirmay-Kalos as suggested in [Bekaert et al. 2002, Szirmay-Kalos 2002]2002]

• Improving the efficiency of visibility tests for Improving the efficiency of visibility tests for reprojected samples using a shaft culling approach.reprojected samples using a shaft culling approach.

• Skipping some visibility tests based on the spatio-Skipping some visibility tests based on the spatio-temporal coherence of neighboring samples as temporal coherence of neighboring samples as implemented in the Maya Rendering System [Sung et implemented in the Maya Rendering System [Sung et al. 2002]al. 2002]

INFORMATIKINFORMATIKThanks!Thanks!

• Polina Kondratieva for rewriting RenderMan ShadersPolina Kondratieva for rewriting RenderMan Shaders

• Markus Weber for help with preparing the scenesMarkus Weber for help with preparing the scenes

• Partial support of IST-2001-34744 Partial support of IST-2001-34744 project RealReflectproject RealReflect

• Polina Kondratieva for rewriting RenderMan ShadersPolina Kondratieva for rewriting RenderMan Shaders

• Markus Weber for help with preparing the scenesMarkus Weber for help with preparing the scenes

• Partial support of IST-2001-34744 Partial support of IST-2001-34744 project RealReflectproject RealReflect

• All anonymous reviewers for their commentsAll anonymous reviewers for their comments• All anonymous reviewers for their commentsAll anonymous reviewers for their comments

INFORMATIKINFORMATIK

Space-Time Architecture: Space-Time Architecture: ReprojectionReprojection

INFORMATIKINFORMATIK

Rendering animations – Rendering animations – ray tracing with shadersray tracing with shaders

Speedup ray tracingSpeedup ray tracing• Moving camera, moving objects: 2.6Moving camera, moving objects: 2.6

Speedup ray tracingSpeedup ray tracing• Moving camera, moving objects: 2.6Moving camera, moving objects: 2.6

( deterministic reflections must be recomputed! )( deterministic reflections must be recomputed! ) ( deterministic reflections must be recomputed! )( deterministic reflections must be recomputed! )

Time per frame 770 sec.Time per frame 770 sec.Time per frame 770 sec.Time per frame 770 sec.