![Page 1: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/1.jpg)
CanvoX: High-Resolution VR Painting for Large Volumetric Canvas
Yeojin Kim1, Byungmoon Kim2, Jiyang Kim1 and Young J. Kim1
Ewha Womans University1, Adobe Research2
S7698:
http://graphics.ewha.ac.kr/canvox/
![Page 2: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/2.jpg)
![Page 3: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/3.jpg)
• Can we recolor, erase, and mix the color?
• Can we draw and mix transparent object?
• How much can we extend canvas?
• How detail can we draw?
• How can we navigate 3D canvas?
Fundamental Questions in VR Painting
![Page 4: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/4.jpg)
Voxels?
![Page 5: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/5.jpg)
Challenges
• Large Canvas with high detail • Deep Level Octree
• Expensive refinement and coarsening
• Dynamic Tree on GPU• Random access → need complex data structure
• CPU-GPU transfer cost
• Rendering • Real-time ray casting (resolution 1680 × 1512 × 2, 90fps~)
• Tree traversal time
• Accumulated error along the ray
![Page 6: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/6.jpg)
CPU
CanvoX Model
One-level refine/coarsen marked cell
Color and mark cell
Paint Thread
GPU
Update 3-neighbor texture
Render Scene(Ray Casting)
Render heat map
Update Scene Quad Tree
Interpolate Scene Quad Tree
HMDController
/Haptic Device
• View• Position
• Brush Position
Main Thread
Initialize Octree
Stroke Data
Octree
Child ID
Parent ID
flag
RGBA
Temp0
Segment ID
Stroke ID
Update Block Ordered Set
block ID
Update view matrix & controller pos.
Render
Store stroke data
Update texture block to GPU
Octree
![Page 7: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/7.jpg)
CPU Side Octree
[Kim15] Byungmoon Kim, Panagiotis Tsiotras, Jeong-Mo Hong , and Oh-young Song,Interpolation and parallel adjustment of center-sampled trees with new balancing constraints
• Strong 2-to-1 Balanced Tree [Kim15]• Root array(Uniform grid) + Tree
• Simple primal-only tree
• Maximum depth level : 26
• Physical Unit : 0.3mm3~ 40𝑘𝑚3
• Each cell has • Parent ID
• Child #0 ID
• Flags – depth, refine, coarsen, etc. …
• RGBA
![Page 8: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/8.jpg)
GPU Side Octree
• GPU has shadow octree of CPU octree• Memory management benefits from CPU
• Convert 1D Array Fields → 2D Array Texture
• Size of texture image : 30MB
• Only updates blocks of texture• Block : 𝑀 ×𝑁 Texels
• Brush causes only local changes with tree
• Tree Index is located on same texelregardless of cell
InterpolationTable Index16bit INT
InterpolationWeight Table16bit FLOAT
IDs & Flag32bit INT
RGBA8bit FLOAT
![Page 9: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/9.jpg)
Refinement and Coarsening
Frame 𝑡0
0 0 1 1
0 1 2 1
1 2 1 0
1 1 0 0
Frame 𝑡𝑛Frame 𝑡1
0 0
0 0
0
0
2
2 1 1
1 0
0 1 1 1
1 1 1
1
1 1 1 1
1 1
1 2
2
2 2
0
0 0 0 0 0 0
0 : outside1 : boundary2 : inside
• At each frame, do only one-level refinement/coarsening • Refinement/Coarsening will finished less than #Max Depth frames• While tree traversal, color the cells and find cells to be refined simultaneously• “Outside” cell helps to reduce tree traversal cost
![Page 10: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/10.jpg)
… and Update Tree on GPU
Update Block Ordered Set
0 1 2 7 3 11 block ID
Frame 𝑡𝑛
Main Thread:Update one block
in every frame
Frame 𝑡0
0 0 1 1
0 1 2 1
1 2 1 0
1 1 0 0
Frame 𝑡1
0 0
0 0
0
0
2
2 1 1
1 0
0 1 1 1
1 1 1
1
1 1 1 1
1 1
1 2
2
2 2
0
0 0 0 0 0 0
Cell to be updated
Push block ID : 0 Push block ID : 1
Pop block ID
![Page 11: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/11.jpg)
![Page 12: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/12.jpg)
Ray Casting in Large Canvas
![Page 13: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/13.jpg)
Ray Casting in Large Canvas
Problem 1Tree traversal from root to leafat every sample points
Problem 2Sampling at empty space is wasting time
Problem 3Error increases along the ray
![Page 14: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/14.jpg)
Octree : 3-Neighbors• Tree traversal from root to leaf at every sample points→ Tree traversal using neighbor cells with ray direction
𝑵𝟎
𝑵𝟏
𝑵𝟐
𝑵𝟑
𝑪
• Thanks to strong 2-to-1 balance tree,• A cell always has 6 neighbors• 3 neighbors share the parent
( = Their ID can be computed by using offset)
• 3 neighbors have different parent
→ If we precompute only 3 Neighbors,we can move to next neighbor directly
Given Cell
Neighbors whichshares the parent
Neighbors whichhave different parent
![Page 15: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/15.jpg)
![Page 16: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/16.jpg)
Ray Casting with Local Coordinates
X
Y
Z
-0.5 0.5
0.5
-0.5
(0., 0., 0.)
vec3 rayCastWithLocalCoord(in vec3 rayDir,in float rayMaxLength,in uint initCellID, in vec3 initRelativeVec)
{posCellID = initCellID;posRelVec = initRelativeVec;
while(rayLength <= rayMaxLength){
//find neighbor cell(nbDir, minDist) = rayBoxIntersection(rayDir, posRelVec)neighborID = getNextNeighbor(posCellID, neighborDir)rayLength += minDist*cellWidth;
if(cellHasColor) colorSum = blendColor(colorVolume);
//update current cell and relative vectorposCellID = neighborID;posRelVec = computeRelativeVectorInNextFrame(posRelVec);
}}
![Page 17: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/17.jpg)
Foveated Rendering
Final Image
NeedImage
Screen Quad Tree
With Screen resolution 𝑾×𝑯,
Heat Map
(𝑾
𝟒×
𝑯
𝟒)
NeedImage
Screen Quad TreeSamples
Level 0(𝑾
𝟒×
𝑯
𝟒)
Level 1(𝑾
𝟐×
𝑯
𝟐)
Level 2(𝑾×𝑯)
Volume Ray Casting
• Convert Screen coordinatesto Tree Texture coordinates
• Make Orphan Nodes according to level
Interpolation
![Page 18: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/18.jpg)
Summary• Dynamic and Simple Octree both on CPU and GPU
• Shadow octree on GPU and local updates
• One-level refine/coarsen strategy
• Ray Casting in Large Canvas• 3-neighbor and ray casting with local coordinates
• Foveated Rendering
• Future work• Performance optimization
• Improve assistive tools
• Isosurface Rendering
![Page 19: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/19.jpg)
![Page 20: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/20.jpg)
![Page 21: CanvoX: High-Resolution VR Painting for Large Volumetric ...graphics.ewha.ac.kr/canvox/files/GTC17.pdf · Young J. Kim, kimy@ewha.ac.kr. Title: CanvoX: High-Resolution VR Painting](https://reader034.vdocuments.us/reader034/viewer/2022052008/601d87d1ca128e1b4a30d9b4/html5/thumbnails/21.jpg)
Thank you
Ack. :
Project Webpage : http://graphics.ewha.ac.kr/canvox/
Yeojin Kim, [email protected]
Byungmoon Kim, [email protected]
Jiyang Kim, [email protected]
Young J. Kim, [email protected]