Synthesis of Compact Textures for real-time Terrain Rendering

Nader Salman22 juin 2007

Encadrant : Sylvain Lefebvre

Motivations• Need for large textures

– Simple repetition: low quality

• Texture synthesis from example– Pre-computation:

• Need to store large result

• Limited memory!

– Image Compression:• Rendering: Hinders sequential coding

• Block based: Fixed rate (÷ 6)

– Our approach :

Synthesize compact textures, that can be efficiently decoded at rendering time

[Garber 1981, Popat & Picard 1993, Efros & Leung 1999, Wei & Levoy 2000, Ashikhmin 2001, Hertzmann et al 2001, Tong et al 2002 …]

Our approach

• Inspired by patch-based synthesis– Patch-based Synthesis

[Praun et al 2000, Liang et al 2001, Efros and Freeman 2001, Kwatra et al 2003]

Our approach

• Inspired by patch-based synthesis– Record patch positioning and stitching

(instead of final image)– Decode at rendering time

• Main difficulties:– Adapt synthesis for compact encoding– Prepare for efficient decoding on GPU

ExemplarExemplar

Positioning

1 2

Cuts

3

Stitch

Our approach

A first example

• Memory size:– Full image: 805 MB ( result size: 16K x16K )– Our encoding: 26 MB ( patch size: 128x128 )

Exemplar(1024x1024)

Synthesis Result

Compact representation

Overview

• Introduction• Our approach

– Patch positioning– Optimal cuts– Seam removal

• Results and performance• Future Work

– Multiple materials– Slope– GPU Decoding

Our approach

ExemplarExemplar

Positioning

1 2

Cuts

3

Stitch

1. Positioning

ExemplarExemplar

• Scan exemplar• Choose best fit patch• Place patch• Memorize position

?

Patch positionPatch position

Compact representationCompact representation

ExemplarExemplar

Positioning

1 2

Cuts

3

Stitch

Our approach

Why do we cut?

Graph cut

1 4 72 5 83 6 9

1 4 72 5 83 6 9

A

B Construct graph such that:

Graph Mincut Best Seam

A

∞

∞

∞B

∞

∞

∞

1 4

2 5

3 6

7

8

9

Potential Seam Cost Constraint

20 100

3010

1585

40

35

1 4

2 5

3 6

7

8

9

A B

20

3010

15

Seam

12 53 6

4 789

A

B

[Kwatra 2003]

Graph cut

• Problem with encoding:– Cut can be arbitraryExpensive to store!

• Our approach– Constrain cuts to be 1D functions– Still search for optimal!

0

0

2

2

2

2

2

1

1

1

1

0

Our approach

ExemplarExemplar

Positioning

1 2

Cuts

3

Stitch

2. Optimal Cuts

Current outputCurrent output

A

B

B

Constrained arcsConstrained arcs

Bidirectional arcsBidirectional arcs

One D. SeamOne D. Seam

Horizontal Overlap ImageHorizontal Overlap Image 00 02 2 2 2 1 1 0 0

Vertical Overlap ImageVertical Overlap Image 2 2 2 1 1 0 1 1

Compact representationsCompact representations

ExemplarExemplar

Positioning

1 2

Cuts

3

Stitch

Our approach

Why stitching ?

• Seams might still be visible in some cases

Avoid visible discontinuities

Our approach3. Stitching

New patch Existing texture

PoissonImageEditing

Current positionMask

[Perez et al. 2003]

Encoding of Poisson field (1)

Before Poisson After Poisson Deformation Field(x8 for visualisation)

Encoding of Poisson field (2)

• Problem: – Deformation is an image as big as output!

Approx ?

• Compression:– Per patch

• Approximate as sum of 2 fields• Each is a ‘flattened 1D pyramid’

– Optimize with least square• argmin || Ax - b ||2

– Fast reconstruction on the GPU• 1D pyramid = 1D MIP-map

Encoding of Poisson field (3)

High Frequency

Smooth

Encoding of Poisson field (4)

pSNR = 43,7 dB

Output with Poisson Approximated Output

High Frequencies

Output with Poisson

pSNR = 50,2 dBpSNR = 43,2 dB pSNR = 46,12dB

Approx. outputs

Overview• Introduction• Our approach

– Patch positioning– Optimal cuts– Seam removal

• Results and performance• Upcoming Work

– Multiple materials– Slope– GPU Decoding

Algorithm performance

Patch size(pixels)

Output size (MB)

Patch Size 16x16Output 2,4 GB

Patch Size 32x32Output 542 MB

Patch Size 128x128Output 90 MB

Input size: 1024 x 1024 Output size: 32K x 32K

32K x 32K 3GB uncompressed

Quick comparison

Our approach

[Kwatra 2003]

Overview

• Introduction• Our approach

– Patch positioning– Optimal cuts– Seam removal

• Results and performance• Upcoming Work

– Multiple materials– Slope– GPU Decoding

Conclusion• Need for detailed large textures

– Limited memory

• Contributions– High quality texture synthesis

• Results comparable to existing schemes– Directly into compact representation

• Reduced storage– Decoded at rendering time only

• Never stored ‘uncompressed’

• First step in more general approach– Data amplification:

• Details at runtime from compact representation

Thank you

Compression value

Output size

4k x 4k 8k x 8k 16k x 16k 131k x 131k

Uncompressed 50 MB 201 MB 805 MB 51 GB

Our technique 4 MB 9 MB 26 MB 1.4 GB

Exemplar size: 1024 x 1024 pixelsPatch Size: 128 x 128 pixels

Ratio improves as output gets largerVaries in this case from 12 up to 36

Compression

• Defauts des compression JPEG/PNG– Pas effectuable au rendu– Necessite de decompresser sur GPU

• Division en blocs – Compression maximale facteur 6– Entraine des pertes significatives

• Or nous voulons– Facteur plus important que 6– Pouvoir etre realise au rendu sur GPU

Calculating the compact size• Output side size : S• Exemplar side size : E• Patch side size : P• Overlap size : O

• Number of patches per side :

n = S ÷ (P – O)

• Number of patches :

nPatches = ceil (n) * ceil(n)

• Total size :

total = nPatches*2 + 2*p*nPatches + e*e*3 + 2*(2*P-1)*2*nPatches

Run-time synthesis

Tiles

Combined to form a texture

• Synthesized at rendering time– Pixel-based

Fine-scale variety

Fast and parallel

Unique material

Complex to implement

– Tile-based

On-demand evaluation

Variety limited by number of tiles

Too many tiles if multiple materials

Distinctive features reveal tiling structure

[Lefebvre and Hoppe 2005]

[Cohen etal. 2003, Lefebvre and Neyret 2003, Wei 2004]

Example

Run-time synthesis

Tiles

Combined to form a texture

• Pixel-based Fine-scale variety

Fast

Still not fast enough

Unique material

Complex to implement

• Tile-based On-demand evaluation

Variety limited by number of tiles

Distinctive features reveal tiling structure

Exemplar Synthesized

[Lefebvre and Hoppe 2005]

[Wei 2004]

g

g

over div vf

[Perez et al 2003]

Poisson image editing

f*

Ω

g

PoissonImageEditing

f

*2 with min ffgf

f

© Disney/Pixar 2003

Oblivion ©2006 Bethesda Softworks

Bad transitions

Distortion

Repetitions

Multiple materials• User guided• Edge map• Color or edge depending on material

(In progress)

Steep patches

• Difficulty:– Adapt each step for steep patches

(In progress)

• Input size: 1024 x 1024 px

• Output size: 32K x 32K px

• Uncompressed output :

32K x 32K x 3 = 3 GB !!

Our algorithm : 90 MB !!

Algorithm performance (1)

Contributions

• High quality texture synthesis– Results comparable to existing schemes

• Directly into compact representation– Reduced storage

• Decoded at rendering time only– Never stored ‘uncompressed’