cs380: introduction to computer graphics reconstruction

Min H. Kim (KAIST) Foundations of 3D Computer Graphics, S. Gortler, MIT Press, 2012

CS380: Introduction to Computer GraphicsReconstruction & Resampling

Chapter 17 & 18

Min H. KimKAIST School of Computing


SUMMARYSampling (continuous à discrete)

2


Aliasing• Scene made up of black and white triangles: jaggies

at boundaries– Jaggies will crawl during motion

• If triangles are small enough then we get random values or weird patterns– Will flicker during motion

• The heart of the problem: too much information in one pixel

3


Anti-aliasing• We can also model this as an optimization problem.• These approaches lead to:

• where is some function that tells us how strongly the continuous image value at

should influence the pixel value

4

I[i][j]← I(x , y)Ω∫∫ Fi , j(x , y)dxdy

Fi , j(x , y)

[x , y]t i,j

I[i][j]

I(xw , yw )

I(x , y)←

ΩFi , jxw = i , yw = j

Fi , j(x , y)

I(x , y)

Vertical section view


• Even that integral is not really reasonable to compute• Instead, it is approximated by some sum of the form:

where k indexes some set of locations called the sample locations, so-called over-sampling.

• The renderer first produces a “high resolution” colorand z-buffer “image”,– where we will use the term sample to refer to each of

these high resolution pixels.

Over-sampling

5

I[i][j]← 1

nI(xk , yk )

k=1

n

∑

(xk , yk )

a pixel


Premultiplied Alpha definition• More specifically, let be a continuous

image, and let be a binary valued coverage function over the continuousdomain, with a value of 1 at any point where the image is “occupied” and 0 where it is not.

• Let us store in our discrete image the values:

6

I(x, y)C(x, y)

(x, y)

I[i][j]← I(x , y)

Ωi , j∫∫ C(x , y)dxdy

α[i][j]← C(x , y)dxdy

Ωi , j∫∫

x ⊂ xw = i , y ⊂ yw = j

I[i][j] I(xw , yw )

I(x , y)←

Ωi , j Ωi , j


Premultiplied alpha: Over operation• To compose , we compute

the composite image colors, , using

• That is, the amount of observed background color at a pixel is proportional to the transparency of the foreground layer at that pixel.

• Likewise, alpha (coverage) for the composite image can be computed as:

7

If[i][j] over Ib[i][j]Ic[i][j]

Ic[i][j]← If [i][j]+Ib[i][j](1-α f [i][j])

αc[i][j]← α f [i][j]+α b[i][j](1-α f [i][j])

Including alpha


Alpha blending error• The difference between two

• Therefore the error is the difference between the integral of a product and a product of integrals.

• It is the amount of covariance between the distribution of foreground coverage in some pixel and the distribution of the background data within that pixel.

8

error= Ib(x , y)Cb(x , y)C f (x , y)dxdyΩi , j∫∫ −

Ib(x , y)Cb(x , y)dxdy× C f (x , y)dxdyΩi , j∫∫Ωi , j

∫∫

cov(X ,Y )=E[XY ]−E[X ]E[Y ]

Real alpha blending

Ideal continuous composition


RECONSTRUCTION(DISCRETE à CONTINUOUS)

Chapter 17

9


Reconstruction• Given a discrete image I[i][j], how do we create

a continuous image I(x,y)?• It is the key problem for resizing images and

texture mapping.– How to get texture colors that fall in between texels.

• This process is called reconstruction.

10


Constant reconstruction• A real valued image coordinate is assumed to

have the color of the closest discrete pixel. This method can be described by the following pseudo-code:

• The (int) typecast rounds a number p (pixel position) to the nearest integer not larger than p.

11

color constantReconstruction(float x, float y, color image[][]){int i = (int) (x + .5);int j = (int) (y + .5);return image[i][j];

}


Constant reconstruction• The resulting continuous image is made up of

little squares of constant color.• Each pixel has an influence region of 1-by-1

12


Bilinear properties• At integer coordinates, we have I(x,y)=I[i][j]; the

reconstructed continuous image I agrees with the discrete image I.

• In between integer coordinates, the color values are blended continuously.

• Each pixel in the discrete image influences, to a varying degree, each point within a 2-by-2 square region of the continuous image.

• The horizontal/vertical ordering is irrelevant.• Color over a square is bilinear function of (x,y).

13


Bilinear function• 1 by 1 square with coordinates

for some fixed .

• where are the fracx and fracy above.

14

j < y < j +1i < x < i +1

i and j

I(i+ x f , j+ y f )←(1− y f ) (1− x f )I[i][j]+(x f )I[i+1][j]{ }+( y f ) (1− x f )I[i][j+1]+(x f )I[i+1][j+1]{ } ,

x f and yf

x f

y f


Bilinear reconstruction• Can create a smoother looking reconstruction using

bilinear interpolation.• Bilinear interpolation is obtained by applying linear

interpolation in both the horizontal and verticaldirections.

15

color bilinearReconstruction(float x, float y, color image[][]){int intx = (int) x;int inty = (int) y;float fracx = x - intx;float fracy = y - inty;

color colorx1 = (1-fracx) * image[intx][inty] +(fracx) * image[intx+1][inty];

color colorx2 = (1-fracx) * image[intx][inty+1] +(fracx) * image[intx+1][inty+1];

color colorxy = (1-fracy) * colorx1 +(fracy) * colorx2;

return(colorxy);}


Bilinear function• Rearranging the terms,

• This function has terms that are constant, linear, and bilinear terms in the variables

16

I(i+ x f , j+ y f )← I[i][j]+ −I[i][j]+I[i+1][j]( )x f + −I[i][j]+I[i][j+1]( ) y f + I[i][j]-I[i][j+1]-I[i+1][j]+I[i+1][j+1]( )x f y f

(x f , yf )


Bilinear basis function• Rearrange the bilinear function to obtain:

• For a fixed position , the color of the continuous reconstruction is linear in the discrete pixel values of I:

17

I(i+ x f , j+ y f )← 1− x f − y f + x f y f( )I[i][j]+ x f − x f y f( )I[i+1][j]+ y f − x f y f( )I[i][j+1]+ x f y f( )I[i+1][j+1]

(x f , yf )

I(x , y)← Bi , j(x , y)I[i][j]

i , j∑


Bilinear basis function• These B are called basis functions (tent

functions)• They describe how much pixel i, j influences the

continuous image at .• In 1D, we can define a univariate hat function

.

18

[x, y]t

Hi (x)

Hi(x)=x − i+1fori−1< x < i−x+ i+1fori < x < i+1

0else

⎧

⎨⎪

⎩⎪


• In 2D (bilinear function), let be a bivariate function:

• This is called a tent function

Bilinear basis function

19

Ti, j (x, y)

Ti , j(x , y)=Hi(x)Hj( y).


Bilinear basis function• In constant reconstruction, is a box

function that is zero everywhere except for the unit square surrounding the coordinates (i,j), where it has constant value 1.

20

I[i][j]

I(xw , yw )

I(x , y)←Ω

Fi , j

xw = i , yw = j

Bi, j (x, y)

I[i][j]← I(x , y)Ω∫∫ Fi , j(x , y)dxdy


RESAMPLING(RECONSTRUCTION+SAMPLING,DISCRETEàCONTINUOUSàDISCRETE)

Chapter 18

21


Resampling• Let’s revisit texture mapping• We start with a discrete image and end with a

discrete image.• The mapping technically involves both a

reconstruction and sampling stage.• In this context, we will explain the technique of

mip mapping used for anti-aliased texture mapping.

22


Resampling equation• Suppose we start with a texture image (discrete)

T[k][l] and apply some 2D warp to this image to obtain an output image I[i][j].

23

T[k][l]← T(x , y)Ω∫∫ Fk ,l(x , y)dxdy


Resampling equation• Reconstruct a continuous texture using

a set of basis functions .

• Apply the geometric wrap (at the view point) to the continuous image.

24

T(xt , yt )Bk ,l(xt , yt )

texturecoordinates(asVV): v

wn

and 1wn

T(xt , yt )← Bk ,l(xt , yt )T[k][l]

k ,l∑

T '(xw , yw )←T(M(xw , yw ))(xt , yt )=M(xw , yw )

M (Ωi, j )


Resampling equation• Integrate against a set of filters (e.g.,

a box filter) to obtain the discrete output image.

25

Fk ,l(xw , yw )

I[i][j]← T '(xw , yw )Ω∫∫ Fi , j(xw , yw )dxwdywM (Ωi, j )


Resampling equation• Let the geometric transform be described by a warp

mapping function , which maps from continuous window to texture coordinates.

• We obtain:

26

(xt , yt )=M(xw , yw )

I[i][j]← Fi, j (xw , yw )

Ω∫∫ Bk ,l M (xw , yw )( )k ,l∑ T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxwdyw

= T[k][l] Fi, j (xw , yw )

Ω∫∫ Bk ,l M (xw , yw )( )( )dxwdyw( )k ,l∑

I(x , y)← Bi , j(x , y)I[i][j]

i , j∑

(we could obtain an output pixel as a linear combination of the input texture pixels.)


Resampling equation

• where is the inverse mapping

is the Jacobian matrix of N (changes of area sizes)(continuity of composition of F and N)

27

I[i][j]← det(D

M −1 ) Fi, j M −1(xt , yt )( )( )M (Ωi , j )∫∫ Bk ,lk ,l∑ (xt , yt )T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxtdyt

DN

ʹF =F !N

N =M−1

= det(DN ) ʹFi, j (xt , yt )( )M (Ωi , j )∫∫ Bk ,l

k ,l∑ (xt , yt )T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxtdyt

1( , ) ( , )w w t tx y M x y-=

• Instead, we can rewrite the integration over the texture domain, instead of the window domain.


Ω∫∫ Bk ,l M (xw , yw )( )k ,l∑ T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxwdyw



Resampling equation

• In summary, when F is a box filter

• where is how much texel k,l blends to obtain the value at continuous texture location p, from reconstruction step.

28

= det(DN )

M (Ωi , j )∫∫ Bk ,lk ,l∑ (xt , yt )T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxtdyt

I[i][j]← Bk ,l M (xw , yw )( )

k ,l∑ T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxw dywΩ∫∫

Bk ,l (p)



Ω∫∫ Bk ,l M (xw , yw )( )k ,l∑ T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxwdyw

= det(DN ) ʹFi, j (xt , yt )( )M (Ωi , j )∫∫ Bk ,l

k ,l∑ (xt , yt )T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxtdyt


Texture mapping cases• That is, we need to integrate over the region

on the texture domain, and blend that data together.

• If M is enlarging the texture, then has a very narrow footprint over

• Conversely, if our transformation M effectively shrinks the texture, then has a large footprint over

• During texture mapping, M can also do funnier things, like shrink in one direction only.

29

M(Ωi , j )

M(Ωi , j )T(xt , yt )

M (Ωi, j ) T (xt , yt )


Enlarging texture• In the case that we are enlarging the texture

(narrow footprint in windows), the filteringcomponent has minimal impact on the output.

• In particular, the footprint of may be smaller than a pixel unit in texture space, and thus there is not much detail that needs blurring/averaging.

• As such, the integration step can be dropped, and the resampling can be implemented as

30

M (Ωi, j )

I[i][j]← Bk ,l(xt , yt )

k ,l∑ T[k][l]⇐ det(DN )M(Ωi , j )

∫∫ Bk ,l(xt , yt )k ,l∑ T[k][l]⎛

⎝⎜⎜

⎞

⎠⎟⎟dxtdyt


Enlarging texture• We tell OpenGL to do this using the call

glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR).

• For a single texture lookup in a fragment shader, the hardware needs to fetch 4 texture pixels and blend them appropriately.

31


Shrinking texture• In the case that a texture is getting shrunk

down, then, to avoid aliasing, the filtercomponent should not be ignored.

• Unfortunately, there may be numerous texture pixels under the footprint of , and we may not be able to do our texture lookup in constant time.

32

M (Ωi, j )


MIP mapping• An acronym of the Latin phrase, multum in

parvo (much in little).• Mipmaps are precalculated scaled versions of an

original texture.

33


MIP mapping• Observe the pattern of each mipmap as it decreases

in size; it is half the dimension of the previous one. • This pattern is repeated until the last mipmap’s

dimension is 1×1. • Interpolate multi-resolution textures via trilinear

interpolation

34


MIP mapping in detail• In mip mapping, one starts with an original

texture and then creates a series of lower and lower resolution (blurrier) texture .

• Each successive texture is twice as blurry. And because they have successively less detail, they can be represented with ½ the number of pixels in both the horizontal and vertical directions.

35

T 0

T i


MIP mapping in detail• This collection, called a mip map, is built before

the any triangle rendering is done.• Thus, the simplest way to construct a mip map is

to average two-by-two pixel blocks to produce each pixel in a lower resolution image.

• During texture mapping, for each texture coordinate , the hardware estimates how much shrinking is going on.– How big is the pixel footprint on the geometry.

36

(xt , yt )


MIP mapping in detail• This shrinking factor is then used to select from an

appropriate resolution texture from the mipmap. Since we pick a suitably low resolution texture, additional filtering is not needed, and again, we can just use reconstruction.

• To avoid spatial or temporal discontinuities where/where the texture mip map switches between levels, we can so-called trilinearinterpolation. We use bilinear interpolation to reconstruct one color from and another reconstruction from . These two colors are then linearly interpolated. This third interpolation factor is based on how close we are to choosing level i or i+1

37

T i

T i

T i+1


MIP mapping in detail• Mip mapping with trilinear interpolation is

specified with the callglTexParameteri(GL TEXTURE 2D, GL TEXTURE MIN FILTER,GL LINEAR MIPMAP LINEAR)

• Trilinear interpolation requires OpenGL to fetch 8 texture pixels and blend them appropriately for every requested texture access.

38


Trilinear interpolation• Trilinear interpolation (3D):

39

p(x, y, z) = c0 + c1Δx + c2Δy + c3Δz + c4ΔxΔy + c5ΔxΔzc6ΔyΔz + c7ΔxΔyΔz

p0 + [0,1,n,n +1,n2,

n2 +1,n2 + n,n2 + n +1]The dimension of CLUT is (2n+1)x(2n+1)x(2n+1)


Trilinear interpolation• where

40

c1 = (p100 − p000 ) / (x1 − x0 )c2 = (p010 − p000 ) / (y1 − y0 )c3 = (p001 − p000 ) / (z1 − z0 )c4 = (p110 − p010 − p100 + p000 ) / [(x1 − x0 )(y1 − y0 )]c5 = (p101 − p001 − p100 + p000 ) / [(x1 − x0 )(z1 − z0 )]c6 = (p011 − p001 − p010 + p000 ) / [(y1 − y0 )(z1 − z0 )]c7 = (p111 − p011 − p101 − p110 + p100 + p001 + p010 − p000 ) /

[(x1 − x0 )(y1 − y0 )(z1 − z0 )].

Δx = x − x0Δy = y − y0Δz = z − z0c0 = p000


Properties• It is easy to see that mip

mapping does not do the exactly correct computation.

• First of all, each lower resolution image in the mipmap is obtained by isotropic shrinking, equally in every direction. But, during texture mapping, some region of texture space may get shrunk in only one direction.

41

No mipmapping

Mipmapping

Anisotropic mipmapping


Properties

42

M (Ωi, j )

• Even for isotropic shrinking, the data in the low resolution image only represents a very specific, dyadic, pattern of pixel averages from the original image.

• Filtering can be better approximated at the expense of more fetches from various levels of the mip map to approximately cover the area

on the texture.• This approach is often called ansiotropic filtering

and can be abled in an API or using the driver control panel.

cs380: introduction to computer graphics reconstruction

Documents