Keys To Graphics Progamming

May 31, 2011

Graphics: The End Goal

• The goal of graphics programming is entirely perceptual.– For games/film:

• Does it look good?• Does it produce the right emotional response?

– Visualization:• Does it convey the information you want.

• A bunch of fancy algorithms may or may not help with this.

The Trap of Photorealism

• What is photo-real anyway?

A specific example• On a sunny day, objects in shadow are illuminated

by the blue sky, hence they have a blue tint compared to objects illuminated by the sun.

The human eye is amazing…

• The human eye corrects for this automatically.• Photographers know this, and eschew working in

bright sun.• You eye corrects in many other ways:

– High dynamic range scenes.– Depth of field – humans don’t see it!

• Yet is sees other artifacts (e.g. Cornsweet illusion).

• Recommend: study photography. Good art form if you can’t draw.

Non-photorealistic rendering

• We have just started to explore the possibilities of non-photorealistic rendering.

Performance...

• From a performance perspective, we are a very long way from photorealism anyway.

• Consider the scale of the problem compared to other CS areas:– Web search: 4 words produces a page of text.– Scene description produces 1920x1080x24 bit image at

60hz! 373,248,000 bytes/sec.

• The result is we use a collection of tricks, approximations.

• New techniques are being developed all the time.

Light Mapping• Light mapping:

– Idea is to store the light intensity in a low-res texture.– Light map can be pre-computed, and so can be as

expensive as you want.• Soft shadows• Global illumination

• Final fragment color is (Tc * Lm)– Very cheap per pixel.

• Big downside: lighting is constant– Can’t use on characters or moving props

• Also, no specular.

Dynamic Lights

• Give lights position, direction, attenuation, and calculate intensity.

• Final fragment color is (Tc * (sum of lights))• With deferred rendering, even a large number

of lights can be handled.• Problems:

– Shadows are hard. Soft shadows even harder.– Global illumination.– Even with deferred rendering, lights are limited.

Can’t we have both?• Light maps for things that don’t move, dynamic lights

for the others?• But then…how do you mix them?

– Fragment color = Tc * (Lm + (sum of lights))

• Dynamic lights won’t be shadowed.• What about characters?

– Pull light from the nearest light map sample– Sometime make an irradiance grid.

• Still no specular.• Have to match preprocess and realtime calculations.

Programmer’s Role

• Part of the graphics programmer’s job is to pick a set of techniques for the project at hand.

• Many techniques fight with each other, or interact in weird ways, so they must be balanced.

• Artists often don’t understand technical details..that’s not their job.

Hardware…

• Actually, the performance problem is __so__ bad we get hardware support.– The general purpose CPU has

taken over every problem…except for rendering.

• Even Intel has given up…

Nothing is free..

• Hardware helps, but adds a number of problems:– More low level technical details to track.– Compatibility.– Large performance spread.– Precision issues.– Graphics programmer has to be cognizant of

hardware capabilities.• Example: Z buffer compression enables fast Z clear.

Unity Web Player Hardware Statistics - 2011 Q1

Graphics memory bandwidth

< 1.0 GB/s 2.6%

< 3.0 GB/s 4.9%

< 4.5 GB/s 6.1%

< 6.0 GB/s 23.3%

< 8.0 GB/s 17.8%

< 11.0 GB/s 18.9%

< 15.0 GB/s 5.1%

< 20.0 GB/s 4.1%

< 30.0 GB/s 4.5%

< 50.0 GB/s 3.2%

< 90.0 GB/s 3.8%

Lots 0.8%

Unknown 4.9%

Graphics memory bandwidth

< 0.1 GP/s 2.4%

< 0.2 GP/s 0.2%

< 0.3 GP/s 5.5%

< 0.5 GP/s 7.2%

< 0.7 GP/s 2.7%

< 1.0 GP/s 3.7%

< 1.5 GP/s 21.4%

< 2.0 GP/s 21.5%

< 3.0 GP/s 14.7%

< 4.0 GP/s 1.8%

< 6.0 GP/s 5.7%

< 8.0 GP/s 0.4%

<10.0 GP/s 2.0%

Lots 6.0%

Unknown 4.9

Memory Bandwidth Fill Rate

In short…

• Good art + bad/old tech can still look good.• Fancy tech + bad art will look like crap.

– This is why academic demos all look terrible.• E.g. Prince of Persia – Sands of Time (PS2)

The art Pipeline• Models are made in Max/Maya.• Exported with a modified Fcollada exporter.• Models, textures and shaders are processed to a

platform specific format or “bundled.”– Models are stripified.– Textures compressed.– Shader precompiled.

• At runtime, data is loaded off disk, bound together and sent off the renderer.

• Goal is to preprocess everything.• Recommend: Learn a DCC tool (Max/Maya/Blender).

Pipeline…it’s complicated…

• Many many steps. There are many chances to introduces errors or artifacts.

• Involves UI and many different pieces of software.

• Graphics programmer must be able to work with the art pipeline.– Follow all the math to look for artifacts.– Add features/UI at each step as needed.

Artifacts…• Its very easy to have bugs and

not know about them. • Sometimes, it’s not a bug, but

an artifact. Often these turn on details of the technique.– You must be able to work through

them.

A Shader example• Some shaders are very complicated, but not

always.• Needed to render some terrain from high

altitude with a mix of terrain types.– Should be quick and low cost for artist.– Should give the impression of high-detail.

Solution• One of our engineers created 3-texture blend.• Use low-res blend texture to switch between 3

detail textures.– Requires 2 sets of texture coordinates.

• Add a scrolling cloud texture to give the appearance of shadows.

• Use vertex lighting from DCC Tool (Max).

Pixel Shaderfloat4 pshader(const VS_OUT vi) : COLOR{

float3 blend = tex2D(sampler_Blend, vi.Tex1).rgb;float3 texr = tex2D(sampler_BlendRed, vi.Tex0).rgb;float3 texg = tex2D(sampler_BlendGreen, vi.Tex0).rgb;float3 texb = tex2D(sampler_BlendBlue, vi.Tex0).rgb;float4 color;color.rgb = vi.Color.rgb *

(blend.r * texr + blend.g * texg + blend.b * texb);color.a = vi.Color.a;

// attenuate by cloud coveragecolor.rgb *= tex2D(sampler_Cloud,

vi.Tex1 + float2(fCloudOffsetU, fCloudOffsetV)).rgb;

float3 veye = vi.ObjPos - vEyePos;color.rgb = ApplyFog( color, length( veye ) );

return color;}

Vertex Shader

VS_OUT vshader(const VS_IN vi){

VS_OUT vo;vo.Pos = mul( float4( vi.Pos, 1 ), mOTP );

#if defined(DX9) || defined(X360)vo.Color = vi.Color;

#elsevo.Color = float4( vi.Color, vi.Alpha );

#endifvo.Tex0 = vi.Tex0;vo.Tex1 = vi.Tex1;vo.ObjPos = vi.Pos;return vo;

}

Result

Maps

Blend Map

CloudRed

Blue Green

Actually, it was more complicated…

• Artist wanted more than three textures.

• He broke the scene into many pieces, each with three textures.

Keys to being a good Graphics Programmer

• A passion for graphics and what looks good.• Must be able to read about and implement

new techniques.• Low level programming skills.• Ability to work with artists.• Ability to push around lots of code, and work

with other people’s code.• Strong math skills.