method and apparatus for performing tangent space lighting and

(12) United States Patent Duluk et al.

US006771264B1

(10) Patent N0.: (45) Date of Patent:

US 6,771,264 B1 Aug. 3, 2004

(54) METHOD AND APPARATUS FOR PERFORMING TANGENT SPACE LIGHTING AND BUMP MAPPING IN A DEFERRED SHADING GRAPHICS PROCESSOR

(75) Inventors: Jerome F. Duluk, Palo Alto, CA (US); Stephen L. Dodgen, Boulder Creek, CA (US); Joseph P. Bratt, San Jose, CA (US); Matthew Papakipos, Menlo Park, CA (US); Nathan Tuck, San Diego, CA (US); Richard E. Hesse], Pleasanton, CA (US)

(73) Assignee: Apple Computer, Inc., Cupertino, CA (Us)

( * ) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 U.S.C. 154(b) by 0 days.

(21) Appl. No.: 09/213,990

(22) Filed: Dec. 17, 1999

(Under 37 CFR 1.47)

(51) Int. Cl.7 .............................................. .. G06T 15/60

(52) U.S. Cl. ...................................... .. 345/426; 345/584

(58) Field of Search ............................... .. 345/426, 582,

345/583, 584

(56) References Cited

U.S. PATENT DOCUMENTS

4,783,829 A 11/ 1988 Miyakawa et 211. 4,945,500 A 7/1990 Deering 4,970,636 A 11/1990 Snodgrass et 81. 5,083,287 A 1/1992 Obata et 81. 5,123,085 A 6/1992 Wells et 81. 5,402,532 A 3/1995 Epstein et 81. 5,448,690 A 9/1995 Shiraishi et 81. 5,455,900 A 10/1995 Shiraishi et 81. 5,481,669 A 1/1996 Poulton et 81. 5,574,836 A 11/1996 Broemmelsiek .......... .. 395/127

5,579,455 A 11/1996 Greene et 81. 5,596,686 A 1/1997 Duluk, Jr.

(List continued on next page.)

SM

OTHER PUBLICATIONS

Angel (interactive computer graphics: a top—doWn approach With OpenGl: ISBN: 0—201—85571—2—sections 6.8 &

7.7.2). Schilling et al., “Texram: a smart memory for texturing,” IEEE computer graphics and applications, 5/96, 32—41.

Primary Examiner—MattheW Luu (74) Attorney, Agent, or Firm—R. Michael Ananian; Dorsey & Whitney LLP

(57) ABSTRACT

A system and method for performing tangent space lighting in a deferred shading graphics processor (DSGP) encom passes blocks of the DSGP that preprocess data and a Phong shader that executes only after all fragments have been preprocessed. A preprocessor block receives texture maps speci?ed in a variety of formats and converts those texture maps to a common format for use by the Phong shader. The preprocessor blocks provide the Phong shader With interpo lated surface basis vectors (v5, v,, n), a vector Tb that represents in tangen/obj ect space the texture/bump data from the texture maps, light data, material data, eye coordinates and other information used by the Phong shader to perform the lighting and bump mapping computations. The data from the preprocessor is provided for each fragment for Which lighting effects need to be computed. The Phong shader computes the color of a fragment using the information provided by the preprocessor. The Phong shader performs all lighting computations in eye space, Which requires it ?rst to transform bump data from tangent space to eye space. In one embodiment the Phong hardWare does this by multiplying a matrix M Whose columns comprise eye space basis vectors (b5, b,, n) derived from the surface basis vectors (v5, v,, n) and the vector Tb of bump map data. The eye space basis vectors are derived by the DSGP preprocessor so that the multiplication (M><Tb) gives the perturbed surface normal N‘ in eye space, re?ecting the bump effects. The perturbed surface normal N‘ is subsequently used in the lighting computations.

35 Claims, 41 Drawing Sheets

US 6,771,264 B1

US. PATENT DOCUMENTS 5,920,326 A 7/1999 Rentschler et a1. ....... .. 345/503 _ 5,949,424 A * 9/1999 Cabral et a1. ............. .. 345/426

576647071 A 9/1997 Nagashlma 5,949,428 A 9/1999 Toelle et 211.

27233723? 2 15/133; g‘?alllllet al-l 5,977,977 A 11/1999 Kajiya et a1. 7 7 / I a e‘ a‘ 5,990,904 A 11/1999 Grif?n

5,710,876 A 1/1998 Peercy et a1. ............. .. 395/126 . 6,002,410 A 12/1999 Battle

5,767,859 A 6/1998 ROSSlIl et a1. _ . 6,002,412 A 12/1999 Schlnnerer 5,798,770 A 8/1998 Baldwln

. . . 6,118,452 A 9/2000 Gannett 5,828,378 A 10/1998 Shlralshl _

6,128,000 A 10/2000 Jouppl et 211. 5,841,447 A 11/1998 DreWs 6,167,486 A 12/2000 Lee et 211.

5,850,225 A 12/1998 Cosman , 6,204,859 B1 3/2001 Jouppl et 211. 5,854,631 A 12/1998 Akeley et a1.

. 6,229,553 B1 5/2001 Duluk, Jr. et 211. 5,860,158 A 1/1999 Pal et a1. .. 6,246,415 B1 6/2001 Grossman et 211.

5,864,342 A 1/1999 Kapya et a1. 6259 452 B1 72001 C t 1 RE36,145 E 3/1999 DeAguiar et a1. 7 7 / 0mg 6 a'

5,880,736 A 3/1999 Peercy et a1. ............. .. 345/426 5,889,997 A 3/1999 Strunk * cited by examiner

U.S. Patent

GENERIC 3D GRAPHICS PIPELINE

Aug. 3, 2004 Sheet 2 0f 41

l

US 6,771,264 B1

>_ FLOATING-POINT INTENSIVE EUNCT IONS

I >_ DRAWING INTENSIVE FUNCTIONS

\( DATA INPUT ) 2|4 2|6\’ musroamnou ‘ m\ cum“ m\ FACE ommmmou 224\ vmaxusmme m\ CLIP(|FNEEDED) \ PERSPECTIVE DIVIDE

228\ SCREEN SPACE 232 common \ SET UPFOR

KINCREMENTAL moan J 234 m\r EDGEWALK “ m\ SPAN INTERPOLATION 24 \\ Z-BUFFERED BLEND J

z \ FRAME BUFFER

244\rnouBLE BUFFERED Mux“ k oumn LOOKUP TABLE J

“K DIGITALTO ANALOG CONVERSION

(moa ART)

U.S. Patent Aug. 3, 2004 Sheet 3 0f 41 US 6,771,264 B1

US 6,771,264 B1

525

m mass: W595;

m

M A E222: =25: 56.55

We

=22: w :35;

m “II n

e l"

m Ea .:

P SH U


52 A .5:

5:2;

#252 #255 Ill: maze. :25: IL u n

:25; | m; _m;.

5.5;. H 22.5: | _II 5:22. I + I“? gang .2 H a

“52;: If m.


1010 2000 10000 / I /

' IIIIIHO§TII'.-- __T\COMMAND 2D :>‘ .............. -. >20 APPLICATION OGUD3D DRNER GHC ___MEM0RY___ GHC , g COMMANDS &

""""""" " GUI

VERTEX

3000\ GEOMETRY 20 ENGINE PIXELS

GOURAUD SHADED, CLIPPED PRINITIvEs AND MODES 10000 . 3D ~ DISPLAY

________ __ /5°°° / 150$ PIxELs CONTROL

EXTRACT MODES /...MEMORY-_. MODES / MODE‘ VSP * """"""" " ' INJECT g-?’zmm \ 4000

SPATIAL VSP INFO T000 _/ COLOR

TILE :::I'soR'T'.'.'.. VSPCOVER SORT SORTED -------------- -- PRIM. COLOR

, PR'M- :MEMQEYZ: A MODES

MM‘ \6000 11000 PER 9000 FRAGMENT /

TILE / INTEIITPE0LTATI0N PHgNG - _OR

I PRIM.|NFO ' BUMP sETuP “mp5s CULL VSPX,Y.Z 19,0

3000 TEXTURE 'IITEXTURE--

12000 / IINE'NI6RY5-T

FIG. 7

U.S. Patent

DEFERRED SHADING GRAPHICS PROCESSOR VERSION l

Aug. 3, 2004

m -

\( DATA INPUT ) 2|4 “?r TRANSFORMATION ‘ m\ CLIPTEST m\ FACEDETERMINATION 22} vmixusmmo m\ CLIP(|FNEEDED) \ PERSPECTIVE own):

228\ SCREEN SPACE m CONVERSION \ SETUPFOR I

LINCREMENIAL among a

25o\ SCENEMEMORY zsl\? EXACT HIDDEN

SURFACE REMOVAL 25.2 253 \?mmm comma ‘ \ BLENDING J

z42\ FRAME BUFFER

244Ynoum: BUFFERED nux? L ouwur LOOKUP mu;

246\ DIGITAL T0 ANALOG CONVERSION

FIG. 8

Sheet 9 0f 41 US 6,771,264 B1

F FLOATING-POINT INTENSIVE FUNCTIONS

U.S. Patent Aug. 3, 2004 Sheet 10 0f 41

I000

2000

COMMAND FETCH AND DECODE

IN FROM l/O BUS

FRAGMENT

TEXTURE IN FROM |/0 ' FRAMEBUFFER MEMORY BUS 2” GMPH'“ MEMORY

/ / moo laooo nooo

FIG. 9

US 6,771,264 B1

7000 6000 (J

SORT MEMORY

OUT TO DISPLAY


7 FOUR

FRAGMENTS

WHITE BLACK FILLED FRAGMENTS FRAGMENT BACKGROUND PIXELS

FIG. 11A FIG. 118 FIG. 11C FIG. 11D


2010

20l2 I cm BLOCK ”

zooo BUS INTERFACE

r l + $ ‘ READ/WRITE 20l6 lNTERRlIPT

m8 ’ comm \zom 2m \ summon L \ / -+ DNA CONTROLLER

BACKE 0mg? ‘ ‘ I DNA DATA ‘

BUS commas VIA Pl0 ' l 20g? 202\2 ‘ 2031 ' 202\6 v 2035

Bus DNA DNA WRITE 2n REQUEST 20 3o mum 30

[2036 TO HOST cm: azsrouss cm: RESPONSE QUEUE QUEUE QUEUE QUEUE QIIEUE

‘7 “k k

|8000 1034 l l R V V R

= ‘ 2o ENGINE mtfgc'gwn ' l 2

F“" _ * i "M = > m 3000

noool BUFFER H5002 W [6004 GEO f

y DISPLAY =


a1&‘ 31m 3112 ZLiR- 31% N Nx N , 315B

8 6 28 CURRENT N° 3x3 NORMALTRANSFO?iMElWiEYQCOORDINATES ) 3m

INVERSE 312a ' ' n

NORMAL Go MODEL- G - 3138 3154 TANGENT VIEW 9 NORMAUZE 3160

MATRIX Gt BINORMAL Bo & 6 m MMV' Be SCALE ' 9

3120 31% 3116 311*“3130 3152 m 3162 Ve Xe, Ye, Ze We AUTOMATIC e Bt VER‘IEX EYE (£0022. SURFACE SI mb

Se Te TANGENT TEXTURE COOR GENERATION B9 _

3122 3132 3136

FIG. 14


BUMP VECTOR

TRIVIAL REJECT BACK FACE CULLING USER DEFINED CLIP VIEW VOLUME CLIP

PERSPECTIVE DIVIDE VIEWPOEAT PROJECT AR SIGN (FRONT 0R BACK)

LINE STIPPLE DEPTH OFFSET

FIG. 15

US 6,771,264 B1 US‘ Patent Aug- 3, 2004 Sheet 16 0f 41

SORT MEMORY

LARRY

WATER‘ \' Y cm STP ->

T ——> MEX | m] I

+

AA ‘ CURLY _> [FD PIX ‘‘ +

Y PHB <- TEX MOE m

[B MEMORY TEXTURE MEMORY

3000 \

GEOMETRY

SPATIAL CULL_MODE, COLOR AND MLM

COLOR AMD MLM DATA 500R‘) DATA

MODE & 4000f EXTRAC'HON POLYGON MEMORY

SPATIAL DATA WITH COLOR POINTERS, CULL__MODE DATA

.__> P 6000/ 30'“ T0 57

Sheet 17 0f 41 US 6,771,264 B1

REUSE TYPE FRONTFACE CULLMODE1 MATFRONT1 MATBACK1

VTXMODE

Aug. 3, 2004

THE FOLLOWING PACKETS WILL BE SENT T0 MEX IN TIME ORDER:

VERTEX PACKET

M. KEEP MIDDLE O: KEEP OLDEST

N:NOTCOMPLETE P: POINT LI LINE T:TR|ANGLE

I 0000000 FIG. 18

NNNNT TTNNTTT T NT NNNNTTTTT MMMMM MMMMOOMMMO OOOOOOMMV 2 2T.

E wmw D 00% DOuTWTWOTWTTTTWTTTMRFHMWTWTWTMWT mmmmmmwmmmmmmmmmmmmwmwmmmmmm CSCSC SCSSSSCSSSCMTUCSCSCSCS WWAVIWW WWWAVEWWWWWW WWWWWWWW 4 W V ..

U.S. Patent


STATE APPROXIMATE PARAMETER DESCRIPTION SIZE IN I44

SET BIT ORDS) IF THE FRONT OR BACK FACING PRIMITIIIES ARE NOT CULLED OUT,

SOFTWARE MAV C OOSE T ATTACH DIFFERENT TEXTURE AND MATERIAL TEXT UREAFRONT PARAMETER TO THE FRONT AND BACK FACING PRIMITIVES. Z

TEXT UREAFRONT PARTITION OF THE STATE VECTOR CONTAINS THE TEXA STATE APPLICABLE IF THE CURRENT PRIMITIVE IS FRONT FACING.

TEXTUREBFRONT PARTITION 0T THE STATE VECTOR CONTAINS THE TEXB IEXIUREBFRONT sTATE APPLICABLE IF THE CURRENT PRIMITIVE IS FRONT FACING. 6

TEXTUREABALK TEXTUREABACK PARTITION OF THE STATE VECTOR CONTAINS THE TEXA STATE APPLICABLE IF THE CURRENT PRIMITIVE IS BACK FACING.

TEXTUREBBACK TEXTUREBBACK PARTITION OR THE STATE vEcroR CONTAINS THE TEXB 6 STATE APPLICABLE IF THE CURRENT PRIMITIVE IS BACK FACING. MATERIALFRONT PARTITION OF THE STATE VECTOR CONTAINS THE

N

"ATEIIIMTRONT MATERIAL STATE APPLICABLE IF THE cuRREMT PRIMITIVE IS FRONT FACING. '0

MATERIMBACK MATERIALBACK PARTITION or THE mm mm CONTAINS THE MATERIAL ,0 WE APPLICABLE IF THE CURRENT PRIMITIVE IS BACK FACING.

LIGHT CONTAINS THE CURRENT LIGHT sTATE. I8 PIXELMODES CONTAINS THE CURRENT PIXEL MODES. 2

STIPPLE CONTAINS CURRENT STIPPLE PATTERN a CULLMODES CONTAINSCURRENTCULLMODES 0.33

"('2 FIG. 19 DIRTY POLVGON MEMORY 4m ‘198 _

MExsTATEvEcToR FLAGS POINTER WHOM MLMPOINTERS

4'°Z\_FR0NTMATER|AL l2] IFRONTMATERIALPTRI- Mmmuo h FACE POLYGON MEMORY, MATERIALPTR

4'°‘'\_ W mm [2 BACIIMATERIALPTII. UPDATE PTR,AND ‘"1 4m CLEARDIRTYBIT 4l46 4'48

4I°T\FRONUEXTUREA [:l IFRONTTEXTUREAPTRI- 4M0 _ 4|24 TAcE TEXTUREAPTR

4 ' 08X IZI WKTEXT'JREAPTR sAvE FRONTTEXTB TO BMKIEXTUREA 4m POLYGON MEMORY

4I'°\FRONHEXTUREB E] FRONTTEXTUREBPTR UPDATE PTR,AHO’ IEXIUREBPIR mg m; CLEARDIRTVBIT 4'50

4"“- IZI BMKTEXTUREBPTR SAVELIGIITTO 4M2 4'52 BACIITEXTUREB f 4|3o POLVGON MEMORY, m

4"4\_ LIGHT |Z| | LIGHTPTR t- UPDATE PTR,AND 4“, ‘ mLAiwfm 4m IXE E D~ PIXELMODES IZI P'XMODEPTI 4'34 SHIILIGOHQROITIO PIXMODEPTR 4| l8 UPDATE PTR. AND

\_ SHPPLE I] STIPPLE4PIT3R6 CLEARDIRWB" STIPPLEPTR ASSUMES THAT POLVGONS ARE FRONT FACING. 4'56

FIG. 20

method and apparatus for performing tangent space lighting and

Documents