b y B r i a n H o o k G R A P H I C C O N T E N T

the procedures in place at id are notperfect, they have resulted in the time-ly shipment of several popular prod-ucts, so take this for what you will.This column is pretty dry and matter-of-fact — it’s more like a laundry list,to be honest — so while it may not beamusing and entertaining, I hopemany of you will find it informative.And just to be clear, this is not a recipefor success.

This month, I’ll be talking aboutsome of the programming methodolo-gies that we’ve used during QUAKE 2’Sdevelopment. Next month, I plan todiscuss the tools that we employ, bothsoftware and hardware, and some relat-ed issues.

Team Programming

T he programming staff at id consistsof three programmers: John

Carmack, John Cash, and myself.Programming tasks are split into threedistinct groups: graphics, game logic,and “glue.” The graphics subsystems(OpenGL and software rendering) con-sist of the actual code used to render ascene and are encapsulated into two.DLLs, REF_GL.DLL andREF_SOFT.DLL. The game logic is alsoput in a .DLL, GAMEX86.DLL, which

handles all game-specificstuff such as monster intelli-gence, weapon behavior,physics, and power-upeffects. Finally, the “glue”code, which consists of win-dow system interaction,input management, sound,CD management, networkprotocols, and other not-easy-to-categorize crud, islocated in the executable,QUAKE2.EXE.

The sweet spot for ourteam size is working out tobe three programmers. Thegraphics subsystems are mydomain; the game logic isJohn Cash’s responsibility;and the glue code is usuallymodified by any of us. JohnCarmack is the grand dicta-tor and architect — he modifies broadexpanses of all the code at any giventime and is responsible for the overallarchitecture and making sure that thepieces of QUAKE 2 fit together logicallyand efficiently.

The current triangular hierarchy thatwe have in place is extremely efficientbecause John Carmack is the absoluteruler of the programming team. Eventhough Carmack is the undisputedboss because of his position within id,both Cash and I have extreme respectfor him, and it is this respect thatallows Carmack to manage the devel-opment process effectively. It is farmore important to have respect from

your employees than arbitrary authori-ty over them. Also, by taking care ofimplementation details and minutiae,Cash and I allow Carmack to concen-trate almost exclusively on large,sweeping architectural issues.

Also, a key to making the team pro-gramming approach work, at least forid, is that John Carmack is responsiblefor both the architecture and the initialimplementation of any new technolo-gy. This lets him reconcile any unfore-seen implementation and design inter-actions that may have globalrepercussions within the code base.

Another nice thing about the delega-tion of responsibilities is that there is

h t t p : / / w w w . g d m a g . c o m J A N U A R Y 1 9 9 8 G A M E D E V E L O P E R

11

How I Spent My Summer Vacation,

or What I Learned While Working on

QUAKE 2

lot of people in the game industry have asked me about the software devel-

opment processes used at id software, so I thought I’d take the time to

write an article about the processes and philosophies used at id software

while interspersing my own opinions and reflections on them. WhileA

Brian Hook’s last Game Developer col-umn will appear in next month’s issue.Tell him how much you’ll miss him viae-mail at bwh@wksoftware.com.

very little adversarial competitionbetween programmers. Neither Cashnor I is presumptuous enough to chal-lenge Carmack’s dominance, and sinceCash and I work on separate subsys-tems, our work is complementaryinstead of competitive in nature. Thelines of code ownership are clearlydefined and are something with whichwe’re all very comfortable — and werespect each other enough that wedon’t feel any urge to edit someoneelse’s code. This lets us work in a realteam atmosphere, and we manage toavoid the whole “Who’s The Man?”jockeying that is so common amongcomputer programmers.

One problem that team program-ming presents is source control. Asashamed as I am to admit it, id softwaredoes not use source code control. Rightnow, this discrepancy is largely theresult of expediency. We recognize theneed for proper source control, but wehave enough of a working system thatuntil source control becomes a crisis,we won’t address the problem — espe-cially when we’re this close to shippinga product. Our next generation soft-ware hopefully will be developed com-pletely within the framework of a large-scale source control system. Personally,

however, I use MicrosoftVisual SourceSafe on mymain workstation simplybecause I like to have ahistory of my changes atall times.

Another issue that ariseswhen multiple program-mers work together is cod-ing style. It’s importantnot to get wrapped up inreligious issues such as tabsize, brace and parenthesis

placement, or indentation style— you should be able to adjustto any style, even if it irks you.Fighting battles over something as per-sonal as this simply is not worth theeffort — make some compromises andmove on.

Other coding style issues, however,are worth specifying at the outset of aproduct’s development. Standard-ization and consistency are very impor-tant when working on large projects.Files, data structures, APIs, and vari-ables should have a clean, consistent,and intuitive naming convention sothat there is little room for confusionwhen looking at someone else’s code.Static and global variables should betagged as such, be it by a prefix, suffix,or some other convention. Parameterordering should be consistent: Is a des-tination address the first or last para-meter? Are prefixes used in ssttrruucctt mem-bers? Where are global variablesdeclared and under what conditions?Are globals stuffed into a single globalstructure or just tossed out into theglobal namespace? How are manifestconstants differentiated from ccoonnsstt dec-larations? How are directory structuresorganized?

NIH

F or quite some time (over adecade now), computer scien-

tists have been talking about mod-ular software, component soft-ware, or “software ICs.” Thetheory is that programmers shouldbe able to purchase a thoroughlydebugged and optimized prepack-aged software library (who are wekidding?) from some third-partydevelopment house and just dropit into a program — voila, instantnew features and functionality.

Obviously, there is a differencebetween that theory and the harshrealities of creating a product that hasto ship to real people on a real calen-dar. Libraries are written by program-mers, and programmers are human,and humans make mistakes. Manytimes, these programmers will evenhave a different set of priorities thanyour own.

This is the crux of the problem. Thesoftware component that you pur-chased might look great on paper, butwhen you drop it into your programand then spend a week looking for abug that turns out to be a part of yournew magic software IC, well, you tendto snap out of your dream world prettyquickly. Anyone who has wanted tofirebomb Redmond, Washington, afterusing Microsoft’s DirectX knows whatI’m talking about. And when a fix forthat bug isn’t going to arrive in a time-ly fashion, you’re suddenly in the posi-tion of hacking around broken code, aprocess pleasantly known as “comingup with a workaround.” Deal withenough “workarounds,” and you’lleventually reach a cross-over pointwhere you realize that you may havebeen better off if you’d just written thecode yourself.

And bugs aren’t the only problemyou’ll encounter — there are perfor-mance issues to contend with also.With WINQUAKE, GLQUAKE, and QUAKE

2, we had to work around some prettyserious performance problems inMicrosoft’s DirectSound. DirectSoundworks the way it’s supposed to, but it’sslow enough that you get all teary-eyedremembering the days of Sound Blaster16 programming under DOS.

Finally, not only do you have tocontend with bugs and performance

G R A P H I C C O N T E N T

G A M E D E V E L O P E R J A N U A R Y 1 9 9 8 h t t p : / / w w w . g d m a g . c o m

12

issues, but there’s always the specterof flexibility. That nifty new librarymight do everything you need now,but when you’re six months intousing that library, and you must havea couple new features implemented,and the library owner isn’t amenableto adding those features… well, you’rein trouble. You now have the optionof undoing months of work andrewriting everything from scratch, atwhich point you’ve tossed awaymonths of effort, or you forego theextra functionality, which may not bea feasible alternative. And, of course,if you want to port to a developmentplatform or operating system onwhich the library is not available,you’re in deep trouble. You canaddress many of these problems bylicensing the source code to whateverlibrary you’re using, but at that pointyou’re in the position of actuallylearning someone else’s code, not tomention maintaining, extending, anddebugging it. At some point, you mayfind that you’d have been better offwriting everything yourself fromscratch.

Don’t get me wrong — I’m not say-ing that using externally developedlibraries is absolutely a bad thing, butsome tradeoffs are definitely involved.We have a hard enough time dealingwith bugs in our compiler, the Win32API, Microsoft DirectX, and hardwaredrivers without adding someone else’scode to the mix. So the unofficial poli-cy at id is that we engineer all of ourown code unless we absolutely have nochoice, such as the necessity ofdepending on DirectX. It may not be

the most effective use of our resources,but it leaves our destiny in our hands,which has a certain warm and fuzzyappeal to it.

Programming Languages

A s technology-oriented as id soft-ware is perceived, we’re actually

knuckle-dragging primates when itcomes to our programming languageof choice. We use good old ANSI C forthe majority of our development.Objective-C, a version of C withobject-oriented extensions, was thelanguage of choice for tool develop-ment back when id was usingNextStep. However, during the subse-quent move to Windows NT, id wasforced to abandon Objective-C infavor of ANSI C for these tasks. We’recurrently evaluating the performanceand robustness of OpenStep forWindows NT, and if it turns out that itdoesn’t suck, we may switch back tousing Objective-C and OpenStep fortool development.

id software still uses ANSI C for itscore development; we have severalcompelling reasons why. We stressportability, and ANSI C is about asportable as a language can get — it’savailable across a wide range of plat-forms, and most ANSI C compilers areextremely stable. ANSI C is no longerevolving at a frantic pace, so it’s stablein terms of syntax, feature set, andbehavior. Mechanisms for interfacingANSI C with other languages, such asassembler, are well-defined and pre-dictable. Compilers and developmenttools support ANSI C more than anyother language. Finally, ANSI C isa pretty WYSIWYG language —when you look at a chunk of C,you can be reasonably certainwhat kind of machine code willbe generated.

C++, on the other hand, doesnot share these wonderful fea-tures. C++ is stuck on top of Cusing the programming languageequivalent of duct tape andtwine. It’s still evolving at a dis-turbing rate. It’s being designedby a committee. Compilers andtools that support C++ are con-stantly missing features or incor-rectly implementing them, andthe language, as a whole, is so

large that understanding all of it isnearly impossible. Any given chunk ofC++ code, assuming it uses even asmall portion of the language, can gen-erate seemingly random assemblycode. While you can pick up a booksuch as Bjarne Stroustrup’s Design andEvolution of C++ to help you under-stand why C++ is such a screwed uplanguage, it still doesn’t address theissue that C++ is a screwed up lan-guage. It’s constantly evolving, gettingbigger and uglier, and pretty soon it’sgoing to implode under its ownweight.

Seven years ago, I bought BorlandTurbo C++ 1.00 the day it wasreleased (yes, I’m that big a geek), andover the course of the ensuing five orsix years I used C++ as my only pro-gramming language. In that time, Ilearned most of its weird intricacies,adjusted to them, and accepted themas necessary evils, the price I had topay for object-oriented programming.When I started working at 3DfxInteractive, I had to start using ANSIC again because I was developing aprogramming library, Glide, thatneeded to be used by a lot of develop-ers, many of whom would not befamiliar with C++.

The amazing thing to me was thatwhen I switched back to ANSI C, I wasactually happier — I discovered a new-found appreciation for ANSI C’s sim-plicity (at least compared to C++).Sure, I lost some nice syntactic sugar,and I ended up missing classes and vir-tual functions a bit, but I was willingto eschew these niceties in return forsimplicity. By using ANSI C, I neverhad to crack open a reference book,

G R A P H I C C O N T E N T

G A M E D E V E L O P E R J A N U A R Y 1 9 9 8 h t t p : / / w w w . g d m a g . c o m

14

and I rarely had to work around lan-guage quirks. Since then, I’ve donevery little serious C++ programming,and the last time I looked at the spec,it was a completely different language.The language has mutated so muchover time that it’s become theHighlander 2 of programming lan-guages — you can see similar themesand names, but somehow the new ver-sion screws up so badly that it sulliesthe name of its parent.

id’s use of assembly language hasvaried over the years. DOOM had verylittle assembly language in it, but itwas primarily targeted at 486-classprocessors, where scheduling, pipelin-ing, and memory bandwidth issueswere not nearly as relevant as on latergenerations of processors. QUAKE, onthe other hand, was targeted at IntelPentium-class processors, and as aresult, there was significant room forhand-coded assembly optimization. Italso helped that Michael Abrash, Mr.Assembly Optimization Dude, wasworking at id then, and could devotelarge chunks of time to tweakinginner loops.

QUAKE 2 is also targeted at thePentium, and thus benefits fromassembly coding. However, there is avery significant chance that there

will be little to no assem-bly code in post-QUAKE 2products from id software.There are several reasonsfor this, the primary onebeing that hand codingfor advanced processorssuch as the Intel PentiumPro and Pentium II isoften a lost cause. At anygiven time, a Pentium Procan have a large amountof non-deterministicinternal state that radical-ly affects the efficiency ofhand-coded assembly lan-guage. With theseadvanced, superscalar,and superpipelinedprocessors that supportfeatures such as specula-tive, out-of-order execu-tion and branch predic-tion, it makes far moresense to use CPU-friendlyalgorithms as opposed toCPU-optimized code.

Optimization

O ur optimization rules are quitesimple: make sure that what

you’re optimizing makes a difference,don’t optimize before you’re doneimplementing features, and learn tooptimize up to the point of diminish-ing returns but no further. It’s amazinghow badly our intuition can deceive uswhen it comes to finding execution hotspots. You’ll often look at a programand intuitively label certain areas asdefinite bottlenecks only to find out,after analytical profiling, that someother heretofore-unknown hot spot isactually consuming all of your execu-tion time. Before diving into “prob-lem” routines, use a profilersuch as Intel VTune,Tracepoint HiProf, orRational Visual Quantify(skip the one in MicrosoftVisual C++, it's pretty muchuseless) to make sure thatyou are, in fact, going toedit code that makes a dif-ference in your program’soverall execution time.QUAKE had a hand-opti-mized routine responsiblefor clearing the screen to a

flat color. This routine only would becalled if the game had texture mappingdisabled. Optimizing that clear routineprobably wasn’t time well spent.

It’s easy to get sidetracked into opti-mizing a chunk of code that you’reworking on while you’re still thinkingabout it, then tossing away all of thateffort when you change your algo-rithms yet again. Until your overalldesign is set in stone, it’s wise to avoiddoing any optimization at all. I had achunk of code that I was pretty confi-dent wouldn’t need to be changedagain — it was responsible for trans-forming the points within an Aliasmodel — but as luck would have it, twomonths after I optimized that routine,we ended up needing a new featurethat required editing the optimizedsource. Luckily, this wasn’t overly trau-matic, but if it had been even a bitmore complex, it would have con-sumed far more time than if I had justdone all the optimization closer to theend of QUAKE 2’s development.

Premature optimization also hasanother, more sinister, side effect — itmakes you reluctant to make majorchanges to your overall design if itmeans rendering your previous opti-mization work irrelevant. A similaradage exists in the world of creativewriting — don’t keep a bad paragraphjust because it has a great sentence. Theonset of this mentality can often bevery subtle as you start unconsciouslyweighing the benefits of a potentiallybetter design against the hassle ofrewriting your cherished code.

Finally, make sure that you aren’tspending more time optimizing a spe-cific chunk of code than is truly neces-sary. In my experience, the biggestgains during optimization come withinthe first 25% or so of the time spent —after that the increases are only incre-

G R A P H I C C O N T E N T

G A M E D E V E L O P E R J A N U A R Y 1 9 9 8 h t t p : / / w w w . g d m a g . c o m

16

mental. There’s a cross-over pointwhere the extra effort spent optimizinga piece of code is not reaping commen-surate increases in performance. Ourblanket policy is that we wait until thefinal stretch — all features implement-ed — before doing serious “no retreat,no surrender” optimization work —stuff that requires lots of work and thatwill end up being wasted time if wechange higher-level algorithms anddata structures.

Portability

O ne of the nice benefits of usingANSI C is that our product porta-

bility is limited only by our coding dis-cipline. Throughout the developmentof QUAKE 2 a lot of thought was putinto the issue of portability, and byadhering to some general rules, wemake our lives a lot simpler when per-forming a port: segregate OS-specificcode from the rest of the code base,always maintain a C-only version ofyour code, avoid compiler and operat-ing system dependencies, and watchfor endianess assumptions.

The first step in writing portablecode is recognizing appropriate levelsof abstraction and managing your codeappropriately. The bulk of our softwarerendering and sound code only oper-ates on memory addresses — the con-cepts of DIB sections, DirectDraw, wavesound, and DirectSound aren’t knownto the actual graphics and sound-gen-eration code, and are instead handledby abstracted glue code. All OS-specificcode is sequestered into a set of welldefined files — porting QUAKE 2 to annew platform involves touching lessthan a dozen files, each dealing withsome OS-specific subsystem (CD audio,

OpenGL, software render-ing, window system man-agement, input handling,sound output, and OS-spe-cific utility routines, such astime routines). By doingabstractions at this level, wecan greatly minimize thenumber of conditional com-pilation directives in ourmain code body. As a matterof fact, the statement ##iiffddeeffWWIINN3322 only occurs twice in allof our OpenGL code, and it

doesn’t occur at all in thesoftware rendering subsystem.

We always keep up-to-date C-onlyversions of our assembly code, both toassist in porting and also because itmakes debugging the assembly codeextremely easy. It’s really easy to forgetto maintain your C-only paths, but itdoes pay off the moment you try to getthings up and running on anotheroperating system or CPU.

It’s easy to get sucked into compilerand operating system dependencies —for example, utilizing a compiler’s##pprraaggmmaa directives, or maybe an operat-ing system’s message box or memorymanagement functions. Unfortunately,##pprraaggmmaa directives are extremely usefulfor a lot of reasons, and it’s easy to for-get to bracket them with the appropri-ate preprocessor directives. The easiestway to avoid operating system depen-dencies in your main body of code is tomake sure that you’re not includingany OS-specific header files (for exam-ple, WINDOWS.H) — the compilershould complain when you start mak-ing calls that it doesn’t recognize, suchas MMeessssaaggeeBBooxx or VViirrttuuaallAAlllloocc.

Bugs related to CPU endianess can bepretty hard to find, so the best way toprevent them is to recognize codethat’s doing bad things — for example,casting between different sizetypes. A beneficial side effect ofporting to many platforms isthat it also forces you to writemuch cleaner and more robustcode, since hidden bugs in yourcode may only manifest them-selves on another compiler,operating system, or CPU. Youderive a feeling of confidencefrom knowing that your codehas compiled and run success-fully across a wide range ofoperating systems, compilers,

and CPUs. We had a divide-by-zero bugthat only generated an exception onthe DEC Alpha processor, and it wouldhave been a very long time before thisbug was detected in our code under anIntel x86 processor.

However, there are some prettysolid business reasons not to port agame to multiple platforms. WithQUAKE 2 we plan on supportingWin32 (x86), Win32 (DEC Alpha),Linux (various CPUs), SGI Irix (MIPS),and Rhapsody (x86 and PowerPC).Our publisher will likely receive a dis-proportionately higher number ofsupport calls for the non-Intel/non-Win32 versions of QUAKE 2 if werelease these versions on the QUAKE 2CD. Couple this with the fact thatports will probably account for lessthan 3% of our overall revenue, andthe argument for supporting a pletho-ra of system architectures becomespretty flimsy.

We do it anyway, though, becauseit’s cool.

In the end, porting to alternatearchitectures simply doesn’t make verygood business sense for most gamecompanies. We do our ports for onlythree simple reasons: it’s easy to do, welike seeing our games played by asmany people as possible, and we gainthe intangible benefit of havingextremely loyal consumers on systemswith poor mainstream support.

Stay Tuned

B ecause the story is just so darnbig, I’ve had to save some for

later. Tune in to this space next monthfor more on QUAKE 2. I’ll be explainingthe tool choices that we made, compli-menting some vendors, and denigrat-ing a few more. ■

G R A P H I C C O N T E N T

G A M E D E V E L O P E R J A N U A R Y 1 9 9 8 h t t p : / / w w w . g d m a g . c o m

18