COMPUTI<R GRAPHICS AND IMAGE PROCESSING 10, 183-193 (19%)

NOTE

Image Processing and Computer Graphics

ROBIN WILLIAMS

IBM Research Laboratory, San Jose, California 95193

Received December 18, 1978

For a number of years image processing and computer graphics have been treated as separate subjects. Conferences and journals exist for one group or the other but rarely for both groups. However, the hardware for computer graphics and even for alphanumeric displays more and more frequently uses raster-scan techniques, and such displays are equally suitable for image processing. Likewise, image-processing equipment can be used for drawing vectors and writing text as well as for outputting images. Recently, therefore, there has been interest in combining interactive graphics techniques and image processing techniques on the same display screen. This paper explores “image- mixing” techniques and their use in graphics and the corresponding use of graphics in image processing. Several examples are discussed.

1. COLOR GRAPHICS TERMINAL

As a result of some earlier work on decision making using graphic displays of data on maps, a need arose in our laboratory for a terminal that could display vectors and text in color. We wanted to be able to distinguish between two or more similar maps (line drawings) and to display numerical values on these maps. Techniques using dotted lines or different gray scales were not good enough, hence the need for color. Also, we wanted to be able to display images derived tither from co*mputation or from a video scanner operating on-line in real time.

Few display technologies are suitable for making a color graphics terminal. By far the simplest, cheapest, and most versatile is the cathode-ray-tube (CRT) display [l]. There is a vector-beam-deflection CRT system that uses beam penetration into multiple layers of phosphors. l This technique provides a few colors (green, yellow, orange, red). However, only a raster-scan color TV display is capable of providing a full range of colors. Recently several raster-scan display systems, consisting of a display processor, a refresh memory, and a CRT TV monibor, have appeared on the market.2 The display processor contains a vector

1 Beam penetration CRT, “Penetron,” used in CPS-8001 Monitor, CPS Inc. 2 Genisco, Ramtek, Data-disk, Aydin, Evans and Sutherland, and Tektronix are examples of

industrial supplies of color display systems. There are many others.

183

0146-664X/79/060183-11$02.00/0 Copyright @ 1979 by Academic Press, Inc.

All rights of reproduction in any form reserved.

184 ROBIN W’ILLIA~IS

generat’or and a character generator that gcneratc lines or characters in the form of bit patterns which are then &orcd in the refresh memory. The refresh memory logically consists of several two-dimensional bit planes. Corresponding bits in each plane control the color/intensity of a corresponding picture element (pcl) on the display screen. Thus a system wit,h t’hrcc planes can be used to provide on/off cont’rol of the red, blue, and green pels; the display processor cycles through the refresh memory to paint the images in the refresh buffer ont)o the TV screen.

We asscmblcd a graphics terminal which we called “RAINBOW” using a raster-scan CRT display, a minicomputer, and several input devices (Fig. 1). Our first display device had seven refresh memory planes, 512 X 512 bits each, and a video look-up t)ablc (sometimes called a “color-mapper”) (Fig. 2). The display processor was modified to allow the vector/character generators to simultaneously generate bit patterns in any combination of the seven planes of memory. Each pel position effectively has a 7-bit number t,o store color/intensity information. This number is used to address a table (12 bits wide) t’hat contains the actual color/intensity specifications. Four bits specify the brightness for each of the green, red, and blue signals, giving a possible 16 shades for each primary color, or 16 X 16 X 16 = 4096 possible colors. The programmer (or end-user) can specify the contents of the video look-up table (VLT) and hence assign a color for each of the 7-bit pel values. Thus one can choose any 128 colors from the 4096 possible colors. This provides extremely flexible color specifications.

Currently the most cxpensivc component in raster-scan display systems is the

Keyboard

FIG. 1. The display hardware.

IXlA(:E PROCESSING AND COMPUTER GRAPHICS 1X5

512

FIG. 2. Video look-up table organization.

refresh memory. Seven planes of memory is expensive. We built two more experi- mental display systems, one with three refresh memory planes and one with four. Three seems to be a reasonable number for most graphics applications and one can still use a VLT to get eight out of 2” possible colors for flexibility, where n is the number of bits per word in the VLT. Four planes of 512 X 512 bits each were chosen for one system so that it could be used as a monochromatic display with one plane of 1024 X 1024 bits or as a color display with four planes of 512 X 512 bits. Wit,h more memory one can have a higher-resolution full-color system. Color CRTs with sufficient shadow-mask resolution are made,3 but, there is difficulty in obtaining good convergence at higher resolutions.

Raster-scan display systems are very flexible. For example, we have used a TV camera as a scanner to provide an image-mixing capabilit,y. The camera signal is fed through a mixer to one of the red, green, or blue inputs of the TV monitor. Users of the graphics terminal can place objects in front of the TV camera and display a scanned image of the object and then draw lines or write text on the same screen under control of the computer. We expect that TV cameras with at least 512 X 512 resolution using CCD arrays will soon be available. These scan- ners promise to be cheap, reliable, and stable, with much less distortion than current TV cameras. The display memory/scanner technology is very suitable for LSI techniques.

Because the display system is compatible wit’h television technology we can also easily connect several monitors in parallel as slave screens, and we use a projection system (ont)o a large screen) for demonstrations, group meetings, etc. The complet,c color/image-mixed signal is displayed in each case.

3 Mitsubishi, Chuo-Musen, and Conrac make high-resolution CRT Monitors, Matsushita makes high-resolution CRTs. There are probably others.

186 ROBIN WILLIAMS

For hardcopy output we arc constructing an int’erface between t’he refresh memory and raster output printer/plotters. The data is already organized in a form that is very close to that required by raster output devices, which is another advantage of this type of display system. There are many black/white raster out- put devices, and one color ink-jet device was demonstrated by JET-AB, Lund, Sweden, in 1974 [2].”

The first “RAINBOW” terminal used an IBM System 7 computer as a con- troller for a modified display processor and for handling t)he peripheral I/O devices. Alt,hough a general-purpose minicomputer was used for convenience in program- ming, a microcomputer or special-purpose hardware could be used. The two ncwcr display terminals use the IBM 5100 computer and a diffcrcnt display processor. The main advantages of thcsc systems over our first terminal are that they arc portable and less expensive. The System 7 has convenient’ digit)al and analog I/O interfaces which were used to interface the display processor, kcy- board, joystick, local-control switches and data t,ablct. The minicomputer is programmed to communicate via an RS232 telecommunications interface wit,h a host (S/360 or S/370) computer, at speeds ranging from 110 to 9600 baud (bits/xec). It performs buffering and format’ting of data received from the host, and then outputs commands to the display processor. Input,s from the keyboard, joystick, etc., may cause out,put of commands to t’hc display processor, communi- cations to t,he host, and/or control of t,hc cursor generator. Our graphics support software is written so that all our application programs, written for storage tube terminals or for IBM 2250 or 3250 displays, can br run on the color terminal without changing the application code.

2. SIMILARITIES AND DIFFERENCES BETWEEN COMPUTER GRAPHICS AND IMAGE PROCESSING

Ge7,eral Observations

Computer graphics generally involves synthesis techniques. A display terminal is used to input 2, y points and draw lines to specify complete drawings. Oc- casionally mathematical functions are used to calculate arcs, radii, distances, etc., and topological constraint handling functions are used to force objects to be connected. Three-dimensional objects arc input by turning an object on its side (by softwarc or special hardware) and then by drawing the side views. Thus one is t#rying to represent objects in a computer form that allows the objects to be viewed and analyzed easily. For example, t,hese tcchniqucs can be used to draw a map and calculate areas and distances, or to make a mechanical design and to calculate stresses, volumes, and weights. In some cases t,he picture itself is the end result; for example, in making the figures for this paper.

Image processing by cont,rast is mostly analysis, and numerous techniques for analyzing an image in order to extract information from it have been dcviscd and studied [3]. An image is somehow scanned and stored in t)he computer memory. It may be thought of as stored as a matrix of pels (picture elements). Each pel may be from 1 to n bits and may represent gray scale images in black and white or color. Programs then process the image; for example, to find th(b outline of

4 Applicon $ Mead also make color ink-jet printers.

IMAGE PROCESSING AND COMPUTER GRAPHICS 187

regions such as lakes in an image of the earth’s surface, to determine the features of a character-for-character recognition, or to analyze the contents of an aerial photograph to determine the condition of wheat in a wheat-growing area. Much work has been expended on algorithms and heuristics for such tasks, with the overall goal of understanding automatic images. Pictures can also be the end result of image processing ; for example, in image enhancement an image is proc- essed to enhance the contrast in the image. For example, LANDSAT images from satcllitcs arc processed to correct for geometric distortions; medical X-ray images arc processed to product enhanced gray scale images to aid human study.

Hardware Systems

For computer graphics displays several types of techniques have been used [4]. These include random-scan or beam-deflcct’ion CRTs, storage CRTs, and raster- scan CRTs. It is the move toward raster-scan tcchnologics, made possible by cheaper memory prices and new technologies (liquid crystals), that make it possible to display images as well as vectors and text on the same screen. Many output printers also use raster-scan tcchniques.5 Input scanners by their nature arc raster-scan devices (although a few special-purpose scanners have been made that are not simple raster-scan devices).

Image-processing displays are raster-scan devices (generally CRTs). A long time ago most image output was encoded and output very crudely on line printers, but not refreshed displays and raster printers are commonly used. Film is used for static output or for final form. As with computer graphics, regular RAM memory is oft,en used as a refresh store for CRT raster-scan displays, and so the technologies used for graphics and image processing are now very similar. By simply adding a vector and text generator to image-processing equipment and an input xy device, one can do fully interactive graphics too. Likewise in a graphics system using frame buffers, by simply adding a sequential data path to the refresh memory one can display images on the same equipment with reasonable speed. Hence the obvious mixing of computer graphics and images.

Software and Data Structures

In computer graphics systems, graphical objects arc represented in a variety of ways in a data base. Frequently data are represented as a hierarchy of line drawings and are highly structured to represent geometric and topological rela- Gonships among objects. To make processing efficient in applications where speed is important, the data are mapped into other specialized structures to represent pictures displayed on a screen. With some kinds of hardware a third representa- tion of the data is frequently needed in the form of a display file for the hardware to refresh the CRT [5]. In beam-directed CRT systems the display file is a sequential set of graphic orders for drawing lines and for outputing text, whereas in a raster system the refresh mechanism is from a bit buffer. Some raster systems have used a list of graphic orders and text strings when the hardware was fast

b Vcrsatcc, Gould, and IBM are examples of manufacturers of raster printers.

enough to gencrat P the bit, data during the raster scan. Storage t,ubc displays do not require such a display file, but, for compatibility among display systc>ms some people uw display files anyway.

Software packages, language extJcnsions, and wcn special-purpose language have been created for graphics [(il. Thwc arc over 50 similar packages (only differing in detail). They allow one t,o output lines and text, input points and text, and identify entities on a scrwn by pointing to them. Entiw or part~ial drawings may be scaled, rotatctd, and positioned anywhcw on the scrwn, cwn in real time (animation), and perspcctivc views of objcrcts can bc crcatcld. Thcrc have rcccntly bwn c>fforts to standardize thwc tc&niqucs, and proposals exist for standard graphics soft,\varo [7].

Image processing, by and large, employs simphhr data reprcwntations : plants of bits, or arrays of pcls. The display of the data on a raster printer or a CRT also is in the sarn(’ form. The functions required of the software t,o display and process the> data arc pcl or area oriented and quite diffcrcnt from t#hosc required for graphics. Thus image-processing software packages have also been creat,ed, but th(w is virt,ually nothing in common hrtwccn ‘Lpur(~ graphic” software and ‘Lpurc image” software [Z]. Only whore mixed tcchniqucs arc employed (drawing lines on imagw, outputting images with drawings) is one likely to find any owrlap.

Concrally speaking intcractiw graphics applications (cbxcctpt, for thr simple plotting of graphs) rcquiw wry sophist,icated programming techniques to build, manipulate, and edit data structures rcprcsctnting grap&al data and inter- wlationships bctwcrn objects. Image processing is simpler in this rwpwt but rrquircs sophistication of quitcl a diffwwt natuw (namc>ly, analytical rwognition heuristics).

3. CORWUTER GRAPHICS URINC: 1RlAC:E:S

Baclixpvunrl Overlay

Rastchr-scan graphics systJcms, mhct,her color or black and whitso, allow one to display image as ~~11 as graphics. Thus it is nat,ural to SW what one’ can gain bl adding images to traditional graphics applications. Tho most obvious thing one can do is to add backgrounds to graphic displays. Thus when a st#rr:et map is being displayed one can add dat,a other than st’reet s (hospital symbols, parks, railways, rivers, c+c.) by scanning and displaying a regular paper map of thr same ar(‘a with all this cxt,ra information on it. If t’hc: scanner \vorks in real time, as in tht: case of a TV cam(‘ra, then the image from t’ho scannw can be mixed &cttronically with the signal to the display monitor and the resultant composit c signal (strwt map and scanned map) can bc displayed. Tho maps can bo rcgistcwd with rcspcct to each other by adjusting thr camera position (which is difficult) or by scaling and shifting t,he computer data and redisplaying it,. OIW method of wgistcring th(> computer data with a scanned image is to ident,ify thrcr point)s on the image and then throc corresponding points on t,hc lint drawing, from which tha proper position, size, and oriontation of thcl graphic output can be calculat,cd. Th(k graphical data aw then redrawn correctly superimpowd on t,he image. When a real-t imc> image cannot bc crcat’cd it’ is possible to scan tJlc> imagcl, st,ortr it in the

computer, and output it as an image. Of course this requires extra storage and also slows down program execution speed considerably.

The background-image idea is quite useful in practice. We have shown that such a technique can be used for map editing and map digit’izing wit,h 2-4 to lo- fold gains in overall productivity [S].

Other applications include the gencrat,ion of business forms, the input of data on a form, decision making in which an image provides helpful background data (c.g., utility mapping, planning for new telephone cables, ct’c.), and artwork generation. In general background images arc useful for providing reference in- formation; for aiding input of point,s, lines, or text ; or for generat,ing composite pict,urcs to be photographed or output,.

Graphic Primitives

Images can also be used as additional primitives in graphics systems. Some pictures are best represented as bit patterns, (e.g., company logos, map symbols, and photographs), and in our Picture Building System for example, such images can bc scaled, rot)atcd, and positioned in the manner of lines and text [9]. The use of images expands the range of applications, or the range of functions for existing applications, in obvious ways. A most useful application of this technique is for defining new character sets or for representing different, printer-type font’s.

A third major USC of image-processing (or bit-processing) techniques is in the gencrat’ion of shaded pictures. Starting from lint drawings one can fill in areas with colors, cross-hatching, or various other patterns. Thus one can shade a 3-D picture t,o make it appear more real; one can even add highlights and shadows [lo, 111. This is useful when the end result is a picture (e.g., a new building design, or a new car body, or art for publications). Pattern filling is useful for t’he design of materials, carpets and drapes, wallpaper, and clothing.

Realism and Animation

There is also an int’ercst in creating realistic-looking graphic output of near- photographic quality. Images are synthesized with shading and illumination properly rendered [lo, 111. Sometimes the output is recorded on film, slightly altered, recorded again, etc., to make an animated movie [la]. A dream of com- puter graphics workers has been to create films that, are so realis& as to be in- distinguishable from the real thing, and furthermore to produce animation that, runs in real time. Such animation is useful for preparing people for difficult exercises before they undertake the real exercise, e.g., training pilot,s, landing a lunar module, and driving a car; for showing otherwise impossible things (such as how a downtown area will look if a certain proposed building is built) ; or for training and assisting doctors and surgeons wit,h animated movies of medical operst’ions. The big problem, of course, is processing vast amounts of information representing images. In graphics special hardware for line drawings and text has

190 ROBIK WILLIAMS

bec>n built by scvcral companies to rotate, scale, shift, and clip 3-D “wire-frame” lines in real time, and equipment has been built to aid the processing of color and shaded images.6

4. IMAGE PROCESSING USING COMPUTER GRAPHICS

Feature Extraction

Workers in the image-processing field have tackled the problems of cxt’racting “features” from images [3], which can range from finding a straight edge, to identifying a 3D solid object, or to classifying a pattern. Frequently the feature is then represented in a new way, different from the original image, as vectors, as text>, or in special encodings. It is then possible to show the feature as an overlay on the image, in either a different brightness or color, if a display is used. Image processing can become interactive too, and human operators can observe t,hc feat,ures found in the image and direct the comput,ation to test for other features or cause t,he features t,o be adjust’ed and reexamined. This interaction is done with satellite LANDSAT images where the aim is to find and identify wheat crops or enemy missile sites, or to calculate the st,ate of water resources or the spread of urban developmcntf. One can envisage t,he sat’cllitc data as a form of input for regional planning decisions. Images arc analyzed and the data ext#racted from t,hrm arc input to planning models that produce out)put which can be displayed superimposed on the original image. For example, the county boundaries, major highways, and railroads can be superimposed on an aerial image of the urban areas of a larger &ate or count’ry. In every count,y one can display numeric dat,a dclrived from t,hc image, such as building densit,y, pcrcent,age of vegetation, per- centage of pollution in t’hc air, etc. Then planning models can be used to help make decisions about major development8 plans such as building new roads and new towns, locating factories, dams, etc.

Optical Ch.aracter Recognition

Another application is in optical charact,cr rccognit.ion with human assistance. Text is scanned and rccognit’ion performed whenever possible. Typically 997, recognition might bc achieved, and unrecognized text can be displayed as an image together with surrounding recognized characters. From the context an operator can identify an unrecognized character and type it on a keyboard. The character can be displayed under the image and if correct the operator continues to the next unrecognized character. If the operator makes a mistake or cannot recognize the image then scrolling and backup mechanics can be used to further aid context recognition. One can set the day when lct,tcrs and memorandums can be spoken into an office computer, recognized, displayed as images, corrected, and then distributed electronically. Such a useful device would need a signal (patt,crn) recognizcr in a text display terminal. Input documents might be coded text or images from a scanner. Hardware dcvrlopmctnts and price reductions are beginning to make such t,ext, graphics, image, and speech syst,ems possible.

6 Adage, Evans and Sutherland, Vector-General, and HUMRRO.

IMAGE PROCESMIVG AND CORIPUTI~;R GRAPJIICS 191

5. APPLICATION EXAMPLES

Two applications using combinations of graphics and images on a color display system were reported previously [S]. A summary is prescnt’ed here. A third example in which raster data (image) arc used as graphic primitives t>oget,her with vcct,ors and t,ext is in our Pict,urc Building System, also reported earlier [Y], but not summarized here.

Map E&tiny

Map editing involves the entry, error detection, and error correction of digitized maps. WC have developed a prototype system for this application and have studied its USC on graphics terminals with and without image mixing. A map- editing study was made with 12 maps containing from 1500 to more than 10 000 line segments each. The map editing was equally divided between terminals wit’h and wit,hout image mixing. The map-editing soft,ware was the same for each terminal. Six users of approximately equal map-editing skills did t,he work; maps and terminals were randomly assigned to the users. The improvements in user productivity due to image mixing were:

Task Improvement factor Base _____

Map entry 47 10 segments/hr Error detection 2-4 Number of errors Error correction 2-10 5 min/error

Maps are maintained by every utility, most government agencies, most com- panies dealing with natural resources (e.g., timber), and most companies with geographically dispersed customers or property. One user WC worked with wanted to prepare 100 maps averaging about 3000 line segment’s each. On the basis of a factor of 5 productivity improvement over a base of 10 segments/hr, we estimate that the potential labor savings from using an image-mixing terminal rather than manual or non-image-mixing graphics to creabe t’hese 100 maps is approximately 25 000 man-hr. Another enterprise is trying t’o detect and correct errors in 250 maps containing over 50 000 line segments each, averaging about 107o errors/map. Again using our productivity improvement estimates, t,he potential saving with image mixing is between 250 and 400 man-hr/map, or between 62 500 and 100 000 man-hr.

Slide Making

In this application the terminal hardware is used in a stand-alone mode and the complete application runs on the “cont’rol” microprocessor. The user sclect#s functions from a menu and colors from a palette or paint,box. A joystick provides 2, 2/ input for drawing lines, horizontally, vertically, or at any angle; for drawing solid or open rectangles by specifying a diagonal ; or for drawing solid or open circles by specifying a center and a point on the circumference. Text, can be positioned and typed anywhere on the screen with or without a background sur- rounding color or on top of the existing picture. An image from a TV camera can

19’1 ROBIN WILLIAIVlS

also be mixed in and used to form a composite picture. The menu and palcttc can be turned off by a single keystroke. The displayed picture, exactly as constructed by the user, can be photographed, and the resulting pictures or 35-mm slides can be used for technical presentat,ions. Better-quality out,put could be obtained by driving a higher-resolution precision film exposing dcviw to produce a final slide or output. There are other useful fcat,ures of the program : Colors can bc selcctjed and changed at, any time, for cxamplc, for every line, while typing tc>xt, et#c. ; a prompt system can be turned on/off to help a uwr learn how t>o oporatc the program; a picture can be stored on a cassette tape for lat>cr playback or mixing with subsequent pictures; a backtrack feature allows the last entry to bc erased; and some freehand sketching modes arc provided. This is the second version of a program that was first written about 4 years ago on the first color terminal at approximately the same time that R. Schoup first, dcvclopcd his painting syst,cm (indepcndcnt)ly) [13]. 0 ur first system was written to test out hardware and provide a simple demonstration capability, but thr second system was built, to assist, the st,udy of the human factors of man-machine intcrfaccs and is intended to be wry easy to use; almost like a copier. Cost analysis and productivit,y gains arc’ given in [S]. One can typically make slides for $3 oath at a rate of about lO/hr with little expericncc. This cost, includw amortizing cquipmwt, owr 4 years and using it cont,inuously.

6. CONCLUSIONS

For a long t)imc image processing and comput,er graphics have been soparat’e fields of endeavor. Individuals have been in one sphere or the other, but rarely in both. Xow common equipment for both graphics and image processing is bc- coming more widely available and workers are beginning to combine the ad- vantages of both disciplines. Because this offers advantages to bot,h groups one can expect t,he overlap to increase in the future. In some applications this is already happening, as evidenced by some of the examples described.

REFERENCES

1. H. H. Poole, Fu&ame&& of Displays, Spartan, Washington, D.C., 1966. 2. C. H. Hertz and A. Mansson, Color Plotter for Computer Graphics using Electrically Con-

trolled Ink Jets, in Proceedings, IFIP Corqrrss 74, Stockholm, 1974, pp. 85-88, North- Holland, Amsterdam.

3. A. Rosenfeld, Picture processing: 1977, Computer Graphics ad Image Prowssing 7, 1978, 211-242. (609 papers categorized.)

4. U. W. Pooch, Computer graphics, interactive techniques and image processing 1970-1975: A bibliography, Computer 9 (8), 1976, 46-64.

5. R. Williams, A survey of data structure for computer graphics systems, ACM Computirlg surveys 3, 1971, 1-21.

6. R. II. Ewald and R. Fryer, Final Report of t,he GSPC State-of-the-Art Subcommittee, ACM SIGGRAPH Computer Graphics 12(1, 2), 1978, 14-169.

7. ACM Computing Surveys 10(4), 1978. 8. II:. D. Carlson, G. M. Giddings, and R. Williams, Multiple colors and image-mixing in graphics

terminals, in Proceedings IFIP Congress 77, Torouto, August 1977, pp. 179-182, North- Holland, Amsterdam.

9. D. L. Weller and R. Williams, Graphics and relational database support, for problrtm solving, ACM SIGGRAPH Computer graphics 10(a), 1976, 183-189.

IMAGE PROCESSIKG AND COMPUTEX GRAPHICS 193

10. T. P. Bui and F. C. Crow, Improved rendition of polygonal models of curved surfaces, in Proceedings, Second USA-Japan Conference, Tokyo, 1975.

11. F. C. Crow, Shadow algorithms for computer graphics, ACM SIGGRAPH Computer Graphics, 11 (a), 1977; Proceedings of SIGGRAPH 77, San Jose, pp. 242-248.

12. C. Csuri, Computer graphics and art, Proc. IEEE 62(4), 1974, 503. (Special issue on com- puter graphics.)

13. R. G. Schoup, Towards a unified approach to 2-D picture manipulation, ACM SZGGRAPH Graph,&, 11(2), 1977, 178.