images and texts

Springer is collaborating with JSTOR to digitize, preserve and extend access to Computers and the Humanities.

http://www.jstor.org

The Place of Images in a World of Text Author(s): Mary Keeler Source: Computers and the Humanities, Vol. 36, No. 1, Image-Based Humanities Computing (Feb.,

2002), pp. 75-93Published by: SpringerStable URL: http://www.jstor.org/stable/30204697Accessed: 26-04-2015 12:21 UTC

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at http://www.jstor.org/page/info/about/policies/terms.jsp

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of contentin a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship.For more information about JSTOR, please contact [email protected].

This content downloaded from 130.104.77.66 on Sun, 26 Apr 2015 12:21:25 UTCAll use subject to JSTOR Terms and Conditions

Computers and the Humanities 36: 75-93, 2002. a 2001 Kluwer Academic Publishers. Printed in the Netherlands.

75

The Place of Images in a World of Text

MARY KEELER Center for Advanced Research and Technology in the Arts and Humanities, University of Washington, Seattle, WA 98195-3680, USA E-mail: mkeeler@ u.washington.edu

As a veteran graphic artist, technical writer, and professor of radio and television, currently working in Semantic Web research and development, I have been pleased to observe the digital convergence of traditional media of communication (from paper publishing to cinema), and am urged to hope that humanists must soon come to appreciate the place of text in a world of images. I do not come to this topic to bury text, nor to praise images, but I come with the conviction that we must learn to render in text (as linguistic, graphical compositions) when text serves best, leaving to images what they serve better, and to appreciate the useful rela- tionships between description and depiction, and between images as evidence and images as expression. I attempt to indicate why that distinction is never simply a matter of objective (authoritative) judgment, to be automated in an algorithm or in disciplinary conventions (as standards), and how that realization should affect our principles and, more significantly, our methods of image-based software design. New computer-mediated representation capabilities should invite us to consider what sort of augmentation (as tools or instruments) could improve the place images in the well-established, humanist world of text. I will give some indication of how that improvement might occur, after I sketch the necessary theoretical background to appreciate the need for that pragmatic method. Neil Gershenfeld, in When Things Start to Think, poses the challenge to humanists in computing: "It has been a battle for the last decade to argue that a digital representation lets computers be equally adept at handling text, audio, or video. That's been won; what matters now is what is done with these capabilities .... The words we usually use to describe computers have a hard time capturing the profound difference between things that work, and things that work very well" (p. 110).

At Cornell in the 1970s, James J. Gibson began to construct a new approach in perceptual theory toward explaining "the human habit of picture-making," in terms of devising and displaying optical information for perception by others. Although humans had been drawing for at least 300 centuries before writing was invented (as ideographs, syllabaries, and words recorded on a surface), we still have no theory to explain either representation capability nor to distinguish them effectively as modes of communication. Nineteenth-century scientist and philosopher Charles S. Peirce left in manuscript fragments a comprehensive theory of signs that


76 MARY KEELER

conceives representation and communication in terms of the pragmatic evolution of inquiry. Together, Gibson's and Peirce's metatheories anticipate a new humanistic appreciation of representation and new methodology in the design of software as communication media. Presenting a careful enough account of those theories to be useful in hinting what their implications might be will fill the space allowed here. Using only the few concepts I can explicate within these space constraints, I will suggest implications and indicate their potential effects in the context of hypothetical scenarios of computer-supported humanist research operation, at the end.

I cannot in this context review the literature of other possibly relevant theories; I have chosen Gibson and Peirce for their comprehensive theoretical grasp of conditions that have not otherwise been adequately considered and explained. Other theories, of visual or picture perception and of signs and representation, do not even attempt the depth and breadth of these two which together offer better footing for advancement. I will spend considerably more space carefully explaining Gibson's theory, because it proposes a necessary (ecological) perceptual approach to representation, as a basis for appreciating how Peirce's semiotic differs from other so-named theories. For the focus and purpose of this effort, Peirce's theory need be only suggestively related to Gibson's, ultimate, but not-fully-developed concept of affordance. More space would be required to argue that contention adequately but the suggestion is useful in the scenarios that follow. Primarily, I attempt to suggest why pragmatic methodology could be the truly humanist approach to computing needed to advance beyond the current text-based, technology-driven mode of development.

1. Natural Visual Perception Distinguished from Pictorial Perception

Since the early Greeks, we have been told that eye is easily deceived and that our faith in the reality of what pictures display therefore leads to deception and fallacious judgments. Because viewing a picture is somehow more like directly perceiving than is viewing a verbal-text description, depiction can be used to mislead us more easily than can description. Gibson's perceptual research exposes fallacies in this traditional doctrine: "The deception is possible only for a single eye at a fixed point of observation with a constricted field of view, for what I call aper- ture vision ... Only the eye considered as a fixed camera can be deceived" (p. 280). We continue to believe the myth, Gibson argues, because it goes along with traditional theories of retinal image optics, perpetuated by psychological experiments that artificially restrict observers to just those fixed-eye conditions using pictures as stimulus displays. Natural perceptual awareness of the surrounding environment cannot be explained by that "projective theory" of one-to-one correspondence; instead, the visual system has evolved to pickup information while the viewer moves through the flux of information in the natural environment. Even the awareness of objects in space cannot be explained by a "picture theory" of perception,


THE PLACE OF IMAGES IN A WORLD OF TEXT 77

"because for any given solid object that moves there exists an infinite set of possible forms. [The 'frozen form' of an image] does not specify the solid shape of an object, only some of the invariant features that a solid object must have" (p. 269). Instead of a form, he maintains, we see a selected sample of the ambient optic array.

Gibson explains that painters invented the "cues for depth" to construct pictures in formal perspective, then psychologists simply began using "cues" to talk about visual perception. "The notions of counterbalanced cues, of figure-ground reversals, of equivocal perspectives, of different perspectives on the same object, of 'impossible' objects - all these come from artists who were experimenting with frozen optical information" (p. 244). Gibson contrasts the pictorial display of optical information with "the inexhaustible reservoir of information in an illumi- nated medium" of natural perception. The observer's perceptual system (two eyes that function binocularly, in a head that freely moves, on a body that walks around and looks at things from many angles) will always pickup information to specify the display as a surface in the environment (whether canvas, screen, or any other surface) that has been marked or shaded in some respect. Under normal viewing conditions, two kinds of apprehension occur: "a direct perceiving of the picture surface along with an indirect awareness of what it depicts," and the viewer is always aware of this dual apprehension (p. 291). Pictures do not induce an illusion of reality, but rather, "an awareness of being in the world. This is no illusion. It is a legitimate goal of depiction, if not the only one" (p. 284).

His theory of information pickup explains persistence and change in perception: how we notice the change along with the continuing identity of things. "The perceiver extracts the invariants of structure from the flux of stimulation while still noticing the flux. A frozen form does not specify the solid shape of an object, only some of the invariant features that a solid object must have" (p. 249). We never see just a form; we see a sample of the ambient optic array. We are aware of our existence in a persisting environment as we are aware of our movements, along with the movements of objects and nonrigid surfaces, relative to that environment. Gibson uses the term awareness to imply a direct pickup of the information, not necessarily to imply consciousness, as assumed in traditional perceptual theory. "The congruence of the array with itself or the disparity of the array with itself, as the case may be, is continually picked up. The perception of the persisting identity of things is fundamental to other kinds of perception" (p. 249). The puzzle of self-awareness is resolved incidentally; self-awareness is the necessary basis for awareness of the invariants in the world.

In Gibson's ecological theory of perception, information refers to specification of the qualities of objects in the observer's environment. Picking up information must not be thought of as communicating knowledge.

The world does not speak to the observer. Animals and humans communicate with cries, gestures, speech, pictures, writing and television, but we cannot hope to understand perception in terms of these channels; it is quite the other


78 MARY KEELER

way around. Words and pictures convey information, carry it, or transmit it, but the information in the sea of energy around each of us, luminous or mechanical or chemical energy, is not conveyed. It is simply there. The assumption that information can be transmitted and the assumption that it can be stored are appropriate for the [engineering] theory of communication, not for the theory of perception. (p. 242)

If information does not have to be stored in memory because it is always available, the increasing capacity of the perceptual system to pick up information through learning does not in itself constitute gaining information; however, since the process of pickup depends on the input-output loop familiar to engineering theory, information transmitted form one person to another and stored becomes a special case of the more general theory of perception. The picking-up of information becomes more and more subtle, elaborate, and precise with practice. We can keep on learning to perceive throughout life, as artists do, for example.

Gibson defines a picture as "a surface so treated as to make available an arrested optic array, of limited scope, with information about other things than the surface itself' (p. 292). The arrested optic array of a still image is an unusual view of the changing array; it depicts a frozen world as viewed by an observer who holds still and uses one eye, but "the eye developed to register change and transformation" (p. 293). Counter to all we have been told about optics, we should consider the motion picture to be the basic form of depiction and the painting or photograph as a special form of it. These "motions," he contends, "are thoroughly saturated with meaning" (p. 294). His research indicates that the viewer's awareness of motion-picture events is achieved by segmenting the flow of the pictorial optic array so that it specifies the same kinds of subordinate and superordinate structures that are specified in the natural optic array. These segments (or parts and the transients between them) are crucial and should not be simply called "motions." "The ordinary picture is not so much 'still' as it is 'stopped.' The invariants to specify the places, objects, and persons emerge more clearly in the [motion picture's] transforming array than they would in a frozen array" (pp. 301- 302). The ecological theory of visual perception postulates that the invariants are weaker and the ambiguities stronger when the point of observation is motionless. The media of information that most of us have relied on, for centuries before this one, have accentuated the very "frozen conditions" under which Gibson says our perception has not evolved to operate most effectively. (One reviewer of this essay suggests that interactive virtual reality simulation may be a step even further toward representing conditions of the natural visual array).

2. Images Distinguished from Text as Representations

If it is true that a drawing, painting, or photograph is actually an arrest of the normally changing array, Gibson claims that we must revise our theories of cognition and knowledge to reflect how such an arrest affects our "higher processes."



The abrupt noncontinuation of an event, with a continuing nontransformation of the optic array, must be artificial, an "unnatural stoppage of the flow." Responding to the paradox of the painter's claim that motion can be portrayed in a still picture, Gibson explains: "If it is true that the perception of a detached object is not compounded from a series of discrete forms of that object but depends instead on the invariant features of that family of forms over time, it follows that an arrested member of that unique family will have at least some of those invariants. If object perception depends on invariant detection instead of form perception, then form perception itself must entail some invariant detection" (p. 271). The skilled artist knows how to select and display the detected invariants for optimum form perception and object identification from the flux of the natural optic array (see Hochberg's discussions of "cannonical forms," 1977: 165).

Furthermore, the artist's technique in a still image motivates the viewer's attention in many respects, and perhaps even more does a filmmaker's technique in a moving image, but the viewer always maintains some control by remaining aware of the medium as a medium (a surface or display of a previously constructed optic array). Of course, a narration or description can also display information for such awareness at second hand, and the writer then controls the reader, as the filmmaker controls the film viewer. To the extent of that control, Gibson says, the reader or viewer "is at the mercy of the artificer, the artist, the maker, of one-who-knows." But here he cautions: depiction cannot be considered a language with a grammar, as some (including most semiotics theorists) believe; "graphic depiction is not an explicit description" (p. 295). Depiction is not fundamentally a form of description; we do not learn to read pictures just as we learn to read a language, and perspective is not merely a convention. "But the essence of a picture is just that its information is not explicit. The invariants cannot be put into words or symbols, and theories of mental encoding does not explain how a depiction captures an awareness without describing it. The record has not been forced into predications and propositions" (pp. 284-285; my emphasis). There is no way of describing the awareness of being in the environment at a certain place, although great writers can describe their feelings of being there. Writers "cannot put you in the picture anything like the way a painter can" (p. 285); and that's not their purpose, anyway; their purpose is to convey their feeling of being there. (The style of a painting also conveys the artist's feeling, which blurs Gibson's distinction somewhat, but the point he makes remains valid.)

The science of language is well established, Gibson reminds us, but we have nothing even approaching a science of depiction: "artists, critics, and philosophers seem not to discuss the same topic as photographers, opticists, and geometers" (p. 270). To find common ground, he says they need to understand how an observer can see something from no point of observation as well as from a given point, or "from a path of observation as well as a position." He explains that modern painters are trying to paint invariants, not abstractions, concepts, space, or motion. Separat- ing the invariant structure from perspective structure is the heart of the problem:


80 MARY KEELER

"invariants display a world with nobody in it, and the perspective displays where the observer is in that world. One can depict without a fixed point of observation [or perspective], just as one can visualize without a fixed point. But depiction with a point of observation is the more natural sort, and the photographic picture is necessarily of this sort" (p. 284).

A photograph is more analogous to natural perception, as a record of the photog- rapher's selected attention. But any picture preserves, in some respect, what its creator considers worth noticing. Even a painting of a fiction or fantasy preserves invariants that have been noticed in the course of an artist's learning to perceive them esthetically (explained in the next section). According to Gibson's theory, then, a natural perspective picture is "a surface so treated that it makes available a limited optic array of some sort at a point of observation," and the array is "an arrangement of invariants of structure" (p. 270), which is sometimes considered a record of the evidence. "It enables the invariants that have been extracted by an observer - at least, some of them - to be stored, saved, put away and retrieved, or exchanged. Pictures are like writing inasmuch as they can be looked at again and again by the same observer and looked at by many observers. But most 'form- less invariants' in the array from a picture can be captured by the artist but not described" (pp. 273-274).

On this understanding of pictures, Gibson maintains that information about the environment in terms of words has the disadvantage of missing the "reality test- ability" that the pickup of natural information makes possible. We cannot discover a new and surprising feature of an object by merely scrutinizing its textual description, because someone's visual system has already selected the features by which to represent it, and gives us only an expression of what was picked up as evidence. Whether you can discover any new features is "the most decisive test for reality," claims Gibson (p. 257). "Descriptions, spoken or written, do not permit the flowing stimulus array to be scrutinized. ... The invariants have already been extracted. You have to trust the original perceiver; you must 'take his word for it,' as we say" (p. 261). What words represent may be fact or may be fiction, and the same is true of a depiction, but we can also distinguish them on that basis. "Verbal descriptions can be true or false as predications. Visual depictions can be correct or incorrect in an entirely different way. A picture cannot be true in the sense that a proposition is true, but it may or may not be true to life" (pp. 261-262). In real scrutiny, information is inexhaustible, and Gibson speculates how perceiving, knowing, recalling, expecting, and imagining can all be induced by pictorial mediation more readily than by words.

The efficiency that Gibson attributes to picture perception might explain why images have long been considered a threat, both to authority and to the notion of "seeking the truth." Those who want to maintain authority cannot as vividly manipulate the sense of reality by means of written words, without extensive education in the conventions of textual composition and interpretation. Pictures can make fictional representations as easily perceived and effortlessly believed as



factual ones, but Gibson's theory helps us begin to examine these conditions of representation more carefully. First, all fictions are not necessarily fantasies (that automatically lead us astray, as do hallucinations); they can "promote creative plans and permit vicarious learning" (p. 262).

Gibson contends that, if his theory is correct, the common understanding of the term representation is misleading.

There is no such thing as a literal re-presentation of an earlier optic array. The scene cannot be re-established; the array cannot be reconstituted. Some of its invariants can be preserved, but that is all. Even a photograph, a color photograph at its technological best, cannot preserve all the information at a point of observation in a natural environment, for that information is unlimited. As for re-presenting the stimulation in the sense of reimposing an old pattern of light energies on the retina, that is quite impossible. The full energies of wavelengths of light cannot be preserved on film. Some of the ratios, the contrasts or relations of light, can be captured but not the sensations of brightness and color. (p. 279)

He suspects that philosophers and psychologists have failed to clarify what is meant by representation due to the misconception that a picture is an imitation of past seeing, a substitute for going back and looking again. "What it records, registers, or consolidates is information, not sense data" (pp. 279-820). Gibson's theory suggests that pictorial information might be viewed as evidence or it might be viewed as an expression of the artist's feelings. Evidence apprehended by the picture-maker could have a different purpose from expression derived from a picture-maker's apprehension. A picture as an expression might then have a function much more comparable to verbal description.

3. Extended Apprehension, Media Evolution, and Esthetic Attitude

Gibson urges that his redefinition of perception implies the need to reconsider our understanding of "higher mental processes" or "intellectual processes" that are assumed to occur in the brain as "operations of the mind." He suggests implications of his theory, with regard to what we currently consider remembering, thinking, conceiving, inferring, judging, expecting, and above all, knowing. If both perceiving and knowing happen by extracting and abstracting invariants, he argues: "Our reasons for supposing that seeing something is quite unlike knowing something come from the old doctrine that seeing is having temporary sensations one after another at the passing moment of the present time, whereas knowing is having permanent concepts stored in memory" (p. 258). Instead, perceiving is continuous with knowing in the awareness of persisting structure.

Pictures, stories, toys, models, instruments, tools, books, and so forth are all cases of information picked-up by extended modes of apprehension. In the use of these media, "the learner has to hear the speech in order to pick up the message; to


82 MARY KEELER

see the model, the picture, or the writing; to manipulate the instrument in order to extract the information" (p. 258). Gibson speculates (in 1979) that the many modes of communication (including words, pictures, and instruments of many kinds) are merging to extend the limits of human apprehension and comprehension. Although written language can be copied and accumulated in libraries, he says, "we should never forget that this is information that has been put into words. It is not the limitless information available in a flowing stimulus array" (p. 260). Knowledge that has been put into words can be said to be explicit instead of tacit. But Gibson's hypothesis is that there has to be an awareness of the world before it can be put into words. "Perceiving precedes predicating" (p. 260). However skilled an explicator may become, "one can always see more than one can say" (if not so, one ceases to learn).

Gibson's ecological theory has profoundly shaped psychology, especially with regard to controlled experiments in perceptual science (see Cutting, 1993, pp. 234- 235), and his followers continue to test and extend it. Margaret Hagen, for example, argues that invariants do not conceptually account for much of the imaginary and artificial perceptual phenomena that have become commonplace in human experience. "Both the variant and invariant types of optical structure are determined by the common projective rule which generates them as a function of object character and transformation. Neither has priority logically or developmentally or artisti- cally. Each is available to the perceiver as the need arises" (p. 29). However, she agrees with Gibson that ordinary vision is naive in contrast to the "esthetic attitude" achieved by the artist in learning to draw. Through picture-making, we learn "to attend to the characteristics of the momentary visual field in addition to the permanent properties of the visual world" (Hagen, 1980, pp. 21-22).

Gibson conjectures that humans no doubt made and observed traces on surfaces long before they realized that lines could delineate something for which there were yet no words. In learning to draw, we learn how to identify "graphic invariants," which words cannot describe to anyone who has not observed them. As examples, he lists some qualities of lines:

How a straight line (or trace) looks different from the line called "curved"; How a trace can begin and end, or be continuous; How a continuous line can change direction abruptly (zigzag); How a line can be made between existing marks to connect them, and marks can be "lined-up"; How a continuous trace can come back to where it began, and the feature of "closure" emerges; How a continuous trace is apt to produce an invariant called "intersection," making "connections." (see p. 276) Gibson's theory postulates that a line drawing carries information by the quality

of connected lines, not by the lines as such, and that the invariants are found in the ways that the areas are nested, not in theforms of these areas. Again, he says, "these ways are difficult to describe in words. These connections can be manipulated to



create such "ecological contradictions" as M.C. Escher's drawings demonstrate. Gibson is tempted to consider the objects displayed in a picture as virtual objects. "They are not perceived, and yet they are perceived. The duality of the information in the array is what causes the dual experience." In such examples of picture- making as impressionist paintings we clearly see the distinction between virtual and real: "we can see the differences between the illumination of the picture and the illumination in the picture. The two sets of surfaces are not comparable, and the two kinds of illumination are not commensurable" (p. 282). Furthermore, a surface can be ornamented, decorated, embellished, and its reflectance can be altered; that is, "its texture can be changed - all without causing it to specify something other than what it is, a surface. ... The surface that displays information may also be an esthetic object, but..,. a picture is something that always specifies something other than what it is [as a treated surface]" (p. 273).

In that observation, Gibson's theory of ecological perception approaches an explanation of meaning. "If the visual system is capable of extracting invariants from a changing array, there is no reason why it should not extract invariants that seem to us highly complex" (p. 141). A compound invariant (an invariant combi- nation of invariants) specifies an affordance of the environment for someone. Any substance and surface, any layout, has some affordance for the benefit or injury of someone. "The perceiving of an affordance is not a process of perceiving a value- free physical object to which meaning is somehow added in a way that no one has been able to agree upon; it is a process of perceiving a value-rich ecological object. Physics may be value-free, but ecology is not" (p. 140).

4. Affordance and Representation Gibson says the information to specify an affordance points in two ways: "to the environment and the observer," which is only to say that "information to specify the utilities of the environment is accompanied by information to specify the observer himself" (p. 141) The apparent dualism, he insists, is neither mind-matter nor mind-body, but awareness that ones complementary relations to the world are not separable.

The affordance of something does not change as the need of the observer changes. The observer may or may not perceive or attend to the affordance, according to his needs, but the affordance, being invariant, is always there to be perceived. An affordance is not bestowed upon an object by a need of an observer and his act of perceiving it. The object offers what it does because it is what it is. To be sure, we define what it is in terms of ecological physics instead of physical physics, and it therefore possesses meaning and value to begin with. But this is meaning and value of a new sort. (pp. 138-139) James Cutting extends Gibson's theory (calling it an "argument from evolu-

tion") to derive his theory of directed perception, contending that our inventions


84 MARY KEELER

of technology create a "cultural ecology," which we did not evolve to perceive. The resulting affordances bring ecological and cultural ecologies into tension. "Modern technological objects are either multiply affording or they afford things that didn't exist before. Indeed, one can say the driving force of invention is to increase the number of affordances around us" (1993, p. 239). The flat sheet of paper "is humankind's most deeply troubling invention" from the perspective of the ecological approach, he contends. Our behavior is virtually unconstrained by its affordances, and "the most troubling aspect of paper is that we can make marks on it" (1993, pp. 245-246). (We now make virtual marks on computer screens that are potentially even more unconstrained.) Cutting's research distinguishes between line drawings, "which seem unequivocally to invoke cognition in their perception," and the quite recent technical developments of linear perspective and photography, which "seem to make the role of cognition less apparent, if not less important" (p. 163). How the perceiver resolves the duality of a picture (as depiction or as object depicted, that is, as expression or as evidence) depends on the context of its use, in semiotic terms: who uses what sort of image (or sign) for what purpose. A line drawing might reduce the "noise" in an assessment of information; but that "noise" (in a photograph for example) is the source of more information perhaps not yet noticed there, for further discovery, interpretation, and learning - or inquiry.

The primary challenge of any inquiry, according to Peirce's theory, is the self- critical effort to represent and validate the many different observations in the experience of some phenomenon, by continuing to formulate and test hypotheses, with some hope that these expressions can be brought together to form a reliable interpretation of the evidence from which investigation can proceed. Collabora- tive inquiry proceeds only through cooperation of individuals working to represent their experience (or express their views of the evidence), thereby creating a viable community that can give each member a broader self-awareness (in a collective experience) but which can remain vital only through continued individual contributions, to be compared and comprehended in an evolving representation of what we call knowledge. Peirce (based on his own experience as a working scientist) developed his semiotic and pragmatism to account theoretically for a community of inquirers as what makes knowing possible. "The real, then, is that which, sooner or later, information and reasoning would finally result in, and which is therefore independent of the vagaries of me and you. Thus, the very origin of the conception of reality shows that this conception essentially involves the notion of a COMMUNITY, without definite limits, and capable of an indefinite increase of knowledge" (W2, p. 239).

Peirce intended his theory of representation to explain how valid inquiry evolves by verifying the fitness of anything to serve as a sign, which is to say anything "that can be interpreted - by a feeling, an action, or a thought" - in our experience of something. We can study sign-fitness, as the potential for anything to serve as a sign, in three ways: "in respect of the general conditions whereby [signs] have meaning, in respect of the conditions of their truth, and in respect of the conditions



whereby their meaning is transferred to other signs" (NEM, 4, p. 331). The first and most fundamental of the three divisions, called pure grammar, Peirce defines as "the general theory of the nature and meanings of signs whether they be icons, indices, or symbols" (CP, 1.191).

Pure grammar is an analytic conception of the relation between sign and object, which does not divide the world into two mutually exclusive kinds of things: signs and objects. Anything has sign-potential in mediation, just as anything can be an object in the phenomenal sense that Peirce uses the term. Even so, a sign actually serving in semiosis is not an arbitrary occurrence.

Anything whatsoever can be a sign, i.e. can function in that role; but in order to do so it must have some character in virtue of which it can so function. This character is what constitutes the ground or reason of its being capable of being a sign, though it is not actually a sign until it is interpreted as such. (Ransdell, 1966, p. 80)

A representation is not simply a re-presentation, but a necessarily tri-relative (potentially multiply affording) occurrence of something referring to something else for someone, in the continuing evolution of meaning (in semiosis).

Jakobson and many other modern linguists, anthropologists, and semioti- cians have concentrated almost exclusively on these tri-relative concepts of sign-potential. That focus is consistent with their attempt to apply the empirical research model, which can only describe a sign (one thing) standing in a dyadic relation to an object (another thing). For Peirce, this is semiosis considered from someone's momentary point of view: a sign stands for an object to someone in some respect or capacity. We can take this "objective perspective" to investi- gate "sign capacity," or potential for meaning to occur, if we remember that the sign-object relation is not really dyadic action (simply causal), but that we are "taking the place" (so to speak) of the someone for whom the sign might stand for the object, and only in some respect. Any conscientious communicator must achieve this "Golden Rule perspective."

While the ground conditions of anything that serves as a sign are the particular respects which make it inherently fit or capable to represent an object, no signs are purely iconic or indexical; all signs are to some extent symbolic (see CP, 2.228). On the other hand, a convention tells us generally how to use a given sign, but it does not afford us the referential means actually to do so. We have general conventions to interpret the indexical function of a pointing finger, not a special convention for each pointing finger, which we first must recognize and identify as a pointing finger. Peirce defines a genuine sign in terms of iconic, indexical and symbolic functions; each is essential to a complete and workable language for representing the course of inquiry.

In Peirce's theory of signs, the concept of affordance would pertain to the iconic and indexical power of signs to refer, the suggestive and indicative qualities we might perceive in actual ecological circumstances that make effective symbolic reference and meaning possible but not strictly determined (see CP, 2.304). They


86 MARY KEELER

are the experienced aspects of anything that might lead someone to interpret it as a representation, aspects which anything has (even an imaginary thing) whether or not it is interpreted. This sign-potential of something, however, can only effect meaning if someone realizes its affordance (or unless it "points in two ways" for that someone, as Gibson says) in using it to refer to something else.

Peirce's theory can easily be distinguished from most of current semiotics, semiology and linguistics which assume signs function only within a system, or within a code of the language type. In these theories, the structure of the code system defines its function by means of an underlying grammar (or rules for the regularity) of symbolism, such as coded pairs of signifiers and signifieds (or forms and meanings). For Peirce, symbols of any sort are "the most general signs," but they are not reducible to rules among primitive elements, because these primitives are functional relations occurring outside systems of convention or habit.

Many philosophers have mis-construed Peirce's pragmatism as an empiricism (see, for example, Ernst Nagel in Buchler, 1939, p. xvi). But Christopher Hookway explains that Peirce does not merely engage in the controversies of traditional philosophy (such as rationalism vs. empiricism); instead, his pragmatism makes these intellectual oppositions "tractable," by reconciling their complementary roles as thought and observation in human experience which make knowledge possible. Pragmatism, for Peirce, is the method of inquiry implied by his tri-relational sign theory of the conditions required for semiosis - the conditions that afford the growth of meaning. These conditions conceptually correspond to the compound invariants that Gibson's theory of perception identifies as affordances.

Peirce's and Gibson's metatheories might be productively related in the semiotic study of computer-mediation tool development. Gibson insists: "The experimental psychologist should realize that he cannot truly control the perception of an observer, for the reason that it is not caused by stimuli. ... The theory of psychophysical parallelism that assumes that the dimensions of consciousness are in correspondence with the dimensions of physics, and that the equations of such correspondence can be established, is an expression of Cartesian dualism. Perceivers are not aware of the dimensions of physics. They are aware of the dimensions of the information in the flowing array of stimulation that are relevant to their lives" (p. 303).

Segmentation is considered to be a fundamental problem in designing compu- tation systems to augment automated processing of images: how to break down an image into codable units for processing, units that are meaningful to humans and useful in human pursuits? While Gibson's perceptual theory implies that such segments might be identified as compound structural invariants (affordances), Peirce's theory of representation maintains that the symbolic meaning of these phenomenal qualities of what might be used as a sign (the iconic and indexical aspects of an image, in this case) is always virtual (or potential), and must be identified in any particular case by considering segments in terms of the tri-relative circumstances at hand. Peirce distinguishes psychological from logical concerns on



that basis (see CP, 4.479) and created a graphical notation for logical analysis of language, called Existential Graphs, to "put before us moving pictures of thought, I mean of thought in its essence free from physiological and other accidents" (CP, 4.8). A Gibsonian theory of affordance in perceptual systems may be a technically adequate approach to the problem, but it would require the resourceful development of segmentation methods within and for specific contexts of use, as Peirce's theory and pragmatic method instruct.

Nearing the end of my allotted space, I will introduce two contexts in which to demonstrate the implications of Gibsonian and Peircean perspectives for human- computer partnerships in creative inquiry. Peirce's theory would explain Gibson's theory of invariants in terms of relative invariance. Such findings as Cutting reports from experiments dealing with nominal invariants for heading and path information (Cutting, 1997, 1999) appear to indicate that invariants are not individually isolatable perceptual phenomena, at any level of analysis. I will implicate the significance of this point in the use scenarios that follow, which are based upon research undertaken to represent Peirce's archived manuscript collection in a digital image-base, as the primary evidence for a network-based community of scholars and researchers to reconstruct his work from archived fragments.

5. A New Approach to Computer-mediated Image Processing Tools

The largest collection of Peirce's manuscript material, some 100,000 pages archived in the Houghton Library at Harvard, has remained essentially inacces- sible for 80 years, on rapidly deteriorating high-acid paper. Even if traditional methods could provide effective access, the character of his philosophical explorations would defy representation in any discrete medium such as sheets of paper. Reading and editing Peirce's work is an extremely difficult bibliographical task, and he was a polymath whose work cannot be fully comprehended or critically edited by anyone working alone, or even in a small editorial group. His writings are also filled with graphics and color crucial to interpretation of his philosophy, science, and methodology. His discussions, no matter what their nominal topic, are a vast matrix (he called them "knarled twine") of ideas cutting across many disciplines. Mathematics might occur significantly in a paper on phenomenology, or chemistry in one on pragmatism, or logic in a mathematical paper, or geodesy in one on metaphysics. Most of his philosophical work was never composed for or consolidated into a published account during his life-time, and only highly- selected editions have been attempted since. Very few, hardy, scholars have studied the 40,000 pages he wrote from 1900 until his death in 1914, which include his mature work on semiotic, pragmatism, and a graphical notation for logic. Many scholars in collaborative inquiry will be needed even to reconstruct his corpus in some useful condition.

Peirce's semiotic view of inquiry treats scholarly communication as a continuing logical argument, with premises, conclusions, and an account of the


88 MARY KEELER

interpretational procedure by which any result is reached from the evidence; but his pragmatism cautions that judgment in inquiry should proceed heuristically - not algorithmically (maintained by permanent authority or habit of mind). The conceptual basis for judgments established by any group of inquirers may, at any time, be mistaken. Inquiry conceived as a continuing representational challenge, depends on an "editorial role" in its procedure to stabilize collective progress by tracking the possible directions (that is, conceptual tendencies based on paths of relative invariance in the argumentation) it might be taking with respect to the manuscript evidence. Editorial augmentation would construct and continually update a meta-representation (a sort of conceptual map of collective self-reflection) of all individual contributions - without determining its results; as Peirce might say, to keep the matter of inquiry ("texts" or mediational products) from becoming the hide-bound substitute for the continuity (growth) of mind. The following scenarios briefly describe two fundamental computer-augmented capabilities required for scholars (remotely located from one another and from the archived collection) to engage in collaborative argumentation using digitized images of Peirce's manuscript pages, in a community of inquiry. These two capabilities are under development in the Peirce On-line Resource Testbeds (PORT) project (see Keeler et al., 1997-2000).

5.1. IMAGE-QUALITY MEDIATION: IMAGES AS THE PRIMARY EVIDENCE FOR INTERPRETATION

Digital images of the manuscript pages must serve reliably as accurate representations of the original pages, for evidence of what Peirce rendered in graphical forms and in handwritten text, but also for evidence of other bibliographic features (color and kind of paper, of editors' and others commentary that is not in Peirce's hand, and of unintended effects of wear and aging). We might say that these images must exhibit to each scholar and researcher a high relative bibliographical invariance of all critical iconic and indexical features. Collections such as Peirce's manuscripts will require special attention in this regard, because many notebooks have pages missing from their original order, because early editors of the archived material actually marked the pages in various ways that are not immediately distinguish- able from Peirce's own marks, and because of Peirce's significant use of color in text and images. Any visible feature on a page may need to serve as evidence in judgments as to where the page belongs, what was its significance to Peirce, and what insights it might convey. Computer-based augmentation applied in the process of developing effective digital representation of the archived source material must help us determine what image quality is required for transcription of the manuscripts. In virtual working contexts, created on-line, transcribers and editors who jointly perform the transcription task must critically examine the effects of different displayed levels of image quality with respect to such file and display system constraints as compression ratio, screen size and resolution, and color calibration.



To provide uniform image quality to the heterogeneous display systems of all participants, some perceptual mediation must occur to standardize the different views (or to establish relative invariance), and to notify participants when their views do not optimally correspond, and to maintain a reference quality of image for a given file (page) for all viewers.

The augmentation must mediate among collaborating users' requests to establish the optimum perceived attribute values. The match is especially important for collaboration on detailed features, such as tone of paper (for page location identification) or to distinguish between pencil and pen marks in establishing what lines were rendered by Peirce. Subtle iconic and indexical qualities, often not immediately recognized as important (or affording), can become crucial in collaborative interpretation. Collectively, all participants must have access to reliably comparable views of the same artifact, while individually, they must be able to explore this primary data for its potential affordances in each disciplinary pursuit. And any such mediation service must avoid the problem of simply serving the lowest system capabilities. Mediation must include human intervention in the process, at any point needed to resolve discrepancies among diverse views of the evidence. The lowest quality display system should not prevent those with higher quality systems from seeking and finding crucial evidence, and bringing it to the collaboration context as the object of discourse. The image quality issue must be carefully considered in terms of complexly interrelated system variance factors, including resolution and color accuracy, compressed file quality and speed of transmission, along with storage and management for efficient serving. The augmentation must include a means of monitoring this mediation process (or this reconciliation of competing factors), as participants test the augmentation method itself, to facilitate group-critical and self-critical awareness (and control) in viewing the images as the evidence of what Peirce rendered on paper. The relative bibliographic invariance identified in the evidence would then serve as the basis for progressively meaningful segmentation, expressed by means of various indexes, in catalogue form.

5.2. CONCEPTUAL CATALOGUE CONSTRUCTION: IMAGES AS INTERPRETIVE EXPRESSION

A Concept-based catalogue for the Peirce archive can evolve from the data of current print finding aids, which can be digitally linked to both transcribed text and digital images of the original manuscript pages, as they are produced and indexed. Under collaborative development through network communication, scholars and editors can correct these catalogue entries to include additions and alterations accumulated during the decades of print-edition editing. Concurrently, an ontol- ogy of the collection (or a comprehensive framework capable of relating the archive data as a conceptual structure, or logical representation of inference relations) can interrelate the catalogue entries on many conceptual bases. Special


90 MARY KEELER

ontologies can serve as a specialized database views, functionally representing the conceptual perspectives of each group of users, in terms of their disciplinary conventions (or affordances), with respect to the archive and any related data. These affordance-related indexes must continue to evolve as each realm of inquiry evolves. Graphical views (or visualizations) of this data-under-evolution become imperative, if those collaborating are to remain critically aware of the complex implications of each contribution with respect to other contributions and the scheme as a whole. Methods under development in the realm of knowledge representation and processing make possible the three essential functions required in the process of collaborative conceptual catalogue evolution: reporting, tracking, and mapping.

Groups of scholars or researchers in any particular discipline must be able to report their reconstructions of Peirce's complex arguments, to serve as their hypotheses based on their interpretations of the represented (multiply affording) manuscript page content. Editors and scholars must be able to track these contributions, as arguments (from the image evidence supporting them, through their systemic implications), for comparative and expository purposes. Even the order of pages in a manuscript will frequently be hypothetical, since many have been lost or misplaced, but also because of Peirce's complex composition style. He sometimes used the same page in several versions of elaborate discussion, the course of which sometimes even doubles-back on itself to pick up an unexplored path.

One Peirce scholar (Shea Zellweger) has mapped the page-by-page course of some of his "explorations" in several manuscripts. Each scholar contributing to the digital catalogue could create such a map or line-diagram, as a computer-generated image, to record each (possibly unique) reconstruction of Peirce's multi-path arguments. These "S-diagrams" could then be matched (by either human perception or computer-generated graph methods) Detected relative iconic and indexical invariance among maps would reveal discrepancies in sequence across scholar's hypothetical (symbolic) reconstructions, to pin-point where controversies in reconstructions lie and further investigation should take place. As reconstructions and interpretations proceed and scholars' arguments evolve beyond the "frozen page," as primary evidence of what Peirce actually wrote, a "motion picture" master map must be generated, which continues to diagram the courses of different scholars' arguments, creating Peirce's moving pictures of thought.

The highest kind of symbol is one which signifies a growth, or self- development, of thought, and it is of that alone that a moving representation is possible; and accordingly, the central problem of logic is to say whether one given thought is truly, i.e., is adapted to be, a development of a given other or not. In other words, it is the critic of arguments. (CP, 4.9)

Diagrammatic correlation (by automated link-tracking between the manuscript page-images and each scholar's represented arguments) would make it possible for all researchers to trace their interpretations (or expressions) back to the original



evidence, to look again, and even possibly to discover something not noticed before.

6. Conclusions

The two scenarios indicate the convenience of the evidence and expression dichotomy in referring to the role of images, but the continuity between these two is what gives images their great value in communication. Pictures can range from the highest quality photograph to the most abstract diagram. An algebraic expression is an example of how abstract diagrams can be, and how the iconic and indexical empower language. Any iconic and indexical significance in the layout of an expression has diagrammatic potential to be recognized by someone who understands the conventions of its use. Even words on a page can be considered diagrams! Peirce concludes, "the most perfect of signs are those in which the iconic, indicative, and symbolic characters are blended as equally as possible" (CP, 4.448). The problem in understanding representation of any sort is to explain how we perceive or conceive that something is related to something else for someone. The traditional theories of recognition, association, or inference cannot explain the conditions of representation comprehensively enough for us to specify any sort of augmentation that might improve it - and why should we care about that technological improvement? Communication media will continue to improve, whether we learn to engage effectively in its technological evolution or not, but significant perceptual and representational puzzles that remain to be solved lie properly in humanist research.

They are the puzzle of the relation of signs to minds, and of their communication from one mind to another, and the puzzle of the composition of concepts and the nature of the judgment, or, as we of the antipsychological school say, of the proposition. (MS 498, p. 29; an unpublished MS, entitled by Peirce, "On Existential Graphs as an Instrument of Logical Research") An initial theoretical question might be: what does change from one end of the

theoretical image-text continuum to the other? Peirce hints that empirical science, as we have known it in the twentieth century, will not sufficiently respond: "I hear you say: 'All that is not fact; it is poetry.' Nonsense! Bad poetry is false, I grant; but nothing is truer than true poetry. And let me tell the scientific men that the artists are much finer and more accurate observers than they are, except of the special minutiae that the scientific man is looking for" (1.315). Software designer Paul Heckel suggests what I think is an example of a promising humanist research hypothesis:

Words are like pointers that single out significant peaks from the unbroken contour line of a mountain range on the horizon. The peaks are not created by the pointers. They are given objectively; but the pointers fortify the observer's urge to discriminate them. ... Although we may think visually, we can only


92 MARY KEELER

focus on one concept at a time. A writer is constrained by the nature of language to focus the reader's attention on one concept at a time. By ordering words, the writer focuses the reader's attention at a series of places. (pp. 39-40) If humanists cannot bring full human appreciation to the capabilities of new

media in computing, they will fail to lead in the creation of pragmatic methods of improving these augmentation capabilities, technologists will continue to judge what constitutes improvement, and journal titles will continue to be "Computers and ..." Gershenfeld invites us forward in the mode of pragmatic methodology: "Far more interesting than declaring a revolution is to ask how to capture the essence of what works well in the present in order to improve the future. The real challenge is to figure out how to create systems with many components that can work together and change" (p. 10).

Note "MS" references are to Peirce's manuscripts archived at the Houghton Library, Harvard; for "CP" references, see Collected Papers of Charles Sanders Peirce, 8 vols. Ed. Arthur W. Burks, Charles Hartshorne, and Paul Weiss. Belnap Press, Cambridge, 1931-1958 (now available on CD-ROM from InteLex, Inc.). "W" refers to Writings of Charles Sanders Peirce. Ed. Nathan Houser and Christian Kloesel. Indiana University Press, Bloomington, 1977. "NEM" refers to The New Elements of Mathematics by Charles S. Peirce, 4 vols. Ed. C. Eisele. Mouton, The Hague, 1976.

References Buchler, Justus. Charles Peirce's Empiricism. Harcourt, Brace, New York, 1939. Cutting, James E. "Four Ways to Reject Directed Perception". Ecological Psychology, 3(1) (1991),

pp. 25-34. Cutting, James E. In Gibson's Legacy. Ed. S.C. Masin. Elsevier Science Publishers, 1993, pp. 231-

261. Cutting, James E. "Where We Go with a Little Good Information". Journal of the American

Psychological Society, 10(1) (1999), pp. 71-75. Cutting, James E. and M. Massironi. "Pictures and Their Special Status in Perceptual and Cognitive

Inquiry". In Perception and Cognition at Century's End. Ed. J. Hochberg. Academic Press, San Diego, 1998, pp. 137-168.

Gershenfeld, Neil. When Things Start to Think. Henry Holt, 1999. Gibson, James, J. The Ecological Approach to Visual Perception. Houghton Mifflin Co, 1979. Hagen, Margaret A. The Perception of Pictures, Volume 2: Durer's Devices, Beyond the Projective

Model of Pictures. Academic Press, 1980. Heckel, Paul. The Elements of Friendly Software Design, 2nd edn. SYBEX, Alameda, CA, 1991. Hochberg, Julian E. Perception, 2nd edn. Prentice-Hall, 1978. Hochberg, Julian E. "The Construction of Pictorial Meaning". In Advances in Visual Semiotics Ed.

T.A. Sebeok and D.J. Umiker-Sebeok. Mouton de Gruyter, Amsterdam, 1995, pp. 111-162. Hochberg, Julian E. "The Perception of Pictures and Pictorial Art". In Cognitive Ecology. Ed. M.P.

Friedman and E.C. Carterette. Academic Press, San Diego, CA, 1996, pp. 151-203. Hookway, Christopher. Peirce. Routledge and Kegan Paul, London, 1985. Keeler, Mary A. "Pragmatically Yours". In Lecture Notes in Artificial Intelligence, Vol. 1867.

Springer-Verlag, 2000, pp. 82-99.



Keeler, Mary A. "Iconic Indeterminacy and Human Creativity in C.S. Peirce's Manuscripts". In The Iconic Page: In Manuscript, Print, and Digital Cultures. Ed. G. Bornstein and T. Tinkle. University of Michigan Press, 1998.

Keeler, Mary A., C. Kloesel and L. Searle. "PORT: A Testbed Paradigm for Knowledge Processing in the Humanities". In Lecture Notes in Artificial Intelligence, Vol. 1257. Springer-Verlag, 1997, pp. 100-113.

Keeler, Mary A. and Christian Kloesel. "Communication, Semiotic Continuity, and the Margins of the Peircean Text". In Margins of the Text. Ed. David Greetham. University of Michigan Press, Ann Arbor, 1996.

Keeler, Mary A. and Christian Kloesel, Eds. Casting the Net: Towards a Model for Communication and Collaboration on the Electronic Network. Report of an Invitational Symposium at George Washington University, 4-5 June 1992 (Supported by the National Science Foundation).

Keeler, Mary A. and Susan Denning. "The Challenge of Interface Design for Communication Theory: From Interaction Metaphor to Contexts of Discovery". Interacting with Computers, 3(3) (1991), pp. 283-301.

Keeler, Mary A. "Investigating Transparency in the Conditions of Mediation". Semiotica 82(1-2) (1990), pp. 15-41.

Ransdell, Joseph M. Charles S. Peirce: The Idea ofRepresentation. Columbia University dissertation, 1966.


Article Contentsp. 75p. 76p. 77p. 78p. 79p. 80p. 81p. 82p. 83p. 84p. 85p. 86p. 87p. 88p. 89p. 90p. 91p. 92p. 93

Issue Table of ContentsComputers and the Humanities, Vol. 36, No. 1, Image-Based Humanities Computing (Feb., 2002), pp. 1-140Front MatterThe ACH Page: The Job Market for Humanities Computing [pp. 1-2]Editor's Introduction: Image-Based Humanities Computing [pp. 3-6]The Reappearances of St. Basil the Great in British Library MS Cotton Otho B. x [pp. 7-26]Digital Facsimiles: Reading the William Blake Archive [pp. 27-48]Text-Image Coupling for Editing Literary Sources [pp. 49-73]The Place of Images in a World of Text [pp. 75-93]Dialogue and Interpretation at the Interface of Man and Machine. Reflections on Textuality and a Proposal for an Experiment in Machine Reading [pp. 95-107]Select Resources for Image-Based Humanities Computing [pp. 109-131]Back Matter

images and texts

Documents

place of text

world of images

place of images

place images

handling text

world of text authors

humanist world of text

utcyour use