Evans

Brian EvansUniversity of AlabamaDepartment of Art and Art History103 Garland Hall, Box 870270Tuscaloosa, Alabama 35487 USAbrian.evans@ua.edu

Foundations of a VisualMusic

11

In the moving image we have a synthesis—thespatial counterpoint of graphic art, and the tem-poral counterpoint of music.

—Sergei Eisenstein

There is a form of film that is trying to evolvethat area of thinking which I call “moving vi-sual thinking.” And it is intrinsically a visualmusic . . .

—Stan Brakhage

Abstract animation, often called visual music, canhave a structural base similar to that of absolutemusic. In this article, I propose a groundwork for apractical theory of visual music composition. Theapproach builds from the simple premise that theresolution of tension moves us through time. Thearticle will review some fundamentals from art anddesign and apply them to temporal structure. To es-tablish the idea of visual consonance, the designprinciple of proportion will be presented. From this,I will develop and codify the idea that it is possibleto resolve visual dissonance to consonance, and somove a viewer through time in a way similar totonal harmony in music. We will then briefly con-sider problems of color. The article will reviewsome traditional approaches to color harmony andsuggest a simple hierarchical approach to workingwith color in time.

Expanding to a more comprehensive system oftime-based design, the writings of Russian film-maker Sergei Eisenstein will be mined for raw mate-rial (Eisenstein 1942, 1949). Eisenstein believed that“art is always conflict” (Eisenstein 1949, p. 46). Forthis article, conflict is defined as the opposition offorces that motivates and shapes action. The op-posed forces are dissonance/consonance or tension/release. Eisenstein’s writings contain many ideasthat are useful in the development of a theory of vi-sual music—especially his definition of cinematicmontage. From the concepts of montage and time-

design, a tentative but useable grammar for structur-ing time-based visual art can be developed and even-tually applied to the integration of sound and image.

For this article, I created some simple video ex-amples as illustrations. Where useful, I will also re-fer to the works of some visual music pioneers andother filmmakers. (Pieces were selected based onrelevance and availability, and the list is by nomeans exhaustive.)

Visual Music and its History

Visual music can be defined as time-based visualimagery that establishes a temporal architecture ina way similar to absolute music. It is typically non-narrative and non-representational (although it neednot be either). Visual music can be accompanied bysound but can also be silent.

One can imagine the origins of visual music goingback to the discovery of fire and dancing shadowson cave walls, or perhaps reflections of clouds whenfirst seen on the surface of rippling water. In morerecent history, color organs (instruments that pro-jected colored light) were seen by the public as earlyas the 18th century with Castel’s Ocular Harpsi-chord. In the 1920s Thomas Wilfred toured theUnited States and Europe performing on his Clav-ilux, an early electrical instrument that createdclouds and streams of continuous color. This his-tory intensified over the last century, parallel withthe development of cinema.

While the Hollywood-style narrative dominated(and dominates) cinema, pioneering filmmakerswere working to develop a non-narrative languageof light. Typically non-academics, these pioneersworked in an experimental tradition similar to thatof music composers such as Eric Satie, John Cage,and Karlheinz Stockhausen. Among the early film-makers were Germans Walter Ruttman, VikingEggeling, and Hans Richter. Oskar Fischinger, alsofrom Germany and moving later to the UnitedStates, worked over thirty years creating abstract

Computer Music Journal, 29:4, pp. 11–24, Winter 2005© 2005 Massachusetts Institute of Technology.

animations (Moritz 2004). The career of NewZealander Len Lye, working mainly in London,spanned decades. American pioneers included Johnand James Whitney, Mary Ellen Butte, Stan Brakhage,and Jordan Belson. Norman McLaren, Evelyn Lam-bart, and their colleagues at the National FilmBoard of Canada also created significant works.There were many others (Russet and Starr 1976).

They all struggled against financial, institutional,and technical barriers, yet they left a body of workthat provides a base from which to construct a the-ory of visual music. This work is finding renewed in-terest as evidenced by recent exhibitions. The 2003Sonic Light Festival in Amsterdam featured manyof the pioneering animations, as did Sons & Lu-mières, Une histoire du son dans l’art du 20e siècleat the Centre Pompidou in Paris, closing in January2005 (Cruse 2004; see also www.sonicacts.com/03).Also in 2005, the Los Angeles Museum of Contem-porary Art opened the exhibit Visual Music: Synaes-thesia in Art and Music Since 1900, which thenmoved to the Hirshorn Museum in Washington, D.C.(Brougher et al. 2005). Much of this work is becom-ing easier to find on video and DVD at online sourcessuch as the Iota Center (www.iotacenter.org), theCenter for Visual Music (centerforvisualmusic.org),and even Amazon.com.

Visual Music/Absolute Music

The concept of absolute music can be constructedfrom Igor Stravinsky’s statement, “Music meansnothing outside itself” (Stravinsky 1956). This illus-trates a common mindset held by many composersfor centuries. Form and content were often one ininstrumental music. (This of course does not includeprogrammatic music and especially the occasionalattempts to imitate real-world sounds, such as birdcalls, thunderstorms, etc.) Today, acoustic and elec-tronic instruments can be used as generators of ab-stract sounds without a referent in the real worldbeyond the instruments themselves. These non-referential sounds are used to create abstract temporalstructures, from traditional musical forms to thoseof a more experimental nature. (Many of these newforms also find correspondences in visual music.)

A simple definition of absolute music is thestructuring of time with the materials of sound pat-terns (excluding again program music and literaryforms such as song, opera, and theater). Stravinsky’scomment refers to music composition as the mak-ing of “art for art’s sake.” A musical work seeksfrom the listener an aesthetic response to the per-ception of sonic pattern—the appreciation of “sig-nificant form.” This was a primary focus ofmodernism and its formalist leanings (Bell 1914).

When visual artists discuss composition, they aregenerally referring to static design, the formal distri-bution of objects in the picture plane or in three-dimensional space. (This article will dealexclusively with two-dimensional space.) In the20th century, artists sought to express design in theabstract—significant form in visual space. They be-gan to consider seriously their work in a musicalway, and the abstract form of the work was the con-tent. Kandinsky, considered by many to be the fa-ther of abstract painting, said:

A painter . . . in his longing to express his innerlife cannot but envy the ease with which mu-sic, the most non-material of the arts today,achieves this end. He naturally seeks to applythe methods of music to his own art. And fromthese results that modern desire for rhythm inpainting, mathematical, abstract construction,for repeated notes of color, for setting color inmotion. (Kandinsky 1914)

Using music as a basis for visual work was a con-cept explored by many artists in the early 20th cen-tury. Notable is the work by Sergei Eisenstein, aRussian avant-garde filmmaker who put these ideasinto practice and, fortunately for us, wrote downmany of his theories of cinema. These ideas stillhave potency and are especially relevant in discus-sions of visual music, time design, and the combin-ing of sound and image.

Visual Consonance

For the visual artist, composition is “the arrange-ment of elements and characteristics within a de-fined area . . . a grouping of related components that

12 Computer Music Journal

Evans

make sense together . . . balanced by an overall ap-pearance of continuity” (Bowers 1999). Good vi-sual composition achieves harmony. Art theoristRudolph Arnheim said, “One of the basic visual ex-periences is that of right and wrong” (1966). Thisidea forms the basis of design foundations as taughtin most art schools. First-year art students studytwo-dimensional design and learn the principle ofvisual “rightness” and how to achieve it with theconceptual visual elements of point, line, plane, andvolume. These elements manifest themselves as ob-jects in a defined space, expressed as size, shape,color, and texture, establishing dynamic relation-ships as these elements interact.

Visual “rightness” is visual consonance. This be-comes an axiom from which we can build a gram-mar of visual music. From this premise, composingvisual music is a simple process. If rightness is codi-fied and understood, wrongness is easily defined bynot being right. We might call this wrongness visualdissonance, that is, visually active moments of ten-sion in a temporal design. Progressing from visuallywrong to visually right moves us from dissonance toconsonance or tension to release, just as in music adominant harmony resolves to a tonic (e.g., V–I). Asin music, much of the interest and energy is in themoments of tension. (In tonal music theory texts,how many more pages are devoted to the dominantchord and all its variations versus the tonic chord?)

Using the tension/release construction, we canmove dynamically—some might say musically—through time with visual materials. We can estab-lish visual cadence points. With cadences, we canarticulate units of time and so develop larger tem-poral units such as motifs, periods, and phrases.

Proportion

Consider a well-known design principle, the goldensection (usually symbolized as the Greek letter f orphi). A proportion of roughly 1:1.618, it is prevalentin the visual arts worldwide and throughout history(Huntley 1970). Figure 1 shows my brief analysis(what is sometimes called an harmonic analysis) ofHokusai’s famous The Great Wave off Kanagawa(ca. 1831). The cut of the golden section both hori-

zontally and vertically illustrates its utility as acompositional device. Many important points of at-tention lie on these formal lines: the highest waveis placed on a vertical, the ocean level on a horizon-tal, and Mount Fuji at an intersection of a horizon-tal and vertical.

The visual composition has a dynamic balance orsymmetry in the picture plane. We see elements ofrepetition. (Note the wave at left-middle matchingthe shape of Mount Fuji, and the foreground shapeof the big arching wave mirrored with the back-ground sky.) We are not concerned here with thesubject matter, but rather the underlying principleof composition or design.

Figure 2 shows my analysis of a drawing of an elkfrom the walls of the Caves of Lascaux in France.(Dating from approximately 13,000 BCE, it is doubt-ful that the golden section was applied consciously,although the drawing of the rectangle on the wall iscurious as it predates Euclid by over 10,000 years.Even more unusual is the fact that it is a golden rec-tangle!) The animal seems to balance on the lowerleft intersection of phi as it divides the boundingrectangle on both the horizontal and vertical.

Whether by instinct or design, use of universalproportions is a common and effective formal de-vice in visual and music composition. Proceedingfrom simple symmetry to a balanced but activecomposition is a useful skill in creating visual mu-sic. Phi is just one of many proportions that hasbeen explored over the centuries. Jay Hambidge’sstudies in dynamic symmetry showed the applica-

13

Figure 1. My harmonicanalysis of Hokusai’s TheGreat Wave off Kanagawa(ca. 1831).

tion of many mathematical constants, including √2,√3, and various other geometric constructions, illus-trating their application in everything from Grecianurns to furniture to typography (Hambidge 1919,1932). The “secret” of the golden section was re-vealed during the Renaissance in Luca Pacioli’sbook Da Divina Proportione, which was widelyread by artists of the time.

Also at this time, Leon Battista Alberti publishedhis writings on painting and architecture. He de-scribed the system of one-point perspective and theapplication of the Pythagorean proportions of musicto visual construction. Figure 3 is taken from a har-monic analysis of Botticelli’s Birth of Venus byCharles Bouleau (1963). Here, we see an applicationof phi on the cut of the horizon line along with acomposition based on the simple harmonic propor-tions of 9:12:16. Botticelli here used Alberti’s musi-cal consonances. For Alberti, proportion was a visualmusic. In Botticelli’s application of these propor-tions, we see the interval of the perfect fourth (9:12,or 3:4) and what Alberti called the duple diatessaron,or double fourth (9:16, or 32:42) (Alberti 1955).

Simple-number proportions are a foundation ofWestern music intervals, but we can find applica-

tion of proportions such as phi in temporal con-structions as well. Bartók’s and Debussy’s uses ofphi are well documented (e.g., Lendvai 1971; Howat1983). Stockhausen often used the Fibonacci series(a number series from which phi can be derived) inseveral of his early works (Griffiths 1981). Evidenceof phi can even be found in the music of Mozart,which should not be surprising as he was a Mason,and the divine proportion was part of the Masonic“secret knowledge” (Evans 1993).

It should be stressed here that these proportionsare not a panacea for the creation of good composi-tion or visual consonance. They are, however, un-doubtedly useful and often used. Good visualcomposition requires sensitivity to all elements inthe picture plane. Anyone interested in composingvisual music should study two-dimensional designin all its various aspects, much as any first-year artfoundations course will provide.

From an understanding of proportion and compo-sition in the picture plane, ideas can expand intovariations, and larger structures can build on rela-tions of design elements. Figure 4 is from a designlesson in a book by Arthur Wesley Dow written overa century ago. The figure illustrates variations of asingle design, which he calls the motif. Dow said,“Little can be expressed until lines are arranged in aSpace [which] I shall call PRINCIPLES OF COMPO-SITION . . . all being dependent upon a great generalprinciple, PROPORTION or GOOD SPACING”(Dow 1899).

14 Computer Music Journal

Figure 2. My harmonicanalysis of a drawing of anelk from the walls of theCaves of Lascaux in France(ca. 13,000 BCE).

Figure 3. From a harmonicanalysis of Botticelli’sBirth of Venus by CharlesBouleau (1963).

Evans

Temporal Design

In temporal design, we move the viewer from visualdissonance to cadences of visually balanced, well-composed moments. Time passes musically throughpatterns of tension/release. Filmmaker Ken Burns’suse of slow pan and zoom of a camera over old stillphotographs is a simple technique that allows ex-ploration of this basic approach to time design. Us-ing only camera motion, we can see this phrasing infilms such as The Civil War—A Film by Ken Burns(Burns 1990). Now a common technique in histori-cal documentaries, camera motion over still imagescan also tell stories. See for example French film-maker Chris Marker’s 1962 short film La Jetée, anearly and effective application of this device alongwith montage and visual composition (Marker 1962).

I created simple illustrations of camera move-ment with still images, included here as Video Ex-amples 1 and 2. [Editor’s note: video figures for thisarticle can be found on the DVD accompanyingthis issue.] A virtual camera moves over digitallyprocessed photos of flowers. Key frames are seen inFigures 5 and 6. Each example starts in balance witha dynamic or simple symmetry. Once motion starts,a tension is established that seeks resolution instillness and a return to a balanced composition.

Each example can be considered as a phrase. Wecan eventually combine phrases, seeking points ofcommonality and difference in the visual dimen-sions of motion (direction and speed), composition,

and color. We see repetition. The two examples areof equal duration of 8 seconds. They zoom in atroughly the same rate, with a central focus of twoflowers moving to a single flower, focused left ofcenter. The examples feel similar, but there is varia-tion. They contrast in their vertical motion, anglesof rotation, and colors; there is repetition, contrast,and variation. We have the beginnings of a syntaxfrom which to construct larger forms and composevisual music.

The construction of a pattern of tension/releasemoves a viewer through time, and so we seek tobuild tension. The desire to resolve the tension nat-urally follows. Time passes dynamically. Video Ex-ample 3 is an exercise illustrating a single shot withelements entering and exiting the picture plane inan overall rounded binary form (AABA). The ex-ample has four phrases that land on well-composed,visually “right” moments. Phrase 1 introducesMondrian-like graphic materials with red and bluesquares, shifting in color and moving into a com-posed cadence, seen as a small yellow square spin-ning to combine with two thick black lines. Phrase2 follows with a repetition of blue and red squaresspinning in from the opposite direction, cadencingby coming to rest on a balanced design. A new, con-trasting phrase follows with thin lines rotatingaround an axis and a return of the small yellowsquare and thicker black lines. This overlaps into afinal phrase that again repeats the spinning red andblue squares while echoing the rotating lines that fi-nally articulate closure, coming to rest on anothercomplete and balanced design. The example illus-trates how visual time can be musical, using motifsand phrases with the “camera” locked in place andthe whole work unfolding in an single “shot.”

Developing a visual music piece in a single shotwith little or no editing is a common practice in vi-sual music composition; many of John Whitney’sfilms do this. His multi-movement work Arabesqueis built on the development of 360 points of lightmoving in the picture plane (Whitney 1975). Thepoints move independently through various numericprocesses, in and out of visual resonances where thepoints resolve into circles or symmetries in simplenumeric proportions. Whitney called this differential

15

Figure 4. From a design les-son in a book by ArthurWesley Dow (1899).

dynamics or harmonic procession (Whitney 1980).These moments of resonance provide points of vi-sual consonance. Moving away from and back to theconsonance creates tension and resolution.

Norman McLaren’s Pas De Deux (1968) showsmostly single-shot action using time-lapse filmingof two dancers. The piece presents another approachto Whitney’s idea of differential dynamics. Definitepoints of resolution occur when all the time-lapsed

movements of the dancers resolve into single fig-ures, reminiscent of a musical canon at the cadencewhen all voices come together.

Oskar Fischinger’s Study No. 10 (1932), in blackand white, is a wonderful example of this single-shot approach with movement through moments ofvisual consonance. In this piece, Fischinger createsabstract materials that directly follow and expressthe phrases and feeling of the music.

16 Computer Music Journal

Figure 5. Key frames fromVideo Example 1. Dotsshow points of focus, andlines indicate the basic tra-jectory of camera motion.The camera moves up androtates counterclockwise.

Figure 6. Key frames fromVideo Example 2, illustrat-ing points of focus and ba-sic camera trajectory. Thecamera is moving downand rotates clockwise.

Figure 5

Figure 6

Evans

Considering Color

Color can also be used to create a tension that seeksresolution, although it is not an exact science.Artists have developed many theories of color overthe past centuries. A common idea in these theoriesis harmony. In discussing harmony, theorist LuiginaDe Grandis said, “All theories have in common anassumption that the sensation of harmony, or con-cord of parts of a whole . . . results exclusively fromthe relationships and proportions of its chromaticcomponents” (De Grandis 1984).

The problem lies in how those chromatic compo-nents should be organized, as there is no single uni-versal categorization of color. Margaret Livingstone,Professor of Neurobiology at Harvard University,studies how the eye and brain process color. Shenotes that “much of what has been written aboutcolor in art is nonsense.” She points out that thebrain processes hue and luminance (also calledvalue) separately. Some aspects of vision, such asdepth perception and the separation of figure andground, are colorblind, meaning that we get our vi-sual cues from value alone. And the human percep-

tion of color, especially hue, can vary widely basedon genetics and experience (Livingstone 2002).

At the very least, color in art is confusing. Takefor example the two color circles seen in Figure 7.Both are used to determine complementary colors.A basic tenet of color theory is that complementarycolors (those directly opposite on the circle) createcolor harmony. The circle on the left shows thestandard color primaries as understood by artistsover the centuries: red, yellow and blue (“RYB”). Jo-hannes Itten formalized this color circle in develop-ing the color theories he taught at the Bauhaus.This approach to color is still taught in many artschools today (Itten 1973).

The circle on the right is the one used in com-puter graphics displays as they project light. It usesthe additive primaries of red, green, and blue (RGB).When added together, these primaries will makewhite light. The complementary colors in this circleare cyan, magenta, and yellow (CMY), also calledthe subtractive primaries, which theoretically pro-duce black (the absence of light) when combined. Asone mixes them, more and more light is absorbed orsubtracted. Commercial and digital printing pro-

17

Figure 7. Two color circles:(a) Itten’s (“painter’s”)RYB color circle; (b) addi-tive (“computer”) RGBcolor circle.

(a) (b)

cesses use the subtractive primaries with black inkadded (denoted K), as real-world pigments cannotcreate a true black, but rather a somewhat muddybrown.

In Itten’s system, red and green are complemen-tary and concordant in combination. In additivecolor, red and green are both primary and so are dis-cordant. The two systems will create contradictoryharmonies. What is the measure we use to decidewhich is right? Color continues to be problematic inseeking visual harmony. Furthermore, the percep-tion of color is inexact, culturally influenced, andpersonal. I might use a variation of green that re-minds me of the zest of a key lime. The same colormight remind someone else of rotting meat, trigger-ing a very different response. One must be carefulwith color.

There are some generalizations we can draw,however. First, color space is a three-dimensionalconstruct. Familiar color spaces, all hue-based, in-clude RGB, RYB and CMY. Some color spaces in-clude other visual dimensions. One artisticallyuseful space maps a hue-only approach to one wherecolor is defined as hue, saturation, and value (HSV)(Smith 1978). There is no perceptual hierarchy ofhue, but there are of value (bright to dark) and satu-ration (full chroma to grayscale). Movement throughthese hierarchies provides areas of exploration.

Chemist and colorist Maurice Chevreul in the19th century suggested that the eyes seek the bal-ance of grayscale, as evidenced through the visualphenomena of simultaneous and successive con-trast (Chevreul 1854). A sensation of harmony or“chromatic equilibrium” should occur with com-plementary colors, as they would mix to creategrayscale, a sort of visual tonic. Gray is balanced,whereas emphasis on one hue or another createstension. Chevreul also points out the difficulty inpractice using pigments or dyes as “we know of nosubstance which represents a primary color.”

Newton showed us that we see white when allcolors of the spectrum of visible light are combined;lower the value or brightness and we see gray. Digi-tal technology simulates this with the additive RGBcolor space. We can measure the equilibrium ofcolor combinations without concern for color circlesor complementarity. Computer monitors work in

an RGB space designed to create grayscale when theRGB component values are equal.

With digital images, it is possible to calculatesums of the amounts of red, green, and blue in animage. The closer the RGB sums are to being equal,the more “balanced” the image is with respect tocolor, as equal means gray. Color can move usthrough time by resolving an emphasis of a singlehue to a coloring that balances or equalizes RGBsummation values (Evans 1990). Tension/releasewith color can be established in this way, but it re-quires a detailed measurement of digital color thatmay not be practical. What is more practical is ageneral application of the perceptual hierarchies ofbrightness resolving to dark, and high saturation re-solving to low saturation or gray.

A familiar example is mentioned in Bordwell andThompson’s analysis of the 1939 film version of TheWizard of Oz (Baum 1939; Bordwell and Thompson2001). The larger structure of the film is an ABAform based on the settings Kansas-Oz-Kansas.Dorothy is home in Kansas in the beginning, filmedin grayscale. Through her adventures in Oz, whereconflict builds, the film is in full color. Conflict re-solves back to grayscale when she returns home.Shifting from color to grayscale is now a commondevice on television, used to articulate or simplyend the action. A transition to grayscale or simpledesaturation often takes us to a commercial.

Working with color in HSV space, it is useful tothink of brightness as visual amplitude and satura-tion as spectral intensity. Bright, full, saturated col-ors can be tiring to a viewer when used in excess,just as loud, full-spectrum sounds can become tiringto a listener.

Oskar Fischinger’s Motion Painting 1 (1947) is agood example of a general approach to using satura-tion and brightness structurally. In the first sectionespecially, the movement of color is important as itbuilds large phrases. In the film, we see a singlepainting develop layer upon layer over eleven min-utes. We feel tensions resolve in short phrases as wesee shapes developing as motives, begun and thenbrought to completion. Spirals coil and unfold; mo-saics of diamonds appear and cover what is under-neath. Larger structures are perceived as repetitionand contrast of hue choices. Tension resolves as

18 Computer Music Journal

Evans

high saturation, and bright colors give way to darkercolors and lower saturations.

Montage and the Musical Phrase

Cinematographer Joseph Mascelli lists the “Five C’s”of cinematography as camera angles, close-ups, cut-ting, continuity, and composition (Mascelli 1968).All but the cutting require some degree of pre-production planning. Through the practice of story-boarding, we see this planning in action (Begleiter2001). With storyboards, filmmakers illustrate andarticulate many of the decisions to be made whenshooting a scene. The principles of compositionused in support of the story are key to this process.

A director plans a scene as a series of moments,composed as a series of drawings. The drawingsguide the eventual motion, framing, and design ofthe picture plane over time. Continuity, cameraangle, and composition in a scene delineate phrasesthat cadence with moments of visual rightness.Time passes musically.

With visual time passing under compositionalcontrol, a larger temporal structure can be built. Vi-sual phrases are combined in film in a process calledmontage, the cutting or editing of camera shots intoa sequence. Eisenstein’s theory of montage can bevaluable to our developing grammar of visual music.He believed that it was montage that placed film atthe “pinnacle of the arts.” The unit of montage,what Eisenstein called the cell, combines with otherunits and creates a gestalt, expressing more than justthe sequence of images. The power for Eisensteinlay in the heights of expression possible. “To deter-mine the nature of montage is to solve the specificproblem of cinema” (1949, p. 48). More simply, hestates “montage is conflict” (1949, p. 38). Montageis a multi-tiered construct of tension and release.

We can think of each cell as a musical phrase.From good visual composition, we can build phrasesand expand into a larger musical form. Temporal de-sign at the level of the shot moves us forward to ca-dence points. Through montage, we constructrelationships from shot to shot, building a largerform and expressing our ideas, formally or other-wise. Eisenstein lists five basic types of montage:

metric, rhythmic, tonal, overtonal, and intellectual.In discussing the five types, we will focus here onthe first three.

Metric Montage

In defining metric montage, Eisenstein writes that“pieces are joined together according to theirlengths in a formula scheme corresponding to ameasure of music” (1949, p. 72). We are only con-cerned with the duration of each shot, cutting eachshot to a beat without regard for the content of theshot. We establish meter by making each cell thesame length—the same number of frames. Each cellis like a phrase of music. With shots of short dura-tion, cells are like bars of music—sometimes evenbeats within a bar.

A time signature can be loosely establishedthrough accent of some visual dimension. Occa-sionally, the filmmaker will actually put an all-black or all-white frame at the end of each cell,making the visual meter obvious. Sometimes cellsare edited to match musical phrasing—a commondevice in music videos. Because the frame rate offilm or video is constant, we can accelerate or slowdown the meter by changing the length of the cellsas they sequence.

Metric montage follows the “law of simple num-bers,” and it is applied for clarity when based on amusical time passage (3/4, 2/4, 4/4, etc.), where itcan “bring into unison the ‘pulsing’ of the film withthe ‘pulsing’ of the audience” (Eisenstein 1949,p. 73). Using an irregular measure, “16/17, 22/57,etc. . . . brings a degeneration of the method” (Eisen-stein 1949, p. 72). From the extremes of simple toirregular is a middle ground where lengths are var-ied, alternating between two different contexts, jux-taposed temporally back and forth. What he refersto here are irregular-to-metered durations, such asrubato to a tempo, which is itself a basic process oftension/release.

Norman McLaren and Evelyn Lambart’s film Be-gone Dull Care is an enjoyable example of metricmontage. Strips of clear celluloid were painted,scratched, textured, and processed in a variety ofways. They cut the strips into cell lengths to match

19

with music by the Oscar Peterson Trio (Lambartand McLaren 1949).

Rhythmic Montage

In rhythmic montage, the length of the cell is deter-mined by what we see. “Abstract determination ofthe piece-lengths gives way to a flexible relation-ship of the actual lengths” (Eisenstein 1949, p. 74).Durations of phrases support the content of thecells, and most specifically “movement within theframe impels movement from frame to frame . . . ,objects in motion, or of the spectator’s eye directedalong the lines of some immobile object” (Eisen-stein 1949, p. 75). This is apparent in basic continu-ity editing in which a long shot, usually containinglots of image detail, lasts longer than a close-up.The logic is that it takes longer to see all the detailsin a long shot. The passage of time becomes morepsychological than strictly measured; experientialtime is a consideration. Events and images withinthe picture frame determine duration.

Video Example 4 takes the simple camera motionwith the flowers seen earlier and extends it intofour phrases. The camera motion zooms in withcells 1, 2, and 8, slowly panning and rotating fromand to a balanced composition. Consider these cellsas phrases, cadencing with a similar design struc-ture and a focus left-of-center.

Cells 3–7 contrast this and together make up acontrasting phrase. They are much shorter in dura-tion, with centered images and the camera zoomingin slightly. We see a basic rounded binary form: A(cell 1), A (cell 2), B (cells 3–7), A (cell 8). The dura-tion of each cell reflects the amount of cameramovement within the cell.

Metric montage is illustrated with cells 1 and 2(each 8 sec long) and cells 3 through 7 (each 2.4 seclong). As the camera moves over more space in cell8, it is consequently slightly longer in duration thancells 1 and 2, yet similar in camera motion and finalcomposition. This keeps the perception of speedmore or less constant in these cells. Rhythmic mon-tage is developed so the durational feel of cells 1, 2,and 8 is similar, and the total duration of the B sec-tion (cells 3–7), feels in balance.

Camera motion and zoom come to rest at the endof each cell. Cells ease-in and ease-out at endpointsin the larger phrases. Cells enter with cross-fades,the cross-fade itself a visual equivalent to musicalelision.

Tonal Montage

Eisenstein calls his third montage type tonal mon-tage. Continuing his borrowing of musical function,he describes tonal montage as “melodic-emotive.”We must be careful here to avoid misunderstandinghis use of the terms tonal and dominant, as they donot relate very well to the musical definitions of theterms. The tone of a work is its expressive tone.The dominant is the primary emotional feel of theshots. He here begins to bring the thematic contentof the film and the meaning of the images into thediscussion. He attempts to maintain an objectivemeasure by focusing on the graphic or plastic as-pects of the moving image: color and shape. Hecalls it “graphic tonality”—the design elements ofcolor, line, and shape, and what they can represent.

The time design begins to consider the subjectmatter and its signification. As Eisenstein explainshis ideas, he presages developments in semiotics andstructuralist thought. This goes beyond our discus-sion, as we are focusing on the plastic elements alonein composing visual music. The aim is to develop asimple design syntax built from a basic design idea.We are sidestepping issues of content that go beyondformal concerns. Bordwell and Thompson (2001)write, “Since a camera movement consumes timeon screen, it can create an arc of expectation andfulfillment.” This restates the premise: as cameramovement alone fulfills an expectation, the vieweris moved through time to a resolution of tension.

Thus, while in tonal montage, color and shape dobegin a move into connotation, we can reduce animage-in-motion to its design elements. From thisdistilled level of montage, we articulate phraseswith movement in color (saturate to neutral, brightto dark) and movement from and to a harmoniousdistribution of materials on the picture plane.

In the flower example, the graphic tonality is seenin the repetition of points of visual focus and in rep-

20 Computer Music Journal

Evans

etition or contrast of direction, rotation, and speedof camera zoom and pan. Centered images of the Bsection, with little motion, are shorter, and they setup expectations for a return to rotation and a longerphrase. This is another iteration of the basicrounded binary structure, this time at the level oftonal montage.

Many of Stan Brakhage’s works, usually with nosound score, are good examples of tonal montage.Mothlight (1963), for example, is an interestingwork that uses dead moths and other organic debrishe collected from light fixtures. He used no camerabut instead collaged twigs, blades of grass, dust, andmoth parts onto sticky tape, and then he printed itto celluloid for viewing through a film projector.Cells were then cut to create a fast-paced andhaunting montage.

Chris Marker’s La Jetée and Ken Burns’s A CivilWar are good studies of tonal montage with mini-mal sources. For example, battle scenes in A CivilWar may have shots of short duration, in contrast tolonger duration pans over battlefields in the after-math. Both films show how much can be done indeveloping large temporal structures with simplecomposition and effective editing.

Overtonal and Intellectual Montage

The last two types of montage are not really mon-tage in the strict sense. Eisenstein calls his fourthtype of montage overtonal montage. He begins todevelop the relationship between the previous threetypes and the cinematic work as a whole. He seesthe elements of montage metaphorically as har-monic partials of a musical timbre. Conflict devel-ops and resolves as these elements wrestle and thenagree. Eisenstein himself admits that overtonalmontage is “not an altogether exact term” (1942,p. 86), but he is concerned with the larger structuresof the film and its impact on the audience. Fromthis, he takes us to his fifth level, intellectual mon-tage, which will eventually build a “completelynew form of cinematography” (1949, p. 83). In laterwriting, he calls this intellectual cinema and is re-ally considering the movement of thought ratherthan moving light.

Returning to our simple, syntactical point ofview, we can think about montage working at thethree basic levels when building phrases of movingimages. First is metric montage, which deals withthe simple duration of shots. This is expanded torhythmic montage, in which the duration of cellssupports the plastic elements in motion within theshot. Lastly, the graphic elements such as color,line, and shape and the dominant expression ofthe shot build phrases in tonal montage. Each shotor phrase articulates cadences through a constructof tension/release. Phrases are combined and growto a larger form through repetition, contrast, andvariation.

Visual Counterpoint

An artwork can be defined as a complex relation-ship of elements; it is multi-dimensional. In devel-oping the elements particular to an image in motion,it is possible to build the unfolding of these elementsin a contrapuntal structure (Evans 1992). Variousgraphic elements can work independently in sup-port of a larger structure. Eisenstein calls this visualcounterpoint.

Montage is conflict. Cinematic conflicts unfold inthe picture plane. These conflicts work within thecell or shot and in “colliding” cells or montage.Eisenstein lists the conflicts (which deal mainlywith the compositional elements) as follows (theitalics are his):

conflict of graphic directions (lines either staticor dynamic); conflict of scales; conflict of vol-umes; conflict of masses (volumes filled withvarious intensities of light); and conflict ofdepth. With a bit more “intensification,” therecan also be conflicts of close shots and longshots; pieces of graphically varied directions(pieces resolved in volume with pieces resolvedin area); pieces of darkness and pieces of light-ness; and conflicts between an object and itsdimension (and conflicts between an event andits duration).

[In this final conflict, he refers to optical distortion,stop-motion, or slow-motion.]

21

What we have here is a comprehensive list of thedesign materials. With an understanding of two-dimensional design and tension/release, we can de-velop all of these materials independently within ashot and integrate them through many shots into a“unified system” of montage, the basis of a “tenta-tive film syntax” (Eisenstein 1949, p. 55)

Again, consider the first two video examples.They repeat the horizontal direction of focus fromleft to right. They contrast in vertical direction (onewith the camera going up, the other with the cam-era going down). They repeat the relative scale ofzoom, and they repeat the duration. They contrastin rotation angle (clockwise vs. counter-clockwise).Different elements with different temporal struc-tures—some in agreement and some opposed—cre-ate a counterpoint.

Vertical Montage (And the Sound Image)

Eisenstein expanded his counterpoint to combinethe “visual image with the sound image, in theprocess of creating a single, unifying sound-pictureimage” (1942, p. 73). He considered this a newkind of montage, naming it vertical montage.Again, he uses the term loosely, referring to the in-teractions of various elements as they are synchro-nized in time.

He saw vertical montage as an orchestral scorewith several staves of activity unfolding on the hori-zontal yet with all elements interrelating on thevertical, all moving forward with “intricate har-monic musical movement” (1942, p. 74). The mon-tage structure of the visuals is just one more line onthe musical score. The varied techniques of musiccomposition now can be applied to build a soundscore that fits the overall contrapuntal texture ofthe work.

For Eisenstein, it was important that this be atrue, polyphonic texture. He was not interested indirect synchronization of sound and image, butrather the “inner synchronization between the tan-gible picture and the differently perceived sounds . . .the ‘hidden’ inner synchronization where the plas-tic and tonal elements will find complete fusion . . .[where] we find a natural language common to both—

movement” (1942, p. 81). This inner connection isamong the images, the sounds, and the meaning ofthe work. For the audience it is the gestalt of thework as a whole, with all elements contributing tothe overall meaning. In moving beyond the visualmusic foundations, much can be learned from thetechniques of opera, musical theater, film scoring,and sound design.

When we use digital tools to create time-basedwork, layers of material are usually visualized on atimeline. Events are placed graphically like notes onstaves in a musical score. Events unfold horizon-tally, but vertical relationships are also important—multiple lines moving independently yet inharmony. Eisenstein’s image of the orchestral scoreis realized in the user interface of current graphicssoftware designed for time-based work. (Good ex-amples include Adobe After Effects, Digidesign ProTools, Macromedia Flash, and even Apple’s iMovie.)Horizontal layers represent video, audio, still im-ages, camera motion, interpolation paths, and more.These lines can interact and mix both horizontallyand vertically.

Toward a Visual Music Theory

Eisenstein believed that every artwork is conflictaccording to its social mission, according to its na-ture, and/or according to its methodology. Themethodology of cinema is montage. At the graphi-cal level, montage can move us in time through thevisual “rightness” one learns in two-dimensionaldesign. With experience and sensitivity to designfoundations, the artist, filmmaker, or visual musiccomposer can explore the higher levels of artisticconflict, its nature and social mission. Even withstudies in composition and movement using staticimages, a design sense can be exercised. Sensitivityto musical time can be developed and practicedwith visual materials. A sound score can be addedand musical sensibilities applied to a unified sound-image expression.

Literacy these days means speaking the languageof new media, which is in many ways the languageof cinema. As it is a temporal medium, we can findat the foundations of cinema the concept of music.

22 Computer Music Journal

Evans

The tenets of musical expression give us a methodfor moving the viewer/listener through time. Withvisual composition, we create consonance and dis-sonance. The ability to resolve dissonance to conso-nance provides a syntactical basis for temporaldesign and the composition of visual music. Ourcompositions can be a single shot moving throughphrases. We build phrases through repetition, con-trast, and variation of the many visual dimensionsincluding direction, speed, shape, size, and color.Eisenstein’s montage provides a conceptual frame-work for putting the phrases together into largerforms. New media may require a new literacy, yettraditional ideas still have much to offer.

Acknowledgments

This work was done with the support of a ResearchAdvisory Council Grant, University of Alabama.

References

Arnheim, R. 1966. “A Review of Proportion.” In G.Kepes, ed. Module, Proportion, Symmetry, Rhythm.New York: George Braziller, pp 218–230.

Alberti, L. B. 1955. Ten Books on Architecture. London:Tiranti.

Baum, L. F., director. 1939. The Wizard of Oz. 2004. (1939)Motion picture. Hollywood, California: Warner Stu-dios, ASIN: B00000JS62.

Bell, C. 1914. Art. London: Chatto and Windus.Begleiter, M. 2001. From Word to Image: Storyboarding

and the Filmmaking Process. Studio City, California:Michael Wiese Productions.

Bordwell, D., and K. Thompson. 2001. Film Art: An Intro-duction, 6th ed. New York: McGraw-Hill.

Bouleau, C. 1963. The Painter’s Secret Geometry. NewYork: Harcourt, Brace, and World.

Bowers, J. 1999. Introduction to Two-Dimensional De-sign: Understanding Form and Function. New York:Wiley.

Brakhage, S. 1963. Mothlight. Recorded on By Brakhage—Anthology—The Criterion Edition. DVD. New York:Criterion DVD CC1590D (2003).

Brougher, K., et al. 2005. Visual Music: Synaesthesia inArt and Music Since 1900. New York: Thames andHudson.

Burns, K. 1990. A Civil War—A Film by Ken Burns.Alexandria, Virginia: PBS Home Video.

Cheuvreul, M. E. 1854. The Principles of Harmony andContrast of Colors and their Applications to the Artist.West Chester, Pennsylvania: Schiffer Publishing.

Cruse, V. 2004. Sons & Lumières, Une histoire du sondans l’art du 20e siècle. Paris: Centre Pompidou.

De Grandis, L. 1984. Theory and Use of Color. New York:Harry N. Abrams.

Dow, A. W. 1899. Composition. Garden City, New York:Doubleday.

Eisenstein, S. 1942. Film Sense. New York: Harcourt.Eisenstein, S. 1949. Film Form. New York: Harcourt.Evans, B. 1990. “Temporal Coherence with Digital

Color.” Digital Image—Digital Cinema, SIGGRAPH’90, Art Show Catalog, published in Leonardo supple-mental issue. Oxford: Pergamon Press, pp. 43–49.

Evans, B. 1992. “Elemental Counterpoint with DigitalImagery.” Leonardo Music Journal 25(4):13–18.

Evans, B. 1993. “Number as Form and Content: A Com-posers Path of Inquiry.” In M. Emmer, ed. The VisualMind. Cambridge, Massachusetts: MIT Press.

Fischinger, O. 1947. Motion Painting No. 1. Recorded onOskar Fischinger: Pioneer of Abstract and AdvertisingAnimation. LaserDisc. Carson, California: LaserDiscCorporation (1986) Pioneer Special Interests, PSI-88-012.

Fischinger, O. 1932. Study No. 10. Recorded on OskarFischinger: Pioneer of Abstract and Advertising Ani-mation. LaserDisc. Carson, California: LaserDisc Cor-poration (1986) Pioneer Special Interests, PSI-88-012.

Griffiths, Paul. 1981. Modern Music, the Avante GardeSince 1945. New York: Braziller. p. 85.

Hambidge, J. 1919. The Elements of Dynamic Symmetry.New York: Dover.

Hambidge, J. 1932. Practical Applications of DynamicSymmetry. New Haven, Connecticut: Yale UniversityPress.

Howat, R. 1983. Debussy in Proportion: A Musical Anal-ysis. Cambridge: Cambridge University Press.

Huntley, H. E. 1970. The Divine Proportion. New York:Dover.

Itten, J. 1973. The Art of Color. New York: Van NostrandReinhold.

Kandinsky, W. 1914. Concerning the Spiritual in Art.New York: Dover.

Lambart, E., and N. McLaren. 1949. Begone Dull Care.Recorded on Norman McLaren: The Collector’s Edi-tion. DVD. Harrington Park, NJ: Milestone Film andVideo (2003) ASIN: B0000AOV4F.

Lendvai, E. 1971. Béla Bartók: An Analysis of His Music.London: Kahn and Averill.

23

Livingstone, M. 2002. Vision and Art: The Biology of See-ing. New York: Harry Abrams.

Marker, C. 1962. La Jetée. Recorded on Short 2-Dreams.DVD. Hollywood: Warner Studios (1997) ASIN:B000031VPS.

Mascelli, J. 1968. The Five C’s of Cinematography. Holly-wood, California: Cine/Grafic Publications.

McLaren, N. 1968. Pas De Deux. Recorded on NormanMcLaren: The Collector’s Edition. DVD. HarringtonPark, New Jersey: Milestone Film & Video (2003)ASIN: B0000AOV4F.

Moritz, W. 2004. Optical Poetry: The Life and Work ofOskar Fischinger. Eastleigh, UK: John Libbey Publishing.

Russet R., and C. Starr. 1976. Experimental Animation:an Illustrated Anthology. New York: Van NostrandReinhold.

Smith, A. R. 1978. “Color Gamut Transform Pairs.”Computer Graphics 12(3):12–18.

Stravinsky, I. 1956. The Poetics of Music in the Form ofSix Lessons. New York: Vintage Books.

Whitney, J. 1975. Arabesque. Recorded on The World ofJohn Whitney. LaserDisc. Carson, California: LaserDiscCorporation (1984) Pioneer Special Interests, PSI-88-003.

Whitney, J. 1980. Digital Harmony: On the Complementar-ity of Music and Visual Art. New York: McGraw-Hill.

24 Computer Music Journal