Computational Aesthetics in Graphics, Visualization, and Imaging EXPRESSIVE 2015Paul Rosin (Editor)

Video Granular Synthesis

Angus Graeme Forbes1† and Javier Villegas2

1Electronic Visualization Laboratory, University of Illinois at Chicago, USA2School of Information, University of Arizona, USA

AbstractThis paper introduces a technique that enables the creative reshaping of one or more video signals based ongranular synthesis techniques, normally applied only to audio signals. We demonstrate that a wide range of novelvideo processing effects can be generated through conceptualizing a video signal as being composed of a largenumber of video grains. These grains can be manipulated and maneuvered in a variety of ways, and a new videosignal can then be created through the resynthesis of these altered grains; effects include cloning, rotating, andresizing the video grains, as well as repositioning them in space and time. These effects have been used successfullyin a series of interactive multimedia performances, leading us to believe that our approach has significant artisticpotential.

Categories and Subject Descriptors (according to ACM CCS): I.3.6 [Computer Graphics]: Methodology andTechniques—Graphics data structures and data types; I.3.7 [Computer Graphics]: Three-Dimensional Graphicsand Realism—Animation.

1. Introduction to Video Granular Synthesis

This paper extends granular synthesis [Roa96, Roa04a], apopular technique used for electroacoustic composition, tothe visual domain. Creative possibilities emerge from con-ceiving of a video signal as being composed of a large num-ber of video grains within a cube (or, more accurately, anorthogonal parallelepiped) that bounds the spatial and tem-poral range of the video data. These grains— analogousto the way in which audio grains are defined in granularsynthesis— are small perceptual elements with distinct fea-tures that can be programmatically or interactively manipu-lated. Our method, video granular synthesis, provides a con-ceptual approach to video processing that enables a varietyof creative effects. Each of the effects involve a remapping ofdata, first from dicing the original video signal into a set ofwindowed video grains within the new geometrized space,where space and time are conflated, and then back onto aseries of image planes, presented as a new video signal.

The main effects are based on three types of operations:creating the grains; manipulating those grains; and reposi-tioning the grains. We define each grain as a cube of a small

† The contact author is Angus Forbes, email: aforbes@uic.edu,web: http://evl.uic.edu/creativecoding

number of pixels and then apply an envelope so that eachgrain can seamlessly overlap with the others; we then canmanipulate various individual grain characteristics. For in-stance, we can clone some grains, delete others, scale thegrains, and use image processing effects to alter their pixelvalues. We also can reposition the grains in time and space:we can rotate the grains in place; we can shuffle the grains sothat particular regions appear in different orders; or we canpin a grain in place so that it appears frozen in time.

Additional effects are made possible through reinterpret-ing a video as a cube of grains. The perspective of the cameracan be resituated so that the viewer traverses the video ge-ometry in an unexpected temporal trajectory, for example,by swapping the temporal axis with one of the spatial axes.Another effect involves mixing together video grains frommultiple signals, blending them together, or choosing onevideo’s grain over the other’s based on particular parametersof the grains. Multiple video granular synthesis effects canbe applied simultaneously. The techniques can, for the mostpart, be applied in real-time, although some, for obvious rea-sons, have a greater range if the video signal is already cap-tured (as future grains cannot be repositioned if they havenot been captured yet). This conceptualization of the videosignal as a cube full of grains, coupled with our methodsfor manipulating them, provides us with a palette of creative

c© The Eurographics Association 2015.

A. G. Forbes and J. Villegas / Video Granular Synthesis

Figure 1: Four frames from a video portraiture project. The video portrait captures the subject as she is singing, and the audiofeatures (pitch and amplitude) change aspects of the temporal shuffling of the grain, including the grain size and the graindelay.

possibilities. In addition to enabling novel explorations oftime and space, our system makes it easy to define and ma-nipulate grains in real-time, to synchronize audio signals tofeatures of video grains, and to mix multiple video signalstogether. We have designed software that makes it easy touse video granular synthesis to manipulate videos, and wehave linked some operations to controllers so that these ef-fects can be explored interactively.

Granular synthesis techniques are commonly used bycomposers to create new sonic textures. For instance, gran-ular synthesis can be used to manipulate the duration of ex-isting sounds without changing their pitch, or changing thepitch without affecting their length. The fundamental ele-ments of a granular synthesizer are small acoustic objects,called grains, that are of such short duration that they arenearly imperceptible as individual sonic events. A composercan manipulate the grains in order to create particular atmo-spheres, including: the temporal location of the grain, theoverlapping factor of adjacent grains, and individual char-acteristics (such as amplitude or frequency) of each of thegrains. Interesting transformations can also be obtained withgranular techniques if the sound grains are captured fromreal-world signals, rather than computationally generated.Grains extracted from the source signal can be rearranged,eliminated, repeated, or otherwise manipulated in order tocreate compelling effects. This approach is known as mi-cromontage or granulation [Roa96, Z0̈2]. Granular synthe-sis, based on granulation, and applied to real-world signals,is an extremely successful technique; many of the most pop-ular audio manipulation software suites now include toolsfor granular synthesis techniques [Opi13].

Similar to an audio grain, we define a video grain to bea portion of an input video signal windowed by an enve-lope so that the video signal is stronger in the center of thegrain and attenuated toward the edges of the grains. In audiogranular synthesis, different envelopes are used to overlap

regions of the input signal, and can be chosen by a composerfor particular effects [Roa96]. In adapting granular synthe-sis to the video domain, we apply a Hann window to creategrains with a uniform overlap characteristic [HRS02]. Al-though our methods for processing video are based on audiogranular synthesis techniques, how humans perceive audiosignals is quite different from how video signals are per-ceived. Extending the concept of granulation to video de-mands a new exploration of the creative possibilities of suchtechniques.

2. Related Work

While creative approaches using granular synthesis strate-gies are popular in music composition, they are not as com-mon in the video domain. However, some artworks do usevideo to accompany compositions created with granular syn-thesis techniques. For example, Kurt Hentschlaeger and UlfLangheinrich have presented a series of installations show-ing videos of expressions and forms cut-up and displayedout of order along with a soundtrack based on granular syn-thesis [HL96]. Curtis Roads, the primary proponent of gran-ular synthesis, has collaborated with video artists, includ-ing Brian O’Reilly, Woon Seung Yeo, and Garry Kling, ona series of compositions collected in his Point Line Cloudproject [Roa04b]. More recently, the Belgian artist Axel Ri-jpers has explored similar techniques to create narrative ten-sion [Rij13]. The audiovisual composer John Keston has de-veloped a touchscreen instrument that links short segmentsof loops of cranes, train engines, and metal cutters with agranulated composition based on the sounds of those ma-chines [Kes13]. By and large, these examples rely on abruptjump cuts, limiting their expressivity. A recent applicationof video manipulation, Eulerian Video Magnification, en-ables subtle motions and changes in color to be accentu-ated [WRS∗12]. This technique however operates on sin-gle pixels, and while it does provide interesting effects (po-

c© The Eurographics Association 2015.

A. G. Forbes and J. Villegas / Video Granular Synthesis

tentially useful in medical applications), it does not directlyenable the manipulation of video signals for creative videocompositions.

Other previous work, not based on the generation of videograins, presents spatiotemporal manipulations of video sig-nals which are related to certain effects we introduce here.For instance, the slit-scan technique is also based on a con-flation of the time and space axes. A manually generated ver-sion of slit-scan was first popularized by the special-effectsartist Douglas Trumbull [Tru68]. Contemporary photogra-pher Adam Magyar stitches together photos of urban crowdsand subway passengers taken with a slit-scan camera intolonger video sequences [Mag09]. Golan Levin has collated arepository of computationally-generated artworks based onthe slit-scan technique [Lev10]. For example, Camille Ut-terback’s Liquid Time allows a user to position his or herbody to interactively fragment the temporal aspects of a pre-recorded video clip [Utt02]. Our approach can be used toproduce visual outputs similar to the ones obtained with slit-scan, but also enables a range of additional effects. Fels etal. [FLM00] also reinterpret the video signal as a space-time cube and present alternative renderings that shufflethese two domains. Our technique differs in that it ma-nipulates a larger perceptual entity, the video grain, ratherthan individual pixels. Alvaro Cassinelli describes a pixel-based interactive piece that allows spatiotemporal “contor-tions” using a tangible surface [CI05]. A combination ofspace-time analysis and non-photorealistic synthesis is pre-sented by Klein et al. [KSFC02], which uses a set of dif-ferent “rendering solids” to recreate a transformed versionof the input. This approach is somewhat similar to the time-varying envelope that we use to create spatiotemporal videograins, but the rendering solids do not extend granulationtechniques. Techniques dependent on the conflation of spaceand time have been developed for a range of visualizationtechniques [BCD∗12, BDA∗14] and video analyses, such ascomparing motions [SI07], identifying actions [BGS∗05],and correcting corrupted frames [WSI07]. The work de-scribed in this paper is focused on generating creativeoutputs, and relates to previous work by the authors onvideo analysis [FJP14,FV14,VF14a,VEF15] and media artsprojects involving video processing [FO12,FHL13,VF14b].

3. System Overview

Our video granular synthesis software can transmute one ormore input video signals into a creatively manipulated out-put video signal. At Step 1, an input video is interpreted asa video cube of three dimensional data, made up of videoframes (the x and y coordinates) extended through time (thet coordinate). That is, based on parameters that define the in-put window size and various factors that define the amountof overlap, the original video sequence is transformed into aset of video grains. At Step 2, the video grains can be man-ually or programmatically selected (discussed below), and a

wide range of grain-manipulation effects can be generated,such as cloning, skipping, shuffling, or repositioning grains.Additionally, the pixel values of the grains can be changed,for example by applying image processing effects on theselected grains. At Step 3, the scale, orientation, and spa-tiotemporal position of each grain is calculated and storedin a grain map. The grain map links the pixel data storedin a grain to its position (x, y, and t) its size (r) and its 3Dorientation (θ and φ) The grain map can be interactively up-dated in real-time on a per-frame basis. This grain map alsois used to determine how the position of each grain createdfrom the input video cube is mapped to an output signal. AtStep 4, a scheduler component interrogates the output of thegrain map to determine what portion of each grain will beused to generate the current output frame. Our implemen-tation was created in OpenGL, and uses a GLSL geometryshader to create the actual individual slices of each grain thatis currently visible, based on the the current location of thecamera and the individual grains, thus speeding up the ren-dering process by ignoring data not relevant to the currentframe. A GLSL fragment shader is responsible for drawingeach video grain based on how the current frame intersectsit and the current windowing parameters. Fig. 3 provides ageneral overview illustrating the functionality of the videogranular synthesis system and the grain map data structure.

Our system also includes components that make it easierto compose videos and to manipulate video signals interac-tively. We have developed software that allows a user to ma-neuver through the cube of video grains and manually selectonly particular grains. Once these grains are chosen, we canprogrammatically attach particular effects to them, such as achange in their scale, rotation, or position. Additionally, wehave set up keyboard mappings so that effects can be appliedto real-time signals, including mixing together video grainsfrom multiple videos. We have also used other types of con-trollers to select effects and dynamically update parameters.Our system also makes it easy to map audio inputs ontovideo grains, allowing us to select particular effects basedon features of the audio signal, or to change parameters ofthe currently selected effects. For example, for a particularset of grains we can map the detected pitch of the audio sig-nal to the rotation of those grains. We can also synchronizethe mapping of parameters for manipulating the audio signal(using audio granular synthesis) to the video signal (usingvideo granular synthesis).

4. Creative Applications of Video Granular Synthesis

The various techniques described by our video granular syn-thesis method are currently being incorporated into a se-ries of creative projects. In this section, we briefly describeexamples these projects as examples of some of the cre-ative artistic and interactive possibilities that can be gener-ated using video granular synthesis, which include a fixedaudio-visual piece, a choreographed multimedia piece with

c© The Eurographics Association 2015.

A. G. Forbes and J. Villegas / Video Granular Synthesis

Figure 2: Three examples of real-time effects created using video granular synthesis, including, from left to right, (a) signalmixing, (b) spatial repositioning of grains, and (c) grain rotation.

dancers, and an interactive video performance with livevideo manipulation.

Audio-based Video Manipulation For a recent multimediaevent, we used multiple cameras to capture the performanceof an electroacoustic composition; by applying video gran-ular synthesis techniques to these videos we augmented thelive-performance, controlling the video grain parameters viaa laptop. The audio signal was used to control how the grainsfrom each signal were mapped to the output image. That is,we interactively chose particular positions in the grain map,and blended the input video signals programmatically, de-pending on the pitch and amplitude of the audio signal. Thisoutput image was projected on a large screen to one side ofthe performers. One of the effects featured in this perfor-mance was grain cloning. In our system, specific grains canbe cloned, that is, added multiple times to the grain map.In capturing the performance, we created video grains fromthe input videos using a 50% overlapping factor, effectivelychanging the size of the image while preserving the spa-tiotemporal frequencies. That is, granular effects applied tothe audio signal can also be simultaneously applied to videosignals. Fig. 2 shows other effects that were used, for in-stance, Fig. 2c shows example of grain rotation applied to avideo of the performance.

Video Portraiture Using Video Grains In another (ongoing)project, we used grain cloning as a technique for exploringconcepts in video portraiture. Fig. 4 shows a frame from oneof these experiments; the image has a curious resemblanceto the “op art” artwork created by Julio Le Parc [GP05]. Wealso explored the use of temporal shuffling to show fluctu-ations in a video of subjects expressing themselves in dif-ferent ways, through dance or through singing. For instance,in Fig. 5, we utilized video granular synthesis to overlay thesame person from different parts of a video in order to havemultiple copies of the person appear simultaneously in thesame frame. That is, we sample backwards in time and bringthose previous grains to the present. By using different num-bers of grain delays, we can control the number of differentforms that are presented in the current frame. Fig. 1 shows avideo portrait of a female vocalist performing traditional Ira-

nian songs. For this project, we used more fine-grained ma-nipulations of delays and spatial repositioning to create a se-quence of subtler expressivity in order to capture the singer’semotions. We also experimented with synchronizing the au-dio to the video effects by linking the pitch and amplitude ofthe audio signal to the intensity of the grain manipulation.

Spatiotemporal Reinterpretation By considering the inputvideo as a cubic array of grains, new interpretations of thedata can be created simply by relocating the point of viewof the array. That is, we can imagine the video being playedfrom a different direction. Fig. 5c shows a frame obtainedwhile looking at the video array from one corner (where aspace axis and the time axis, the x and z axes, are partiallyinterchanged). Since the grain map can be updated in real-time, more complex traversals could be created as well, i.e.,that do not simply move in a straight line toward the camera.For instance, Fig. 2b shows a new shape forming throughshrinking the grains and creating non-uniform spacing be-tween them. Our custom software that can be used to in-teractively update parameters for navigating the video fromalternative vantage points.

Image Dependent Manipulations The position of grains canalso be modified by other, non-procedural strategies. For ex-ample, higher level information from the video stream can beused to determine the behavior of the grains. In one of ourexplorations we changed the spatial position of the grains ac-cording to the amount of temporal variance detected betweenimage frames. The squared root of the temporal variance fora grain spatially placed at x0,y0 is:

σ̄G (x0,y0) =1

G2s

x0+Gs

∑x=x0

y0+Gs

∑y=y0

√VAR [px,y (t)] (1)

Where Gs is the grain size (assumed to be equal in all dimen-sions), px,y(t) is the gray-scaled pixel value at position x0,y0and time frame t, and the variance is calculated over time.Grains that have a variance greater than a predefined thresh-old move in a random direction by an amount proportionalto the square of the averaged temporal variance of the grain.For instance, a hand waving in front of the camera was usedto add effects to only those grains that are in the same spatial

c© The Eurographics Association 2015.

A. G. Forbes and J. Villegas / Video Granular Synthesis

frame 1

frame n

grain ID x,y,t,r,ø,φ

g1 7,0,0,5,0,90g2 7,0,6,2,0,0g3 0,0,0,5,45,0

grain ID x,y,t,r,ø,φ

g1 7,0,0,5,0,90g2 7,0,6,2,0,0g3 0,0,0,5,45,0

frame 1

frame n

1.

2.

3.

4.

Figure 3: A system overview of our Video Granular Synthe-sis software. At Step 1, a sequence of video frames is turnedinto a set of grains using a particular windowing filter andradius. At Step 2, a user can programmatically or manuallyselect grains and alter them by changing their spatiotempo-ral position, their size, or their orientation, or by modifyingthe pixels within the grains. At Step 3, the grains are storedin a “grain map,” an internal data structure that keeps trackof the parameters and content of the grains. At Step 4, thecurrent image frame uses the grain map to determine whichportion of the currently active grains (i.e., based on theirposition, size, and orientation) will be remapped onto thedisplay.

position as the detected motion of the waving. Dynamicallyupdating grain characteristics via high-level features intro-duces many possibilities for interactive systems. A range ofanalysis techniques could be used to drive this interactivity,such as optical flow [HS81]. Other detected image character-istics that can be used to modify grain behaviors include lu-minance, chrominance, frequency, spatial position, and facerecognition, among others. This approach could be an excit-ing way to compose or interact with audio-visual pieces viadance or other gesture based performance.

Figure 4: This image shows a frame (left) and a detail fromthat frame (right) where the video grains are cloned usinga Hann window with a 50% overlap factor. The overlappingfactor and windowing functions can be updated in real-time.

Figure 5: The left and center frames show the temporal shuf-fling of video grains. The left frame uses two different set-tings for the grain delay, enabling portions of two figures(the same person at different times) to be shown simultane-ously; the center frame uses three grain delays. The rightframe shows an effect similar to those produced using slit-scan cameras, where the video clip is being traversed at a45◦angle around the y-axis. Other paths through the videospace could also be defined.

5. Discussion and Future Work

We have worked with different artists to produce interac-tive multimedia projects that make use of the techniques de-scribed in this paper. For example, a performance with ac-companying video, titled v→t→d, has been well-received ata number of venues, including the University of Arizona’sConfluencenter for Creative Inquiry and at the 2014 Interna-tional Computer Music Conference in Athens, Greece, withover 5,000 attendees. For this project, live sound inputs af-fected the position and size of the grains, captured from alive video feed with a 30-second buffer. Additionally, a per-former used our software to change rotation and cloning pa-rameters it in response to the varying intensity of the musicalperformance. A second camera was affixed to the bell of asaxophone, and the laptop performer also was able to blendin grains from both signals simultaneously [JTVF14].

Another multimedia performance that used video granu-lar synthesis was performed as part of an experimental artsfestival at Exploded View Microcinema in Tucson, Arizonain 2014. For this piece, titled Coming or Going, video grainsgenerated from a computationally-generated animation anda sequence of video footage were blended together by ma-nipulating the grain map. The size and movement of thegrains were choreographed to follow a percussion track. Forthis presentation there was no live interaction, but the artists

c© The Eurographics Association 2015.

A. G. Forbes and J. Villegas / Video Granular Synthesis

are planning to create a new version where the volume andpitch of the percussion instruments create a dynamic visualaccompaniment without the need for prior choreography.

The projects described above present initial forays intousing granular synthesis techniques applied to video. We be-lieve that this method will enable new artistic possibilitiesbased on introducing “pieces of time” into a video sequence.Much of the previous work involving temporal manipula-tions of video has an abrupt, disjointed sensibility. Our cur-rent explorations successfully investigate the technical fea-sibility of mixing grains of time and space in a more subtlemanner, but a more coherent artistic focus could generateeven more compelling aesthetic results.

A principal artistic effect that uses video granular synthe-sis involves thinking of a single frame as being composed ofmultiple moments simultaneously. Using our method, artistsare able to frame particular regions of the image as beingwindows into other times (or only previous times if the videosignal is a live recording). In addition to experimenting withhow different conflations of time could introduce novel nar-rative effects, artists could also manipulate the spatial place-ment of where these times occur for additional narrative im-pact. An additional artistic effect that we investigate utilizesour method in order to think of a video as if it were a blockof information. Current explorations that involve travers-ing a video from unusual vantage points produce somewhat“glitchy” looking videos (as in Fig. 5). We believe that amore nuanced use of this technique could provide interest-ing perspectives that could augment more traditional spatialrepresentations.

We also are continuing to explore the idea of mixing sig-nals, that is, of interweaving multiple videos at multiplepoints in time. While further research needs to be done to de-termine how multiple viewpoints could be most effectivelyintegrated into a single video sequence, this technique seemsrich with possibility. For example, the artistry of many filmsinvolves providing insight into how different people see thesame situation, providing empathy with a range of charac-ters. Video granular synthesis, which enables a superimposi-tion of scenes, could promote new explorations of perspec-tive in narrative videos.

Our work shows that creative manipulations using agranular approach are also possible with video signals; ourartistic explorations lead us to believe that this approach toprocessing video signals has significant narrative potential.Different video atmospheres can be created by introducingthese effects and applying them to spatial or temporaldimensions. By conflating these dimensions, diverseinterpretations of the same block of visual informationcan be generated via unorthodox trajectories through thevideo data, revealing interesting relationships between theperception of time and space. In sum, our video granularsynthesis method presents the following contributions:

• It provides a way to transform a video stream into a set ofvideo grains, each of which has a range of attributes thatcan be modified either individually or collectively;

• It makes it possible to conflate the spatial and temporalelements of a video;

• It makes it easy to mix together multiple video signals innovel ways;

• It enables novel video processing techniques that can beused to manipulate video streams in real-time;

• It encourages the mapping of these video processing tech-niques to audio signals or to a performer’s live control.

Thus far, our creative explorations have focused on appli-cations of video granular synthesis to relatively simple videocompositions. Many possibilities exist for exploring effec-tive ways to use the techniques we introduced in this paperto augment more complex compositions. We are especiallyintrigued by the idea of mixing multiple viewpoints to createnovel approaches to augment storytelling. Future work willfocus on a more sophisticated integration of video granularsynthesis techniques and narrative elements.

We also believe that there is a natural synergy betweengranular synthesis in the audio and video domains. Althoughthere are strong differences between audio and video repre-sentations and in the way in which human perception worksin the two domains, some of the strategies can be extended ina very straightforward way. This is the case when we cloneor skip grains. In the audio domain, this strategy is com-monly used to modify the length of an audio signal with-out changing its pitch [Roa96]. But this trade off is not asmeaningful when processing visual information. While, forinstance, relatively small changes on the reproduction rateof a voice signal can immediately sound artificial, we areused to seeing faces at different scales. However, althoughthis pitch-preserving time-modification is not as important inimages (an exception might involve periodic textures), evena straightforward application incorporating cloning and/orskipping grains produces visually appealing results.

Another interesting area for future exploration is the inter-action between both streams. Some of the effects introducedin this paper, including cloning and randomization, can bestraightforwardly applied to audio and video signals. Highlycoupled audio-visual pieces can be generated if both streamsundergo similar operations simultaneously. Moreover, an au-dio granular synthesizer can be used to generate the sound-track of a granular video, and audio events can be triggeredby the video grains. Characteristics such as the density orfrequency content of the audio grains can be mapped to otherfeatures in the video grains. Finally, spatialized sound and3D audio compositions that place audio grains at differentpositions within a virtual space could be complemented byvideo grains moving with similar dynamics throughout animmersive environment.

c© The Eurographics Association 2015.

A. G. Forbes and J. Villegas / Video Granular Synthesis

Acknowledgements

We thank Curtis Roads of the Media Arts and Technologyprogram at UC Santa Barbara for his inspiration. We alsothank each of the artists involved in the projects described inthis paper for their valuable feedback, including especiallyChristopher Jette, Kelland Thomas, and Ronak Etemadpour.

References

[BCD∗12] BORGO R., CHEN M., DAUBNEY B., GRUNDY E.,HEIDEMANN G., HÖFERLIN B., HÖFERLIN M., LEITTE H.,WEISKOPF D., XIE X.: State of the art report on video-basedgraphics and video visualization. Computer Graphics Forum 31,8 (Dec. 2012), 2450–2477. 3

[BDA∗14] BACH B., DRAGICEVIC P., ARCHAMBAULT D.,HURTER C., CARPENDALE S.: A review of temporal data visu-alizations based on space-time cube operations. In EurographicsConference on Visualization (2014), Borgo R., Maciejewski R.,Viola I., (Eds.), The Eurographics Association. 3

[BGS∗05] BLANK M., GORELICK L., SHECHTMAN E., IRANIM., BASRI R.: Actions as space-time shapes. In Computer Vi-sion, 2005. ICCV 2005. Tenth IEEE International Conference on(2005), vol. 2, IEEE, pp. 1395–1402. 3

[CI05] CASSINELLI A., ISHIKAWA M.: Khronos projector. InEmerging Technologies, SIGGRAPH 2005 (Los Angeles, CA,2005), ACM. 3

[FHL13] FORBES A. G., HÖLLERER T., LEGRADY G.: Gen-erative fluid profiles for interactive media arts projects. In Pro-ceedings of the International Symposium on Computational Aes-thetics in Graphics, Visualization, and Imaging (CAe) (Anaheim,California, July 2013), pp. 123–129. 3

[FJP14] FORBES A. G., JETTE C., PREDOEHL A.: Analyzing in-trinsic motion textures created from naturalistic video captures.In Proceedings of the International Conference on InformationVisualization Theory and Applications (IVAPP) (Lisbon, Portu-gal, January 2014). 3

[FLM00] FELS S., LEE E., MASE K.: Techniques for interactivevideo cubism. In Proceedings of the eighth ACM internationalconference on Multimedia (MM) (2000), ACM, pp. 368–370. 3

[FO12] FORBES A. G., ODAI K.: Iterative synaesthetic com-posing with multimedia signals. In Proceedings of the Interna-tional Computer Music Conference (ICMC) (Ljubjiana, Slovenia,September 2012), pp. 573–578. 3

[FV14] FORBES A. G., VILLEGAS J.: Creative applications ofmicrovideos. In Proceedings of the the Sixth International Con-ferences on Advances in Multimedia (MMEDIA) (Nice, France,February 2014), pp. 108–111. 3

[GP05] GUIGON E., PIERRE A.: L’oeil moteur: Art optique etcinétique, 1950-1976. Musées de Strasbourg, Strasbourg, Ger-many, 2005. 4

[HL96] HENTSCHLAEGER K., LANGHEINRICH U.: Granular-synthesis, modell5. http://www.granularsynthesis.info/ns, 1996. Retrieved March 25, 2015. 2

[HRS02] HEINZEL G., RUDIGER A., SCHILLING R.: Spectrumand spectral density estimation by the Discrete Fourier transform(DFT), including a comprehensive list of window functions andsome new at-top windows. Tech. Rep. 395068.0, Max-Planck-Institut fur Gravitationsphysik, February 2002. 2

[HS81] HORN B. K., SCHUNCK B. G.: Determining optical flow.In 1981 Technical Symposium East (1981), International Societyfor Optics and Photonics, pp. 319–331. 5

[JTVF14] JETTE C., THOMAS K., VILLEGAS J., FORBES A. G.:Translation as technique: Collaboratively creating an electro-acoustic composition for saxophone and live video projection.In Joint Proceedings of the of the 40th International ComputerMusic Conference (ICMC) and the 11th Sound and Music Com-puting Conference (SMC) (Athens, Greece, September 2014),pp. 463–468. 5

[Kes13] KESTON J.: Machine Machine. http://johnkeston.com, 2013. Retrieved March 25, 2015. 2

[KSFC02] KLEIN A. W., SLOAN P.-P. J., FINKELSTEIN A.,COHEN M. F.: Stylized video cubes. In Proceedings of the 2002ACM SIGGRAPH/Eurographics Symposium on Computer Ani-mation (SCA) (New York, NY, USA, 2002), ACM, pp. 15–22.3

[Lev10] LEVIN G.: An informal catalogue of slit-scan videoartworks and research. http://www.flong.com/texts/lists/slit_scan, 2010. Retrieved March 25, 2015. 3

[Mag09] MAGYAR A.: Urban Flow: New York. http://www.magyaradam.com, 2009. Retrieved March 25, 2015. 3

[Opi13] OPIE T.: Granular synthesis: A granular synthesisresource website. http://granularsynthesis.com/software.php, 2013. Retrieved April 11, 2015. 2

[Rij13] RIJPERS A.: Revive. https://vimeo.com/82177087, 2013. Retrieved March 25, 2015. 2

[Roa96] ROADS C.: The computer music tutorial. MIT press,1996. 1, 2, 6

[Roa04a] ROADS C.: Microsound. The MIT Press, Sept. 2004. 1

[Roa04b] ROADS C.: Point Line Cloud. Asphodel, 2004. 2

[SI07] SHECHTMAN E., IRANI M.: Space-time behavior-basedcorrelation –OR– How to tell if two underlying motion fields aresimilar without computing them? Pattern Analysis and MachineIntelligence, IEEE Transactions on 29, 11 (2007), 2045–2056. 3

[Tru68] TRUMBULL D.: Creating special effects for 2001. Amer-ican Cinematographer 49, 6 (June 1968), 416–420,451–453. 3

[Utt02] UTTERBACK C.: Liquid Time Series. http://camilleutterback.com/projects, 2002. RetrievedMarch 25, 2015. 3

[VEF15] VILLEGAS J., ETEMADPOUR R., FORBES A. G.: Eval-uating the perception of different matching strategies for time-coherent animations. In Human Vision and Electronic ImagingXX (HVEI), vol. 9394 of Proceedings of SPIE-IS&T ElectronicImaging. San Francisco, California, February 2015, pp. 939434–1–13. 3

[VF14a] VILLEGAS J., FORBES A. G.: Analysis/synthesis ap-proaches for creatively processing video signals. In Proceedingsof the ACM International Conference on Multimedia (MM) (Or-lando, Florida, November 2014), pp. 37–46. 3

[VF14b] VILLEGAS J., FORBES A. G.: Interactive non-photorealistic video synthesis for artistic user experience on mo-bile devices. In Proceedings of the International Workshop onVideo Processing and Quality Metrics for Consumer Electronics(VPQM) (Chandler, Arizona, January 2014). 3

[WRS∗12] WU H.-Y., RUBINSTEIN M., SHIH E., GUTTAG J.,DURAND F., FREEMAN W.: Eulerian video magnification forrevealing subtle changes in the world. ACM Transactions onGraphics (TOG) 31, 4 (2012), 65. 2

[WSI07] WEXLER Y., SHECHTMAN E., IRANI M.: Space-timecompletion of video. Pattern Analysis and Machine Intelligence,IEEE Transactions on 29, 3 (2007), 463–476. 3

[Z0̈2] ZÖLZER U.: DAFX: Digital audio effects. John Wiley &Sons, Ltd, 2002. 2

c© The Eurographics Association 2015.