Transitioning Tradition? Examining the Digitization of Cultural Practices

DANIELA ROSNER, Stanford University, CA

MARCO ROCCETTI, University of Bologna, Italy GUSTAVO MARFIA, University of Bologna, Italy

Over the last few decades, computing researchers have developed a wealth of new systems to maintain particular cultural resources. Computing platforms allow people to access high-resolution images of ancient cave paintings and Google-artwork; 3D scanning systems digitize Michelangelo’s sculptures and Pompeii’s remains [Levoy et al. 2000]; and haptic devices allow users to touch and feel ancient sculptural forms [Bergamasco et al. 2001]. Most recently, researchers have employed collaborative and crowd-sourced software in the restoration of specific artifacts and environments (e.g., paintings and archeological sites) [Mazzeo et al. 2007]. These approaches tend to fall within one of three categories: (a) digitally reconstructing objects and landscapes from the past, (b) broadening access to cultural resources through remote distribution platforms, and, (c) digitally representing and archiving cultural artifacts and media. In each case, computing researchers position techniques of digitization as a means of cultural preservation.

Yet, in this line up of technological developments, researchers also miss a key player. Even as our cultural artifacts and media carry on digitally for future generations, our reasons for reading books, our contexts for understanding artworks, and our ways of sharing and celebrating everyday practices remain relatively far afield. And for good reason: computing researchers find cultural practices hard to identify, codify and, in turn, digitally record. Cultural practices, in other words, do not figure as squarely into the language of computation. They emerge as forms of memory practice, wherein we remember and forget the past as a lived and constantly mutating collective practice [Esposito 2008], [Mayer-Schonberger 2011]. In this regard, we do not conserve cultural practices in the

This work is supported by the ALTER-NET Italian Project. Author’s addresses: Daniela Rosner is with the Stanford University and can be reached at danielar@stanford.edu. Marco Roccetti and Gustavo Marfia are with the University of Bologna and can be reached at {marco.roccetti, gustavo.marfia}@unibo.it respectively. Permission to make digital or hardcopies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies show this notice on the first page or initial screen of a display along with the full citation. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credits permitted. To copy otherwise, to republish, to post on servers, to redistribute to lists, or to use any component of this work in other works requires prior specific permission and/or a fee. Permissions may be requested from Publications Dept., ACM, Inc., 2 Penn Plaza, Suite 701, New York, NY 10121-0701 USA, fax +1 (212) 869-0481, or permissions@acm.org. © 2010 ACM 1539-9087/2010/03-ART39 $15.00 DOI:http://dx.doi.org/10.1145/0000000.0000000

39

sense that we freeze them in time by making them explicit. On the contrary, we preserve cultural practices by enacting tradition and experience. Digitization can enable and extend this work of preservation. In making this argument, we look at practices, after [Schatzki 2012], as arrays of human activity: temporally situated events that involve rehearsed, materially mediated actions and embodied social relations. The world heritage group United Nations Educational, Scientific and Cultural Organization (UNESCO) has identified practices as “very fragile by their very nature.” While they bring to life some of our most celebrated cultural forms, practices are easily overlooked [Bouchenaki 2003].

Figure 1. Systems designed to sustain cultural resources: (a) digitally reconstructing objects and landscapes from the past, (b) broadening access to cultural resources through remote distribution platforms, and, (c) digitally representing and archiving cultural artifacts and media.

This article considers the role that computing techniques play in sustaining cultural practices. By tracing cultural practices in relation to developments in computing research, we argue that our widely accepted techniques of preservation through digitization need revision. Inspired by a large body of literature from the digital humanities, we suggest understanding and recognizing the dynamic nature of cultural forms as an important strategy for preservation. This view of preservation does not fit within the dominant conceptual framework defined by computing researchers, a rubric grounded in the assumption that cultural histories are stabilized by the material forms on which they are encoded, stored, and replicated.

To take a recent example, consider the Artist’s View iPad application [Artist’s View 2012]. Netribe of the Palazzo Strozzi in Florence designed the application to familiarize museum visitors with the paintings exhibited in the 2012 “Americans in Florence – Sargent and the American Impressionists” show. The designers took advantage of the museum’s close proximity to many of the Florentine and Tuscan landscapes portrayed in the exhibition’s paintings by guiding visitors through the locations in which exhibiting artists painted or found their inspiration. When visitors reach any of the locations, the application provides them with the possibility of juxtaposing the painting with that scene the painting represented. The application does more than simply reconstruct objects from the past. It leverages geo-location and augmented reality techniques to let

users reconstruct the spark that ignited a given artist. Applications like Artist’s View provide people with a rich and imaginative context with which to interpret and engage historical artifacts. They provide access to aesthetic resources and uncanny moments hidden from view on the museum floor. By extending the practice of visiting, paintings, books, and other historical artifacts may become part of the visitor’s broader museum experience.

The Palazzo Strozzi has put in place an intriguing yet limited interaction with cultural history. We can observe the same ecological environments that once inspired the creators of Renaissance paintings, yet our views are shaped by the modern contexts in which they develop. Visitors can discover Frank Duveneck’s tranquil rendering of the Ponte Vecchio bridge as they walk along its contemporary street, full of bustling commerce and international tourism. The painterly depictions of landscapes are at once drawn near and given distance. Situated in the same space at another time, they can feel physically familiar but socially out of reach. Sometimes evocative, other times pedagogical, such software renderings share a specific and partial view of the artist’s brushwork.

In combining modern computer vision techniques with historical objects, applications like Artist’s View invite new questions. What is the place of computing in facilitating our experience with the past? What are the limits of cultural engagement through modern technological developments, such as gesture recognition, networking, and miniaturized displays? How might computing techniques inform what it means to preserve cultural practices alongside artifacts? To begin addressing these questions, let us consider two projects that begin to extend and disrupt the work of computational preservation today.

1. EXPLORING CULTURAL PRACTICES WITH COMPUTATION

One of the authors, Professor Marco Roccetti, discovered his allergy to glutton when he was in his 40s, resulting in a disconnection with the culinary tradition of his Italian family, traditionally linked to such type of food.

Though it was not particularly unusual, Marco’s allergy introduced a real challenge: he could no longer consume pasta, a prerequisite to Italian culinary life. As a lifelong resident of Bologna, the gastronomically (and politically) vibrant city south of Venice, Roccetti struggled to revisit a celebrated aspect of his past. Bolognese tortellini, the small stomach-shaped dumplings traditionally prepared in chicken broth, had pervaded his childhood and that of generations of family members before him.

Working with another author, Gustavo Marfia, and others at the University of Bologna, Roccetti saw the possibility of exploring tortellini as not only a delicacy to consume but also as a cultural pastime to preserve – the technique of producing pasta. Out of this interest in sustaining Bolognese practices came Tortellino X-

perience, a game designed to teach people how to make the Northern Italian tortellini (Figure 2 represents part of the storyboard) [Roccetti et al. 2010]. Designed for the International Expo of Shanghai, the system uses a webcam and simple gesture recognition techniques to enable people to learn and repeat the basic gestures that compose Tortellini. Projecting an interactive video above an empty counter surface, the system prompts people to repeat a pre-specified set of actions in preparation of a tortellini recipe. Specifically, the video parses the actions in the form of a game. Particular actions are required during each preparation phase of Tortellini and shown to a player. When the player reproduces the actions shown in the video, a set of vision algorithms compares the person’s gestures against a set of encoded demonstrated actions, and finally the system presents the subsequent step in the tortellini-making process.

In practice, the system leads a user through a step-by-step process to perform all the actions necessary to prepare Tortellini Pasta, starting from an empty cooking board. In the first step, users pour flour, eggs and water at the center of the cooking board. In the second step, users mixes all the ingredients together, making circular gestures, until a ball of dough appears. Users performs the third step with a rolling pin: they grab its ends and rolls it forward and backward until a thin foil of dough appears. In the fourth step, users cut the thin foil of dough into squares, simply using a knife. The fifth step amounts to stuffing the resulting squares, and the two final steps lead to the final result: the stuffed squares of dough are finally closed and ready to be cooked. Et voila’, virtual Tortellini are now ready!

Figure 2. Tortellino X-Perience: a few storyboard steps.

Thousands of miles west of Bologna, a third author, Daniela Rosner, designed interactive media for the Adler Planetarium and Astronomy Museum in Chicago, Illinois. In collaboration with educators and astronomers at the museum, she used Adobe Macromedia software to develop interactive games for exploring scientific data. The software was presented to museum visitors via kiosks scattered

throughout the exhibit halls. Producing and maintaining the software — and the resulting science narratives —involved installing frequent software updates, fixing broken hardware, and monitoring networking errors. This design work felt different from creative activities she had practiced since childhood such as knitting. Knitting, as a process of interlocking loops of yarn with two needles, presented tangible histories of production by indexing her handwork in the stitches. While Rosner worked at the Adler, a growing number of people were rediscovering or newly embracing the activity of knitting. Around knitting, she discovered various disruptive politico-spatial practices, such as yarnbombing, which refers to the practice of applying old knitted garments (scarves and sweaters) to public infrastructure, such as lamp poles, fire hydrants, or trees. She also discovered the use of digital enhancements to the knitting project: knitters were using online videos and how-to instructions to learn new techniques, connecting with other knitters on social-networking sites for craft, and posting updates of their progress on knitting blogs. Rosner was struck by the qualitative difference between the vibrant social relationships emerging around tools for knitting and the digital interactive displays she had designed for the museum environment. Knitting seemed to embody concerns for cultural practice that contrasted with conventional digital content presentation and codification tools. Knitting privileged the means of production alongside the artifacts produced.

Figure 3. Spyn is a mobile phone software that associates digital records of the creative process (captured through audio/visual media, text, and geographic data) with physical locations on handmade fabric. Image credited to James Jordan at California Magazine.

Taking the practice of knitting as a focus, Rosner began working with Professor Kimiko Ryokai at the University of California, Berkeley, to develop Spyn [Rosner and Ryokai 2009], a system designed to enable the association of digital records with locations on hand-knit fabric. Over three years, the system evolved from an Adobe Flex application running on a Mobile PC to an Android

application running on a G1 smartphone. While knitting, people could mark a position on fabric, and link digital media (images, text, video) to that location on the garment. By pointing a smartphone’s camera to a desired position, people could later retrieve the associated information. The original system used infrared ink patterns printed on yarn to associate locations on fabric with the collected media. The pattern could be recognized by a simple visual edge detection algorithm and used as an index (i.e., virtual link) to retrieve the digital content linked to that position (e.g., a home made video, a song, etc.). The system allowed knitters to store associations between digital media and fabric using their smartphone.

In bringing together Tortellino X-perience and Spyn, we find that the two projects enable people to enact tradition and past experiences by creating new cultural resources. Tortellini X-perience engages practices of tortellini-making by teaching embodied techniques, tacit knowledge, and presuppositions that might, in turn, impart an excitement for Bolognese culture and incite further engagement. Spyn extends opportunities for social annotation and storytelling around the highly feminized handcraft of knitting by allowing particular social relations to emerge. Taking a step back, both projects inform how we think about the digitization of cultural practices. The kinds of information that become most relevant, useful, and evocative while making tortellini or knitting differed among the people using our tools. Some visitors to the Tortellino X-perience exhibit found the sequence of actions depicted on screen a compelling dimension of learning to cook. Others narrowed their focus on the resemblance between their own gestures and those on the screen, gaining social value from accomplishing a similar move. With Spyn, the volume of recorded media associated with their gifted item overwhelmed some recipients. The pressure for reciprocity could feel intensified to the point of discomfort, as when one person referred to the Spyn knit as “emotional blackmail.” Other times the digital records associated with the knit presented a sense of timelessness, or, as one recipient explained, a “4D object.” The system shifted what the work of knitting could accomplish: it enabled the recipient of the knit to gain access to stories of its making, sometimes changing the meaning of the knitted item— turning a pair of gloves into a travel journal, a hat into a mix tape, or a vest into a puzzle. Such differing responses illustrate the fickle nature of the digital; how social environments in which cultural practices are situated inform and importantly transform how digital information is interpreted and disclosed over time.

The projects also highlight new opportunities for sustaining the kinds of data we might want to preserve. Tortellini apprentices may not concern themselves with the precision of gesture replication, as gestures are often readjusted and reoriented while working. Knitters may not care how many stitches it took to make a particular hat, or what a knitter was thinking while making it, since the hat may already embody shared values and ideals. In this sense, the data collected through cameras or other sensors has little value in isolation. Rather, the patterns of social

activity that unfold as part of engaging the devices — rhythms of interaction expressed through annotated stitches or moments of reflection afforded by video queues — may reveal more about the history and collective memory of a craft practice than a gesture alone. In enabling people to extend the work of the hand with digital media, the system simultaneously invites the conservation and the symbolic transformation of a given memory practice.

2. THE TECHNOLOGICAL PARADOX

Here we reach a familiar paradox of preservation: things change in order to remain the same. As we aim to sustain cultural practices – enabling the “replication” of techniques through digitization – we simultaneously change the replicated forms. Roccetti and Marfia designed Tortellino X-Perience to emphasize the technical process — guiding the succession of actions that lead to the dumplings — yet largely neglected the cultural resources and communities of practice in which tortellini-making developed. Further, the Tortellino system separated the means of production from the embodied and tactile engagements particular to its tools (rolling pin, knife, spoon) and ingredients (flour, eggs, salt, water). Rosner designed Spyn to focus on the social relations tightly bound with fragments of fabric (i.e., a woman pinning the music she listens to while knitting with stitches on her mother’s scarf). However, Spyn was not tracking the pattern or gestures with which the stitches were made. Each system was built to sustain critical modes of cultural engagement but, in the process, also left some out. Practices of preservation inevitably came with transformation.

Beyond limitations of codification, cultural practices rely on digital infrastructures whose hardware and software may ultimately wear down. As a result, cultural practices not only change, they also deteriorate. With Spyn, for example, the Android phones began to wear down and the software used to store the data became incompatible with subsequent versions of the Android operating system. Over time, users could no longer access the data. Preservation drew disruption.

From cave painting to cataloguing, people have built cultural records of ritual activity and embodied knowledge since the dawn of humanity. By giving practices material form, they make performance into imagery, text, and audio to hold onto some aspects of their tacit experience and throw away others. A person may not fully grasp an oration from an heirloom painting of a sermon, and the painting may not exist for the next generation if it is not properly stored. In this sense, the question of what to preserve (and how) should strike us as nothing new. Yet, with digitization comes something more. We face possibilities for learning and interacting that weren’t available to us before. As with Tortellini X-perience and Spyn, we can perform an activity by not only following the gestures displayed on a video, but also by interlacing what we learn from the video with additional practices (e.g., digital annotation using computer vision). These added resources for linking digital forms and patterns in turn shift the very purpose of the practice

itself (using Spyn, the knit hat becomes a mixtape). We see preservation as not the work of encoding our lived experience, but as that of enabling memory practices, the modes of performance with which we meet and re-envision our past.

3. WHERE DO WE GO FROM HERE ?

As we have discussed, a wealth of digitization projects aim to support the capture, storage, and representation of material resources, whether books, archeological sites, or works of art [Kallinikos et al 2010]. Levoy et al.’s Google books project, for example, designed techniques for scanning delicate texts, allowing pages from an antiquarian book to be carefully captured, parsed and stored without requiring the book to be opened completely. Opening and flattening antiquarian volumes, as necessary with conventional scanners, endangers the books’ bindings (glued or sewn bands that hold a book’s pages together). Injuring the pages, in turn, disrupts the process of digitization, as it threatens to degrade what is conventionally considered the original artifact — the physical book. Yet, when originality depends on the edition of a volume or to whom the manuscript belonged, different issues of scanning arise: did the scanner capture the mottling on the page margins or the evidence of dog-eared pages? The Google Books project offers a method of digitization that applies advanced imaging techniques to digital archiving. In this, it also suggests questioning what exactly we aim to preserve.

In a second project, [Proudfoot and Levoy 2012] developed, what they have termed, a “3D computer archive” of Renaissance artist Michelangelo’s sculptures by scanning the original forms. They sought to create a lasting archive of the artworks by stitching together data from multiple sources: planar light field scanner, handheld lightfield scanner, and low-res models for planning purposes. In the process, they accommodated for variable ambient light: filling in holes using space carving techniques, and aligning scans from multiple gantry angles and positions. This increased access to the statues, has the additional affect of shifting the rarity of such artifacts as well as displacing their cultural context.

Computing researchers conceive of digitization, in these settings, as a project of computer science and engineering. Engineers built the scanners and engineers use them to scan and archive the physical artifacts. The people using the digital scans have little influence on what books are available to them and how. While an engineer could decide not to scan the first edition of Miguel de Cervantes Saavedra’s Don Quixote, believing the fifth and sixth editions were sufficient, a scholar of Latin literature might require the first edition to contextualize 17th century interpretations. In Tortellino X-perience and Spyn, not only the engineers, but also the craftspeople and interested persons producing the items —Italian chefs and knitters — take up the work of digitization. Players of Tortellino X-perience shape the vision algorithm’s diagnosis of events, and the subsequent sequence of actions depicted on the video screen. Knitters using Spyn choose the kinds of stories they want to tell through digital media collected while knitting and control the degree to which the fabric is legible to vision algorithms. By

creating, using and revisiting the digitized files, participants have the agency to take part in the digitization and, in turn, the preservation of cultural practices.

We have found through our studies of Tortellini X-perience and Spyn that people use computing resources to connect present experiences to past events. They compare their unique gestures to the delicate pinch of the tortellini dough rendered by a digital display to remember a given technique. They use media associated with knitted fabric to recall lost stitches at particular moments in time. Even though the revisited pinching and knitting techniques will never emerge exactly the same as originally performed, the events help people construct memories of the past and recognize particular practices (tortellini making or knitting) as part of a cultural tradition. As such, preserving cultural practices is about moving beyond digital historiography and its institutionalized requirements for explicit evidence, documentation, authenticity and provenance. Viewing cultural practices from the perspective of memory practices, we find them “preserved” through histories are in constant flux, intertwined with collective memories.

Our discussion of computing technologies around cultural practices has revealed the complex nature of digitization and degradation and enabled us to comment on the status of the ephemeral in these contexts — how when we trace the ephemeral, it changes the behavior we hope to trace. We view modern technologies as capable of registering and reifying the practices central to our cultural heritage, but also capable of reconfiguring them [Ferretti et al. 2010]. Hence, we face the enormous challenge of understanding the kinds of transformations made possible through digitization and the patterns they yield.

ACKNOWLEDGMENTS The authors would like to thank the Italian ALTER-NET Project for supporting this work.

REFERENCES Artist’s View, available, online: https://itunes.apple.com/mg/app/artists-view/id504132267?mt=8, 2012. Brown, J. S., and Duguid P., Borderline issues: social and material aspects of design. Human–Computer

Interaction 9.1 (1994): 3-36. Bergamasco, M., Avizzano, C., Di Pietro, G., Barbagli, F., and Frisoli, A., The museum of pure form:

system architecture, Robot and Human Interactive Communication, 2001. Proceedings. 10th IEEE International Workshop on , vol., no., pp.112-117, 2001.

Bouchenaki, M., The interdependency of the tangible and intangible cultural heritage, (2003): 1-5. Ferretti, S., Furini, M., Palazzi, C. E., Roccetti, M., and Salomoni, P. (2010). WWW recycling for a better

world. Communications of the ACM, 53(4), 139-143. Kallinikos, J., A. Aaltonen and A. Marton (2010). "A theory of digital objects." First Monday 15(6). Levoy, M., Pulli, K., Curless, B., Rusinkiewicz, S., Koller, D., Pereira, L., ... and Fulk, D. (2000, July). The

digital Michelangelo project: 3D scanning of large statues. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques (pp. 131-144). ACM Press/Addison-Wesley Publishing Co.

McIntosh, A., and Prentice, R.C., Affirming authenticity – Consuming cultural heritage, Annals of Tourism Research.

Mayer-Schonberger, V., Delete: The virtue of forgetting in the digital age (new in paper). Princeton University Press, 2011.

Mazzeo, R., Palazzi, C.E., Roccetti, M., and Sciutto, G., Computer-assisted pigment identification in artworks. In Proceedings of the Third IASTED European Conference on Internet and Multimedia Systems and Applications (EurolMSA '07)

Proudfoot, R.A., and Levoy, M., Systems and methods for glare removal using polarized filtering in document scanning. U.S. Patent No. 8,174,739. 8 May 2012.

Roccetti, M., Marfia, G., and Zanichelli, M. (2010). The art and craft of making the Tortellino: playing with a digital gesture recognizer for preparing pasta culinary recipes. Computers in Entertainment (CIE), 8(4), 28.

Rosner, D.K., and Ryokai, K. (2008, September). Spyn: augmenting knitting to support storytelling and reflection. In Proceedings of the 10th international conference on Ubiquitous computing (pp. 340-349). ACM.

Schatzki, T. R. (2012). A primer on practices: Theory and research. Practice-based education: Perspectives and strategies. Rotterdam, The Netherlands: Sense.