Paint with Your Voice: An Interactive, Sonic Installation

Abstract In this paper, we describe the design and implementa-tion of an interactive, sonic installation that lets visitors paint pictures with their own voice. Through speaking into an augmented megaphone and moving it, visitors control a digital paintbrush that paints onto a projected canvas. Depending on both pitch and volume of the visitor’s voice, the paintbrush changes its stroke and color. Once the painting is finished, a printout is pro-duced as a tangible artifact that completes and trans-cends the visitor’s interactive experience. We also re-port on a deployment of the installation at a regional science fair and depict salient insights from observa-tions that will guide further development of the exhibit.

Author Keywords Sonic Installation, Interactive installation, Embodied interaction, Urban HCI, Exhibition, Sound, Painting

ACM Classification Keywords H.5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.

Introduction The advent of highly capable technology such as unob-trusive tracking systems or ultra-bright projectors em-powers creative coders to create novel ways of using the physical space as a digital canvas. Artists and re-

Copyright is held by the author/owner(s). TEI 2014, Feb 16 – 19, 2014, Munich, Germany.

Benjamin Böhm 1 benboehm86@gmail.com Julian Hermann 1 julian.hermann@img.fh-mainz.de Tim Rizzo 1 tim.rizzo@img.fh-mainz.de Anja Stöffler 1 anja.stoeffler@img.fh-mainz.de Jochen Huber 2,3 jhuber@mit.edu

1 Institut für Mediengestaltung Fachhochschule Mainz Mainz, Germany

2 Singapore University of Technology and Design Singapore

3 MIT Media Lab USA

searchers alike have investigated painting digital art and simultaneously situating it within a projected can-vas. Mobispray [4], for instance, uses a smartphone to control a digital paintbrush through gestures while pro-jecting the digital imagery onto urban places. VR/Urban’s SMSlingshot [1] utilizes an augmented slingshot to shoot and project texts onto buildings.

In this ongoing work, we created an interactive installa-tion that explores a cross-modal approach to interactive painting: Through speaking into an augmented mega-phone and physically moving it, visitors control a digital paintbrush that paints onto a projected canvas. De-pending on both pitch and volume of the voice, the paintbrush changes its stroke and color.

We describe the design and implementation of the in-teractive installation in the following. We also report on

an exhibition of the installation at a regional science fair and depict the most relevant insights from observations that will guide further development of the exhibit.

Interactive Installation The conceptual framing, the envisioned visitor experi-ence and the implementation of the installation are de-scribed in the following.

General Concept The general concept is that of leveraging the human voice as input to paint onto a projected, large-scale canvas (cf. Figure 1). The characteristics of the voice are used to control the expressiveness of the paint-brush. An augmented megaphone captures the voice. The visitor can move it in physical space to control the position of the brush.

The expressiveness of the brush depends on two fea-tures: the pitch and the volume of the voice.

• Pitch: The pitch of the voice determines the color of the brush. The actual pitch frequency is hence mapped to the hue scale.

• Volume: The stroke of the brush is modeled as 14 individual pixels. The volume is mapped to the spacing of the pixels respectively and im-pacts the thickness (or density) of the stroke.

The expressive nature of the stroke is inspired by [5].

Envisioned Visitor Experience The overall exhibit is designed to provide a seamless and also serendipitous experience: a visitor walks up to the augmented megaphone and grabs it. When direct-

Figure 1. General concept of the installation: a visitor paints onto a projected canvas through speaking into an augmented megaphone and physically moving it.

ing the megaphone towards the projected canvas, the brush appears and the visitor can interact with the megaphone to paint on the projected canvas.

Once the visitor finished a painting, a printout is gener-ated. The print serves as a tangible representation of the painting that transcends the interactive experience. The canvas can then be cleared to start a new painting.

Implementation The hardware setup of the exhibit consists of two es-sential parts: (i) the augmented megaphone and (ii) a silver screen used as the digital canvas through back projection. The implementation is also illustrated in this video figure: http://vimeo.com/78633545.

The augmented megaphone (cf. Figure 2) contains a wireless microphone located at the mouthpiece that

records the voice input. Also, a Wiimote is mounted at the open end of the megaphone, facing the canvas. The infrared camera of the Wiimote is used to track infrared markers embedded in the silver screen, therefore esti-mating the pose of the megaphone (inspired by [3]).

A screenshot of the simplistic interface is shown in Fig-ure 3. The paintbrush is visualized in the center. The interface also features two buttons: one for clearing the canvas (left) and another for printing the current paint-ing (right). Continuously pointing at them with the megaphone for about three seconds triggers them. The activation of the button is visualized through a progress bar. The overall system is implemented using VVVV.

Exhibition The installation was exhibited over the course of two days at a regional science fair. During these days, sev-eral hundred visitors interacted with the exhibit. An exemplary set of pictures created by visitors is shown in Figure 5.

Figure 2. Design of the megaphone used in the installa-tion. A wireless microphone is located at the mouthpiece of the megaphone and the Wiimote at the end is used as an infrared camera.

Figure 3. Screenshot of the interface.

Although the primary focus was to solely expose the installation to the public, we also observed how visitors approached the exhibit and how they interacted with it. In the following, we report on two particularly pertinent observations.

Involvement of Bystanders The current implementation is a single user setup. In general, we observed very similar space patterns as noted in [2]. In particular, we observed visitors at-tracting bystanders that either joined the interaction space as spectators or tried to be actively involved in the creation process by generating noise, e.g. through yelling. Visitors also passed the megaphone to co-create a painting and share experiences together.

Appropriation of the Megaphone We observed another form of co-creating shared expe-riences: visitors also worked together to create unique sounds and therefore unique strokes. Figure 4 shows a

particularly interesting example where one visitor used his wind instrument to create a tone and another visitor held the megaphone to control the paintbrush.

Conclusion and Outlook We described the design and implementation of an in-teractive, sonic installation that lets visitors paint pic-tures with their own voice. Through speaking into an augmented megaphone and moving it, visitors control a digital paintbrush that paints onto a projected canvas.

The exhibition at a regional science fair revealed inter-esting phenomena. Visitors appropriated the exhibit in various ways, particularly focusing on (i) multi-user interaction, and (ii) combining the megaphone with different sound sources, such as instruments, to co-create unique brushes and paintings, respectively. Fu-ture designs of the exhibit will focus on these aspects to create novel ways of co-creating shared experiences.

References 1. Fischer, P.T., Hornecker, E., and Zoellner, C. SMSling-

shot: An Expert Amateur DIY Case Study. In Proc. TEI 2013, ACM (2013), 9–16.

2. Fischer, P.T. and Hornecker, E. Urban HCI: Spatial As-pects in the Design of Shared Encounters for Media Fa-cades. In Proc. CHI 2012, ACM (2012), 307–316.

3. Lee, J.C. Hacking the nintendo wii remote. Pervasive Computing, IEEE 7, 3 (2008), 39–45.

4. Scheible, J. and Ojala, T. MobiSpray: Mobile Phone as Virtual Spray Can for Painting BIG Anytime Anywhere on Anything. Leonardo 42, 4 (2009), 332–341.

5. Willis, K.D.D. and Hina, J. Alchemy: Experiments in Interactive Drawing, Creativity, & Serendipity. Proceed-ings of the Seventh ACM Conference on Creativity and Cognition, ACM (2009), 441–442.

Figure 4. Visitors at the science fair join an instrument and the megaphone to create unique strokes.

Figure 5. Exemplary paintings created by visitors of the regional science fair.