This article was downloaded by: [175.143.160.248]On: 05 January 2013, At: 18:12Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office:Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Digital CreativityPublication details, including instructions for authors and subscriptioninformation:http://www.tandfonline.com/loi/ndcr20

Full-body movement as material forinteraction designLise Amy Hansen aa Oslo School of Architecture and DesignVersion of record first published: 04 Jan 2012.

To cite this article: Lise Amy Hansen (2011): Full-body movement as material for interaction design, DigitalCreativity, 22:4, 247-262

To link to this article: http://dx.doi.org/10.1080/14626268.2011.622284

PLEASE SCROLL DOWN FOR ARTICLE

Full terms and conditions of use: http://www.tandfonline.com/page/terms-and-conditions

This article may be used for research, teaching, and private study purposes. Any substantialor systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, ordistribution in any form to anyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representation that thecontents will be complete or accurate or up to date. The accuracy of any instructions, formulae,and drug doses should be independently verified with primary sources. The publisher shall notbe liable for any loss, actions, claims, proceedings, demand, or costs or damages whatsoever orhowsoever caused arising directly or indirectly in connection with or arising out of the use of thismaterial.

Full-body movement asmaterial for interactiondesignLise Amy Hansen

Oslo School of Architecture and Designliseamy.hansen@aho.no

Abstract

This article focuses on the design potential of digitalinteractions where the body is seen as the interface.With computational technology and sensors infiltratingmany aspects of our lives and urban surroundings, inter-action designers’ ability to visualise and generatedesigns are important in order to understand andexplore such design spaces. I propose three con-cepts—accessibility, immediacy and generation—asmeans for analysing movement as a design materialfor interaction design. Drawing on a social semioticsapproach, contemporary choreographic research isstudied where digital tools are used to generate, expli-cate and communicate interactive movement. I arguethat by drawing on the particularities and potentials ofthe moving body as interface such as those exploredthrough choreographic practice, we may avoid imitatingexisting exchanges with technology and create novelinteractions.

Keywords: interaction design, design material, com-munication, choreography

1 Positions

Walking in the hallway, a door shuts snappingat your heels. The pre-programmed time for itsopening ends, regardless of whether you havegone through or not. Your slow and consideredmovements go unnoticed.At work, you have a complex presentation towrite and you spend much time sitting still,contemplating how to clearly present yourthoughts. The lights that were automaticallyturned on as you entered your office go out.The time spent occupying the office is not regis-tered.

These everyday encounters indicate a new designspace for the interaction designer. The traditionalscreen interface between a human and a computeris usually accompanied by buttons on a keyboard,a mouse to click or a surface to touch. In theexamples above, the screen is removed and thetechnology is activated by computer vision, and,by extension, ourselves. Here, the body is theinterface.

The main question I address regarding inter-action design is how may we approach full-bodymovement as a material. This is taken up in anheuristic approach that includes conceptualisingsuch movements, by looking at practice-basedexperimental examples from choreography, aswell as referring to an interdisciplinary body ofrelated research. The focus is on the processof instigating, probing, shaping and changing

Digital Creativity2011, Vol. 22, No. 4, pp. 247–262

ISSN 1462-6268 # 2011 Taylor & Francishttp://dx.doi.org/10.1080/14626268.2011.622284http://www.tandfonline.com

Dow

nloa

ded

by [

175.

143.

160.

248]

at 1

8:12

05

Janu

ary

2013

possible modes of representing and generatingmovements. I argue that by drawing on the particu-larities and potentials of the body as interface, wemay avoid imitating existing interactions (seeFigure 1) and create novel interactions.

In presenting three core concepts, I draw oncommunication and design theory and cognitivepsychology. The development of the conceptsaccessibility, immediacy and generation is given.These are applied as analytical resources forunderstanding meaning-making with movementfor interaction design. The current sophisticationof computer vision coupled with increased compu-tational speed, now allows for more movementdata to be captured, making it possible to furtherexplore aspects and qualities of expressive move-ment. Interaction design may benefit by drawingon how choreographic practice—the shaping offull-body movement—is studied and communi-cated. I critically analyse current research on chor-

eographic resources in the performance field, anddiscuss how they may inform the design of themoving body as an interface.

The fields of interaction design and choreogra-phy are connected in order to explore how we mayunderstand the body as the interface, wheremeaning is generated through bodily movement.

Currently movement is increasingly read bytechnology, and in interaction design, it is mostlyapplied in gaming scenarios such as Nintendo’sWii and more recently Sony’s PlayStation Moveand Microsoft’s Xbox Kinect. These presentexiting applications of movement in interaction.However, they currently interact existing scenarios,akin to those of dance instructors, rafting and golf. Inorder to explore novel applications and communi-cations, it is argued that the interaction designerneeds to further explore the communicative poten-tial of the moving body, in order to allow andenable the material to inform possible applications.

Figure 1. Still from a promotional video for Microsoft’s Kinect Xbox. The interactions, albeit ‘hands free’, are the same as those ofexisting movement actions and scenarios, such as driving and ball games, dance and yoga classes.Source: Image by kind permission # Microsoft Corporation.

Hansen

248

Dig

italC

rea

tivity

,V

ol.

22

,N

o.

4

Dow

nloa

ded

by [

175.

143.

160.

248]

at 1

8:12

05

Janu

ary

2013

Technologies can now go beyond fit and func-tion (Wright et al. 2008), and whilst the guidingdesign principles of ergonomics and efficiencyremain important, an understanding of the possibi-lities of the moving body needs to be developedtogether with the application of digital technology.By visualising and working with movementaccording to the concepts introduced here, thedesigner may be able to conceptualise designs atan early stage. This is important in order toavoid having technology dictate our movementsand, by extension, how we live.

2 Movement research

Bodily movement has been studied in many fieldsrelated to communication. In anthropology, phys-ical movement has been interpreted as non-verbalcommunication in relation to verbal communi-cation. Through inter-cultural studies, the anthro-pologist Edward Hall (1966) found that wecommunicate through the use and position of thebody in proximity to others. Ray Birdwhistell(1971) founded kinesics as a field of researchand developed a deciphering system of facialexpressions, posture, etc., to be interpreted in thecontext of other means of communication. In cog-nitive science, the role of our bodies is increas-ingly taken into account. Alva Noe (2004)argues that our perception and consciousnessdepends on and is a result of, our bodily capacitiesand activities. Rolf Pfeifer and Josh Bongard(2007) show how thought is constrained as wellas enabled by the body, by analysing the makingof artificial intelligence. In an embodied communi-cation perspective Ipke Waschmuth et al. (2008,p. 3) argue that human communications gobeyond verbal communication and that ourbodies enable ‘parallel and highly interactivecouplings between communication partners’.These fields of study show that the body indeedplays a communicative role.

In ‘new media’, the focus is on the bodyframed by technology. Simon Penny (2004) callsfor a framework for interactivity that goesbeyond the theories of visual art, as interactiveimages are procedural, and previous theories do

not include the ‘ensuing activity’. Mark Hansen(2004) proposes a philosophy that encompassesthe development of images in communicative pro-cesses, where their perception is bound to theactivity of the body. And following on from Kathr-ine Hayles’ (1999) disconnect of information froma body or medium, Anna Munster (2006) also dis-cusses new modes of sensory engagement, imply-ing that digital aesthetics have reconfigured bodilyexperience and reconceived materiality.

In human–computer interaction (HCI),attempts to create a framework for evaluatingbodily potential (Loke et al. 2007, Fogtmannet al. 2008) indicate that notions of the bodyincreasingly are considered. As the designprocess is opened up for multiple interpretations(Gaver et al. 2003, Sengers and Gaver 2006),there is an acknowledgement of the communica-tive potential for expressive movement. PaulDourish (2001, p. 126) presents an approach toembodied interaction, defining it as ‘creation,manipulation and sharing of meaning throughengaged interaction with artifacts’. Theseapproaches, based on theories of embodiment,emphasise the role that the body plays in shapingperception and action. However, ‘whilst therehas been substantial advances in human motionreconstruction the visual understanding ofhuman behavior and action remains immaturedespite a surge of recent interest’ (Moeslundet al. 2006, p. 116).

In contrast, performance studies has a long tra-dition of reading movement like a text, as a part ofa mediating scenario, ‘a showing of a doing’(Schechner 2002, p. 141). As traditional perform-ance increasingly makes use of interactive tech-nologies, the performed movements are not onlyexpressive but functional, in that they enableother media, such as computer vision and videoprojection, to express and mediate. In dance,movement is studied as the main mediatingmaterial. The communication focus, however,has been on the role of tacit or bodily knowledgeand whether it may, in fact, be seen as knowledge,as this kind of communication does not fit tightlywithin the structures of language (Sigman 2000).When movement is read by technology it is

Full-body movement as material for interaction design

249

Dig

italC

rea

tivity,V

ol.

22

,N

o.

4

Dow

nloa

ded

by [

175.

143.

160.

248]

at 1

8:12

05

Janu

ary

2013

usually interpreted together with language. In thenon-verbal communication field, ‘typically, ges-tures are thought of as arm and hand movements,but head gestures are also well known’ (Knappand Hall 2006, p. 225). As this article draws onthe practice of choreography, the whole body isthe focus, therefore ‘gesture’ becomes toolimited and ‘movement’ more apt in describingfull-body motion.

All the fields mentioned above study move-ment in relation to communication. However, thestudies only partly inform interaction design intheir study and shaping processes of movement,as interaction design may now draw on the com-municative aspects of movement such as fre-quency and force, repetition and rhythm,alignment and position. Ashbrook and Starner(2010) point to the fact that motion gesturecontrol does not yet appear outside the gameconsole and that the reason for this might be thatinteraction designers are not experts in pattern rec-ognition. It is here that choreographic researchmight have transferable insights to the field ofinteraction design as motion sensing increasinglybecomes available. This presents a challenge forthe interaction field, as the designer now increas-ingly draws on, engages with and shapes physicalmovement as a semantic element in interactions.Here I analyse current choreographic research inrelation to the interaction design process, and for-mulate concepts relevant to such a design process.

3 Approach

In exploring the body, and specifically the movingbody, as an interface, this article adopts a com-munications view, drawing on social semiotics(Kress and van Leeuwen 2001). Research suchas Eikenes (2010) investigates movement indigital interfaces, and draws on such a framework.This article looks at the moving body as the inter-face through which one may control, access, influ-ence and interact with digital technology. Thespatial and temporal qualities of movement com-municate multimodally, for instance throughshape and change (such as frequency, repetitionand scale). In fact, the body is always communi-

cating. ‘We cannot not communicate’ (Watzlawicket al. 1967). This is perhaps why the body as aninterface becomes so complex. We each havelife-long training in the sophisticated way withwhich we read each other. In order for technologyto now do the reading of bodily movement qual-ities beyond touch and click, we need to analysethe moving body as a mediating artefact (Vygotsky1978).

In semiotics a ‘text’ can be any type of objectthat communicates meaning, such as an image, avideo, an artefact, or an interface such as thebody. The concept of text enables a discussion ofmeaning, analysis and close engagement with theobject (Bal 2002). As this approach advocates,signs and texts, as indeed movements and ges-tures, do not have a fixed meaning, rather aresocially influenced and continuously changing(Kress and van Leeuwen 2001, 2006, Noland2009). A communication view, treating themoving body as ‘text’, enables a discussion onthe possibilities of movement as a design material,feeding knowledge back into interaction design.There is research on the moving body and technol-ogy with a phenomenological approach (e.g.Schiller 2006, Broadhurst 2007, Kozel 2007,Manning 2009). However, these studies focus onthe effect of technology on the moving body andnot on how designers may work with, exploreand shape these interactive movements. Asdesigners create transformative spaces (Morrisonet al. 2009), movement is important, as we maygenerate our own mediated meanings and notonly be the recipients of pre-set information.Designers need to work with, alter and augmentmovements as part of the design process. Thenwe may truly interact with, not only react to, thescreens we encounter.

The three concepts to be presented below comeout of a ‘complex field of knowledge production’(e.g. Sevaldson 2010, p. 8). They have been devel-oped where design is explored through both theoryand practice. The focus is on working with andinstigating, probing, shaping and changing poss-ible modes of representing and generating move-ments. Below I focus on the specification of thethree core concepts by the way of reference to

Hansen

250

Dig

italC

rea

tivity

,V

ol.

22

,N

o.

4

Dow

nloa

ded

by [

175.

143.

160.

248]

at 1

8:12

05

Janu

ary

2013

leading design and research projects together withcritical readings of related research.

4 Interaction design

The places, events and scenarios where technologyfacilitates communication, interaction and actionare created by the interaction designer. Lowgrenand Stolterman define interaction design as ‘theprocess that is arranged within existing resourceconstraints to create, shape and decide all use-oriented qualities (structural, functional, ethicaland aesthetic) of a digital artifact for one or manyclients’ (Lowgren and Stolterman 2004, p. 5).Here Lowgren and Stolterman focus on the artefact.I argue that the interaction designer today needs toaddress aspects that go beyond what can be commu-nicated through an object. In this regard, the defi-nition of interaction by Poggenpohl et al. is moreapt:

Interactions are a succession of actions, eachresponding to prior actions and each beingresponded to by succeeding action. By identify-ing and studying interaction patterns in thissuccession, we can design interventions thatprovide material support for desirable inter-action patterns to emerge (Poggenpohl et al.2004, p. 603).

It is the material support for interaction patterns1

that I take up.The interaction designer needs to understand

the possibilities and limitations that are inherentin the new making material discussed here, inorder to make informed choices at a conceptualstage. This may enable the designer to lookbeyond imitations of already existing scenarios,and open up for new interaction possibilities.Bolter and Grusin (1999) argue that digitalmedia achieve their cultural signification though‘remediation’, by paying homage to earliermedia in the same way as photography has refash-ioned (or remediated) painting. However, as ourneed to communicate may be media independent,the mechanism with which we communicate aremedia specific and when one kind of communi-cation only imitates another, it will always be at

a disadvantage (Hollan and Stornetta 1992). Bydrawing on the particularities of the makingmaterial, the designer may create novel inter-actions, such as TV coming into its own when itno longer imitated radio but employed its uniquequalities in the editing of visuals, combiningimage and sound, choice of mis-en-scene, etc. Inexploring movement as communication for inter-action design we need to find the particularitiesof movement in order to make interactions suitedto its making materials.

5 Design material

By exploring movement as a design material Irefer to design material as the constituent elementsthat are formed through a design process. ‘Form isthe way material builds things; to build a thing, weform materials’ (Hallnas et al. 2002, p. 157). Asinteraction design goes from the design ofobjects to the design of experiences, the designers’repertoire expands and designers now also influ-ence aspects such as use of space and how wemove. The transient and ephemeral nature of phys-ical movement existing only in time presents achallenge. When the designer is working on aninteraction at a conceptual stage, the physicalpart is rendered invisible, as interaction designlacks the means to visualise and, in particular, togenerate and augment physical movement.

Hummels et al. (2007) argue for the designersof movement-based interactions to move them-selves, i.e. go beyond imagining physical move-ment to themselves moving, exploring and tryingout actions with their bodies, such as with jug-gling. Hummels et al. also ask how movement isunique as a design material. We have a lifelongexperience of it, both in the reading of it, andalso in the use of it. We all have experience ofhow our own body ‘works’, and in particularhow it communicates.

6 Choreography

I suggest that the creative practice of contemporarychoreography has correlations to these new designspaces. In order to be able to explore and shape this

Full-body movement as material for interaction design

251

Dig

italC

rea

tivity,V

ol.

22

,N

o.

4

Dow

nloa

ded

by [

175.

143.

160.

248]

at 1

8:12

05

Janu

ary

2013

communicative movement material, the designerneeds to become familiar with its qualities, limit-ations and possibilities. It is worthwhile to take astep back from the existing use of physical move-ment in interaction design in order to learn aboutthe potential for expressive movement as it is com-municated to and through technology. Here thepractice of contemporary choreography is relevantdue to its refined, abstracted, aesthetic andrehearsed field of movement, with a history ofexperimentation, collaboration and exploration ofboth new movements as well as emerging technol-ogies (Wildschut and Butterworth 2009).

Contemporary choreography is the practice ofcomposing new movements, working with bodilyrestrictions (such as every part being connected,elbows bending only one way, and so on) and pos-sibilities (a high degree of difference from itsnumerous joints that may function independentlyof each other). Choreographers’ practice mayresemble an interaction designer’s practice, asthey shape, design and through iteration refinetheir material until the composition and expressionof the material has the desired shape and effect.‘During this playful probing of physical andsemantic potential, choreographers’ and dancers’bodies create new images, relationships, conceptsand reflections’ (Foster 1995, p. 15). The choreo-graphic product is usually instantiated in a per-formance. However, ‘choreography and dancingare two distinct and very different practices’ (For-sythe 2008, p. 5). The focus here is on the choreo-graphic resources employed in the makingprocess, rather than the performed product.

7 Choreographic resources

Technology now enables interaction design todraw on a wider scope of movement-based full-body communication. This extends the communi-cative use of movement aspects such as spatial andtemporal qualities of movement: for example fre-quency and force, repetition and rhythm, align-ment and position. The choreographic researchprojects described below are chosen with thisfocus in order to go beyond the use of gestures,e.g. arms in certain positions, and to move from

controlling, instructive movements to includeexpressive movement. We are able to read eachother’s body language; however, so far technologydoes not mirror this sophistication of interpret-ation. ‘We can observe it with the human eye,but methods to extract such information are stillin their infancy’ (Bevilaqua 2007, p. 27).

By resource I refer to meaning-makingmaterials which aid the communication of a crea-tive process, e.g. ‘resources for representation’(Kress 2010, p. 8). The choreographic researchprojects that will be discussed here explore howto visualise movement to aid and multimodallycommunicate the creation process, and do so bydrawing on notation, video and motion captureas resources.

8 Movement concepts

The concept of accessibility informs how thedesigner may start to think about movement withregard to interaction as they select relevantaspects of the movement in the visualisations ofmovement. Immediacy allows the designer toexplore whether the chosen movement aspectscould serve a communicative purpose in a rapid,iterative manner. The concept of generationallows the designer to alter and augment parts ofthe visualisations in order to test, probe andexplore. How these three concepts may help thedesigner include physical movement in thedesign process is further presented below.

8.1 AccessibilityFor the designer to work with movement, thatmovement needs to be accessible, e.g. visualisedin ways that are relevant to interaction design.Sketches or visualisations are used as thinkingtools for the designer (Schon 1983, Goldschmidt1994). With experience, the designer may getnew ideas from inspecting their own sketches asnew and unintended relations, patterns and func-tions emerge (Suwa et al. 2001). Such visualisa-tions are also influenced by a working knowledgeof functionality of the making material (Tsenget al. 2002). Goldschmidt (1994) argues that thesketch amplifies the mechanism of visual cognition

Hansen

252

Dig

italC

rea

tivity

,V

ol.

22

,N

o.

4

Dow

nloa

ded

by [

175.

143.

160.

248]

at 1

8:12

05

Janu

ary

2013

with which computational tools may be furtheramplified. Bill Buxton also argues that for inter-action design, the sketch can extend beyond thatof pen and pencil, to other forms, as ‘they need tobe able to capture the essence of design conceptsaround transition, dynamics, feel, phrasing, andall the other unique attributes of interactivesystems’ (2007, p. 136). The studies point to theimportant role of the visual during the designprocess.

The choreographer William Forsythe hasexplored a wide range of visualisations of interac-tive movement. He instigated the research projectSynchronous Objects with the question ‘is it poss-ible for choreography to generate autonomousexpressions of its principals, a choreographicobject, without the body?’ (Forsythe 2008, p. 5).In this project Forsythe and The Ohio State Uni-versity Advanced Computing Center for the Artsand Design (ACCAD) explore Forsythe’s dancepiece One Flat Thing reproduced (OFTr). Thedance is complex as it is performed to an interac-tive score by seventeen dancers navigating atwenty-table grid (see Figure 2).

In this work, thirty-five themes or phrases areactivated, repeated and used over fifteen minutes.The underlying structural systems, or counter-

points, of movement phrases, cueing and align-ments are visualised by abstracting andvisualising data from a recording of a single per-formance. The project then analysed this footageand produced twenty choreographic objects.Norah Zuniga Shaw (2009) described theprocess as ‘a flow from dance to data toobjects’.2 Compositionally, each object arosefrom a trans-disciplinary process involving geo-graphers, animators, designers and architects, aswell as dancers and choreographers (e.g. Figure 3).

The project clearly demonstrates the complex-ity involved in communicating interactivemovement, and the myriad of ways it may beapproached. The visualisations took the researchteam three years to generate, some through com-putational interpretation, others were animatedframe by frame. For the interaction designer,there is a need for sketches, abstractions andinstructions for each stage of a design develop-ment, so the processes employed in SynchronousObjects would be too extensive.

Some of the visualisations are furtherabstracted, such as the counterpoint object. Itoffers a tool, where the dancers are replaced by‘widgets’, and the viewer or user has slidingscales through which to control the shape, speed,

Figure 2. Still from OFTr with graphic annotations of alignments, e.g. where cues between the dancers are sent and received.Source: Image by kind permission # Synchronous Objects Project, The Ohio State University and The Forsythe Company.

Full-body movement as material for interaction design

253

Dig

italC

rea

tivity,V

ol.

22

,N

o.

4

Dow

nloa

ded

by [

175.

143.

160.

248]

at 1

8:12

05

Janu

ary

2013

horizontal and vertical motion, from unison todifference (see Figure 4). The designer maywork with this to ‘build relationships of actions’.

However, as this choreographic resource focuseson the position of the body in space, it becomesless useful for the interaction designer. This

Figure 3. Still visualising spatial patterns generated by the moving bodies, with a focus on density according to where on stage thedancers spend most of their time.Source: Image by kind permission # Synchronous Objects Project, The Ohio State University and The Forsythe Company.

Figure 4. A still from the Counterpoint Tool where the dancers’ bodies are replaced by widgets and the lines represent visual cues onwhich the other dancers act.Source: Image by kind permission # Synchronous Objects Project, The Ohio State University and The Forsythe Company.

Hansen

254

Dig

italC

rea

tivity

,V

ol.

22

,N

o.

4

Dow

nloa

ded

by [

175.

143.

160.

248]

at 1

8:12

05

Janu

ary

2013

indicates that the visualisations of movements thatare of interest to interaction design need to encom-pass the ‘how-ness’ of the movement as well as themovement itself, e.g. the micro-analysis of tem-poral, communicative aspects of movement.

There are attempts to analyse bodily move-ment with video, such as motiongrams, basedvisually on spectrograms produced for sound (Jen-senius 2007). Here movement is seen in relation tosound and music performance, so in this respect itproduces data that are useful. However, for theinteraction designer the motiongrams are tooabstract to be read as communicative movement.Other real-time visualisations are being developedwith a focus on expressive feature recognition,such as eyesweb (Camurri et al. 2004) andgesture follower (Bevilacqua et al. 2010). Asthese computational tools indicate, the develop-ment of computer vision and computation isbecoming increasingly sophisticated in thereading of movement. Yet, it is the interpretation,recognition and application of these data that arechallenging the further use of temporal communi-cative aspects when interacting with technology.

In semiotic work the first stage is to identifyand name the meaning-making modes involvedin an activity, each with a specific task and func-tion. The design of such modal ensembles ‘makea specific message about a particular issue fora particular audience’ (Kress 2010, p. 28).The Synchronous Objects project shows thewealth of choices that come with communicatingmovement. As we include movement in inter-action design, choices are made as to whichaspects of movement are made meaningful forthe technology. In order to make such choices,the designer needs knowledge of the extensiverange of possibilities of movements and theirvisualization.

8.2 ImmediacyDrawing further on the design process, I argue thatmovement needs to be visualised in ways thatare immediate.3 The interaction designer usuallyworks in a rapid, iterative process, each stageinvolving reflection-in-action and reflection-on-action (Schon 1983). To enable such reflection,

the visualisations of movements need to be athand for each stage. Currently, the analysis ofdance with traditional notation such as BeneshMovement System and Labanotation is a time-consuming and highly skilled process. All thesame, notation has the scope to notate possiblemovements as well as a single instantiation. ‘Thegreatest value of the systems is not necessarilyhow precise they could be, but the possibilitythey have to record more than one stratum of pre-cision’ (Bastien 2007, p. 48). This means, forinstance, that as a viewer of a set of movementsor a movement phrase you will only see the oneversion performed. The notation, however, mayencompass the instruction as well as the instanceas it was performed and thereby reveal the intentor purpose of the movements.

This aspect of possibility was explored in theresearch project Inside Movement Knowledgewhere the choreographer Emio Greco and PieterScholten (EC|PC) used the Double Skin/DoubleMind installation as a ‘test case in documenting,analyzing and re-presenting essential elements ofthe work of EC|PC’ (deLahunta 2007, p. 20).Taking Greco’s own choreographic practice as astarting point, they developed a vocabulary ofover 200 words, which together with the prepara-tory dance workshop formed the basis for an inter-active, multimodal installation. This was in order toexplore ‘systems for the documentation of live andvariable media artworks and to explore how toanalyse and document the dance creation processof EG|PC’.

In this project ‘the interdisciplinary researchteam proceeds on the assumption that thecomplex nature of dance cannot be adequately rep-resented by a single technology’ (Hoogenboom2007, pp. 86–87). As a result the installationexplores multimodal mediation of movement bydrawing on gesture analysis, dance notation, docu-mentary film-making and interactive mediadesign. Designed by Chris Ziegler, it consists ofan aluminium frame construction with one projec-tion screen, four sound speakers and a trackingcamera. The participant ‘takes part’ in the virtualworkshop within this set-up (see Figure 5).Instructed by a virtual Emio Greco, the

Full-body movement as material for interaction design

255

Dig

italC

rea

tivity,V

ol.

22

,N

o.

4

Dow

nloa

ded

by [

175.

143.

160.

248]

at 1

8:12

05

Janu

ary

2013

movements are tracked and visualised by FredericBevilaqua’s software gesture follower. The instal-lation gives you feedback on how you moverelated to pre-captured data or movement algor-ithms, and gives you a visual representation ofown movement and sonically provide feedbackon how you deviate from or align with the algor-ithms.

This is of interest for the interaction designer asit looks at the communicability of the specificitiesof movement, i.e. not only that the arm is raised,but how the arm is raised. This multimodalmediation of movement shows how a rich band-width of movement qualities and intent may becommunicated. The Double Skin/Double Mindinstallation is primarily a teaching tool, but asMarian Bleeker argues, one could look at it as anarchiving tool for the process.

New movement software allows for notationpractices in which the focus is on the multiplepotentialities of movement that the body holds. . . In this context, the computer becomes animportant element for choreographers, not as

an imaging device but as a medium in aprocess of emergence (Bleeker 2010, p. 3).

The installation shows that for a system torecognise, categorise and give meaningful feed-back, there is a need for pre-recorded informationor algorithms set to recognise qualities of move-ments. This is problematic in a communicationframe that is focused on more naturally occurringor unplanned interactions. This is because thecomplex meaning of a movement phrase is inter-preted according to the aforementioned spatialand temporal qualities of movement (frequencyand force, repetition and rhythm, alignment andposition). These are perhaps easy to interpret in ahuman to human context, but as we meet technol-ogy, it lacks the reading of these communicativenuances.

As discussed, there is such a wealth of pos-sibilities in the interpretation of movement,which makes it highly context dependent. Theconcept of immediacy allows the designer toexplore in a rapid, iterative manner whetherthe chosen movement aspects could serve a

Figure 5. The author using EC|PC’s Double Skin/Double Mind installation.

Hansen

256

Dig

italC

rea

tivity

,V

ol.

22

,N

o.

4

Dow

nloa

ded

by [

175.

143.

160.

248]

at 1

8:12

05

Janu

ary

2013

communicative purpose in a specific project orinstallation.

8.3 GenerationThirdly, the visualisations of movements need to begenerative, e.g. an augmentable visualisation thatwould respond to a change to one of its instrumen-tal parts or parameters. Working with visualisa-tions or sketches, the designer aims to find newfeatures, aspects or possibilities. It is argued thatthe designer does this through reorganisingelements of the visualisation with regard to a refer-ence frame (Cornoldi 1996). The detection of unin-tended features is a key element in generation ofcreative designs (Suwa et al. 1999). Stenning andOberlander (1995) discuss specificity and abstrac-tion in visualisation and how the focus on visualis-ing one feature, such as proximity, for example,could reveal unintended relationship with regardto other features. So the focus of the visualisationmay both limit and expand the design possibilities.The ability to discover unintended relationshipsand ideas from the visualisations is a skill and

draws on knowledge and practice in the field. Italso depends on the visuals or sketches being flex-ible or generative in the sense that they can bechanged, edited and re-assembled and still givenew, meaningful visualisations.

Digital technology has increasingly enablednew dissemination of approaches and communi-cations of the choreographic process due to bothease of documentation through video and newvisualisation possibilities from computer visionand computation.4 This development allows for anew kind of access to the making process. Inpart this is due to the multimodal possibilities ofdigital media in communicating physical move-ment. Despite the lack of a universal notationsystem, choreographers have developed theirown methods, practices and strategies. These areincreasingly being communicated within thechoreographic field as well as to related fields(deLahunta and Shaw 2008).

The dance company Siobhan Davies Danceand Coventry University have created a digital,online archive for the collective works of the

Figure 6. Screenshot of creative ‘ingredients’ for the making of ‘Bird Song’.Source: Image by kind permission # Siobhan Davies RePlay 2009.

Full-body movement as material for interaction design

257

Dig

italC

rea

tivity,V

ol.

22

,N

o.

4

Dow

nloa

ded

by [

175.

143.

160.

248]

at 1

8:12

05

Janu

ary

2013

company spanning thirty years (Siobhan DaviesRePlay 2009). They continue to develop tools togain insight into Davies’s choreographic process.For example, selected performances are presentedon the website in an interactive manner, encom-passing all parts, roles and ‘ingredients’ thatmade the piece (see Figure 6). Other key sourcesfor understanding movement are the rehearsaltapes (see Figure 7). The dancers in the companyhave frequently taped each other with handheldcameras during rehearsals, and this becomes docu-mentation, for instance, of a dancer workingthrough a movement ‘problem’ or challenge.When the viewer gains access to such aspects ofdance-making, the archive becomes an edu-cational tool informing creation as well as allow-ing for reflection on existing performances andtheir visual and kinetic documentation.

The site also presents Rotosketch, a softwareprogram which allows drawing on live video tobe saved as a record. This might become a usefultool for analysing and interpreting choreographicphrases (Whatley and Varney 2009). A furtherdevelopment of such a resource is a pen toolfor real-time annotation on video (Cabral andCorreia 2009). Such examples indicate that

digital technologies are increasingly beingapplied in the creative and meditational processesof choreography.

In the creation of archives, scores and edu-cational tools such as those mentioned above,choreographers enhance the communicability ofthe creative practice. These appropriations of tech-nology become interesting for interaction designas they develop from a documentation roletowards a more active, generative role in the crea-tive process. With augmentable visuals, the designof movement as a semiotic resource becomesuseful as it may be altered and explored accordingto the context in which it is applied.

9 Conclusion

In interaction design a new design space is nowreaching beyond the artefact or object, beyondthe screen and actions of touch and click. In thisdesign space, physical movement becomes impor-tant to understand as the designer shapes andresponds to movement as a design material. Ihave argued that in order to make novel inter-actions, the designer needs to be familiar withqualities of their making material and that inter-action design may benefit by drawing on contem-porary choreographic research.

To conceptualise, communicate and explorepossible designs, the interaction designer needsto be able to visualise, augment and try out phys-ical movement. Visualisation of physical move-ment in interaction design enables a conceptualexploration of novel communications. It hasbeen shown that accessibility, immediacy and gen-eration are important concepts in evaluating thisprocess of building knowledge of movement ininteraction design. These concepts enrich thevocabulary of social semiotics and allow forfurther designing of interactions involving full-body movement.

The Synchronous Objects project showed thatphysical movement may be visualised in awealth of modes, from a singular focus to highlycomplex. It also showed that the bodily represen-tation needed to retain the physical outline of thebody, as the widgets showed the communicative

Figure 7. Example of rehearsal tape from Plain Clothes, dancerHenry Montes.Source: Image by kind permission # Siobhan Davies RePlay2009.

Hansen

258

Dig

italC

rea

tivity

,V

ol.

22

,N

o.

4

Dow

nloa

ded

by [

175.

143.

160.

248]

at 1

8:12

05

Janu

ary

2013

aspects of movement was lost. The Inside Move-ment Knowledge project showed that movementcan be represented meaningfully with severalmodes of technology. When movement is commu-nicated multimodally it may become a resource, asit is then able to capture and communicate potenti-ality. The project also showed that the namingprocess in identifying the modes of communi-cation is essential in order to design a modalensemble, e.g. to communicate something mean-ingful.

When movement is visualised, relevantaspects are selected and interpreted. Here theanalysis of choreographic resources may giveus an indication of methods for visualising move-ment for interaction design. As Siobhan Davies’sweb-based archive shows, new digital technol-ogy enables a new immediacy and multimodalitywhere these resources go from critique toresource. In other words, they become generativetools and therefore useful to design practice aswell as research.

As argued above, in exploring how we may seethe particularities of physical movement as adesign material for interaction design, we mayenable novel interactions. By drawing on choreo-graphic research and its related practices, inter-action design may explore the communicativepotential of physical movement and let it informthe design process. Movement matters in oureveryday life; it may supplement, detract or con-tradict what we say, for instance. As these commu-nicative aspects now may be captured and appliedby technology, the interaction designer needs to beable to analyse, visualise and shape physicalmovements. By approaching movement as adesign material for interaction design, we mayallow for an understanding of how designerscould work with movement at a conceptualstage, and acknowledge the choreographicaspects of the interaction designer’s repertoire, asthe body becomes the interface.

Acknowledgements

Many thanks to Professor Andrew Morrison forconstructive feedback and encouragement.

Notes1 The notion of patterns in design comes from

Christopher Alexander’s work in urban planning inthe 1970s, where architectural patterns representways for supporting patterns of events thatfrequently occur in a space (Alexander 1979); ithas been used in interaction design as a way tocommunicate best-practice to recurring problems(Lowgren and Stolterman 2004, Lowgren 2007).

2 Norah Zuniga Shaw is a choreographer and Directorof Dance and Theory at ACCAD, and Co-CreativeDirector of Synchronous Objects.

3 The notion of immediacy here refers to a durationof time in an iterative design process, rather thana distinguishing notion of transparency betweenthe virtual and real (Kickasola 2006). It is closerto Schneiderman’s (1998) use of directmanipulation where an action has an immediatevisual effect (such as when driving a car andturning the wheel).

4 Such as the aforementioned eyesweb and gesturefollower as well as Merce Cunningham’sLifeforms, Troika Ranch’s Isadora and Meso’s vvvv.

ReferencesAlexander, C., 1979. The timeless way of building.

New York: Oxford University Press.

Ashbrook, D. and Starner, T., 2010. Magic: a motiongesture design tool. In: Proceedings of the 28th inter-national conference on human factors in computingsystems, 10–15 April 2010 Atlanta, Georgia. NewYork: ACM, 2159–2168.

Bal, M., 2002. Travelling concepts in the humanities: arough guide. Toronto: University of Toronto Press.

Bastien, M., 2007. Notation-in-progress. In: S. deLa-hunta, ed. Capturing intention. Amsterdam: EmioGreco | PC and Amsterdam School of the Arts,48–55.

Bevilacqua, F., et al., 2010. Continuous realtime gesturefollowing and recognition. In: S. Kopp and I. Wachs-muth, eds. Gesture in embodied communication andhuman-computer interaction. vol. 5934. Heidelberg:Springer Berlin, 73–84.

Bevilaqua, F., 2007. Momentary notes on capturing ges-tures. In: S. deLahunta, ed. Capturing intention.Amsterdam: Emio Greco|PC and AmsterdamSchool of the Arts, 26–31.

Birdwhistell, R.L., 1971. Kinesics and context: essayson body-motion communication. London: AllanLane, Penguin Press.

Full-body movement as material for interaction design

259

Dig

italC

rea

tivity,V

ol.

22

,N

o.

4

Dow

nloa

ded

by [

175.

143.

160.

248]

at 1

8:12

05

Janu

ary

2013

Bleeker, M., 2010. What if this were an archive? In: S.deLahunta, ed. Notation. Amsterdam: ARTI, ArtisticResearch, Theory and Innovation, 3–5.

Bolter, J.D. and Grusin, R., 1999. Remediation: under-standing new media. Cambridge, MA: MIT Press.

Broadhurst, S., 2007. Digital practices: aesthetic andneuroesthetic approaches to performance and tech-nology. New York: Palgrave Macmillan.

Buxton, B., 2007. Sketching user experiences: gettingthe design right and the right design. Amsterdam:Elsevier.

Cabral, D. and Correia, N., 2009. Pen-based video anno-tations: a proposal and a prototype for Tablet PCs.Proceedings of the 12th IFIP TC 13 international con-ference on human-computer interaction: part II, 24–28 August 2009, Uppsala. Springer-Verlag, 17–20.

Camurri, A., Mazzarino, B., and Volpe, G., 2004.Expressive interfaces. Cognition, Technology &Work, 6 (1), 15–22.

Cornoldi, C., 1996. Stretching the imagination: rep-resentation and transformation in mental imagery.New York: Oxford University Press.

deLahunta, S. and Shaw, N.Z., 2008. Choreographicresources agents, archives, scores and installations.Performance Research, 13 (1), 131–133.

deLhunta, S., 2007. The moment to question. . . DoubleSkin/Double Mind. In: S. deLahunta, ed. Capturingintention. Amsterdam: Emio Greco | PC and Amster-dam School of the Arts, 20–21.

Dourish, P., 2001. Where the action is: the foundationsof embodied action. Cambridge, MA: MIT Press.

Eikenes, J.O., 2010. Navimation: a socioculturalexploration of kinetic interface design. Oslo: OsloSchool of Architecture and Design.

eyesweb [online]. Available from: http://www.infomus.org/Eywindex.html [Accessed 30 August 2010].

Fogtmann, M.H., Fritsch, J., and Kortbek, K.J., 2008.Kinesthetic interaction: revealing the bodily potentialin interaction design, In: Proceedings of the 20th Aus-tralasian conference on computer-human interaction:designing for habitus and habitat, 8–12 December2008, Cairns. New York: ACM, 89–96.

Forsythe, W., 2008. Choreographic objects. In: MarkusWeisbeck for Ursula Bickle Stiftung, ed. Suspense.Zurich: JRP | Ringier Kunstverlag, 5–7.

Foster, S.L., 1995. An introduction to moving bodies.In: S.L. Foster, ed. Choreographing history. Bloo-mington, IN: Indiana University Press, 3–21.

Gaver, W.W., Beaver, J., and Benford, S., 2003. Ambi-guity as a resource for design. Proceedings of theSIGCHI conference on human factors in computingsystems, 5–12 April 2003 Fort Lauderdale, Florida.New York: ACM, 233–240.

gesture follower [online]. Available from: http://ftm.ircam.fr/index.php/Gesture_Follower [Accessed 30August 2010].

Goldschmidt, G., 1994. On visual design thinking: the viskids of architecture. Design Studies, 15 (2), 158–174.

Hall, E.T., 1966. The hidden dimension. New York:Anchor Books, Doubleday.

Hallnas, L., Melin, L., and Redstrom, J., 2002. Textiledisplays: using textiles to investigate computationaltechnology as design material. In: Proceedings ofthe second Nordic conference on human–computerinteraction, 19–23 October 2003, Aarhus.New York: ACM, 157–166.

Hansen, M.B.N., 2004. New philosophy for new media.Cambridge, MA: MIT Press.

Hayles, N.K., 1999. How we became posthuman:virtual bodies in cybernetics, literature, and infor-matics. Chicago, IL: University of Chicago Press.

Hollan, J. and Stornetta, S., 1992. Beyond being there.In: Proceedings of the SIGCHI conference onhuman factors in computing systems, 3–7 May1992, Monterey, California. New York; ACM,119–125.

Hoogenboom, M., 2007. Artistic research as anexpanded kind of choreography using theexample of Emio Greco | PC. In: S. Gehm, P.Huseman and K.v. Wilcke, eds. Knowledge inmotion: perspectives of artistic research and scien-tific research in dance, vol. 9. Bielefeld: transcript,81–90.

Hummels, C., Overbeeke, K.C., and Klooster, S., 2007.Move to get moved: a search for methods, tools andknowledge to design for expressive and rich move-ment-based interaction. Personal Ubiquitous Com-puting, 11 (8), 677–690.

ICKamsterdam [online]. Available from: http://www.ickamsterdam.com/index.php?art¼507&page¼2[Accessed 30 August 2010].

Inside Movement Knowledge [online]. Available from:http://insidemovementknowledge.net/ [Accessed 30August 2010].

Isadora [online]. Available from: http://www.troika-tronix.com/isadora.html [Accessed 30 August2010].

Hansen

260

Dig

italC

rea

tivity

,V

ol.

22

,N

o.

4

Dow

nloa

ded

by [

175.

143.

160.

248]

at 1

8:12

05

Janu

ary

2013

Jensenius, A.R., 2007. Action – sound: developingmethods and tools to study music-related bodymovement, Unpublished thesis, University ofOslo.

Kickasola, J., 2006. Contemporary media and the evol-ving notion of immediacy. Quarterly Review of Filmand Video, 23 (4), 299–310.

Knapp, M.L. and Hall, J.A., 2006. Nonverbal communi-cation in human interaction. Belmont, CA:Thomson/Wadsworth

Kozel, S., 2007. Closer: performance, technologies,phenomenology. Cambridge, MA: MIT Press.

Kress, G., 2010. Multimodality: a social semioticapproach to contemporary communication.London: Routledge.

Kress, G. and van Leeuwen, T., 2001. Multimodal dis-course: the modes and media of contemporary com-munication. London: Arnold.

Kress, G. and van Leeuwen, T., 2006. Reading images:the grammar of visual design. London: Routledge.

Lifeforms [online]. Available from: http://www.mer-ce.org/about/danceforms.php [Accessed 30 August2010].

Loke, L., et al., 2007. Understanding movement forinteraction design: frameworks and approaches. Per-sonal Ubiquitous Computing, 11 (8), 691–701.

Lowgren, J., 2007. Inspirational patterns for embodiedinteraction. Knowledge, Technology & Policy, 20(3), 165–177.

Lowgren, J. and Stolterman, E., 2004. Thoughtful inter-action design: a design perspective on informationtechnology. Cambridge, MA: MIT Press.

Manning, E., 2009. Relationscapes: movement, art,philosophy. Cambridge, MA: MIT Press.

Moeslund, T.B., Hilton, A., and Kruger, V., 2006. Asurvey of advances in vision-based human motioncapture and analysis. Computer Vision and ImageUnderstanding, 104 (2), 90–126.

Morrison, A., et al., 2009. Disquiet in the plasma.Digital Creativity, 20 (1), 3–20.

Munster, A., 2006. Materializing new media: embodi-ment in information aesthetics. Hanover, NH: Uni-versity Press of New England.

Noe, A., 2004. Action in perception. Cambridge, MA:MIT Press.

Noland, C., 2009. Agency & embodiment - performinggestures/producing culture. Cambridge, MA:Harvard University Press.

Penny, S., 2004. Representation, enaction, and the ethicsof stimulation. In: P. Harrigan and N. Wardrip-Fruin,eds. First person: new media as story, performance,and game. Cambridge, MA: MIT Press, 73–84.

Pfeifer, R. and Bongard, J., 2007. How the body shapesthe way we think: a new view of intelligence. Cam-bridge, MA: MIT Press.

Poggenpohl, S., Chayutsahakij, P., and Jeamsinkul, C.,2004. Language definition and its role in developing adesign discourse. Design Studies, 25 (6), 579–605.

Schechner, R., 2002. Performance studies: an introduc-tion. London: Routledge.

Schiller, G., 2006. Kinaesthetic traces across materialforms: stretching the screen’s stage. In: S. Broad-hurst, ed. Performance and technology: practicesof virtual embodiment and interactivity. Basing-stoke: Palgrave Macmillan, 100–111.

Schon, D.A., 1983. The reflective practitioner: how pro-fessionals think in action. Aldershot: Avebury.

Sengers, P. and Gaver, B., 2006. Staying open tointerpretation: engaging multiple meanings in designand evaluation. In: Proceedings of the 6th conferenceon designing interactive systems, 26–28 June 2006,University Park, PA. New York: ACM, 99–108.

Sevaldson, B., 2010. Discussions and movements indesign research. FORMakademisk, 3 (1).

Siobhan Davies RePlay [online]. Available at: http://www.siobhandaviesreplay.com [Accessed 30August 2010].

Shaw, N.Z., 2009. Animating choreography, [online].Available from: http://www.youtube.com/watch?v=dHqVJkgFNZY [Accessed 30 August 2010].

Shneiderman, B., 1998. Designing the user interface:strategies for effective human–computer inter-action. Reading, MA: Addison-Wesley.

Sigman, J., 2000. How dances signify: Trio A and themyth of ordinary movement. Journal of Philosophi-cal Research, 25, 489–533.

Stenning, K. and Oberlander, J., 1995. A cognitivetheory of graphical and linguistic reasoning: logicand implementation. Cognitive Science, 19,97–140.

Suwa, M., Gero, J.S., and Purcell, T., 1999. Unex-pected discoveries and s-inventions of designrequirements: a key to creative designs. In: J.S.Gero and M.L. Maher, eds. Computational modelsof creative design IV. Sydney: Centre of DesignComputing and Cognition, University of Sydney,297–320.

Full-body movement as material for interaction design

261

Dig

italC

rea

tivity,V

ol.

22

,N

o.

4

Dow

nloa

ded

by [

175.

143.

160.

248]

at 1

8:12

05

Janu

ary

2013

Suwa, M., et al., 2001. Seeing into sketches: regroupingparts encourages new interpretations. In: J. Gero, B.Tversky and T. Purcel, eds. Visual and spatialreasoning in design II. Sydney: Key Centre ofDesign Computing and Cognition, University ofSydney, 207–219.

Synchronous Objects [online]. Available from: http://synchronousobjects.osu.edu/ [Accessed 30 August2010].

Tseng, W., Scrivener, S., and Ball, L., 2002. The impactof functional knowledge on sketching. In: Proceed-ings of the 4th conference on creativity & cognition,13–16 October 2002, Loughborough. New York:ACM, 57–64.

vvvv [online]. Available at: http://legacy.vvvv.org/tikiindex.php [Accessed 30 August 2010].

Vygotsky, L.S., 1978. Mind in society: the developmentof higher psychological processes. Cambridge, MA:Harvard University Press.

Wachsmuth, I., Lenzen, M., and Knoblich, G., 2008.Embodied communication in humans and machines.Oxford: Oxford University Press.

Watzlawick, P., Bavelas, J.B., and Jackson, D.D., 1967.Pragmatics of human communication: a study ofinteractional patterns, pathologies, and paradoxes.New York: Norton.

Whatley, S. and Varney, R., 2009. Born digital; dance inthe digital age. International Journal of Perform-ance Arts and Digital Media, 5 (1), 51–63.

Wildschut, L. and Butterworth, J., 2009. Contemporarychoreography: a critical reader. London: Routledge.

Wright, P., Blythe, M., and McCarthy, J., 2008. Edi-torial. Personal Ubiquitous Computing, 12 (5),343–346.

Lise Amy Hansen is a designer and a PhDResearch Fellow at Oslo School of Architectureand Design (AHO). She has a BA (Hons),Central Saint Martins (CSM) and a MA in Com-munication Art & Design, Royal College of Art,London. She has taught at CSM and AHO. Thearticle is the first part of a three-phase PhDresearch project at AHO.

Hansen

262

Dig

italC

rea

tivity

,V

ol.

22

,N

o.

4

Dow

nloa

ded

by [

175.

143.

160.

248]

at 1

8:12

05

Janu

ary

2013