heartplotter: visualizing bio-data by drawing on paper · oday, many artis ted by pen plotte ing...

HeartPlotter: Visualizing Bio-data by Drawing on Paper

Abstract This paper reintroduces pen plotting to interaction and visualization design through the project of HeartPlotter. The HeartPlotter collects the user’s heartbeat data, maps the data into the pen movements, then presents the real-time variations in heart rate through its mechanical movements and sounds, and finally delivers the overall HRV information in a compact form as one drawing on paper. In this pilot study, we experimented with three basic mappings between data and visualizations by controlling the pen movement in speed, path and pen-down timing. The results show that the pen’s speed could present changing heart rate data in real-time and the pen’s path mainly affects the data visualization and the aesthetic of the plotted drawings. Finally, we discuss the possibility and limitations of the pen plotter used in information display and interaction design.

Author Keywords Biofeedback; Physical visualization; Digital fabrication; Bio-data

ACM Classification Keywords H.5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. Copyright is held by the owner/author(s). CHI'16 Extended Abstracts, May 07-12, 2016, San Jose, CA, USA ACM 978-1-4503-4082-3/16/05. http://dx.doi.org/10.1145/2851581.2892289

Bin YU Eindhoven University of Technology Eindhoven, the Netherlands [email protected] Rogier Arents 1e Pijnackerstraat 36, 3036 GJ Rotterdam, the Netherlands [email protected] Mathias Funk Eindhoven University of Technology Eindhoven, the Netherlands [email protected]

Jun Hu Eindhoven University of Technology Eindhoven, the Netherlands [email protected] Loe M. G. Feijs Eindhoven University of Technology Eindhoven, the Netherlands [email protected]

Figure 1: Heart Plotter

Late-Breaking Work: Extending User Capabilities #chi4good, CHI 2016, San Jose, CA, USA

1794

Figure 2: S

Figure 3: visu

System of Heart Plo

ualization process ofPlotter

IntroIn theprintincontinpen psprayeven tattractouchsteppfile anmoveexamcommsepar“moveplotterespo

This wbeyonpen phas beHoinkpatterusers’like pa[3], thMetapabstraa systrate dinterevisualwaterperso

tter

f Heart

oduction e 1980s, pen plottng devices. They wnuous lines they clotters use an acting little drops of today, many artiscted by pen plotteing paper. Typicaing through everynd translate the coments. Pen plotteple HPGL (Hewlet

mands from a seriaate movement sue from point A to er can be used as nding to a user's

work expands the nd the screen by dlotter. In many preen visualized in v

kis [1] visualizes trns in the water s’ accumulated actatterns on the wrihe authors develophone, transformiact painting. Furthtem called SweatAdata into 3D printeestingly, they also izing heart rate d

r fountain installatnalized sports drin

ters were popular were beloved becould draw. Unliketual pen to draw, ink over the papests and graphic ders for the effect olly, pen plotters dy command in a veommands into a sers can be controllt-Packard Graphical port in real timech as “pen up”, “point B”. Seen in an interactive dralive input.

view of visualizindrawing the data orevious studies, hvarious novel wayhe heartbeat throurface. Lee et al. tivity logs by engristband of activityoped a drawing mang the user’s bio-hermore, Khot et Atoms that transfoed material artifac proposed a new wata through a pubtion, which createnk [5].

vector-graphics ause of the fine

e inkjet printers, instead of er. As such, esigners are still of actual ink draw slowly by ector graphics series of pen led by for cs Language) e, achieving a “pen down”, or this light, a pen awing machine,

g bio-data on paper with a eart rate data

ys. For instance, ough vibration [2] visualizes raving patina-y trackers. In achine named data into an al. [4] designed orms the heart cts. More way of blic interactive s a

Recently, quirky, cusWilshen ansound devWinterbergbased on aan audiencin art instaexplored tinformatio

HeartPloIn this stuscreen gravariability.signal, calcthen presedrawings o

System The system2. The pulan Arduinothat receivprocesses (HRV) datamapped toHPGL form232C) porplotter, copaper.

Bio-data aThe pulse a samplingcalculationThen the p

pen plotters havestomizable interacnd Quinn [6] devevice based on a peg et.al [7] designa plotter, which dce’s facial expressallations, we founthe feasibility of pon display and pur

otter: a Visualiudy, we use a penaphic display to vi. HeartPlotter meaculates the heart ents the data by ton paper.

m structure of Hese signal is measo board. We deveves the pulse sign the signal to extra. In the programo the parameters

mat commands. Thrt, the commands ontrolling the pen

acquisition and pro signal is acquiredg rate of 256 Hz. ns, the signal is dopulse signal is filte

e been rediscoverective devices. Foreloped an electro-en plotter (Rolanded an interactive raws graphics ression. Except for apd that very few sten plotter as an inrpose-directed int

ization Tool of plotter instead ofsualize the heart asures the user’s rate variability dathe pen movemen

artPlotter is showured by a PPG senloped a Processin

nal from the ampliract Heart Rate Va

m, HRV parameter of the pen movemhen through a ser are passed to the acting on the surf

ocessing d with an optic PPGIn an effort to simown-sampled to 1ered by a 3th orde

ed as r instance, -mechanical d-DXY). installation

sponding to pplications tudies had nterface for eraction.

f Bio-data f an on- rate pulse ata, and nts and its

wn in figure nsor with g sketch ifier and ariability rs are ments in rial (RS-e pen face of the

G sensor at mplify the 128Hz. er


1795

Butter2Hz, sdetectpulse Inter two hsmootan exheartbIBI)/1time Hby therepresstatistmovinthis wfrom tSDNNvisual

MappiDifferfeedbdirectpen’s paperin thehumacan beparamspacespaceprepamappways,and ti

rworth low pass fiso the peaks of thted accurately. Ea peak value it savBeat Interval (IBIeartbeats. For betthen the IBI data ponentially weighbeats. The formul16. The initial valuHRavg (Heart Rate)e value of IBIavg. Isentation of the Htic feature, the stang window of 16 hway, we obtained tthe measured pul

N16, which will be mization.

ing between different from the desigack, the data is nly. In HeartPlotte movements and tr allows deviationse program, which gn control. The mae divided into two

meters spaces: bio and visualization, three parameterred as above mened to the moveme namely speed ofme of pen-down.

ilter with a cut-offhe signal (pulse peach time the systees the time accorI) is the time diffetter control of Hea by averaging theted moving averaa is: IBIavg = (15

ue of IBIavg is 600 ) value will be 600In the time-doma

HRV time series, wandard deviation heartbeats, namelthree clearly definse signal, HRavg, Imapped to pen m

rent parameter spgn of screen-baseot used to create r, the data is mapthe ‘interaction’ bs from the visualizgives rise to aesthapping process in o stages between o-data space, phy space, see figurers (HRavg, IBIavg anntioned. Then, theent space in threef movement, path The physical mov

f frequency of eak) can be em detects the rdingly. The erence between artPlotter, we IBI data with

age filter of 16 × IBIavg + ms. The real-000ms divided in

we calculated of IBI in a y SDNN16. In

ned parameters IBIavg and ovements for

paces ed visual visualization pped to the etween pen and zation designed hetics beyond HeartPlotter three sical movement e 4. In data nd SDNN16) are e data is e different of movement vements of the

pen determalso affectof movemthe paper,

Figure 4:

HeartPlottheartbeat transpareninteractiveact upon ethe extracmovementshould be interval, wprocess ofslow to bewe use ‘dovisualizatiodata can band lines oposition.

Design EWe conducof HRV witmovementand time o

mine the drawing t the user experieent influences the, but also the tone

: Division of mappin

ter is designed to data. To make thnt, we implement e mode for HeartPeach individual hected parameters ‘mts. This also requi finished within thwhich is about 500f a circle, arc, or pe finished within oot’ and ‘line’ as thon design. Througbe visualized on pof different size, t

Exploration cted a pilot study th the plotter. Thrt were experimenof pen-down. Thro

output on the pance. For instance,e thickness of the e of machine soun

ngs between parame

present the variathe interaction imm a heartbeat-triggPlotter, where the eartbeat, but one omodulate’ the penires that each mo

he shortest heartb0ms [8]. The drawpolygon shape wone heartbeat intee basic forms in tgh the pen movemaper with a sequethickness, length a

to explore the visree basic aspects nted, namely speeough the following

per but , the speed lines on nd.

eter spaces

tion of mediate and gered pen will or more of

n vement

beat wing uld be too rval, hence the ments, the ence of dots and

sualization of the pen

ed and path g three


1796

(a).

FigureHeartPlo

deep breanormal re

FigureHeartPlott

normal rdee

(b).

5: The speed-mappotter drawings. (a) dathing exercise (b) dlaxation, (c) after p

exercise

e 6: The path-mappier drawings. (a) (c)

relaxation, (b) (d) dp breathing exercise

explorand covisual

VisualThe spthicknmach40, wand 4pilot usoundpercespeed

In thedrawsspeedspeedcan peintera(beat distingwhite 0.3mmexammeasuconditrelaxadrawipale athickeoscilladrawiheart heart

(c).

ing uring during hysical

ing during

during e

rations, we wouldonstraints of the pization and intera

lization 1: Based opeed of the pen mness of the lines, bine sound. The sphere 0 is the slow0 is the fastest spuser test, we found changes linearly ived by users in a

d 20.

e speed-mapping vs straight lines of d is modulated by d indicates a fast herceive the variat

action, the HRavg v per minute) is maguishable speed ( print paper of 80m tip size for the ple of drawings prured from the samtions: (a) during dation (c). after phyngs, the lines genand the ones with er and darker. Draation in heart rateng (c), the relativ rate after exercis rate, but with a lo

like to see the dyplotter from the paction.

on speed-mappingmovement not onlybut also varies thepeed parameter rawest speed of 13.8peed of 468.7 mmnd that the pitch o with the speed, b

a limited range fro

visualization procsame length (10 m the user’s heart rheart rate. To ensions of heart ratealue ranging fromapped to a hearin(0-20). We selecte g/m2 and a blackuser test. Figure roduced with hearme participant unddeep breathing exysical exercise. Fr

nerated with a fast a slower speed teawing (a) shows ae caused by deep vely pale lines shose. Drawing (b) reow variation.

ynamic qualities perspectives of

g y affects the e tone of the anges from 0 to 8 mm/second, m/second. In a of the machine but can only be om speed 0-

ess, the plotter mm). The pen’s rate. A high sure the user during the

m 50 to 120 bmp g-ed the normal k felt pen of 5 shows one rtbeat data der different xercise (b). in rom the t speed look end to be an obvious breathing. In w an increased

eflects a normal

VisualizatiIn HeartPlthat mighta move is the path omoves, dirmove. BasmovementIn this stucomplexitya simple sdirection adata.

We startedcircular baas the bason a flatbeinteractiveradiating ocenter. WeIBIavg to ttwo adjacedrawings prelaxation perceive thsession dirdrawings. (c), drawinrhythm byshape and

VisualizatiBesides thdraw discrDown-Penthe path o

ion 2: Based on paotter, each heartbt consist of severa a straight-line moof a movement: strection of each msed on the numbet could be a simpl

udy, we do not inty of graphic desigstraight line at oneand length of the

d out with the simar chart. Again, a sic element. As Heed plotter, to keepe process, we madoutwards, and eace tried two mappithe length of the ent lines. Figure 6produced by hear and deep breathihe overall HRV infrectly from the sh Compared to subng (b) and (d) shoy the regular fluctd the denseness o

ion 3: Based on tihe continuous linerete dots by a conn-Up. The positionof movement; unli

ath-mapping beat triggers one al moves. The basotion. Four factorstart position, numove, and length o

er of moves, the ple line or a compleend to explore then; instead, the ple heartbeat, wherlines are modulat

mple visualization single straight lin

eartPlotter is desigp the continuity ofde the bar chat inch line starts fromng methods: mapline or to the spa

6 shows path-maptbeats data duringing exercise. We cformation of the e

hape and the textubtle changes in draow an obvious heuations on the ouf the texture.

ime of pen-down s, the pen plotter

nsecutive moveme of a dot is determike line drawings,

movement sic form of s determine

mber of of each path of a ex shape. e otter draws

re the ed with IBI

design of a e is used gned based f the a circle

m the pping ce between pping g the can entire ure of awings (a) art rate tline of the

r can also ent of Pen-mined by the pen


1797

(a).

FigureHeartPlo

(b) aftduring

(b).

e7: The dot drawingtter. (a) during relaer physical exercise deep breathing exe

does nreachup modot nepen isa largdown size oavaila

Firstlyon theprograthat, dots c(shortlarge heartbdown.to 120state,small immemethomovedown pen isOthera newtest aheart In draprodutime. exercby va

(c).

gs of xation, , (c)

ercise

not touch the papes the specific poovement is quite fearly as small as ts touching the paper dot on the pap without movemef the dot, which b

able for data visua

y, we tested the ee dot’s size. The dam from 100ms tbased on the darkcan only be identifter than 500ms), (longer than 1000beat intervals (IBI. The typical huma00ms during deep fluctuations of IB range of 700ms-1diate and natural od. When HeartPlos the pen to a new on the paper. Whs lifted immediaterwise, the pen keew heartbeat is detere shown in figure rate level with a awing (b), after thces a set of small As we expected, ise, the enhancedried dots, see figu

per during the movsition. Typically, tfast (less than 100the pen tip. And tper, the more ink per. Therefore, thent on the paper dbecomes another palization.

ffect of the pen-dduration is controlo 1800ms. The rekness and size of fied into three typmedium (500ms-0ms). Then, we mIavg) to the duratioan IBI data rangep breathing. And dBI are maintained 1000ms. Here, to mapping, we useotter detects a new position and puhen IBIavg is less thly with the defaul

eps in touch with tected. The resultse 7. Drawing (a) sset of similar med

he exercise, the fal dots by a very shduring a deep bre

d heart rate rhythmure 6(c).

ves until it the pen-down-0ms), leaving a he longer the will spread into e time of pen etermines the parameter

down duration led in the

esult shows the dot, the

pes: small -1000ms) and map the on of pen-es from 500ms during normal within the achieve an

e a simple ew heartbeat, it ts the pen han 700ms, the t speed. the paper until s of the user shows a normal dium-sized dots. ast heart rate hort pen-down eathing m is reflected

DiscussiThe pilot spen movemmovementrate data bThis gives hearing sypitch chanvisualizatioand darknbe dividedthe bold linmovementone moveminterval. Fslowest (s7mm in a interval (5important design, beof the movgreater fleinformatiodirection aThe time ovisualizatiolimited; onsmall, med

Comparedplotter hasinteractivemodal (phthrough thof the macSecondly,

ion study explored thrment and their eft can be an effectbased on the vari users an immediystem can only disnges caused by spon, the speed maess of lines. Gene

d into three groupne can only be get. The speed also ment, which shou

For example, if thepeed value = 0), straight line withi500ms). Therefore factor to be consecause it limits thevement path. Theexibility in visualizon display. The chand length, can beof pen-down mainon. The capacity only three ‘levels’ odium and large do

d with on-screen vs the following vae output. Firstly, ahysical-visual-audihe mechanical mochine, and the dra the working of a

ree basic parametffects on drawingstive way of presened tones of machate audio feedbacstinguish a limitedpeed modulation. Iinly influences the

erally, the lines ons based on thicknenerated with a ve determines the duld be within one he speed were set t the pen would onin the shortest hee, the pen’s speedidered in the intee length and the ce path of movemezation and a good anges of lines, sue easily perceivednly affects the dot of presentation ofof data can be expots.

visual displays, weluable characterisa pen plotter is a itory) output interovements of the pawings on the papplotter is a two-st

ters of the s. Speed of nting heart ine sound. ck. But our d range of In the e thickness n paper can nesses and ery slow duration of heartbeat to the

nly move eartbeat d is an raction complexity nt has a capacity of

uch as the d visually. size in the f dots is pressed by

e found the stics as an multi-rface en, sound per. tage


1798

transformation process: from data to pen movements and from pen movements to the effect of ink on the paper. This gives plotters a good controllability of data representation but also flexibility and versatility in rendering the output artistically. Thirdly, plotters not only display the output data in real time, but also present the overall information with a whole painting after the interaction. We believe there is a great, untapped potential of pen plotters in the HCI field. The pen plotter can be used for purpose-directed interaction, presenting various types of data in a physical-audio-visual modality. It could act upon various inputs that are live and interactive, providing the users with a rich interactive experience. Just as one user said: “It brings back memories of me sitting in my father’s office looking at and listening to this ‘magical’ machine!” As we can see, there are still some weaknesses with the plotters, such as the limited drawing speed, special requirements on pen and paper, and inevitable machine sounds. But we think these difficulties can be understood and used well, can turn out to be valuable factors in interaction design.

Conclusion and future work This paper presents HeartPlotter, a device to visualize heart related information through integration with a pen-plotter device. HeartPlotter presents heart rate variability in a dynamic drawing process, where the pen moves with different paths and touches paper with ink gradually generating visualization. We hope the findings of this study could breathe new life into “obsolete” plotters for their interactive aspects and attract more interest from researchers in the field of HCI. Future research on HeartPlotter will explore more interactive modes and visualization designs based on the findings from this pilot study.

References 1. Hoinkis, M. 2012. Herzfassen: a responsive object.

In CHI'12 Extended Abstracts on Human Factors in Computing Systems . 975-978.

2. Lee, M. H., Cha, S., & Nam, T. J. 2015. Patina Engraver: Visualizing Activity Logs as Patina in Fashionable Trackers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '15) 1172-1182.

3. Šimbelis, V., Lundström, A., Höök, K., Solsona, J., & Lewandowski, V. 2014. Metaphone: machine aesthetics meets interaction design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14) 1-10.

4. Khot, R. A., Hjorth, L., & Mueller, F. F. 2014. Understanding physical activity through 3D printed material artifacts. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems . 3835-3844

5. Khot, R. A., Lee, J., Hjorth, L., & Mueller, F. F. 2015. TastyBeats: Celebrating Heart Rate Data with a Drinkable Spectacle. In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction. 229-232.

6. Oliver Wilshen, Niall Quinn, REMAP (2014), http://www.signal-to-noise.co.uk/portfolio/remap/

7. Michel Winterberg “trial, be a plotter” (2015), http://www.michelwinterberg.ch/

8. Lehrer, P., & Vaschillo, E. 2008. The Future of Heart Rate Variability Biofeedback. Biofeedback 36, 1, 11-14.

Acknowledgements The authors would like to thank the China Scholarship Council, Grafisch Atelier Daglicht, and Creative Industries Fund NL for their support on the project.


1799

heartplotter: visualizing bio-data by drawing on paper · oday, many artis ted by pen plotte ing...

Documents