Grasp Sensing for Human-Computer Interaction

Raphael WimmerUniversity of Munich

Amalienstr 17 80333 Munich Germanyraphaelwimmerifilmude

ABSTRACTThe way we grasp an object depends on several factors egthe intended goal or the handrsquos anatomy Therefore a graspcan convey meaningful information about its context In-ferring these factors from a grasp allows us to enhance in-teraction with grasp-sensitive objects This paper highlightsan grasp as an important source of meaningful context forhuman-computer interaction and gives an overview of priorwork from other disciplines This paper offers a basis andframework for further research and discussion by propos-ing a descriptive model of meaning in grasps The GRASPmodel combines five factors that determine how an object isgrasped goal relationship between user and object anatomysetting and properties of the object The model is validatedboth from an epistemological perspective and by applying itto scenarios from related work

ACM Classification KeywordsH52 Information Interfaces and PresentationUser InterfacesH12 UserMachine Systems Human factors

General TermsHuman Factors Theory

Author Keywordsgrasp recognition tangible user interface grasp meaning

INTRODUCTIONDuring evolution humans gained the unique ability to grasptools and apply them skillfully Nowadays a multitude ofversatile or specialized tools extend our manual abilitiesOftentimes Human-Computer Interaction (HCI) also relieson our ability to grasp objects - be it a computer mouse atangible user interface or a mobile phone Given that wespend a reasonable amount of time grasping objects whileusing them it seems straightforward to use the contact be-tween device and hand for transmitting information to theuser Haptic feedback sends short cues to users via actua-

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page To copy otherwise orrepublish to post on servers or to redistribute to lists requires prior specificpermission andor a feeTEIrsquo11 January 22ndash26 2011 Funchal PortugalCopyright 2011 ACM 978-1-4503-0478-81101$1000

AnatomyAnatomy

Goal

SettingSetting

Relatio

n-

ship

Properties Properties

Figure for Grasp Sensing for Human Computer Interaction - copyright transfer to ACM only for figure with this notice

Figure 1 The way we grasp an object conveys meaningful informationSeveral factors determine a grasp Our GRASP model groups theminto five general factors Goal of the grasp Relationship between userand object (eg knowing that an object is very expensive) Anatomyof the user (including neuropsychological processes) Setting (ie envi-ronment in which the grasp takes place) and Properties of the objectFactors were chosen to represent orthogonal types of meaning

tors that stimulate tactile receptors in the skin Such cuesand status information can also be represented as changes inweight distribution or shape [9 10] However a grasp ac-tually allows bi-directional data transfer The way we graspan object also conveys information about us our goals andother aspects Inferring these aspects from the grasp can al-low us to enhance interaction with graspable user interfacesThe way we grasp an object is determined in part by whatwe want to do with it Knowing the intended goal of a graspa grasp-sensitive object could support this goal For exam-ple a mobile phone would infer from your grasp that youwant to call someone - and would present the call list Abottle might know that you want to drink from it and openitself - because you grasped it near its center of gravity Anda power drill might refuse to power on because you are notholding it safely enough With each grasp we convey in-formation - and given that we grasp objects all the day thisis a lot of information we produce Currently almost all ofthis information is ignored however The aim of this paperis to raise awareness to grasping as an important source ofmeaningful context for human-computer interaction give anoverview of existing work from other disciplines and pro-vide a basis for further research by proposing a descriptivemodel of meaning in grasps (see Figure 1)

This paper is structured as follows In the next section I givean overview of related work from robotics medicine and

neuropsychology Then I discuss technical foundations ofgrasp sensing presenting a three-stage process for capturingidentifying and interpreting grasps Afterwards I present adescriptive model of the meaning of grasps and discuss itsvalidity applications and limitations

RELATED WORKHumanrsquos ability to purposefully grasp arbitrary objects hasbeen subject of intensive research in different disciplinesThe biomechanical properties of the human hand are subjectof medical research [20] In neuropsychology researchersinvestigate how grasps are planned and controlled [6] Inrobotics researchers investigate how robotic hands may re-liably grasp arbitrary objects [25] MacKenzie and Iberall[17] give a comprehensive but slightly dated overview of re-search in these fields

DefinitionsIn this paper a grasp is defined as a ldquofirm hold or griprdquo [1]Accordingly grasping means performing a graspStable grasps can be divided into two different types [21]Force closure - whereby the object is held by opposing fin-gers exerting pressure against the objectrsquos surfaceForm closure - whereby the object is enclosed by the handso that it cannot move even if no force is exerted by the fin-gers Often the terms grasping and prehension are used inter-changeably [17] For others prehension is the coordinatedact of reaching for an object and grasping it [29] Kang andIkeuchi state that a grasping task is composed of pre-graspphase static grasp phase and manipulation phase [13]

Classifications of GraspsSeveral classifications of grasps have been proposed overtime Schlesinger [24] classified grasps based on the shapesof objects they were applied to Napier [20] proposed a sim-ple classification of grasps into power grips which generallyinvolve the palm and precision grips which only employfingers and thumb Cutkosky and Wright [7] use a tree hier-archy for classifying grasps beginning with Napierrsquos powerand precision grasps Kang and Ikeuchi [13] recognized thatthe aforementioned models lack a formal description of thegrasps which makes them unsuitable for automated grasprecognition They proposed the contact web as a formalclassification of grasps that is based on contact points be-tween object and fingers Feix at al condensed 147 graspdescriptions found in a review of 17 publications into 33unique grasps [8] These were classified based on involvedfingers Napierrsquos distinction opposition type and thumb po-sition

MEANING IN A GRASPExisting classifications focus on defining groups of visuallyor functionally distinctive grasps However a multitude offactors contribute to a grasp which are not considered inthese classificationsIberall and MacKenzie offer an abstract model of grasping[12 17] They describe grasping as a black box with agoal and an object as inputs and a specific grasp as out-put Several additional constraints are given that affect theoutcome like the placement and orientation of the object

However MacKenzie and Iberall do not explain the formaldifference between inputs and constraints These constraintsare grouped by MacKenzie and Iberall [17] as follows

bull High Level SocialCultural Motivational InformationalFunctional

bull Physical Object Properties BiomechanicalMechanical

bull Sensorimotor Neural Perceptual AnatomicalPhysiologicalEvolutionaryDevelopmental

However for HCI the question is not how a grasp shouldlook like for a certain combination of object and goal (hu-mans are quite good at figuring it out themselves) Instead itwould be useful to know what a grasp actually means Thatis what can a grasp tell us about the context it is happeningin and what can we do with this context intformation Fora certain grasp - which we can capture by different means -we want to know which factors contributed to it

GRASP SENSINGThis section explains why it is acceptable and helpful to onlycapture stable grasps and defines three interpretative stepsfor grasp-sensing systems capturing the grasp identifyingit and interpreting it (Figure 2)

capture

identify

interpretleft hand Alice

wants to call somebody

grasp signature

grasp classification

Figure for Grasp Sensing for Human Computer Interaction - copyright transfer to ACM only for figure with this notice

Figure 2 General workflow for grasp sensing sensors capture a graspsignature which gets mapped to a specific group of grasps Finallyinterpreting the grasp generates meaningful context

An important simplificationGrasping is a complex action Depending on the point ofview it also includes initial reaching movements and exploratoryadjustments until we find a stable grasp Sometimes wemove through a number of different grasps even involvingdifferent hands - eg when picking up an object and puttingit somewhere else Complex hand movements are neededto transform one stable grasp into another However in thefollowing I will use the term grasp to refer only to singlestable static grasps performed by a human hand This has aformal and a technical reason

bull a series of grasps can be broken up into a number of stablegrasps The transitional hand movements between thosestable grasps primarily depend on the previous and nextgrasp Therefore these transitions add only very little in-formation about the meaning of a grasp 1 Excluding those

1An exception would be a slippery object that is held in a form-closure grasp The (stable) grasp does not need to adress the slip-pery surface However any transitional hand movement wouldneed to take this into account

transitions from our analysis removes much complexitywhile forfeiting only little information

bull the aforementioned complexity of transitional movementsmakes them hard to interpret using heuristics or machine-learning Concentrating on stable grasps makes it mucheasier to extract meaningful patterns

Step 1 Capturing GraspsGrasps can be captured by either capturing the grasping handor by equipping the object with a grasp-sensitive surfaceBoth approaches have unique advantages and disadvantagesFor everyday use objects with grasp-sensitive surfaces areprobably more convenient and socially acceptable however

Instrumenting the UserFor tracking fingers and hand often an optical or magneti-cal tracking system is used A marker or receiver needs tobe attached to each finger This method allows for a high-resolution capture of hand pose and position It does notprovide any information about contact forces between fin-ger and object Therefore it can not detect whether a fingeris touching the object or not Such tracking systems needto instrument the user Additionally capturing grasps worksonly within the tracking infrastructure Therefore such sys-tems are of limited use for real-world applications Howeverthey are very flexible and therefore well suited for labora-tory studies External tracking systems also provide a high-resolution ground truth for other technologiesInstead of using markers for defining finger joints some ap-proaches use computer vision techniques for an image of thehand pose to a grasp [31] Another approach that does notrequire a tracking infrastructure are data gloves These arethin gloves - often with a mesh surface on the palmar side- that are equipped with strain sensors These sensors al-low capturing each finger jointrsquos angle From these a handpose can be derived [11] These gloves are less obtrusivethan tracking markers However they limit tactile percep-tion Force-sensing with gloves is difficult as force sensorsattached to the fingertips obviously result in greatly reducedtactile perception An alternative way of determining graspforces is to capture images of the finger nail and interpretchanges in blood flow that can be observed there [19]As the fingers are controlled by muscles in the forearm someinformation about the hand pose can also be gained frommeasuring changes in these muscles or the wrist shape [22]Another option is not to capture grasps but only detect whichobject is held This can be done by attaching RFID tags toall objects and strapping an RFID reader to the userrsquos wrist[4] This method does not provide any information about theactual grasp howeverInstrumenting the users allows for easily capturing graspsfor a multitude of object shapes The objects need not bemodified However putting markers on hands would severlyimpede everyday manual tasks Additionally in most appli-cations the object itself shall react to a grasp Therefore thegrasp information would first need to be transferred to theobject - either wirelessly or by electrical coupling [34] Thiswould require a standardized communications protocol forgrasp information

Grasp-sensitive SurfacesGrasp information can also be captured by the grasped ob-ject itself To this end the object needs to be equipped witha grasp-sensitive surface that can capture the contact areabetween object and hand Grasp-sensitive surfaces requiremore engineering effort than external tracking solutions andusually do not offer identification of single fingers or digitsImplementing a grasp-sensitive surface is more difficult thanimplementing a touch-screen for two reasonsShape Sensing Instead of single or multiple touch pointsa grasp-sensitive surfaces should capture multiple complexshapes and possibly additional information like pressure There-fore common commercial touchscreen technologies like (ana-log) resistive[16] or self-capacitance [3] touch sensing can-not be used Digital resistive [23] or mutual-capacitancetechnologies are better suited for capturing contact areas shapesAdditionally the driver software needs to be able to repre-sent complex shapesSensor Shape The sensing surface is usually non-planarand continuously wrapped around the whole object Whileit is possible to embed only a few sensors at strategic posi-tions within the surface[5 33] often large parts of the sur-face shall be made grasp-sensitive This requires flexible orvery small sensors Additionally a normal cartesian coor-dinate system - as used in displays and touch input - maynot correctly represent the sensing surface Possible alterna-tives are spherical coordinate systems[15] or a relative map-ping[32]

In research prototypes a multitude of sensor technologies areused for making surfaces grasp-sensitive Often researchersbuild their own capacitive sensors [14 27 33] Other pro-totypes use resistive pressure sensors [15 30] impendancesensors [18] or optical sensors [5 32] A comparison of theinherent properties of the different types of sensors can befound in [32]Capturing a grasp results in a grasp signature a digital rep-resentation of contact points or digit positions

Step 2 Identifying GraspsOnce a grasp signature has been captured the system needsto derive the grasp that caused this signature [13] This iscommonly done by mapping the signature to a certain cat-egory of grasps either using heuristics or machine learn-ing The classification algorithm to be used depends bothon the format of the sensor data and the type of informa-tion that is needed in the interpretation stage Commonlyused classifieres include support vector machines [14 26]Bayesian classifiers [14 26] and Hidden-Markov-Models[27] Grasp categories can be defined formally using egone of the models described above More often grasp cat-egories are defined by example To this end a machine-learning classifier is trained with grasp signatures of graspsthe developer deems similar Without a set of formally de-fined grasps however it is not possible to compare differ-ent classifiers and sensing technologies In several casesgrasps with completely different signatures should actuallybe interpreted as the same grasp For example when grasp-ing a uniform sphere the grasp signature greatly dependson the rotation of the sphere In these cases an appropri-

ate mapping needs to be found that ignores certain featuresof a grasp Wimmer et al [33] recognize different types ofleft-handed and right-handed grasps using slightly modifiedheuristics for both Kry et al [15] use spherical harmonicsfor rotation-invariant grasp classificationsClassifying a grasp signature results in a grasp classification- for example a list of grasps that might have generated thegrasp signature

Step 3 Interpreting GraspsThe final step in graps sensing is to determine what a cap-tured and classified grasp actually means What is to beconsidered meaningful depends on the application For au-thentication purposes [30] it is relevant whether the graspbelongs to a certain category - whereby different categoriesrepresent different users When using grasps to switch an ob-ject between different modes [27] the mapping from a cer-tain grasp category to a mode provides meaning Howeverexisting grasp-sensitive systems capture only one meaning-ful aspect of a grasp (see below) ignoring the other aspectsWhile such systems attribute a certain meaning to a graspthe actual meaning of a grasp is ultimately defined by theuser This leads to unexpected behavior if the system at-tributes a meaning to a grasp that the user did not intendFor example a grasp-sensitive mobile phone might interpreta certain grasp as the intention to switch to camera mode[14] However the user might just hold the phone this wayfor a variety of other reasons - and would be annoyed if thephone suddenly changed into a different mode

Insofar it seems advisable to try to capture as much informa-tion about the meaning of a grasp as possible In the follow-ing section I look at meaningful factors that influence whichgrasp is chosen

GRASP - A MODEL OF MEANING IN GRASPSAs stated above a model of meaning within a grasp is neededThe model I present here is based on the model by MacKen-zie and Iberall that is described in the second section Thenew model considers the constraints and inputs they describedand re-organizes them into meaningful factors An importantsimplification is abstracting the user hiding much of the neu-ropsychological and anatomical complexity This probablydoes not remove meaningful information as most statementswe might be able to make about such sub-factors will nothelp us in improving the user interface Being aware of thisabstraction allows us to replace it with a more fine-grainedrepresentation if needed

In the following I present GRASP a model of human grasp-ing that describes five meaningful factors These factors allinfluence which grasp a user applies and each of them repre-sents a different group of meaningful information Figure 3illustrates these factors All factors are independent of eachother The validity of this model is discussed in the nextsection

GoalA very important factor determining how to grasp a givenobject is the goal that is to be achieved by the grasp Goals

are all factors that cause a grasp to be initiated Goals canbe divided into sub-goals As goals are essentially meaningthey are discussed here in more detail than the other factorsI propose grouping goals by two dimensions implicit vsexplicit grasps and primary vs supportive grasps Table1 shows examples for each combination of implicitexplicitand primarysupportive grasps

implicit explicitprimary user picks up mo-

bile phoneuser holds phonehorizontally to in-voke photo mode

supportive user holds phonewhile typing ontouchscreen withother hand

user holds phonescrolling a mapwith the otherhand zoomingby squeezing thephone

Table 1 Examples illustrating the distinction between implicit and ex-plicit grasp interaction respectively primary and supportive grasps

Usually a grasp is implicit meaning that its goal is to ma-nipulate an object Implicit grasps are similar for grasp-sensitive and non-grasp-sensitive objects Explicit grasps areconducted primarily in order to trigger a certain effect in agrasp-sensitive object For example a user might authenti-cate himself to the device using a certain grasp or switchthe mode of an object by holding it in a certain way Like-wise there is a difference in the userrsquos intention for primaryand supportive grasps A primary grasp is a grasp that isexecuted with the intention to manipulate the grasped ob-ject For example a primary grasp is used for moving acomputer mouse A supportive grasp serves to fixate or po-sition an object in order to interact with it using eg theother hand or fingers that are not involved in grasping Forexample holding a mobile phone while typing a short mes-sage is a supportive grasp These distinctions are not ran-dom but are needed to judge the amount of meaning gainedfrom a grasp An explicit grasp always conveys informa-tion about the userrsquos intention It can and should always beseen as meaningful The user expects a reaction An im-plicit grasp may convey some information about the userrsquosintention However this intention is not necessarily aimedat interacting with the object A user might just want toput away an object and would be annoyed if it reacts to thegrasp Therefore determining whether a grasp is implicitor explicit is necessary for interpreting the grasprsquos meaningLikewise a supportive grasp conveys less meaning than aprimary grasp as the main focus of the interaction lies noton the grasp but on the action that is supported by it Addi-tionally a supportive grasp is restricted by the requirementsof the main interaction and usually can not be modified aseasily as a primary grasp One might use this distinction toassign different meanings to a grasp For example whenholding a mobile phone to the ear (primary grasp) differentgrasps can launch actions When holding the phone whileinteracting with its touchscreen using the other hand differ-ent supportive grasps might only switch between modes oradjust viewport properties

Figure 3 Examples for each of the meaningful factors that contribute to a grasp A user might hold a screwdriver differently depending on the Goalhe wants to achieve with it The mental Relationship to an object determines how we grasp it In the illustrated case a paper towel that belongs tooneself is held differently than one found on the floor Depending on our Anatomy we need to apply different grasps to achieve the same effect Dueto different Settings a bottle in a shelf has to be grasped differently than a bottle in a crate The Properties of an object for example its size have amajor influence on the grasp to be used

RelationshipFeelings like disgust fear or anger can also influence theway we grasp Relationship includes all non-physical fac-tors that apply to a combination of user and object The re-lationship between user and object can be different for everyuser-object combination For example many people seem tofind a used paper towel disgusting They would only pick itup with a pinching grasp trying to avoid contact with any-thing attached to it However if the used paper towel belongsto oneself one might just pick it up like any arbitrary pieceof paper Unlike anatomical properties the relationship canalso change over time like when someone overcomes a fearof spiders and is able to touch one How a feeling changesa grasp is also determined by the knowledge about an ob-ject An angry person might unconsciously grasp the steer-ing wheel more tightly but would not do the same with anegg These effects are not solely properties of the grasp-ing person or the object but depend on the combination ofgrasping person and grasped object Relationship probablyhas little impact on most grasps However this factor hasnot been explored in detail so far

AnatomyNot every hand grasps the same way Anatomy includesall factors that are inherent to the grasping personrsquos bodyObviously differences in palm size finger length or fingercount result in slightly different grasps Additionally differ-ences in sensorimotor control of grasps also causes differ-ent grasps Differences in peoplersquos grasps might be used forautomatically identifying the person grasping an object Apower tool might also detect insufficient grasp forces for astable grasp and warn users that they have to hold it moretightly

SettingThe environment within which a grasp takes place also af-fects its outcome Setting includes all factors that pertain tothe environment in which a grasp takes place ie all physi-cal factors that are independent of user and object For exam-

ple the grasp used for pulling a bottle out of a crate differsfrom the one used for picking the bottle from a shelf Ad-ditional environmental factors might include lighting condi-tions space constraints or temperature

PropertiesThe properties of the object to be grasped naturally have ahuge influence on the grasp to be used Properties are allfactors that are intrinsic to the object This includes shapesize surface texture weight and weight distribution or sur-face temperature

VERIFYING GRASPThis section explains why GRASP is indeed a model andthat it fulfils formal requirements of a model Finally severalexamples highlight how GRASP can be applied

Formal ValidityThere are a variety of views what a model actually is Bailer-Jones offers the definition[2] ldquoA model is an interpretativedescription of a phenomenon that facilitates access to thatphenomenonrdquo GRASP is a model according to this def-inition as it describes a certain aspect (the meaning) of aphenomenon (a grasp) However what are the criteria fora good model I have chosen Vaandragerrsquos definition of agood model [28] While it pertains to models in computersystem analysis it is a concrete checklist that with plausiblerequirements Vaandrager lists seven properties of a goodmodel but notes that some of these contradict each othersometimes These properties have been rephrased in the fol-lowing to keep them succinct

According to Vaandrager a good model should

have a clearly defined object of modelingGRASP pertains to static human grasps

have a clearly specified purposeGRASP defines meaningful factors contributing to a grasp

clearly show the links between model and empiricalworldeach factor is clearly tied either to a clear concept (GoalRelationship) or a physical object (Anatomy Setting Prop-erties)

be truthful ie correctly represent relevant propertiesof the object of modellingAs a model is a simplification of a complex matter there isnot only one correct model Additionally a model inherentlyoversimplifies certain aspects of a phenomenon Thereforea model can not be completely truthful (as acknowledged byVaandrager) Additionally meaning is a property that is at-tributed to a phenomen by humans Therefore it is hard totell if our distinction of meaningful factors in a grasp is cor-rect However we can look at the meaning other researchersattribute to a grasp and see whether it matches to the factorswe defined

bull Veldhuis et al[30] solely try to derive the Anatomy of auser from a grasp

bull Taylor and Bove[27] solely try to derive an explicit Goalfrom a grasp

bull Kim et al[14] solely try to derive an explicit Goal from agrasp

bull Wimmer and Boring[33] solely try to derive an implicitGoal from a grasp In the two cases without a grasp aSetting is guessed from the sensor data

bull SpherUID - a yet unpublished project - tries to determinethe userrsquos position in relation to the object (Setting) froma grasp

bull Kry et al[15] solely tries to derive an explicit Goal fromthe grasp

Therefore our model seems to correctly identify at least GoalAnatomy and Setting as meaningful factors of a grasp I donot know of any application deriving Relationship or Prop-erties from a grasp however

be as simple as possible - but not too simpleFor GRASP this would mean that none of factors can beomitted Otherwise the model could be simplified That isfor each factor at least one situation exists where the grasp isprimarily determined by the factor Examples of such situa-tions are given in the descriptions of the factors The require-ment also indicates that none of the factors may be partiallycovered by another (ie they should be orthogonal) Other-wise the model would need to be more granular and thereforemore complex A goal can be defined without knowing anyof the other factors Therefore it may not contain sub-factorsalso found in the other factors Relationship explicitly coverssub-factors that are independent of the userrsquos and the objectrsquosphysical properties The Relationship also only pertains to acertain user-object combination and is therefore independentof goal and setting Anatomy setting and properties pertainto inherent properties of user environment and object Eachof them is a clearly defined physical object Thus they shareno common sub-factors with each other or the other factors

be extensible and reusableGRASP is intended as a working draft Therefore it is alsointended to be extended For example GRASP could beused as a basis for a predictive model I discuss this ideain the Future Work section

allow interoperability with related modelsGRASP shares the the concepts of Goal and Properties withother models Anatomy combines multiple factors from othermodels Setting - while not explicitly named in the othermodels - seems to be compatible to them Relationship is anewly introduced factor It is not yet completely clear how itrelates to other models

Overall using above criteria GRASP seems to be a usefuland robust model of meaning in grasps

Practical ApplicationsIn the following the GRASP model is exemplarily appliedto two different grasp-sensitive applications which have beenproposed in related work The goal is to identify which is-sues need to be considered when actually implementing suchapplications

Both Taylor and Bove [27] and Kim et al [14] propose to usegrasp recognition for mode switching For example a mo-bile phone might switch between camera mode (two-handedgrasp) and short-message typing mode (one-handed thumbon keypad) depending on the way it is being held TheGRASP model can be used to highlight which factors of agrasp were investigated in the prototype and which impor-tance these factors have in real-life scenariosGoal recognized switch mode (implicit might become ex-plicit with regular use)Relationship ignored in real-life scenarios probably mostlyconstant as a mobile phone is a personal deviceAnatomy varied in real-life scenarios probably mostly con-stant as a mobile phone is a personal deviceSetting assumed constant (lab study) in real-life scenariossetting changes constantlyProperties assumed constant prototype does not changeproperties

GRASP also allows a structured analysis of the design spacerevealing a number of challenges faced when implementingmode-switching functionality for mobile phones

bull Users might get accustomed to the mode-switching be-havior It should be checked - eg with a long-term study -whether this affects their use Persons using such a phonefor the first time might get irritated

bull The phone should achieve good grasp recognition accu-racy for a single user but should not completely fail ifanother person uses it (both systems were trained usinggrasps from multiple persons)

bull It should be tested how different settings affect recogni-tion performance

bull It should be tested how well the system performs if thephonersquos weight distribution or shape change [10 9]

Veldhuis et al [30] present a a prototype of a grasp-sensitivegun It recognizes its user based on the pressure pattern ofthe grasp Only authorized users shall be able to use the gunLike before GRASP can be used to investigate which mean-ingful factors of a grasp are considered by this prototypeGoal assumed constant defined as lsquofire gunldquo (implicit)) inreal-life scenarios probably mostly constantRelationship ignored in real-life scenarios probably mostlyconstant as a gun is a personal deviceAnatomy recognizedSetting controlled (lab study) will vary greatly in real-lifescenariosProperties constant prototype does not change properties

Similarly a structured analysis of the design space reveals anumber of challenges

bull Users might have different goals when grasping the guneg cleaning it The system should not treat this as anunauthorized attempt to use the gun

bull A gun may be used in various different settings eg whilerunning or in tight spaces The system needs to correctlyidentify authorized users in all these conditions

bull Sweat or rain might make the gunrsquos handle slippery re-quiring the user to grasp differently

In summary the GRASP model proposed in this paper canbe used for categorizing grasp-sensitive objects exploringthe design space of grasp-sensing applications and analyz-ing challenges when implementing grasp-sensitive user in-terfaces

LimitationsGRASP is a descriptive model of meaning in grasps Assuch it is based on a subjective concept Different personsand cultures might prefer other groupings GRASP is alsolimited to meaningful interaction Thus it is not necessarilysuited for other domains While I have tried to show that thismodel is formally valid and can be useful for reseachers anddesigners ony time will tell whether this model is actuallyhelpful Finally I am not yet certain whether Relationshipreally is an own factor One might als group it with Anatomyinto a Human factor Some might argue that Relationship isa actually just a modifier to a Goal In the end it mightdepend on whether it is useful to define Relationship as anown factor

CONCLUSION AND FUTURE WORKIn this paper I have argued that grasp sensing is an impor-tant source of meaningful information for human-computerinteraction I have given an overview of existing work fromother disciplines and have shown that existing models arenot sufficient for describing grasps in the context of human-computer interaction Therefore I propose a descriptive modelof meaning in grasps This model is supported by examplesand an epistemological analysis GRASP is intended as aworking model The main goal for GRASP is to serve as avocabulary and basis for discussing meaning in grasps andgrasp interaction utilizing this meaning This model may

also help in structuring research eg by highlighting inter-esting areas of further research It can also be used to deter-mine which factors need to be controlled in a user study Animportant long-term goal would be to automatically extractall factors contributing to a grasp from just the signature ofthe grasp For this a classifying model would be helpfulassociating certain quantitative features of a grasp with themeaningful factors contributing to it

REFERENCES1 The American Heritage Dictionary of the English

Language Fourth Edition Houghton Mifflin 2009

2 D Bailer-Jones Scientific models in philosophy ofscience Univ of Pittsburgh Pr 2009

3 G Barrett and R Omote Projected-Capacitive TouchTechnology Information Display 26(3)16ndash21 2010

4 E Berlin J Liu K van Laerhoven and B SchieleComing to grips with the objects we grasp detectinginteractions with efficient wrist-worn sensors In TEIrsquo10 Proceedings of the fourth international conferenceon Tangible embedded and embodied interactionpages 57ndash64 New York NY USA 2010 ACM

5 A Butler S Izadi and S Hodges Sidesightmulti-rdquotouchrdquo interaction around small devices InProceedings of UIST rsquo08 pages 201ndash204 New YorkNY USA 2008 ACM

6 U Castiello The neuroscience of grasping NatureReviews Neuroscience 6(9)726ndash736 2005

7 M Cutkosky On grasp choice grasp models and thedesign of hands for manufacturing tasks IEEETransactions on Robotics and Automation5(3)269ndash279 1989

8 T Feix R Pawlik H-B Schmiedmayer J Romeroand D Kragic The generation of a comprehensivegrasp taxonomy Technical report 2009

9 F Hemmert S Hamann M Lowe A Wohlauf andG Joost Weight-Shifting Mobiles One-DimensionalGravitational Displays in Mobile Phones AdjunctProceedings of UIST 9 2009

10 F Hemmert G Joost A Knorig and R WettachDynamic knobs shape change as a means ofinteraction on a mobile phone In CHIrsquo08 extendedabstracts pages 2309ndash2314 ACM 2008

11 G Heumer H Amor M Weber and B Jung Grasprecognition with uncalibrated data gloves-Acomparison of classification methods In IEEE VirtualReality Conference 2007 VRrsquo07 pages 19ndash26 2007

12 T Iberall and C MacKenzie Opposition space andhuman prehension In Dextrous robot hands pages32ndash54 Springer-Verlag New York Inc 1990

13 S Kang and K Ikeuchi Grasp recognition using thecontact web In Proc IEEERSJ Int Conf on IntRobots and Sys Raleigh NC 1992

14 K Kim W Chang S Cho J Shim H Lee J ParkY Lee and S Kim Hand Grip Pattern Recognition forMobile User Interfaces In In Proc AAAI rsquo06volume 21 page 1789 2006

15 P G Kry and D K Pai Grasp recognition andmanipulation with the tango In InternationalSymposium on Experimental Robotics volume 10Springer 2006

16 J Loviscach Two-finger input with a standard touchscreen In Proceedings of the 20th annual ACMsymposium on User interface software and technologypage 172 ACM 2007

17 C MacKenzie and T Iberall The grasping hand NorthHolland 1994

18 J Mantyjarvi K Nybergh J Himberg and K HjeltTouch Detection System for Mobile Terminals InProceedings of MobileHCI rsquo05 Springer 2004

19 S Mascaro and H Asada Measurement of fingerposture and three-axis fingertip touch force usingfingernail sensors IEEE Transactions on Robotics andAutomation 20(1)26ndash35 2004

20 J Napier The prehensile movements of the humanhand Journal of Bone amp Joint Surgery British Volume38(4)902 1956

21 J Ponce S Sullivan A Sudsang J Boissonnat andJ Merlet On computing four-finger equilibrium andforce-closure grasps of polyhedral objectsInternational Journal of Robotics Research16(1)11ndash46 1997

22 J Rekimoto Gesturewrist and gesturepad Unobtrusivewearable interaction devices 2001

23 I Rosenberg and K Perlin The UnMousePad aninterpolating multi-touch force-sensing input pad ACMTransactions on Graphics (TOG) 28(3)65 2009

24 G Schlesinger Der Mechanische Aufbau derKunstlichen Glieder Ersatzglieder und Arbeitshilfenpart II 1919

25 K Shimoga Robot grasp synthesis algorithms Asurvey The International Journal of RoboticsResearch 15(3)230 1996

26 B T Taylor and M V Bove The bar of soap a grasprecognition system implemented in a multi-functionalhandheld device In CHI rsquo08 CHI rsquo08 extendedabstracts on Human factors in computing systemspages 3459ndash3464 New York NY USA 2008 ACM

27 B T Taylor and M V Bove Graspablesgrasp-recognition as a user interface In In Proc CHIrsquo09 pages 917ndash926 New York NY USA 2009 ACM

28 F Vaandrager What is a good modelhttpwwwcsrunl fvaanPVwhat is a good modelhtml 2010

29 C van de Kamp and F Zaal Prehension is reallyreaching and grasping Experimental Brain Research182(1)27ndash34 2007

30 R N J Veldhuis A M Bazen J A Kauffman andP H Hartel Biometric verification based ongrip-pattern recognition In E J Delp and P W Wongeditors Security Steganography and Watermarking ofMultimedia Contents volume 5306 of Proceedings ofSPIE pages 634ndash641 SPIE 2004

31 R Y Wang and J Popovic Real-time hand-trackingwith a color glove ACM Transactions on Graphics28(3) 2009

32 R Wimmer FlyEye Grasp-Sensitive Surfaces UsingOptical Fiber In In Proc TEI rsquo10 Jan 2010

33 R Wimmer and S Boring HandSense - DiscriminatingDifferent Ways of Grasping and Holding a TangibleUser Interface In In Proc TEI rsquo09 February 2009

34 T Zimmerman Personal Area Networks Near-fieldintrabody communication IBM SYSTEMS JOURNAL35(3amp4)610 1996

transitions from our analysis removes much complexitywhile forfeiting only little information

bull the aforementioned complexity of transitional movementsmakes them hard to interpret using heuristics or machine-learning Concentrating on stable grasps makes it mucheasier to extract meaningful patterns

Step 1 Capturing GraspsGrasps can be captured by either capturing the grasping handor by equipping the object with a grasp-sensitive surfaceBoth approaches have unique advantages and disadvantagesFor everyday use objects with grasp-sensitive surfaces areprobably more convenient and socially acceptable however

Instrumenting the UserFor tracking fingers and hand often an optical or magneti-cal tracking system is used A marker or receiver needs tobe attached to each finger This method allows for a high-resolution capture of hand pose and position It does notprovide any information about contact forces between fin-ger and object Therefore it can not detect whether a fingeris touching the object or not Such tracking systems needto instrument the user Additionally capturing grasps worksonly within the tracking infrastructure Therefore such sys-tems are of limited use for real-world applications Howeverthey are very flexible and therefore well suited for labora-tory studies External tracking systems also provide a high-resolution ground truth for other technologiesInstead of using markers for defining finger joints some ap-proaches use computer vision techniques for an image of thehand pose to a grasp [31] Another approach that does notrequire a tracking infrastructure are data gloves These arethin gloves - often with a mesh surface on the palmar side- that are equipped with strain sensors These sensors al-low capturing each finger jointrsquos angle From these a handpose can be derived [11] These gloves are less obtrusivethan tracking markers However they limit tactile percep-tion Force-sensing with gloves is difficult as force sensorsattached to the fingertips obviously result in greatly reducedtactile perception An alternative way of determining graspforces is to capture images of the finger nail and interpretchanges in blood flow that can be observed there [19]As the fingers are controlled by muscles in the forearm someinformation about the hand pose can also be gained frommeasuring changes in these muscles or the wrist shape [22]Another option is not to capture grasps but only detect whichobject is held This can be done by attaching RFID tags toall objects and strapping an RFID reader to the userrsquos wrist[4] This method does not provide any information about theactual grasp howeverInstrumenting the users allows for easily capturing graspsfor a multitude of object shapes The objects need not bemodified However putting markers on hands would severlyimpede everyday manual tasks Additionally in most appli-cations the object itself shall react to a grasp Therefore thegrasp information would first need to be transferred to theobject - either wirelessly or by electrical coupling [34] Thiswould require a standardized communications protocol forgrasp information

Grasp-sensitive SurfacesGrasp information can also be captured by the grasped ob-ject itself To this end the object needs to be equipped witha grasp-sensitive surface that can capture the contact areabetween object and hand Grasp-sensitive surfaces requiremore engineering effort than external tracking solutions andusually do not offer identification of single fingers or digitsImplementing a grasp-sensitive surface is more difficult thanimplementing a touch-screen for two reasonsShape Sensing Instead of single or multiple touch pointsa grasp-sensitive surfaces should capture multiple complexshapes and possibly additional information like pressure There-fore common commercial touchscreen technologies like (ana-log) resistive[16] or self-capacitance [3] touch sensing can-not be used Digital resistive [23] or mutual-capacitancetechnologies are better suited for capturing contact areas shapesAdditionally the driver software needs to be able to repre-sent complex shapesSensor Shape The sensing surface is usually non-planarand continuously wrapped around the whole object Whileit is possible to embed only a few sensors at strategic posi-tions within the surface[5 33] often large parts of the sur-face shall be made grasp-sensitive This requires flexible orvery small sensors Additionally a normal cartesian coor-dinate system - as used in displays and touch input - maynot correctly represent the sensing surface Possible alterna-tives are spherical coordinate systems[15] or a relative map-ping[32]

In research prototypes a multitude of sensor technologies areused for making surfaces grasp-sensitive Often researchersbuild their own capacitive sensors [14 27 33] Other pro-totypes use resistive pressure sensors [15 30] impendancesensors [18] or optical sensors [5 32] A comparison of theinherent properties of the different types of sensors can befound in [32]Capturing a grasp results in a grasp signature a digital rep-resentation of contact points or digit positions

Step 2 Identifying GraspsOnce a grasp signature has been captured the system needsto derive the grasp that caused this signature [13] This iscommonly done by mapping the signature to a certain cat-egory of grasps either using heuristics or machine learn-ing The classification algorithm to be used depends bothon the format of the sensor data and the type of informa-tion that is needed in the interpretation stage Commonlyused classifieres include support vector machines [14 26]Bayesian classifiers [14 26] and Hidden-Markov-Models[27] Grasp categories can be defined formally using egone of the models described above More often grasp cat-egories are defined by example To this end a machine-learning classifier is trained with grasp signatures of graspsthe developer deems similar Without a set of formally de-fined grasps however it is not possible to compare differ-ent classifiers and sensing technologies In several casesgrasps with completely different signatures should actuallybe interpreted as the same grasp For example when grasp-ing a uniform sphere the grasp signature greatly dependson the rotation of the sphere In these cases an appropri-

ate mapping needs to be found that ignores certain featuresof a grasp Wimmer et al [33] recognize different types ofleft-handed and right-handed grasps using slightly modifiedheuristics for both Kry et al [15] use spherical harmonicsfor rotation-invariant grasp classificationsClassifying a grasp signature results in a grasp classification- for example a list of grasps that might have generated thegrasp signature

Step 3 Interpreting GraspsThe final step in graps sensing is to determine what a cap-tured and classified grasp actually means What is to beconsidered meaningful depends on the application For au-thentication purposes [30] it is relevant whether the graspbelongs to a certain category - whereby different categoriesrepresent different users When using grasps to switch an ob-ject between different modes [27] the mapping from a cer-tain grasp category to a mode provides meaning Howeverexisting grasp-sensitive systems capture only one meaning-ful aspect of a grasp (see below) ignoring the other aspectsWhile such systems attribute a certain meaning to a graspthe actual meaning of a grasp is ultimately defined by theuser This leads to unexpected behavior if the system at-tributes a meaning to a grasp that the user did not intendFor example a grasp-sensitive mobile phone might interpreta certain grasp as the intention to switch to camera mode[14] However the user might just hold the phone this wayfor a variety of other reasons - and would be annoyed if thephone suddenly changed into a different mode

Insofar it seems advisable to try to capture as much informa-tion about the meaning of a grasp as possible In the follow-ing section I look at meaningful factors that influence whichgrasp is chosen

GRASP - A MODEL OF MEANING IN GRASPSAs stated above a model of meaning within a grasp is neededThe model I present here is based on the model by MacKen-zie and Iberall that is described in the second section Thenew model considers the constraints and inputs they describedand re-organizes them into meaningful factors An importantsimplification is abstracting the user hiding much of the neu-ropsychological and anatomical complexity This probablydoes not remove meaningful information as most statementswe might be able to make about such sub-factors will nothelp us in improving the user interface Being aware of thisabstraction allows us to replace it with a more fine-grainedrepresentation if needed

In the following I present GRASP a model of human grasp-ing that describes five meaningful factors These factors allinfluence which grasp a user applies and each of them repre-sents a different group of meaningful information Figure 3illustrates these factors All factors are independent of eachother The validity of this model is discussed in the nextsection

GoalA very important factor determining how to grasp a givenobject is the goal that is to be achieved by the grasp Goals

are all factors that cause a grasp to be initiated Goals canbe divided into sub-goals As goals are essentially meaningthey are discussed here in more detail than the other factorsI propose grouping goals by two dimensions implicit vsexplicit grasps and primary vs supportive grasps Table1 shows examples for each combination of implicitexplicitand primarysupportive grasps

implicit explicitprimary user picks up mo-

bile phoneuser holds phonehorizontally to in-voke photo mode

supportive user holds phonewhile typing ontouchscreen withother hand

user holds phonescrolling a mapwith the otherhand zoomingby squeezing thephone

Table 1 Examples illustrating the distinction between implicit and ex-plicit grasp interaction respectively primary and supportive grasps

Usually a grasp is implicit meaning that its goal is to ma-nipulate an object Implicit grasps are similar for grasp-sensitive and non-grasp-sensitive objects Explicit grasps areconducted primarily in order to trigger a certain effect in agrasp-sensitive object For example a user might authenti-cate himself to the device using a certain grasp or switchthe mode of an object by holding it in a certain way Like-wise there is a difference in the userrsquos intention for primaryand supportive grasps A primary grasp is a grasp that isexecuted with the intention to manipulate the grasped ob-ject For example a primary grasp is used for moving acomputer mouse A supportive grasp serves to fixate or po-sition an object in order to interact with it using eg theother hand or fingers that are not involved in grasping Forexample holding a mobile phone while typing a short mes-sage is a supportive grasp These distinctions are not ran-dom but are needed to judge the amount of meaning gainedfrom a grasp An explicit grasp always conveys informa-tion about the userrsquos intention It can and should always beseen as meaningful The user expects a reaction An im-plicit grasp may convey some information about the userrsquosintention However this intention is not necessarily aimedat interacting with the object A user might just want toput away an object and would be annoyed if it reacts to thegrasp Therefore determining whether a grasp is implicitor explicit is necessary for interpreting the grasprsquos meaningLikewise a supportive grasp conveys less meaning than aprimary grasp as the main focus of the interaction lies noton the grasp but on the action that is supported by it Addi-tionally a supportive grasp is restricted by the requirementsof the main interaction and usually can not be modified aseasily as a primary grasp One might use this distinction toassign different meanings to a grasp For example whenholding a mobile phone to the ear (primary grasp) differentgrasps can launch actions When holding the phone whileinteracting with its touchscreen using the other hand differ-ent supportive grasps might only switch between modes oradjust viewport properties

Figure 3 Examples for each of the meaningful factors that contribute to a grasp A user might hold a screwdriver differently depending on the Goalhe wants to achieve with it The mental Relationship to an object determines how we grasp it In the illustrated case a paper towel that belongs tooneself is held differently than one found on the floor Depending on our Anatomy we need to apply different grasps to achieve the same effect Dueto different Settings a bottle in a shelf has to be grasped differently than a bottle in a crate The Properties of an object for example its size have amajor influence on the grasp to be used

RelationshipFeelings like disgust fear or anger can also influence theway we grasp Relationship includes all non-physical fac-tors that apply to a combination of user and object The re-lationship between user and object can be different for everyuser-object combination For example many people seem tofind a used paper towel disgusting They would only pick itup with a pinching grasp trying to avoid contact with any-thing attached to it However if the used paper towel belongsto oneself one might just pick it up like any arbitrary pieceof paper Unlike anatomical properties the relationship canalso change over time like when someone overcomes a fearof spiders and is able to touch one How a feeling changesa grasp is also determined by the knowledge about an ob-ject An angry person might unconsciously grasp the steer-ing wheel more tightly but would not do the same with anegg These effects are not solely properties of the grasp-ing person or the object but depend on the combination ofgrasping person and grasped object Relationship probablyhas little impact on most grasps However this factor hasnot been explored in detail so far

AnatomyNot every hand grasps the same way Anatomy includesall factors that are inherent to the grasping personrsquos bodyObviously differences in palm size finger length or fingercount result in slightly different grasps Additionally differ-ences in sensorimotor control of grasps also causes differ-ent grasps Differences in peoplersquos grasps might be used forautomatically identifying the person grasping an object Apower tool might also detect insufficient grasp forces for astable grasp and warn users that they have to hold it moretightly

SettingThe environment within which a grasp takes place also af-fects its outcome Setting includes all factors that pertain tothe environment in which a grasp takes place ie all physi-cal factors that are independent of user and object For exam-

ple the grasp used for pulling a bottle out of a crate differsfrom the one used for picking the bottle from a shelf Ad-ditional environmental factors might include lighting condi-tions space constraints or temperature

PropertiesThe properties of the object to be grasped naturally have ahuge influence on the grasp to be used Properties are allfactors that are intrinsic to the object This includes shapesize surface texture weight and weight distribution or sur-face temperature

VERIFYING GRASPThis section explains why GRASP is indeed a model andthat it fulfils formal requirements of a model Finally severalexamples highlight how GRASP can be applied

Formal ValidityThere are a variety of views what a model actually is Bailer-Jones offers the definition[2] ldquoA model is an interpretativedescription of a phenomenon that facilitates access to thatphenomenonrdquo GRASP is a model according to this def-inition as it describes a certain aspect (the meaning) of aphenomenon (a grasp) However what are the criteria fora good model I have chosen Vaandragerrsquos definition of agood model [28] While it pertains to models in computersystem analysis it is a concrete checklist that with plausiblerequirements Vaandrager lists seven properties of a goodmodel but notes that some of these contradict each othersometimes These properties have been rephrased in the fol-lowing to keep them succinct

According to Vaandrager a good model should

have a clearly defined object of modelingGRASP pertains to static human grasps

have a clearly specified purposeGRASP defines meaningful factors contributing to a grasp

clearly show the links between model and empiricalworldeach factor is clearly tied either to a clear concept (GoalRelationship) or a physical object (Anatomy Setting Prop-erties)

be truthful ie correctly represent relevant propertiesof the object of modellingAs a model is a simplification of a complex matter there isnot only one correct model Additionally a model inherentlyoversimplifies certain aspects of a phenomenon Thereforea model can not be completely truthful (as acknowledged byVaandrager) Additionally meaning is a property that is at-tributed to a phenomen by humans Therefore it is hard totell if our distinction of meaningful factors in a grasp is cor-rect However we can look at the meaning other researchersattribute to a grasp and see whether it matches to the factorswe defined

bull Veldhuis et al[30] solely try to derive the Anatomy of auser from a grasp

bull Taylor and Bove[27] solely try to derive an explicit Goalfrom a grasp

bull Kim et al[14] solely try to derive an explicit Goal from agrasp

bull Wimmer and Boring[33] solely try to derive an implicitGoal from a grasp In the two cases without a grasp aSetting is guessed from the sensor data

bull SpherUID - a yet unpublished project - tries to determinethe userrsquos position in relation to the object (Setting) froma grasp

bull Kry et al[15] solely tries to derive an explicit Goal fromthe grasp

Therefore our model seems to correctly identify at least GoalAnatomy and Setting as meaningful factors of a grasp I donot know of any application deriving Relationship or Prop-erties from a grasp however

be as simple as possible - but not too simpleFor GRASP this would mean that none of factors can beomitted Otherwise the model could be simplified That isfor each factor at least one situation exists where the grasp isprimarily determined by the factor Examples of such situa-tions are given in the descriptions of the factors The require-ment also indicates that none of the factors may be partiallycovered by another (ie they should be orthogonal) Other-wise the model would need to be more granular and thereforemore complex A goal can be defined without knowing anyof the other factors Therefore it may not contain sub-factorsalso found in the other factors Relationship explicitly coverssub-factors that are independent of the userrsquos and the objectrsquosphysical properties The Relationship also only pertains to acertain user-object combination and is therefore independentof goal and setting Anatomy setting and properties pertainto inherent properties of user environment and object Eachof them is a clearly defined physical object Thus they shareno common sub-factors with each other or the other factors

be extensible and reusableGRASP is intended as a working draft Therefore it is alsointended to be extended For example GRASP could beused as a basis for a predictive model I discuss this ideain the Future Work section

allow interoperability with related modelsGRASP shares the the concepts of Goal and Properties withother models Anatomy combines multiple factors from othermodels Setting - while not explicitly named in the othermodels - seems to be compatible to them Relationship is anewly introduced factor It is not yet completely clear how itrelates to other models

Overall using above criteria GRASP seems to be a usefuland robust model of meaning in grasps

Practical ApplicationsIn the following the GRASP model is exemplarily appliedto two different grasp-sensitive applications which have beenproposed in related work The goal is to identify which is-sues need to be considered when actually implementing suchapplications

Both Taylor and Bove [27] and Kim et al [14] propose to usegrasp recognition for mode switching For example a mo-bile phone might switch between camera mode (two-handedgrasp) and short-message typing mode (one-handed thumbon keypad) depending on the way it is being held TheGRASP model can be used to highlight which factors of agrasp were investigated in the prototype and which impor-tance these factors have in real-life scenariosGoal recognized switch mode (implicit might become ex-plicit with regular use)Relationship ignored in real-life scenarios probably mostlyconstant as a mobile phone is a personal deviceAnatomy varied in real-life scenarios probably mostly con-stant as a mobile phone is a personal deviceSetting assumed constant (lab study) in real-life scenariossetting changes constantlyProperties assumed constant prototype does not changeproperties

GRASP also allows a structured analysis of the design spacerevealing a number of challenges faced when implementingmode-switching functionality for mobile phones

bull Users might get accustomed to the mode-switching be-havior It should be checked - eg with a long-term study -whether this affects their use Persons using such a phonefor the first time might get irritated

bull The phone should achieve good grasp recognition accu-racy for a single user but should not completely fail ifanother person uses it (both systems were trained usinggrasps from multiple persons)

bull It should be tested how different settings affect recogni-tion performance

bull It should be tested how well the system performs if thephonersquos weight distribution or shape change [10 9]

Veldhuis et al [30] present a a prototype of a grasp-sensitivegun It recognizes its user based on the pressure pattern ofthe grasp Only authorized users shall be able to use the gunLike before GRASP can be used to investigate which mean-ingful factors of a grasp are considered by this prototypeGoal assumed constant defined as lsquofire gunldquo (implicit)) inreal-life scenarios probably mostly constantRelationship ignored in real-life scenarios probably mostlyconstant as a gun is a personal deviceAnatomy recognizedSetting controlled (lab study) will vary greatly in real-lifescenariosProperties constant prototype does not change properties

Similarly a structured analysis of the design space reveals anumber of challenges

bull Users might have different goals when grasping the guneg cleaning it The system should not treat this as anunauthorized attempt to use the gun

bull A gun may be used in various different settings eg whilerunning or in tight spaces The system needs to correctlyidentify authorized users in all these conditions

bull Sweat or rain might make the gunrsquos handle slippery re-quiring the user to grasp differently

In summary the GRASP model proposed in this paper canbe used for categorizing grasp-sensitive objects exploringthe design space of grasp-sensing applications and analyz-ing challenges when implementing grasp-sensitive user in-terfaces

LimitationsGRASP is a descriptive model of meaning in grasps Assuch it is based on a subjective concept Different personsand cultures might prefer other groupings GRASP is alsolimited to meaningful interaction Thus it is not necessarilysuited for other domains While I have tried to show that thismodel is formally valid and can be useful for reseachers anddesigners ony time will tell whether this model is actuallyhelpful Finally I am not yet certain whether Relationshipreally is an own factor One might als group it with Anatomyinto a Human factor Some might argue that Relationship isa actually just a modifier to a Goal In the end it mightdepend on whether it is useful to define Relationship as anown factor

CONCLUSION AND FUTURE WORKIn this paper I have argued that grasp sensing is an impor-tant source of meaningful information for human-computerinteraction I have given an overview of existing work fromother disciplines and have shown that existing models arenot sufficient for describing grasps in the context of human-computer interaction Therefore I propose a descriptive modelof meaning in grasps This model is supported by examplesand an epistemological analysis GRASP is intended as aworking model The main goal for GRASP is to serve as avocabulary and basis for discussing meaning in grasps andgrasp interaction utilizing this meaning This model may

also help in structuring research eg by highlighting inter-esting areas of further research It can also be used to deter-mine which factors need to be controlled in a user study Animportant long-term goal would be to automatically extractall factors contributing to a grasp from just the signature ofthe grasp For this a classifying model would be helpfulassociating certain quantitative features of a grasp with themeaningful factors contributing to it

REFERENCES1 The American Heritage Dictionary of the English

Language Fourth Edition Houghton Mifflin 2009

2 D Bailer-Jones Scientific models in philosophy ofscience Univ of Pittsburgh Pr 2009

3 G Barrett and R Omote Projected-Capacitive TouchTechnology Information Display 26(3)16ndash21 2010

4 E Berlin J Liu K van Laerhoven and B SchieleComing to grips with the objects we grasp detectinginteractions with efficient wrist-worn sensors In TEIrsquo10 Proceedings of the fourth international conferenceon Tangible embedded and embodied interactionpages 57ndash64 New York NY USA 2010 ACM

5 A Butler S Izadi and S Hodges Sidesightmulti-rdquotouchrdquo interaction around small devices InProceedings of UIST rsquo08 pages 201ndash204 New YorkNY USA 2008 ACM

6 U Castiello The neuroscience of grasping NatureReviews Neuroscience 6(9)726ndash736 2005

7 M Cutkosky On grasp choice grasp models and thedesign of hands for manufacturing tasks IEEETransactions on Robotics and Automation5(3)269ndash279 1989

8 T Feix R Pawlik H-B Schmiedmayer J Romeroand D Kragic The generation of a comprehensivegrasp taxonomy Technical report 2009

9 F Hemmert S Hamann M Lowe A Wohlauf andG Joost Weight-Shifting Mobiles One-DimensionalGravitational Displays in Mobile Phones AdjunctProceedings of UIST 9 2009

10 F Hemmert G Joost A Knorig and R WettachDynamic knobs shape change as a means ofinteraction on a mobile phone In CHIrsquo08 extendedabstracts pages 2309ndash2314 ACM 2008

11 G Heumer H Amor M Weber and B Jung Grasprecognition with uncalibrated data gloves-Acomparison of classification methods In IEEE VirtualReality Conference 2007 VRrsquo07 pages 19ndash26 2007

12 T Iberall and C MacKenzie Opposition space andhuman prehension In Dextrous robot hands pages32ndash54 Springer-Verlag New York Inc 1990

13 S Kang and K Ikeuchi Grasp recognition using thecontact web In Proc IEEERSJ Int Conf on IntRobots and Sys Raleigh NC 1992

14 K Kim W Chang S Cho J Shim H Lee J ParkY Lee and S Kim Hand Grip Pattern Recognition forMobile User Interfaces In In Proc AAAI rsquo06volume 21 page 1789 2006

15 P G Kry and D K Pai Grasp recognition andmanipulation with the tango In InternationalSymposium on Experimental Robotics volume 10Springer 2006

16 J Loviscach Two-finger input with a standard touchscreen In Proceedings of the 20th annual ACMsymposium on User interface software and technologypage 172 ACM 2007

17 C MacKenzie and T Iberall The grasping hand NorthHolland 1994

18 J Mantyjarvi K Nybergh J Himberg and K HjeltTouch Detection System for Mobile Terminals InProceedings of MobileHCI rsquo05 Springer 2004

19 S Mascaro and H Asada Measurement of fingerposture and three-axis fingertip touch force usingfingernail sensors IEEE Transactions on Robotics andAutomation 20(1)26ndash35 2004

20 J Napier The prehensile movements of the humanhand Journal of Bone amp Joint Surgery British Volume38(4)902 1956

21 J Ponce S Sullivan A Sudsang J Boissonnat andJ Merlet On computing four-finger equilibrium andforce-closure grasps of polyhedral objectsInternational Journal of Robotics Research16(1)11ndash46 1997

22 J Rekimoto Gesturewrist and gesturepad Unobtrusivewearable interaction devices 2001

23 I Rosenberg and K Perlin The UnMousePad aninterpolating multi-touch force-sensing input pad ACMTransactions on Graphics (TOG) 28(3)65 2009

24 G Schlesinger Der Mechanische Aufbau derKunstlichen Glieder Ersatzglieder und Arbeitshilfenpart II 1919

25 K Shimoga Robot grasp synthesis algorithms Asurvey The International Journal of RoboticsResearch 15(3)230 1996

26 B T Taylor and M V Bove The bar of soap a grasprecognition system implemented in a multi-functionalhandheld device In CHI rsquo08 CHI rsquo08 extendedabstracts on Human factors in computing systemspages 3459ndash3464 New York NY USA 2008 ACM

27 B T Taylor and M V Bove Graspablesgrasp-recognition as a user interface In In Proc CHIrsquo09 pages 917ndash926 New York NY USA 2009 ACM

28 F Vaandrager What is a good modelhttpwwwcsrunl fvaanPVwhat is a good modelhtml 2010

29 C van de Kamp and F Zaal Prehension is reallyreaching and grasping Experimental Brain Research182(1)27ndash34 2007

30 R N J Veldhuis A M Bazen J A Kauffman andP H Hartel Biometric verification based ongrip-pattern recognition In E J Delp and P W Wongeditors Security Steganography and Watermarking ofMultimedia Contents volume 5306 of Proceedings ofSPIE pages 634ndash641 SPIE 2004

31 R Y Wang and J Popovic Real-time hand-trackingwith a color glove ACM Transactions on Graphics28(3) 2009

32 R Wimmer FlyEye Grasp-Sensitive Surfaces UsingOptical Fiber In In Proc TEI rsquo10 Jan 2010

33 R Wimmer and S Boring HandSense - DiscriminatingDifferent Ways of Grasping and Holding a TangibleUser Interface In In Proc TEI rsquo09 February 2009

34 T Zimmerman Personal Area Networks Near-fieldintrabody communication IBM SYSTEMS JOURNAL35(3amp4)610 1996

ate mapping needs to be found that ignores certain featuresof a grasp Wimmer et al [33] recognize different types ofleft-handed and right-handed grasps using slightly modifiedheuristics for both Kry et al [15] use spherical harmonicsfor rotation-invariant grasp classificationsClassifying a grasp signature results in a grasp classification- for example a list of grasps that might have generated thegrasp signature

Step 3 Interpreting GraspsThe final step in graps sensing is to determine what a cap-tured and classified grasp actually means What is to beconsidered meaningful depends on the application For au-thentication purposes [30] it is relevant whether the graspbelongs to a certain category - whereby different categoriesrepresent different users When using grasps to switch an ob-ject between different modes [27] the mapping from a cer-tain grasp category to a mode provides meaning Howeverexisting grasp-sensitive systems capture only one meaning-ful aspect of a grasp (see below) ignoring the other aspectsWhile such systems attribute a certain meaning to a graspthe actual meaning of a grasp is ultimately defined by theuser This leads to unexpected behavior if the system at-tributes a meaning to a grasp that the user did not intendFor example a grasp-sensitive mobile phone might interpreta certain grasp as the intention to switch to camera mode[14] However the user might just hold the phone this wayfor a variety of other reasons - and would be annoyed if thephone suddenly changed into a different mode

Insofar it seems advisable to try to capture as much informa-tion about the meaning of a grasp as possible In the follow-ing section I look at meaningful factors that influence whichgrasp is chosen

GRASP - A MODEL OF MEANING IN GRASPSAs stated above a model of meaning within a grasp is neededThe model I present here is based on the model by MacKen-zie and Iberall that is described in the second section Thenew model considers the constraints and inputs they describedand re-organizes them into meaningful factors An importantsimplification is abstracting the user hiding much of the neu-ropsychological and anatomical complexity This probablydoes not remove meaningful information as most statementswe might be able to make about such sub-factors will nothelp us in improving the user interface Being aware of thisabstraction allows us to replace it with a more fine-grainedrepresentation if needed

In the following I present GRASP a model of human grasp-ing that describes five meaningful factors These factors allinfluence which grasp a user applies and each of them repre-sents a different group of meaningful information Figure 3illustrates these factors All factors are independent of eachother The validity of this model is discussed in the nextsection

GoalA very important factor determining how to grasp a givenobject is the goal that is to be achieved by the grasp Goals

are all factors that cause a grasp to be initiated Goals canbe divided into sub-goals As goals are essentially meaningthey are discussed here in more detail than the other factorsI propose grouping goals by two dimensions implicit vsexplicit grasps and primary vs supportive grasps Table1 shows examples for each combination of implicitexplicitand primarysupportive grasps

implicit explicitprimary user picks up mo-

bile phoneuser holds phonehorizontally to in-voke photo mode

supportive user holds phonewhile typing ontouchscreen withother hand

user holds phonescrolling a mapwith the otherhand zoomingby squeezing thephone

Table 1 Examples illustrating the distinction between implicit and ex-plicit grasp interaction respectively primary and supportive grasps

Usually a grasp is implicit meaning that its goal is to ma-nipulate an object Implicit grasps are similar for grasp-sensitive and non-grasp-sensitive objects Explicit grasps areconducted primarily in order to trigger a certain effect in agrasp-sensitive object For example a user might authenti-cate himself to the device using a certain grasp or switchthe mode of an object by holding it in a certain way Like-wise there is a difference in the userrsquos intention for primaryand supportive grasps A primary grasp is a grasp that isexecuted with the intention to manipulate the grasped ob-ject For example a primary grasp is used for moving acomputer mouse A supportive grasp serves to fixate or po-sition an object in order to interact with it using eg theother hand or fingers that are not involved in grasping Forexample holding a mobile phone while typing a short mes-sage is a supportive grasp These distinctions are not ran-dom but are needed to judge the amount of meaning gainedfrom a grasp An explicit grasp always conveys informa-tion about the userrsquos intention It can and should always beseen as meaningful The user expects a reaction An im-plicit grasp may convey some information about the userrsquosintention However this intention is not necessarily aimedat interacting with the object A user might just want toput away an object and would be annoyed if it reacts to thegrasp Therefore determining whether a grasp is implicitor explicit is necessary for interpreting the grasprsquos meaningLikewise a supportive grasp conveys less meaning than aprimary grasp as the main focus of the interaction lies noton the grasp but on the action that is supported by it Addi-tionally a supportive grasp is restricted by the requirementsof the main interaction and usually can not be modified aseasily as a primary grasp One might use this distinction toassign different meanings to a grasp For example whenholding a mobile phone to the ear (primary grasp) differentgrasps can launch actions When holding the phone whileinteracting with its touchscreen using the other hand differ-ent supportive grasps might only switch between modes oradjust viewport properties

Figure 3 Examples for each of the meaningful factors that contribute to a grasp A user might hold a screwdriver differently depending on the Goalhe wants to achieve with it The mental Relationship to an object determines how we grasp it In the illustrated case a paper towel that belongs tooneself is held differently than one found on the floor Depending on our Anatomy we need to apply different grasps to achieve the same effect Dueto different Settings a bottle in a shelf has to be grasped differently than a bottle in a crate The Properties of an object for example its size have amajor influence on the grasp to be used

RelationshipFeelings like disgust fear or anger can also influence theway we grasp Relationship includes all non-physical fac-tors that apply to a combination of user and object The re-lationship between user and object can be different for everyuser-object combination For example many people seem tofind a used paper towel disgusting They would only pick itup with a pinching grasp trying to avoid contact with any-thing attached to it However if the used paper towel belongsto oneself one might just pick it up like any arbitrary pieceof paper Unlike anatomical properties the relationship canalso change over time like when someone overcomes a fearof spiders and is able to touch one How a feeling changesa grasp is also determined by the knowledge about an ob-ject An angry person might unconsciously grasp the steer-ing wheel more tightly but would not do the same with anegg These effects are not solely properties of the grasp-ing person or the object but depend on the combination ofgrasping person and grasped object Relationship probablyhas little impact on most grasps However this factor hasnot been explored in detail so far

AnatomyNot every hand grasps the same way Anatomy includesall factors that are inherent to the grasping personrsquos bodyObviously differences in palm size finger length or fingercount result in slightly different grasps Additionally differ-ences in sensorimotor control of grasps also causes differ-ent grasps Differences in peoplersquos grasps might be used forautomatically identifying the person grasping an object Apower tool might also detect insufficient grasp forces for astable grasp and warn users that they have to hold it moretightly

SettingThe environment within which a grasp takes place also af-fects its outcome Setting includes all factors that pertain tothe environment in which a grasp takes place ie all physi-cal factors that are independent of user and object For exam-

ple the grasp used for pulling a bottle out of a crate differsfrom the one used for picking the bottle from a shelf Ad-ditional environmental factors might include lighting condi-tions space constraints or temperature

PropertiesThe properties of the object to be grasped naturally have ahuge influence on the grasp to be used Properties are allfactors that are intrinsic to the object This includes shapesize surface texture weight and weight distribution or sur-face temperature

VERIFYING GRASPThis section explains why GRASP is indeed a model andthat it fulfils formal requirements of a model Finally severalexamples highlight how GRASP can be applied

Formal ValidityThere are a variety of views what a model actually is Bailer-Jones offers the definition[2] ldquoA model is an interpretativedescription of a phenomenon that facilitates access to thatphenomenonrdquo GRASP is a model according to this def-inition as it describes a certain aspect (the meaning) of aphenomenon (a grasp) However what are the criteria fora good model I have chosen Vaandragerrsquos definition of agood model [28] While it pertains to models in computersystem analysis it is a concrete checklist that with plausiblerequirements Vaandrager lists seven properties of a goodmodel but notes that some of these contradict each othersometimes These properties have been rephrased in the fol-lowing to keep them succinct

According to Vaandrager a good model should

have a clearly defined object of modelingGRASP pertains to static human grasps

have a clearly specified purposeGRASP defines meaningful factors contributing to a grasp

clearly show the links between model and empiricalworldeach factor is clearly tied either to a clear concept (GoalRelationship) or a physical object (Anatomy Setting Prop-erties)

be truthful ie correctly represent relevant propertiesof the object of modellingAs a model is a simplification of a complex matter there isnot only one correct model Additionally a model inherentlyoversimplifies certain aspects of a phenomenon Thereforea model can not be completely truthful (as acknowledged byVaandrager) Additionally meaning is a property that is at-tributed to a phenomen by humans Therefore it is hard totell if our distinction of meaningful factors in a grasp is cor-rect However we can look at the meaning other researchersattribute to a grasp and see whether it matches to the factorswe defined

bull Veldhuis et al[30] solely try to derive the Anatomy of auser from a grasp

bull Taylor and Bove[27] solely try to derive an explicit Goalfrom a grasp

bull Kim et al[14] solely try to derive an explicit Goal from agrasp

bull Wimmer and Boring[33] solely try to derive an implicitGoal from a grasp In the two cases without a grasp aSetting is guessed from the sensor data

bull SpherUID - a yet unpublished project - tries to determinethe userrsquos position in relation to the object (Setting) froma grasp

bull Kry et al[15] solely tries to derive an explicit Goal fromthe grasp

Therefore our model seems to correctly identify at least GoalAnatomy and Setting as meaningful factors of a grasp I donot know of any application deriving Relationship or Prop-erties from a grasp however

be as simple as possible - but not too simpleFor GRASP this would mean that none of factors can beomitted Otherwise the model could be simplified That isfor each factor at least one situation exists where the grasp isprimarily determined by the factor Examples of such situa-tions are given in the descriptions of the factors The require-ment also indicates that none of the factors may be partiallycovered by another (ie they should be orthogonal) Other-wise the model would need to be more granular and thereforemore complex A goal can be defined without knowing anyof the other factors Therefore it may not contain sub-factorsalso found in the other factors Relationship explicitly coverssub-factors that are independent of the userrsquos and the objectrsquosphysical properties The Relationship also only pertains to acertain user-object combination and is therefore independentof goal and setting Anatomy setting and properties pertainto inherent properties of user environment and object Eachof them is a clearly defined physical object Thus they shareno common sub-factors with each other or the other factors

be extensible and reusableGRASP is intended as a working draft Therefore it is alsointended to be extended For example GRASP could beused as a basis for a predictive model I discuss this ideain the Future Work section

allow interoperability with related modelsGRASP shares the the concepts of Goal and Properties withother models Anatomy combines multiple factors from othermodels Setting - while not explicitly named in the othermodels - seems to be compatible to them Relationship is anewly introduced factor It is not yet completely clear how itrelates to other models

Overall using above criteria GRASP seems to be a usefuland robust model of meaning in grasps

Practical ApplicationsIn the following the GRASP model is exemplarily appliedto two different grasp-sensitive applications which have beenproposed in related work The goal is to identify which is-sues need to be considered when actually implementing suchapplications

Both Taylor and Bove [27] and Kim et al [14] propose to usegrasp recognition for mode switching For example a mo-bile phone might switch between camera mode (two-handedgrasp) and short-message typing mode (one-handed thumbon keypad) depending on the way it is being held TheGRASP model can be used to highlight which factors of agrasp were investigated in the prototype and which impor-tance these factors have in real-life scenariosGoal recognized switch mode (implicit might become ex-plicit with regular use)Relationship ignored in real-life scenarios probably mostlyconstant as a mobile phone is a personal deviceAnatomy varied in real-life scenarios probably mostly con-stant as a mobile phone is a personal deviceSetting assumed constant (lab study) in real-life scenariossetting changes constantlyProperties assumed constant prototype does not changeproperties

GRASP also allows a structured analysis of the design spacerevealing a number of challenges faced when implementingmode-switching functionality for mobile phones

bull Users might get accustomed to the mode-switching be-havior It should be checked - eg with a long-term study -whether this affects their use Persons using such a phonefor the first time might get irritated

bull The phone should achieve good grasp recognition accu-racy for a single user but should not completely fail ifanother person uses it (both systems were trained usinggrasps from multiple persons)

bull It should be tested how different settings affect recogni-tion performance

bull It should be tested how well the system performs if thephonersquos weight distribution or shape change [10 9]

Veldhuis et al [30] present a a prototype of a grasp-sensitivegun It recognizes its user based on the pressure pattern ofthe grasp Only authorized users shall be able to use the gunLike before GRASP can be used to investigate which mean-ingful factors of a grasp are considered by this prototypeGoal assumed constant defined as lsquofire gunldquo (implicit)) inreal-life scenarios probably mostly constantRelationship ignored in real-life scenarios probably mostlyconstant as a gun is a personal deviceAnatomy recognizedSetting controlled (lab study) will vary greatly in real-lifescenariosProperties constant prototype does not change properties

Similarly a structured analysis of the design space reveals anumber of challenges

bull Users might have different goals when grasping the guneg cleaning it The system should not treat this as anunauthorized attempt to use the gun

bull A gun may be used in various different settings eg whilerunning or in tight spaces The system needs to correctlyidentify authorized users in all these conditions

bull Sweat or rain might make the gunrsquos handle slippery re-quiring the user to grasp differently

In summary the GRASP model proposed in this paper canbe used for categorizing grasp-sensitive objects exploringthe design space of grasp-sensing applications and analyz-ing challenges when implementing grasp-sensitive user in-terfaces

LimitationsGRASP is a descriptive model of meaning in grasps Assuch it is based on a subjective concept Different personsand cultures might prefer other groupings GRASP is alsolimited to meaningful interaction Thus it is not necessarilysuited for other domains While I have tried to show that thismodel is formally valid and can be useful for reseachers anddesigners ony time will tell whether this model is actuallyhelpful Finally I am not yet certain whether Relationshipreally is an own factor One might als group it with Anatomyinto a Human factor Some might argue that Relationship isa actually just a modifier to a Goal In the end it mightdepend on whether it is useful to define Relationship as anown factor

CONCLUSION AND FUTURE WORKIn this paper I have argued that grasp sensing is an impor-tant source of meaningful information for human-computerinteraction I have given an overview of existing work fromother disciplines and have shown that existing models arenot sufficient for describing grasps in the context of human-computer interaction Therefore I propose a descriptive modelof meaning in grasps This model is supported by examplesand an epistemological analysis GRASP is intended as aworking model The main goal for GRASP is to serve as avocabulary and basis for discussing meaning in grasps andgrasp interaction utilizing this meaning This model may

also help in structuring research eg by highlighting inter-esting areas of further research It can also be used to deter-mine which factors need to be controlled in a user study Animportant long-term goal would be to automatically extractall factors contributing to a grasp from just the signature ofthe grasp For this a classifying model would be helpfulassociating certain quantitative features of a grasp with themeaningful factors contributing to it

REFERENCES1 The American Heritage Dictionary of the English

Language Fourth Edition Houghton Mifflin 2009

2 D Bailer-Jones Scientific models in philosophy ofscience Univ of Pittsburgh Pr 2009

3 G Barrett and R Omote Projected-Capacitive TouchTechnology Information Display 26(3)16ndash21 2010

4 E Berlin J Liu K van Laerhoven and B SchieleComing to grips with the objects we grasp detectinginteractions with efficient wrist-worn sensors In TEIrsquo10 Proceedings of the fourth international conferenceon Tangible embedded and embodied interactionpages 57ndash64 New York NY USA 2010 ACM

5 A Butler S Izadi and S Hodges Sidesightmulti-rdquotouchrdquo interaction around small devices InProceedings of UIST rsquo08 pages 201ndash204 New YorkNY USA 2008 ACM

6 U Castiello The neuroscience of grasping NatureReviews Neuroscience 6(9)726ndash736 2005

7 M Cutkosky On grasp choice grasp models and thedesign of hands for manufacturing tasks IEEETransactions on Robotics and Automation5(3)269ndash279 1989

8 T Feix R Pawlik H-B Schmiedmayer J Romeroand D Kragic The generation of a comprehensivegrasp taxonomy Technical report 2009

9 F Hemmert S Hamann M Lowe A Wohlauf andG Joost Weight-Shifting Mobiles One-DimensionalGravitational Displays in Mobile Phones AdjunctProceedings of UIST 9 2009

10 F Hemmert G Joost A Knorig and R WettachDynamic knobs shape change as a means ofinteraction on a mobile phone In CHIrsquo08 extendedabstracts pages 2309ndash2314 ACM 2008

11 G Heumer H Amor M Weber and B Jung Grasprecognition with uncalibrated data gloves-Acomparison of classification methods In IEEE VirtualReality Conference 2007 VRrsquo07 pages 19ndash26 2007

12 T Iberall and C MacKenzie Opposition space andhuman prehension In Dextrous robot hands pages32ndash54 Springer-Verlag New York Inc 1990

13 S Kang and K Ikeuchi Grasp recognition using thecontact web In Proc IEEERSJ Int Conf on IntRobots and Sys Raleigh NC 1992

14 K Kim W Chang S Cho J Shim H Lee J ParkY Lee and S Kim Hand Grip Pattern Recognition forMobile User Interfaces In In Proc AAAI rsquo06volume 21 page 1789 2006

15 P G Kry and D K Pai Grasp recognition andmanipulation with the tango In InternationalSymposium on Experimental Robotics volume 10Springer 2006

16 J Loviscach Two-finger input with a standard touchscreen In Proceedings of the 20th annual ACMsymposium on User interface software and technologypage 172 ACM 2007

17 C MacKenzie and T Iberall The grasping hand NorthHolland 1994

18 J Mantyjarvi K Nybergh J Himberg and K HjeltTouch Detection System for Mobile Terminals InProceedings of MobileHCI rsquo05 Springer 2004

19 S Mascaro and H Asada Measurement of fingerposture and three-axis fingertip touch force usingfingernail sensors IEEE Transactions on Robotics andAutomation 20(1)26ndash35 2004

20 J Napier The prehensile movements of the humanhand Journal of Bone amp Joint Surgery British Volume38(4)902 1956

21 J Ponce S Sullivan A Sudsang J Boissonnat andJ Merlet On computing four-finger equilibrium andforce-closure grasps of polyhedral objectsInternational Journal of Robotics Research16(1)11ndash46 1997

22 J Rekimoto Gesturewrist and gesturepad Unobtrusivewearable interaction devices 2001

23 I Rosenberg and K Perlin The UnMousePad aninterpolating multi-touch force-sensing input pad ACMTransactions on Graphics (TOG) 28(3)65 2009

24 G Schlesinger Der Mechanische Aufbau derKunstlichen Glieder Ersatzglieder und Arbeitshilfenpart II 1919

25 K Shimoga Robot grasp synthesis algorithms Asurvey The International Journal of RoboticsResearch 15(3)230 1996

26 B T Taylor and M V Bove The bar of soap a grasprecognition system implemented in a multi-functionalhandheld device In CHI rsquo08 CHI rsquo08 extendedabstracts on Human factors in computing systemspages 3459ndash3464 New York NY USA 2008 ACM

27 B T Taylor and M V Bove Graspablesgrasp-recognition as a user interface In In Proc CHIrsquo09 pages 917ndash926 New York NY USA 2009 ACM

28 F Vaandrager What is a good modelhttpwwwcsrunl fvaanPVwhat is a good modelhtml 2010

29 C van de Kamp and F Zaal Prehension is reallyreaching and grasping Experimental Brain Research182(1)27ndash34 2007

30 R N J Veldhuis A M Bazen J A Kauffman andP H Hartel Biometric verification based ongrip-pattern recognition In E J Delp and P W Wongeditors Security Steganography and Watermarking ofMultimedia Contents volume 5306 of Proceedings ofSPIE pages 634ndash641 SPIE 2004

31 R Y Wang and J Popovic Real-time hand-trackingwith a color glove ACM Transactions on Graphics28(3) 2009

32 R Wimmer FlyEye Grasp-Sensitive Surfaces UsingOptical Fiber In In Proc TEI rsquo10 Jan 2010

33 R Wimmer and S Boring HandSense - DiscriminatingDifferent Ways of Grasping and Holding a TangibleUser Interface In In Proc TEI rsquo09 February 2009

34 T Zimmerman Personal Area Networks Near-fieldintrabody communication IBM SYSTEMS JOURNAL35(3amp4)610 1996

Figure 3 Examples for each of the meaningful factors that contribute to a grasp A user might hold a screwdriver differently depending on the Goalhe wants to achieve with it The mental Relationship to an object determines how we grasp it In the illustrated case a paper towel that belongs tooneself is held differently than one found on the floor Depending on our Anatomy we need to apply different grasps to achieve the same effect Dueto different Settings a bottle in a shelf has to be grasped differently than a bottle in a crate The Properties of an object for example its size have amajor influence on the grasp to be used

RelationshipFeelings like disgust fear or anger can also influence theway we grasp Relationship includes all non-physical fac-tors that apply to a combination of user and object The re-lationship between user and object can be different for everyuser-object combination For example many people seem tofind a used paper towel disgusting They would only pick itup with a pinching grasp trying to avoid contact with any-thing attached to it However if the used paper towel belongsto oneself one might just pick it up like any arbitrary pieceof paper Unlike anatomical properties the relationship canalso change over time like when someone overcomes a fearof spiders and is able to touch one How a feeling changesa grasp is also determined by the knowledge about an ob-ject An angry person might unconsciously grasp the steer-ing wheel more tightly but would not do the same with anegg These effects are not solely properties of the grasp-ing person or the object but depend on the combination ofgrasping person and grasped object Relationship probablyhas little impact on most grasps However this factor hasnot been explored in detail so far

AnatomyNot every hand grasps the same way Anatomy includesall factors that are inherent to the grasping personrsquos bodyObviously differences in palm size finger length or fingercount result in slightly different grasps Additionally differ-ences in sensorimotor control of grasps also causes differ-ent grasps Differences in peoplersquos grasps might be used forautomatically identifying the person grasping an object Apower tool might also detect insufficient grasp forces for astable grasp and warn users that they have to hold it moretightly

SettingThe environment within which a grasp takes place also af-fects its outcome Setting includes all factors that pertain tothe environment in which a grasp takes place ie all physi-cal factors that are independent of user and object For exam-

ple the grasp used for pulling a bottle out of a crate differsfrom the one used for picking the bottle from a shelf Ad-ditional environmental factors might include lighting condi-tions space constraints or temperature

PropertiesThe properties of the object to be grasped naturally have ahuge influence on the grasp to be used Properties are allfactors that are intrinsic to the object This includes shapesize surface texture weight and weight distribution or sur-face temperature

VERIFYING GRASPThis section explains why GRASP is indeed a model andthat it fulfils formal requirements of a model Finally severalexamples highlight how GRASP can be applied

Formal ValidityThere are a variety of views what a model actually is Bailer-Jones offers the definition[2] ldquoA model is an interpretativedescription of a phenomenon that facilitates access to thatphenomenonrdquo GRASP is a model according to this def-inition as it describes a certain aspect (the meaning) of aphenomenon (a grasp) However what are the criteria fora good model I have chosen Vaandragerrsquos definition of agood model [28] While it pertains to models in computersystem analysis it is a concrete checklist that with plausiblerequirements Vaandrager lists seven properties of a goodmodel but notes that some of these contradict each othersometimes These properties have been rephrased in the fol-lowing to keep them succinct

According to Vaandrager a good model should

have a clearly defined object of modelingGRASP pertains to static human grasps

have a clearly specified purposeGRASP defines meaningful factors contributing to a grasp

clearly show the links between model and empiricalworldeach factor is clearly tied either to a clear concept (GoalRelationship) or a physical object (Anatomy Setting Prop-erties)

be truthful ie correctly represent relevant propertiesof the object of modellingAs a model is a simplification of a complex matter there isnot only one correct model Additionally a model inherentlyoversimplifies certain aspects of a phenomenon Thereforea model can not be completely truthful (as acknowledged byVaandrager) Additionally meaning is a property that is at-tributed to a phenomen by humans Therefore it is hard totell if our distinction of meaningful factors in a grasp is cor-rect However we can look at the meaning other researchersattribute to a grasp and see whether it matches to the factorswe defined

bull Veldhuis et al[30] solely try to derive the Anatomy of auser from a grasp

bull Taylor and Bove[27] solely try to derive an explicit Goalfrom a grasp

bull Kim et al[14] solely try to derive an explicit Goal from agrasp

bull Wimmer and Boring[33] solely try to derive an implicitGoal from a grasp In the two cases without a grasp aSetting is guessed from the sensor data

bull SpherUID - a yet unpublished project - tries to determinethe userrsquos position in relation to the object (Setting) froma grasp

bull Kry et al[15] solely tries to derive an explicit Goal fromthe grasp

Therefore our model seems to correctly identify at least GoalAnatomy and Setting as meaningful factors of a grasp I donot know of any application deriving Relationship or Prop-erties from a grasp however

be as simple as possible - but not too simpleFor GRASP this would mean that none of factors can beomitted Otherwise the model could be simplified That isfor each factor at least one situation exists where the grasp isprimarily determined by the factor Examples of such situa-tions are given in the descriptions of the factors The require-ment also indicates that none of the factors may be partiallycovered by another (ie they should be orthogonal) Other-wise the model would need to be more granular and thereforemore complex A goal can be defined without knowing anyof the other factors Therefore it may not contain sub-factorsalso found in the other factors Relationship explicitly coverssub-factors that are independent of the userrsquos and the objectrsquosphysical properties The Relationship also only pertains to acertain user-object combination and is therefore independentof goal and setting Anatomy setting and properties pertainto inherent properties of user environment and object Eachof them is a clearly defined physical object Thus they shareno common sub-factors with each other or the other factors

be extensible and reusableGRASP is intended as a working draft Therefore it is alsointended to be extended For example GRASP could beused as a basis for a predictive model I discuss this ideain the Future Work section

allow interoperability with related modelsGRASP shares the the concepts of Goal and Properties withother models Anatomy combines multiple factors from othermodels Setting - while not explicitly named in the othermodels - seems to be compatible to them Relationship is anewly introduced factor It is not yet completely clear how itrelates to other models

Overall using above criteria GRASP seems to be a usefuland robust model of meaning in grasps

Practical ApplicationsIn the following the GRASP model is exemplarily appliedto two different grasp-sensitive applications which have beenproposed in related work The goal is to identify which is-sues need to be considered when actually implementing suchapplications

Both Taylor and Bove [27] and Kim et al [14] propose to usegrasp recognition for mode switching For example a mo-bile phone might switch between camera mode (two-handedgrasp) and short-message typing mode (one-handed thumbon keypad) depending on the way it is being held TheGRASP model can be used to highlight which factors of agrasp were investigated in the prototype and which impor-tance these factors have in real-life scenariosGoal recognized switch mode (implicit might become ex-plicit with regular use)Relationship ignored in real-life scenarios probably mostlyconstant as a mobile phone is a personal deviceAnatomy varied in real-life scenarios probably mostly con-stant as a mobile phone is a personal deviceSetting assumed constant (lab study) in real-life scenariossetting changes constantlyProperties assumed constant prototype does not changeproperties

GRASP also allows a structured analysis of the design spacerevealing a number of challenges faced when implementingmode-switching functionality for mobile phones

bull Users might get accustomed to the mode-switching be-havior It should be checked - eg with a long-term study -whether this affects their use Persons using such a phonefor the first time might get irritated

bull The phone should achieve good grasp recognition accu-racy for a single user but should not completely fail ifanother person uses it (both systems were trained usinggrasps from multiple persons)

bull It should be tested how different settings affect recogni-tion performance

bull It should be tested how well the system performs if thephonersquos weight distribution or shape change [10 9]

Veldhuis et al [30] present a a prototype of a grasp-sensitivegun It recognizes its user based on the pressure pattern ofthe grasp Only authorized users shall be able to use the gunLike before GRASP can be used to investigate which mean-ingful factors of a grasp are considered by this prototypeGoal assumed constant defined as lsquofire gunldquo (implicit)) inreal-life scenarios probably mostly constantRelationship ignored in real-life scenarios probably mostlyconstant as a gun is a personal deviceAnatomy recognizedSetting controlled (lab study) will vary greatly in real-lifescenariosProperties constant prototype does not change properties

Similarly a structured analysis of the design space reveals anumber of challenges

bull Users might have different goals when grasping the guneg cleaning it The system should not treat this as anunauthorized attempt to use the gun

bull A gun may be used in various different settings eg whilerunning or in tight spaces The system needs to correctlyidentify authorized users in all these conditions

bull Sweat or rain might make the gunrsquos handle slippery re-quiring the user to grasp differently

In summary the GRASP model proposed in this paper canbe used for categorizing grasp-sensitive objects exploringthe design space of grasp-sensing applications and analyz-ing challenges when implementing grasp-sensitive user in-terfaces

LimitationsGRASP is a descriptive model of meaning in grasps Assuch it is based on a subjective concept Different personsand cultures might prefer other groupings GRASP is alsolimited to meaningful interaction Thus it is not necessarilysuited for other domains While I have tried to show that thismodel is formally valid and can be useful for reseachers anddesigners ony time will tell whether this model is actuallyhelpful Finally I am not yet certain whether Relationshipreally is an own factor One might als group it with Anatomyinto a Human factor Some might argue that Relationship isa actually just a modifier to a Goal In the end it mightdepend on whether it is useful to define Relationship as anown factor

CONCLUSION AND FUTURE WORKIn this paper I have argued that grasp sensing is an impor-tant source of meaningful information for human-computerinteraction I have given an overview of existing work fromother disciplines and have shown that existing models arenot sufficient for describing grasps in the context of human-computer interaction Therefore I propose a descriptive modelof meaning in grasps This model is supported by examplesand an epistemological analysis GRASP is intended as aworking model The main goal for GRASP is to serve as avocabulary and basis for discussing meaning in grasps andgrasp interaction utilizing this meaning This model may

also help in structuring research eg by highlighting inter-esting areas of further research It can also be used to deter-mine which factors need to be controlled in a user study Animportant long-term goal would be to automatically extractall factors contributing to a grasp from just the signature ofthe grasp For this a classifying model would be helpfulassociating certain quantitative features of a grasp with themeaningful factors contributing to it

REFERENCES1 The American Heritage Dictionary of the English

Language Fourth Edition Houghton Mifflin 2009

2 D Bailer-Jones Scientific models in philosophy ofscience Univ of Pittsburgh Pr 2009

3 G Barrett and R Omote Projected-Capacitive TouchTechnology Information Display 26(3)16ndash21 2010

4 E Berlin J Liu K van Laerhoven and B SchieleComing to grips with the objects we grasp detectinginteractions with efficient wrist-worn sensors In TEIrsquo10 Proceedings of the fourth international conferenceon Tangible embedded and embodied interactionpages 57ndash64 New York NY USA 2010 ACM

5 A Butler S Izadi and S Hodges Sidesightmulti-rdquotouchrdquo interaction around small devices InProceedings of UIST rsquo08 pages 201ndash204 New YorkNY USA 2008 ACM

6 U Castiello The neuroscience of grasping NatureReviews Neuroscience 6(9)726ndash736 2005

7 M Cutkosky On grasp choice grasp models and thedesign of hands for manufacturing tasks IEEETransactions on Robotics and Automation5(3)269ndash279 1989

8 T Feix R Pawlik H-B Schmiedmayer J Romeroand D Kragic The generation of a comprehensivegrasp taxonomy Technical report 2009

9 F Hemmert S Hamann M Lowe A Wohlauf andG Joost Weight-Shifting Mobiles One-DimensionalGravitational Displays in Mobile Phones AdjunctProceedings of UIST 9 2009

10 F Hemmert G Joost A Knorig and R WettachDynamic knobs shape change as a means ofinteraction on a mobile phone In CHIrsquo08 extendedabstracts pages 2309ndash2314 ACM 2008

11 G Heumer H Amor M Weber and B Jung Grasprecognition with uncalibrated data gloves-Acomparison of classification methods In IEEE VirtualReality Conference 2007 VRrsquo07 pages 19ndash26 2007

12 T Iberall and C MacKenzie Opposition space andhuman prehension In Dextrous robot hands pages32ndash54 Springer-Verlag New York Inc 1990

13 S Kang and K Ikeuchi Grasp recognition using thecontact web In Proc IEEERSJ Int Conf on IntRobots and Sys Raleigh NC 1992

14 K Kim W Chang S Cho J Shim H Lee J ParkY Lee and S Kim Hand Grip Pattern Recognition forMobile User Interfaces In In Proc AAAI rsquo06volume 21 page 1789 2006

15 P G Kry and D K Pai Grasp recognition andmanipulation with the tango In InternationalSymposium on Experimental Robotics volume 10Springer 2006

16 J Loviscach Two-finger input with a standard touchscreen In Proceedings of the 20th annual ACMsymposium on User interface software and technologypage 172 ACM 2007

17 C MacKenzie and T Iberall The grasping hand NorthHolland 1994

18 J Mantyjarvi K Nybergh J Himberg and K HjeltTouch Detection System for Mobile Terminals InProceedings of MobileHCI rsquo05 Springer 2004

19 S Mascaro and H Asada Measurement of fingerposture and three-axis fingertip touch force usingfingernail sensors IEEE Transactions on Robotics andAutomation 20(1)26ndash35 2004

20 J Napier The prehensile movements of the humanhand Journal of Bone amp Joint Surgery British Volume38(4)902 1956

21 J Ponce S Sullivan A Sudsang J Boissonnat andJ Merlet On computing four-finger equilibrium andforce-closure grasps of polyhedral objectsInternational Journal of Robotics Research16(1)11ndash46 1997

22 J Rekimoto Gesturewrist and gesturepad Unobtrusivewearable interaction devices 2001

23 I Rosenberg and K Perlin The UnMousePad aninterpolating multi-touch force-sensing input pad ACMTransactions on Graphics (TOG) 28(3)65 2009

24 G Schlesinger Der Mechanische Aufbau derKunstlichen Glieder Ersatzglieder und Arbeitshilfenpart II 1919

25 K Shimoga Robot grasp synthesis algorithms Asurvey The International Journal of RoboticsResearch 15(3)230 1996

26 B T Taylor and M V Bove The bar of soap a grasprecognition system implemented in a multi-functionalhandheld device In CHI rsquo08 CHI rsquo08 extendedabstracts on Human factors in computing systemspages 3459ndash3464 New York NY USA 2008 ACM

27 B T Taylor and M V Bove Graspablesgrasp-recognition as a user interface In In Proc CHIrsquo09 pages 917ndash926 New York NY USA 2009 ACM

28 F Vaandrager What is a good modelhttpwwwcsrunl fvaanPVwhat is a good modelhtml 2010

29 C van de Kamp and F Zaal Prehension is reallyreaching and grasping Experimental Brain Research182(1)27ndash34 2007

30 R N J Veldhuis A M Bazen J A Kauffman andP H Hartel Biometric verification based ongrip-pattern recognition In E J Delp and P W Wongeditors Security Steganography and Watermarking ofMultimedia Contents volume 5306 of Proceedings ofSPIE pages 634ndash641 SPIE 2004

31 R Y Wang and J Popovic Real-time hand-trackingwith a color glove ACM Transactions on Graphics28(3) 2009

32 R Wimmer FlyEye Grasp-Sensitive Surfaces UsingOptical Fiber In In Proc TEI rsquo10 Jan 2010

33 R Wimmer and S Boring HandSense - DiscriminatingDifferent Ways of Grasping and Holding a TangibleUser Interface In In Proc TEI rsquo09 February 2009

34 T Zimmerman Personal Area Networks Near-fieldintrabody communication IBM SYSTEMS JOURNAL35(3amp4)610 1996

clearly show the links between model and empiricalworldeach factor is clearly tied either to a clear concept (GoalRelationship) or a physical object (Anatomy Setting Prop-erties)

be truthful ie correctly represent relevant propertiesof the object of modellingAs a model is a simplification of a complex matter there isnot only one correct model Additionally a model inherentlyoversimplifies certain aspects of a phenomenon Thereforea model can not be completely truthful (as acknowledged byVaandrager) Additionally meaning is a property that is at-tributed to a phenomen by humans Therefore it is hard totell if our distinction of meaningful factors in a grasp is cor-rect However we can look at the meaning other researchersattribute to a grasp and see whether it matches to the factorswe defined

bull Veldhuis et al[30] solely try to derive the Anatomy of auser from a grasp

bull Taylor and Bove[27] solely try to derive an explicit Goalfrom a grasp

bull Kim et al[14] solely try to derive an explicit Goal from agrasp

bull Wimmer and Boring[33] solely try to derive an implicitGoal from a grasp In the two cases without a grasp aSetting is guessed from the sensor data

bull SpherUID - a yet unpublished project - tries to determinethe userrsquos position in relation to the object (Setting) froma grasp

bull Kry et al[15] solely tries to derive an explicit Goal fromthe grasp

Therefore our model seems to correctly identify at least GoalAnatomy and Setting as meaningful factors of a grasp I donot know of any application deriving Relationship or Prop-erties from a grasp however

be as simple as possible - but not too simpleFor GRASP this would mean that none of factors can beomitted Otherwise the model could be simplified That isfor each factor at least one situation exists where the grasp isprimarily determined by the factor Examples of such situa-tions are given in the descriptions of the factors The require-ment also indicates that none of the factors may be partiallycovered by another (ie they should be orthogonal) Other-wise the model would need to be more granular and thereforemore complex A goal can be defined without knowing anyof the other factors Therefore it may not contain sub-factorsalso found in the other factors Relationship explicitly coverssub-factors that are independent of the userrsquos and the objectrsquosphysical properties The Relationship also only pertains to acertain user-object combination and is therefore independentof goal and setting Anatomy setting and properties pertainto inherent properties of user environment and object Eachof them is a clearly defined physical object Thus they shareno common sub-factors with each other or the other factors

be extensible and reusableGRASP is intended as a working draft Therefore it is alsointended to be extended For example GRASP could beused as a basis for a predictive model I discuss this ideain the Future Work section

allow interoperability with related modelsGRASP shares the the concepts of Goal and Properties withother models Anatomy combines multiple factors from othermodels Setting - while not explicitly named in the othermodels - seems to be compatible to them Relationship is anewly introduced factor It is not yet completely clear how itrelates to other models

Overall using above criteria GRASP seems to be a usefuland robust model of meaning in grasps

Practical ApplicationsIn the following the GRASP model is exemplarily appliedto two different grasp-sensitive applications which have beenproposed in related work The goal is to identify which is-sues need to be considered when actually implementing suchapplications

Both Taylor and Bove [27] and Kim et al [14] propose to usegrasp recognition for mode switching For example a mo-bile phone might switch between camera mode (two-handedgrasp) and short-message typing mode (one-handed thumbon keypad) depending on the way it is being held TheGRASP model can be used to highlight which factors of agrasp were investigated in the prototype and which impor-tance these factors have in real-life scenariosGoal recognized switch mode (implicit might become ex-plicit with regular use)Relationship ignored in real-life scenarios probably mostlyconstant as a mobile phone is a personal deviceAnatomy varied in real-life scenarios probably mostly con-stant as a mobile phone is a personal deviceSetting assumed constant (lab study) in real-life scenariossetting changes constantlyProperties assumed constant prototype does not changeproperties

GRASP also allows a structured analysis of the design spacerevealing a number of challenges faced when implementingmode-switching functionality for mobile phones

bull Users might get accustomed to the mode-switching be-havior It should be checked - eg with a long-term study -whether this affects their use Persons using such a phonefor the first time might get irritated

bull The phone should achieve good grasp recognition accu-racy for a single user but should not completely fail ifanother person uses it (both systems were trained usinggrasps from multiple persons)

bull It should be tested how different settings affect recogni-tion performance

bull It should be tested how well the system performs if thephonersquos weight distribution or shape change [10 9]

Veldhuis et al [30] present a a prototype of a grasp-sensitivegun It recognizes its user based on the pressure pattern ofthe grasp Only authorized users shall be able to use the gunLike before GRASP can be used to investigate which mean-ingful factors of a grasp are considered by this prototypeGoal assumed constant defined as lsquofire gunldquo (implicit)) inreal-life scenarios probably mostly constantRelationship ignored in real-life scenarios probably mostlyconstant as a gun is a personal deviceAnatomy recognizedSetting controlled (lab study) will vary greatly in real-lifescenariosProperties constant prototype does not change properties

Similarly a structured analysis of the design space reveals anumber of challenges

bull Users might have different goals when grasping the guneg cleaning it The system should not treat this as anunauthorized attempt to use the gun

bull A gun may be used in various different settings eg whilerunning or in tight spaces The system needs to correctlyidentify authorized users in all these conditions

bull Sweat or rain might make the gunrsquos handle slippery re-quiring the user to grasp differently

In summary the GRASP model proposed in this paper canbe used for categorizing grasp-sensitive objects exploringthe design space of grasp-sensing applications and analyz-ing challenges when implementing grasp-sensitive user in-terfaces

LimitationsGRASP is a descriptive model of meaning in grasps Assuch it is based on a subjective concept Different personsand cultures might prefer other groupings GRASP is alsolimited to meaningful interaction Thus it is not necessarilysuited for other domains While I have tried to show that thismodel is formally valid and can be useful for reseachers anddesigners ony time will tell whether this model is actuallyhelpful Finally I am not yet certain whether Relationshipreally is an own factor One might als group it with Anatomyinto a Human factor Some might argue that Relationship isa actually just a modifier to a Goal In the end it mightdepend on whether it is useful to define Relationship as anown factor

CONCLUSION AND FUTURE WORKIn this paper I have argued that grasp sensing is an impor-tant source of meaningful information for human-computerinteraction I have given an overview of existing work fromother disciplines and have shown that existing models arenot sufficient for describing grasps in the context of human-computer interaction Therefore I propose a descriptive modelof meaning in grasps This model is supported by examplesand an epistemological analysis GRASP is intended as aworking model The main goal for GRASP is to serve as avocabulary and basis for discussing meaning in grasps andgrasp interaction utilizing this meaning This model may

also help in structuring research eg by highlighting inter-esting areas of further research It can also be used to deter-mine which factors need to be controlled in a user study Animportant long-term goal would be to automatically extractall factors contributing to a grasp from just the signature ofthe grasp For this a classifying model would be helpfulassociating certain quantitative features of a grasp with themeaningful factors contributing to it

REFERENCES1 The American Heritage Dictionary of the English

Language Fourth Edition Houghton Mifflin 2009

2 D Bailer-Jones Scientific models in philosophy ofscience Univ of Pittsburgh Pr 2009

3 G Barrett and R Omote Projected-Capacitive TouchTechnology Information Display 26(3)16ndash21 2010

4 E Berlin J Liu K van Laerhoven and B SchieleComing to grips with the objects we grasp detectinginteractions with efficient wrist-worn sensors In TEIrsquo10 Proceedings of the fourth international conferenceon Tangible embedded and embodied interactionpages 57ndash64 New York NY USA 2010 ACM

5 A Butler S Izadi and S Hodges Sidesightmulti-rdquotouchrdquo interaction around small devices InProceedings of UIST rsquo08 pages 201ndash204 New YorkNY USA 2008 ACM

6 U Castiello The neuroscience of grasping NatureReviews Neuroscience 6(9)726ndash736 2005

7 M Cutkosky On grasp choice grasp models and thedesign of hands for manufacturing tasks IEEETransactions on Robotics and Automation5(3)269ndash279 1989

8 T Feix R Pawlik H-B Schmiedmayer J Romeroand D Kragic The generation of a comprehensivegrasp taxonomy Technical report 2009

9 F Hemmert S Hamann M Lowe A Wohlauf andG Joost Weight-Shifting Mobiles One-DimensionalGravitational Displays in Mobile Phones AdjunctProceedings of UIST 9 2009

10 F Hemmert G Joost A Knorig and R WettachDynamic knobs shape change as a means ofinteraction on a mobile phone In CHIrsquo08 extendedabstracts pages 2309ndash2314 ACM 2008

11 G Heumer H Amor M Weber and B Jung Grasprecognition with uncalibrated data gloves-Acomparison of classification methods In IEEE VirtualReality Conference 2007 VRrsquo07 pages 19ndash26 2007

12 T Iberall and C MacKenzie Opposition space andhuman prehension In Dextrous robot hands pages32ndash54 Springer-Verlag New York Inc 1990

13 S Kang and K Ikeuchi Grasp recognition using thecontact web In Proc IEEERSJ Int Conf on IntRobots and Sys Raleigh NC 1992

14 K Kim W Chang S Cho J Shim H Lee J ParkY Lee and S Kim Hand Grip Pattern Recognition forMobile User Interfaces In In Proc AAAI rsquo06volume 21 page 1789 2006

15 P G Kry and D K Pai Grasp recognition andmanipulation with the tango In InternationalSymposium on Experimental Robotics volume 10Springer 2006

16 J Loviscach Two-finger input with a standard touchscreen In Proceedings of the 20th annual ACMsymposium on User interface software and technologypage 172 ACM 2007

17 C MacKenzie and T Iberall The grasping hand NorthHolland 1994

18 J Mantyjarvi K Nybergh J Himberg and K HjeltTouch Detection System for Mobile Terminals InProceedings of MobileHCI rsquo05 Springer 2004

19 S Mascaro and H Asada Measurement of fingerposture and three-axis fingertip touch force usingfingernail sensors IEEE Transactions on Robotics andAutomation 20(1)26ndash35 2004

20 J Napier The prehensile movements of the humanhand Journal of Bone amp Joint Surgery British Volume38(4)902 1956

21 J Ponce S Sullivan A Sudsang J Boissonnat andJ Merlet On computing four-finger equilibrium andforce-closure grasps of polyhedral objectsInternational Journal of Robotics Research16(1)11ndash46 1997

22 J Rekimoto Gesturewrist and gesturepad Unobtrusivewearable interaction devices 2001

23 I Rosenberg and K Perlin The UnMousePad aninterpolating multi-touch force-sensing input pad ACMTransactions on Graphics (TOG) 28(3)65 2009

24 G Schlesinger Der Mechanische Aufbau derKunstlichen Glieder Ersatzglieder und Arbeitshilfenpart II 1919

25 K Shimoga Robot grasp synthesis algorithms Asurvey The International Journal of RoboticsResearch 15(3)230 1996

26 B T Taylor and M V Bove The bar of soap a grasprecognition system implemented in a multi-functionalhandheld device In CHI rsquo08 CHI rsquo08 extendedabstracts on Human factors in computing systemspages 3459ndash3464 New York NY USA 2008 ACM

27 B T Taylor and M V Bove Graspablesgrasp-recognition as a user interface In In Proc CHIrsquo09 pages 917ndash926 New York NY USA 2009 ACM

28 F Vaandrager What is a good modelhttpwwwcsrunl fvaanPVwhat is a good modelhtml 2010

29 C van de Kamp and F Zaal Prehension is reallyreaching and grasping Experimental Brain Research182(1)27ndash34 2007

30 R N J Veldhuis A M Bazen J A Kauffman andP H Hartel Biometric verification based ongrip-pattern recognition In E J Delp and P W Wongeditors Security Steganography and Watermarking ofMultimedia Contents volume 5306 of Proceedings ofSPIE pages 634ndash641 SPIE 2004

31 R Y Wang and J Popovic Real-time hand-trackingwith a color glove ACM Transactions on Graphics28(3) 2009

32 R Wimmer FlyEye Grasp-Sensitive Surfaces UsingOptical Fiber In In Proc TEI rsquo10 Jan 2010

33 R Wimmer and S Boring HandSense - DiscriminatingDifferent Ways of Grasping and Holding a TangibleUser Interface In In Proc TEI rsquo09 February 2009

34 T Zimmerman Personal Area Networks Near-fieldintrabody communication IBM SYSTEMS JOURNAL35(3amp4)610 1996

Veldhuis et al [30] present a a prototype of a grasp-sensitivegun It recognizes its user based on the pressure pattern ofthe grasp Only authorized users shall be able to use the gunLike before GRASP can be used to investigate which mean-ingful factors of a grasp are considered by this prototypeGoal assumed constant defined as lsquofire gunldquo (implicit)) inreal-life scenarios probably mostly constantRelationship ignored in real-life scenarios probably mostlyconstant as a gun is a personal deviceAnatomy recognizedSetting controlled (lab study) will vary greatly in real-lifescenariosProperties constant prototype does not change properties

Similarly a structured analysis of the design space reveals anumber of challenges

bull Users might have different goals when grasping the guneg cleaning it The system should not treat this as anunauthorized attempt to use the gun

bull A gun may be used in various different settings eg whilerunning or in tight spaces The system needs to correctlyidentify authorized users in all these conditions

bull Sweat or rain might make the gunrsquos handle slippery re-quiring the user to grasp differently

In summary the GRASP model proposed in this paper canbe used for categorizing grasp-sensitive objects exploringthe design space of grasp-sensing applications and analyz-ing challenges when implementing grasp-sensitive user in-terfaces

LimitationsGRASP is a descriptive model of meaning in grasps Assuch it is based on a subjective concept Different personsand cultures might prefer other groupings GRASP is alsolimited to meaningful interaction Thus it is not necessarilysuited for other domains While I have tried to show that thismodel is formally valid and can be useful for reseachers anddesigners ony time will tell whether this model is actuallyhelpful Finally I am not yet certain whether Relationshipreally is an own factor One might als group it with Anatomyinto a Human factor Some might argue that Relationship isa actually just a modifier to a Goal In the end it mightdepend on whether it is useful to define Relationship as anown factor

CONCLUSION AND FUTURE WORKIn this paper I have argued that grasp sensing is an impor-tant source of meaningful information for human-computerinteraction I have given an overview of existing work fromother disciplines and have shown that existing models arenot sufficient for describing grasps in the context of human-computer interaction Therefore I propose a descriptive modelof meaning in grasps This model is supported by examplesand an epistemological analysis GRASP is intended as aworking model The main goal for GRASP is to serve as avocabulary and basis for discussing meaning in grasps andgrasp interaction utilizing this meaning This model may

also help in structuring research eg by highlighting inter-esting areas of further research It can also be used to deter-mine which factors need to be controlled in a user study Animportant long-term goal would be to automatically extractall factors contributing to a grasp from just the signature ofthe grasp For this a classifying model would be helpfulassociating certain quantitative features of a grasp with themeaningful factors contributing to it

REFERENCES1 The American Heritage Dictionary of the English

Language Fourth Edition Houghton Mifflin 2009

2 D Bailer-Jones Scientific models in philosophy ofscience Univ of Pittsburgh Pr 2009

3 G Barrett and R Omote Projected-Capacitive TouchTechnology Information Display 26(3)16ndash21 2010

4 E Berlin J Liu K van Laerhoven and B SchieleComing to grips with the objects we grasp detectinginteractions with efficient wrist-worn sensors In TEIrsquo10 Proceedings of the fourth international conferenceon Tangible embedded and embodied interactionpages 57ndash64 New York NY USA 2010 ACM

5 A Butler S Izadi and S Hodges Sidesightmulti-rdquotouchrdquo interaction around small devices InProceedings of UIST rsquo08 pages 201ndash204 New YorkNY USA 2008 ACM

6 U Castiello The neuroscience of grasping NatureReviews Neuroscience 6(9)726ndash736 2005

7 M Cutkosky On grasp choice grasp models and thedesign of hands for manufacturing tasks IEEETransactions on Robotics and Automation5(3)269ndash279 1989

8 T Feix R Pawlik H-B Schmiedmayer J Romeroand D Kragic The generation of a comprehensivegrasp taxonomy Technical report 2009

9 F Hemmert S Hamann M Lowe A Wohlauf andG Joost Weight-Shifting Mobiles One-DimensionalGravitational Displays in Mobile Phones AdjunctProceedings of UIST 9 2009

10 F Hemmert G Joost A Knorig and R WettachDynamic knobs shape change as a means ofinteraction on a mobile phone In CHIrsquo08 extendedabstracts pages 2309ndash2314 ACM 2008

11 G Heumer H Amor M Weber and B Jung Grasprecognition with uncalibrated data gloves-Acomparison of classification methods In IEEE VirtualReality Conference 2007 VRrsquo07 pages 19ndash26 2007

12 T Iberall and C MacKenzie Opposition space andhuman prehension In Dextrous robot hands pages32ndash54 Springer-Verlag New York Inc 1990

13 S Kang and K Ikeuchi Grasp recognition using thecontact web In Proc IEEERSJ Int Conf on IntRobots and Sys Raleigh NC 1992

14 K Kim W Chang S Cho J Shim H Lee J ParkY Lee and S Kim Hand Grip Pattern Recognition forMobile User Interfaces In In Proc AAAI rsquo06volume 21 page 1789 2006

15 P G Kry and D K Pai Grasp recognition andmanipulation with the tango In InternationalSymposium on Experimental Robotics volume 10Springer 2006

16 J Loviscach Two-finger input with a standard touchscreen In Proceedings of the 20th annual ACMsymposium on User interface software and technologypage 172 ACM 2007

17 C MacKenzie and T Iberall The grasping hand NorthHolland 1994

18 J Mantyjarvi K Nybergh J Himberg and K HjeltTouch Detection System for Mobile Terminals InProceedings of MobileHCI rsquo05 Springer 2004

19 S Mascaro and H Asada Measurement of fingerposture and three-axis fingertip touch force usingfingernail sensors IEEE Transactions on Robotics andAutomation 20(1)26ndash35 2004

20 J Napier The prehensile movements of the humanhand Journal of Bone amp Joint Surgery British Volume38(4)902 1956

21 J Ponce S Sullivan A Sudsang J Boissonnat andJ Merlet On computing four-finger equilibrium andforce-closure grasps of polyhedral objectsInternational Journal of Robotics Research16(1)11ndash46 1997

22 J Rekimoto Gesturewrist and gesturepad Unobtrusivewearable interaction devices 2001

23 I Rosenberg and K Perlin The UnMousePad aninterpolating multi-touch force-sensing input pad ACMTransactions on Graphics (TOG) 28(3)65 2009

24 G Schlesinger Der Mechanische Aufbau derKunstlichen Glieder Ersatzglieder und Arbeitshilfenpart II 1919

25 K Shimoga Robot grasp synthesis algorithms Asurvey The International Journal of RoboticsResearch 15(3)230 1996

26 B T Taylor and M V Bove The bar of soap a grasprecognition system implemented in a multi-functionalhandheld device In CHI rsquo08 CHI rsquo08 extendedabstracts on Human factors in computing systemspages 3459ndash3464 New York NY USA 2008 ACM

27 B T Taylor and M V Bove Graspablesgrasp-recognition as a user interface In In Proc CHIrsquo09 pages 917ndash926 New York NY USA 2009 ACM

28 F Vaandrager What is a good modelhttpwwwcsrunl fvaanPVwhat is a good modelhtml 2010

29 C van de Kamp and F Zaal Prehension is reallyreaching and grasping Experimental Brain Research182(1)27ndash34 2007

30 R N J Veldhuis A M Bazen J A Kauffman andP H Hartel Biometric verification based ongrip-pattern recognition In E J Delp and P W Wongeditors Security Steganography and Watermarking ofMultimedia Contents volume 5306 of Proceedings ofSPIE pages 634ndash641 SPIE 2004

31 R Y Wang and J Popovic Real-time hand-trackingwith a color glove ACM Transactions on Graphics28(3) 2009

32 R Wimmer FlyEye Grasp-Sensitive Surfaces UsingOptical Fiber In In Proc TEI rsquo10 Jan 2010

33 R Wimmer and S Boring HandSense - DiscriminatingDifferent Ways of Grasping and Holding a TangibleUser Interface In In Proc TEI rsquo09 February 2009

34 T Zimmerman Personal Area Networks Near-fieldintrabody communication IBM SYSTEMS JOURNAL35(3amp4)610 1996

14 K Kim W Chang S Cho J Shim H Lee J ParkY Lee and S Kim Hand Grip Pattern Recognition forMobile User Interfaces In In Proc AAAI rsquo06volume 21 page 1789 2006

15 P G Kry and D K Pai Grasp recognition andmanipulation with the tango In InternationalSymposium on Experimental Robotics volume 10Springer 2006

16 J Loviscach Two-finger input with a standard touchscreen In Proceedings of the 20th annual ACMsymposium on User interface software and technologypage 172 ACM 2007

17 C MacKenzie and T Iberall The grasping hand NorthHolland 1994

18 J Mantyjarvi K Nybergh J Himberg and K HjeltTouch Detection System for Mobile Terminals InProceedings of MobileHCI rsquo05 Springer 2004

19 S Mascaro and H Asada Measurement of fingerposture and three-axis fingertip touch force usingfingernail sensors IEEE Transactions on Robotics andAutomation 20(1)26ndash35 2004

20 J Napier The prehensile movements of the humanhand Journal of Bone amp Joint Surgery British Volume38(4)902 1956

21 J Ponce S Sullivan A Sudsang J Boissonnat andJ Merlet On computing four-finger equilibrium andforce-closure grasps of polyhedral objectsInternational Journal of Robotics Research16(1)11ndash46 1997

22 J Rekimoto Gesturewrist and gesturepad Unobtrusivewearable interaction devices 2001

23 I Rosenberg and K Perlin The UnMousePad aninterpolating multi-touch force-sensing input pad ACMTransactions on Graphics (TOG) 28(3)65 2009

24 G Schlesinger Der Mechanische Aufbau derKunstlichen Glieder Ersatzglieder und Arbeitshilfenpart II 1919

25 K Shimoga Robot grasp synthesis algorithms Asurvey The International Journal of RoboticsResearch 15(3)230 1996

26 B T Taylor and M V Bove The bar of soap a grasprecognition system implemented in a multi-functionalhandheld device In CHI rsquo08 CHI rsquo08 extendedabstracts on Human factors in computing systemspages 3459ndash3464 New York NY USA 2008 ACM

27 B T Taylor and M V Bove Graspablesgrasp-recognition as a user interface In In Proc CHIrsquo09 pages 917ndash926 New York NY USA 2009 ACM

28 F Vaandrager What is a good modelhttpwwwcsrunl fvaanPVwhat is a good modelhtml 2010

29 C van de Kamp and F Zaal Prehension is reallyreaching and grasping Experimental Brain Research182(1)27ndash34 2007

30 R N J Veldhuis A M Bazen J A Kauffman andP H Hartel Biometric verification based ongrip-pattern recognition In E J Delp and P W Wongeditors Security Steganography and Watermarking ofMultimedia Contents volume 5306 of Proceedings ofSPIE pages 634ndash641 SPIE 2004

31 R Y Wang and J Popovic Real-time hand-trackingwith a color glove ACM Transactions on Graphics28(3) 2009

32 R Wimmer FlyEye Grasp-Sensitive Surfaces UsingOptical Fiber In In Proc TEI rsquo10 Jan 2010

33 R Wimmer and S Boring HandSense - DiscriminatingDifferent Ways of Grasping and Holding a TangibleUser Interface In In Proc TEI rsquo09 February 2009

34 T Zimmerman Personal Area Networks Near-fieldintrabody communication IBM SYSTEMS JOURNAL35(3amp4)610 1996