Metrics for Assessing Human Skill When Demonstrating aBimanual Task to a Robot

Ana Lucia Pais UrecheLearning Systems and Algorithms LaboratoryÉcole Polytechnique Fédérale de Lausanne

CH-1015 Lausanne, Switzerlandlucia.pais@epfl.ch

Aude BillardLearning Systems and Algorithms LaboratoryÉcole Polytechnique Fédérale de Lausanne

CH-1015 Lausanne, Switzerlandaude.billard@epfl.ch

ABSTRACTOne of the major challenges in Programming by Demonstra-tion is deciding who to imitate. In this paper we propose aset of metrics for assessing how skilled a user is when demon-strating a bimanual task to a robot, that requires both acoordinated motion of the arms, and proper contact forces.We record successful demonstrations relative to the task goaland evaluate user performance with respect to 3 measures:the ability to maneuver the tool, the consistency in teaching,and the degree of coordination between the two arms. Wepresent preliminary results on a scooping task.

Categories and Subject DescriptorsI.2.9 [Robotics]; H.1.2 [User/Machine Interfaces]: Hu-man Information Processing

General TermsHuman Factors

KeywordsProgramming by demonstration; Human skill assessment

1. INTRODUCTIONWhen naive users perform Programming by Demonstra-

tion (PbD) on a robot various factors might influence theirbehavior. This might lead to low quality demonstrationsfrom which it is hard to learn and to generalize. In thispaper we propose a set of metrics for assessing the humanperformance and skill while providing bimanual demonstra-tions to a robot. We make the assumption that the maintask features are invariant across demonstrators, however auser might be more or less skilled in performing a certaintask. Assessing the level of skill is particularly importantin tasks that require coordination and modeling of contactforces as the resulting model is responsible for the way therobot is able to execute the task.

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage, and that copies bear this notice and the full ci-tation on the first page. Copyrights for third-party components of this work must behonored. For all other uses, contact the owner/author(s). Copyright is held by theauthor/owner(s).HRI’15 Extended Abstracts, March 2–5, 2015, Portland, OR, USA.ACM 978-1-4503-3318-4/15/03.http://dx.doi.org/10.1145/2701973.2702017.

Figure 1: Setup for providing bimanual demonstrations of a task

2. RELATED WORKSuccessfully demonstrating a task to a robot is often sub-

ject to the human understanding of the task, and the abilityto perform it well [4]. Therefore the quality of the demon-strations might vary between users. In this work we proposea set of metrics for automatically determining skilled users,that implicitly provide good demonstrations. Users’ skillhas been previously assessed based on discontinuities in thedata, or the use of corrective motions [7]. However a lot ofinformation is conveyed in the way the user holds the tool,applies forces in contact tasks, or coordinates the motionbetween its arms.

Dealing with poor demonstrations has been assessed be-fore by excluding bad data points [6], asking the teacherfor complementary information [2, 3], or evaluating if thedemonstrated trajectory is optimal in a known environment[5]. Our approach allows the robot to choose learning fromthe most skilled demonstrator.

3. PROPOSED METRICSWe consider bimanual asymmetrical tasks in which both

arms are in contact with the environment and evaluate users’performance based on measurements that relate to using thetool, consistency in teaching, and arm coordination. Weuse a previous approach for extracting task constraints foreach arm [8]. They consist of: determining a frame of refer-ence designating the object of interest at each time step, thevariable of interest on each dimension given by the relativeimportance of force and position and a stiffness modulation.

Maneuvering the Tool. We make the assumption that ifforce was determined to be a variable of interest on a givenaxis with respect to the object being used, then a skilled user

37

Figure 2: (left) GQ computed for each subject, on each axis. Encir-cled are the subjects with high GQ across the important dimensions;(right) Comparison of normal and over segmentation (using [8])

(a) [S01] scooping (b) [S01] trashing (c) [S06] scooping

Figure 3: The grasp configuration changes for each action. The ref-erence frame represents the center of the grasp, while the small ar-rows indicate the contact intensity. When scooping strong contact isrecorded, mainly on the thumb. When trashing it is significantly less.

would also use a grasp suitable for applying strong forcesalong that direction. We evaluate the grasp and its adapt-ability to the task by computing a grasp quality metric (GQ)based on contact points and pressure information [1]. Thuswe quantify how strong a grasp is across a given direction.

Consistency in Teaching. We evaluate user consistencyacross trials. We count the number of changes that occurin the extracted constraints, across each dimension, for agiven action. These determine individual segments with dif-ferent constraints [8]. Often over segmentation occurs whenthere is a disturbance in the motion, for example when theuser reacts poorly to the interaction with the environment,generating a trembling motion in the wrist and fingers.

Arm Coordination. We further study the coordination be-tween the two arms with respect to the variables extractedas important task constraints by estimating the causal re-lationship between them [9]. If the determined structurechanges within a segment then this is mostly caused by auser performing jerky motions. Unskilled people are likelyto have hand oscillations that produce such instances of de-coupling between the arms.

4. PRELIMINARY RESULTSWe considered the scooping task shown in Figure 1 and

conducted a user study with 10 participants. We used a 7degrees of freedom Kuka LWR robot, a data glove coveredwith Tekscan tactile sensors, and a tool with a 6 axis force-torque sensor. We recorded: the robot end effector cartesianpose and wrench; glove joint angles and tactile information,and exerted tool forces. We tracked the pose of the object,wrist and tool using an Optitrack vision system and recorded8 demonstrations per user. All demonstrations were success-ful with respect to the goal, however for some users it waseasy to complete the task, while others struggled.

First we determined the task constraints and analyzed thegrasp quality with respect to the frame of reference of thepassive arm. Results are shown in Fig 2(a). While the GQvaries significantly across users F (10, 80) = 19.354, p < 0.01,effective users (subjects 1, 2, 3 and 10) managed to obtaingood GQ across the directions of interest. Some of the otherusers handled the tool such as to apply forces perpendicular

Figure 4: (left) segments determined by discontinuities in the action;(right) causal connection between the torques exerted by the tool andthe motion of the robot’s end effector (EE), obtained using [9]

to the surface, rather than following the curvature of thefruit. The users did not improve across trials, the effect ofproviding multiple demonstrations on the grasp quality notbeing significant F (8, 80) = 0.488, p = 0.878. Typical graspsused by effective users are shown in Fig 3(a) for scooping and(b) for emptying the scoop. The hand preshape changes verylittle, but the contact intensity decreases significantly, whichshifts the center of the grasp. In contrast, a grasp used bya less successful user is shown in Fig 3(c).

Second we assessed the number of segments NS corre-sponding to the scooping action, for each user. The averagenumber of segments has a significant influence on the GQ,F (4, 79) = 3.631, p = 0.006, the two presenting weak in-verse correlation. Fig 4(a) presents the NS across users,with efficient users having less segments. Fig 2(b) shows acomparison between a skilled and an unskilled user, in thenumber of segments taken to complete the scooping action.

Third we determine the most common causal structurepresent in the data between arms, for the current setup,as shown in Fig 4(b). The variation in arm coordinationcan be quantified as the number of links that change in thestructure when analyzing subparts of a segment.

5. CONCLUSIONS AND FUTURE WORKIn this work we propose an approach for automatically

determining skilled demonstrators in tasks that require armcoordination and contact forces. We present preliminaryresults on a scooping task. Future work concerns the useof unsuccessful demonstrations for detecting what caused afail, and bootstrapping this as information for learning.

6. ACKNOWLEDGMENTSThe research leading to these results has received funding from

the European Union Seventh Framework Programme FP7/2007-2013under grant agreement no 288533 ROBOHOW.COG.

7. REFERENCES[1] C. Borst, M. Fischer, and G. Hirzinger. A fast and robust grasp

planner for arbitrary 3d objects. In ICRA, 1999.

[2] M. Cakmak and A. L. Thomaz. Designing robot learners thatask good questions. In HRI, 2012.

[3] M. Cakmak and A. L. Thomaz. Eliciting good teaching fromhumans for machine learners. Artificial Intelligence, 2014.

[4] S. Calinon and A. Billard. What is the teacher’s role in robotprogramming by demonstration? Interaction Studies, 2007.

[5] J. Chen and A. Zelinsky. Programing by demonstration: Copingwith suboptimal teaching actions, 2003.

[6] D. Grollman and A. Billard. Robot learning from faileddemonstrations. IJSR, 2012.

[7] M. Kaiser, H. Friedrich, and R. Dillmann. Obtaining goodperformance from a bad teacher. In ICML, 1995.

[8] A. L. Pais, K. Umezawa, Y. Nakamura, and A. Billard. Taskparametrization using continuous constraints extracted fromhuman demonstrations. Submitted, 2013.

[9] A. K. Seth. A matlab toolbox for granger causal connectivityanalysis. Journal of Neuroscience Methods, 2010.

38