QoLT Systems: Active Home

Project TeamProject Leader: ??PhD Students: Dmitry Berenson (CMU Robotics), Alvaro Collet (CMU Robotics), Rosen Diankov (CMURobotics), Mehmet Dogar (CMU Robotics), Anca Dragan (CMU Robotics), Mike Vande Weghe (CMU Robotics staff)Industrial Participants: Siddhartha Srinivasa (Intel Labs Pittsburgh)Pittsburgh)Project GoalsOur primary goal is to explore ways to provide physical assistance for Instrumental Activities of Daily Living in the home. We are currently developing a mobile manipulation system as one way to deliver such assistance. The key research challenges are building a safe, easy to operate system that can reliably perform common household tasks in environments that have the uncertainty typical of homes. Currently, we are conducting experiments on a WAM arm and Barrett hand mounted on a Segway mobile base located at Intel Labs Pittsburgh (one of the QoLT industry partners). We call this version of the household robot prototype HERB for Home Exploring Robot Butler.

Achievements The theme of this year’s work was uncertainty. MOPED (Figure 2), a new vision system designed to detect and accurately estimate

the pose of real-world objects in high clutter was developed and deployed on HERB. The novel algorithm achieves real-time performance by exploiting domain knowledge, both in the structure of the scene by adaptive clustering and in the computing architecture by parallelizing critical paths of the algorithm on the GPU.

Uncertainty in identification and in pose estimates is mitigated by an efficient multi-view extension of MOPED (Figure 3.) that combines multiple camera images with a sub-linear performance tradeoff.

A new framework of Task Space Constraints (TSRs) for manipulating objects with task constraints, like uncertain heavy weights or keeping mugs upright, was developed (Figure 4). The algorithm extends traditional bidirectional RRT algorithms to constraint manifolds. The algorithm is also capable of dealing with closed-chain kinematic constraints (Figure 6)

Uncertainty in object and environment pose estimates is taken into consideration in the TSR framework explicitly during planning time (Figure 5) enabling the planner to produce worst-case feasible plans that are driven by the sensor uncertainty and task requirements.

HERB uses every bit of sensor information to reduce its uncertainty about the state of the world. It even uses proprioceptive information when it bumps into objects to update the environment model or its pose estimate in the world (Figure 8). HERB infers a distribution over possible contacts by querying the error torques that it receives when it makes contact.

Tacking the uncertainty and dynamism of real-world environments was a major milestone reached this year. HERB is now outfitted with a spinning Hokuyo laser that provides noisy 3D data at 40,000 points per second. An occupancy grid model is used to filter this uncertain data into a world model for HERB to plan in. Combined with the pose estimate from MOPED, this forms the hallucination of the world that HERB sees. (Figure 9).

Major strides were also taken this year in human-robot interaction. A thorough study on the expectations people have on robots and the failure modes robots should exhibit to elicit a favorable response was published in HRI this year (Figure 7). Detailed scenarios of HERB interacting in a home were sketched out and visualized (Figure 10). A group of HCI students developed numerous prototypes of interaction

devices that were the outcome of a study of an eldercare facility in Pittsburgh and the interactions of the elders with their caregivers (Figure 11).

Six papers were presented at IEEE ICRA 2009, a top-tier robotics conference Five papers were accepted at IEEE ICRA 2010, a top-tier robotics conference Dmitry Berenson was awarded the Intel PhD Fellowship for the 2009-2010 academic

year Alvaro Collet was awarded with a La Caixa fellowship for Graduate Studies for the

2008-2009 academic year Mehmet Dogar, Dmitry Berenson, and Alvaro Collet were awarded the Intel Summer

Fellowship in 2009

Figure 1. HERB, the test platform for the Active Home project, with team members

Project Related Websites

Personal Robotics: http://personalrobotics.intel-research.netQoLT PerMMA:http://www.qolt.org/Research/Project.jsp?projectID=33CMU Robotics Institute: Planning and Autonomy Labhttp://www.ri.cmu.edu/labs/lab_76.htmlPlanning for Manipulation:http://www.ri.cmu.edu/research_project_detail.html?project_id=638&menu_id=261OpenRAVE software:http://openrave.programmingvision.com/ (wiki and information page)http://sourceforge.net/projects/openrave (sourceforge main page)

Future work HERB 2.0 with two arms Infinite loop

Figure 2. The scalability of the object recognition and pose estimation system we developed is shown. Our system handles an object database comprised of 91 objects and 400.000 features with ease, enabling real time detection of objects in extremely complex images.

Figure 3. Object recognition algorithm on a set of three images. (top) Input images. (Bottom) Object recognition system is executed on all images simultaneously. Recognized objects are assigned to simple shapes and placed according to their estimated poses.

Figure 4. (Left) Configuration space of a 3DOF manipulator generated by exhaustive sampling. (Center) 3-Link Manipulator configurations corresponding to several points along a path that moves the weight from one table to the other. (Right) Snapshots from a 7DOF WAM arm with a 8.17kg end-effector mass executing a path found by the CBiRRT to move the dumbbell from one table to the other.

Figure 5. Typical results of tasks three object grasping tasks. The intersecting boxes above show several of the intersecting TSRs for these tasks. In task 2 the TSR for grasping the box from the top is eliminated by the uncertainty (there is no point where all the boxes intersect) while the one for grasping it from the side is not.

Figure 6. Snapshots from several example plans generated using TSR chains. (Left) A closed-chain kinematics constraint for lifting a box. (Center) Simultaneously opening a door and putting a bottle into a refrigerator.

Scene Script Condition ManipulationChris is thirsty, and asks the robot to bring a can of Coke. The robot says, “OK.”

Forewarning: Chris is thirsty, and asks the robot to bring a can of Coke. The robot says, “OK, but it might be hard to identify Coke from other sodas.”

The robot looks at the Coke and Sprite on the counter.

Forewarning: The robot looks at the Coke and Sprite on the counter. The robot is confused because there are multiple cans.

After a few minutes, the robot comes back with a can of Sprite. Chris says, “OK, good. But I wanted a Coke.”

Control: After a few minutes, the robot comes back with a can of Coke. Chris says, “OK, good.”

The robot says, Apology: “I thought this was Coke. I apologize for bringing the wrong one.”Compensation:“I thought this was Coke. I will give you this drink for free.”Options:“I thought this was Coke. I can go back and try to find it. You can also show me a picture of a Coke, so I can recognize what it looks like.”No recovery & Control: “OK.”

Figure 7. Expectancy-setting and recovery strategies for personal robots. The above strategies were tested on 317 participants

Figure 8: Proprioceptive localization: (a) Shows the actual pose of the robot along with a translucent overlay illustrating the estimated localized pose. (b) Due to inaccurate localization, a collision occurs at the illustrated point. (c) Real and estimated robot poses after correction. (d) Task execution with updated pose estimate.

Figure 9: Online environment modeling: (left) Custom-built spinning laser unit (mid) occupancy grid model of environment updated online (right) grasp planning in clutter with environment model

Figure 10: Example scenarios for HERB performing assistive tasks in a home

Figure 11. User interface concepts for HERB suitable for interaction with aging adults