ABOUT    |    MEDIA    |    CONTACT    |    CAREERS
  Autonomous Robotic Manipulation

Andy phone home! The robot aligns the phone (A), grasps and picks it up (B), then checks whether the keypad is facing in the right direction (C). Only then does it hang up the phone (D).

Andy detects the door, plans the best way to open it, and executes its plan. The monitor on the left shows how the robot models its world and evaluates a tree of behavior strategies.


The ARM-S software runs on Andy, CMU’s ARM robot, and controls its every move.  Andy can grasp and manipulate ordinary objects with its versatile manipulator arm and three-fingered hand (or end effector).  The software goes through a three-step process to complete a task:  scene analysis, task planning, and task execution.

Step 1: Scene Analysis
When Andy begins a task, it analyzes the scene to identify the object (or objects) to be manipulated, then plans how it will carry out its task.

Suppose the robot is picking up a hammer by its handle.  This is so easy for humans that even a young child can do it.  It’s much harder for a robot!  The robot must identify the hammer and figure out which part is its handle.  It matches 3D sensor input to a computer graphics mesh model of a hammer. But hammers come in a huge variety of colors, shapes and sizes.  The robot must be able to extract and identify the visual features of many different kinds of hammers. 

Step 2: Task Planning
Next, Andy plans the best way to reach for the hammer, grasp it by the handle, and lift it into the air.  It evaluates different arm movements and chooses the one that efficiently avoids obstacles and approaches the hammer from the best grasping angle.

Step 3: Task Execution
After Andy finishes analyzing and planning its approach, it carries out the task of picking up the hammer. 

The robot gets constant kinematic feedback from its joints and always knows the position of its arm.  Force sensors in its fingers and wrist detect when it touches or grasps an object. 

But what if the robot’s fingers slip?  It must try again or switch to a different strategy.   Each step in the task is represented by a branch on a behavior tree.  If the robot cannot complete a step, it tries again.  If it still cannot complete the step, it tries a different strategy (represented by a different sub-branch in the behavior tree).  Only when it’s exhausted all available strategies does it admit defeat.

National Robotics Engineering Center    SITE MAP    |    ABOUT    |    MEDIA    |    CONTACT    |    CAREERS    © Carnegie Mellon University