1

Vision-Guided Humanoid Walking- Concepts and Experimental Results -

Günther SchmidtR. Cupec J. Denk J. F. Seara O. Lorch

ViGWaM GroupInstitute of Automatic Control Engineering

Technische Universität München

http://www.lsr.ei.tum.de/~vigwam

SPP 1039 Autonomous WalkingProject: Perception-based Walking

VDI/VDE-GMA FA “Robotik”, 16. Juli 2003

2Motivation and Objectives

“Goal-Oriented Autonomous Walkingrequires Cognitive Abilities,

e.g. Merging of Locomotion and Perception”

ViGWaM: Vision Guided Walking Machine

3

BARt-UH with LSR-Guidance System

Experimental Platform #1: 2-D Walker

• Stabilized Walking Machine, Institute of Automatic Control (IRT) Universität Hannover

• Pan-Tilt Stereo Camera Head and Guidance System, Institute of Automatic Control Engineering (LSR) TU München

4Reactive Walking: Obstacle Avoidance Capability

January 2002

Video:Javier4.avi

5

Basic Scheme of Visual Guidance System

6

large

Smooth and Free Biped Locomotionin 3D-Scenario with Obstacles

• start and stop locomotion• change step-length• stride over small obstacles• make direction changes• step on platform, climb stairs• etc.

Robot requires ability to autonomously

barrier

7

large

Step Sequence Planning: Off-line

• start-/stop-primitive• cyclic primitive• transition primitive• obstacle primitive combination• curve primitive combination• stair primitives

Walking Primitives:

“Walking Primitives“for Statically or Dynamically Stable Biped Locomotion

8 Hybrid Optimization based Approach

Synthesis of WalkingPrimitives for “Step Database“pre-swing swing heel-contact

9

Strategy # 1: Formal, “Tree Search“

Step Sequence Selection and Execution: On-line

10

Strategy # 2: Rule-based, “3-Steps-Ahead”

11

12

Architecture of Visual Perception Systemcamera system inclination sensor encoders

edge detection

line extraction

calculation ofcamera poserelative to SF

dead reckoning

object detectionand localization

object classification

pose estimationby

object featuretracking

edge map

straight edges

FTC

detected objects

3D objects in view

Local Environment Map (LEM)

1 image / step

precise objectpose

F(k)TF(k-1)

1 im

age

/ 33

ms

object base edge

scene analysis

VISIONSYSTEM

WALKINGMACHINE

Vision for Walking

13

Phase 1: Line Extraction and Obstacle Detection

14

Phase 2: Obstacle Localization

15

Phase 2: Obstacle Localization, alternative

16

Phase 2: Obstacle Localization

17

Phase 3: Obstacle Classification, preliminarye.g. Barrier or Wall

Analysis of Obstacle Situation

– barrier: robot can step over it– wall: robot can possibly walk around

– stair: robot can climb on it

18

• For each 2D line - two hypotheses: – projection of a vertical 3D edge – projection of a horizontal 3D edge

• Orientation of the camera system relative to the gravity axis:

– pruning of vertical edge hypotheses – orientation horizontal edges

Phase 3: Obstacle Classification, finale.g. Recognition of Rectangular Objects

• Edge grouping Rectangular Objects

Stairs

Step 1

Step 2

19

20

Task-Dependent Selection of View Direction

fi Adaptation of View Direction

“Where to look next ?”: Intention Problem

– Limited Field of View– Active Vision System

StartAreaGoalAreaDesiredPathRealPathLandmarksSelf-LocalizationObjectsObstacle Avoidance

Gaze Control

21

Entscheidungsstrategie

Aufgabenspezifische Blickwinkelsteuerungsstragien

Decision Strategy

Task-SpecificGaze Control Strategy

Agents

Information Management

Hybrid EKF

Task and Situation Dependent Gaze Control

A1 A2 A3 A4 ...

Perception

Dead-Reckoning

PerceptionModel

RobotKin. Model

Vision for Locomotion

ViewDirection

Winner Selection Society

Environment Map

Self-Localization

Exploration, etc. ...

Obstacle Avoidance

22Simulation Results

Self-localizationonly

(xS, yS, qS) = (0,3 m, 0,2 m, 0°)

(sx, sy, sq) = (0,005 m, 0,005 m, 3°)

14 measurements per step

Task-SpecificGaze Control

Obstacle Avoidanceonly

23

24

Bio-inspired Guidance and Control Architecture

Biped Robot Guidance System

25

„Intelligent“ Biped Robot Johnnie

Experimental Platform # 2: 3-D Walker

• Stabilized Walking Machine, Institute for Applied Mechanics (AMM) TU München

• Pan-Tilt Stereo Camera Head and Guidance System, Institute of Automatic Control Engineering (LSR) TU München

26 Obstacle Avoidance and Self-localization

Demonstration of Perception-based Guidance Capabilities

preplannedwithout obstaclemodified path

modified path

27 Hannover Messe, April 2003

Video:johnnie_hannover_3.mpe

28

Major Achievements of Project• Systematic approach to vision-guided biped locomotion

• Successful demonstration of autonomous walking based on

– robust line based image processing techniques

– combination of real-time scene analysis and feature tracking

– offline planning and predictive step sequence selection

• Reactive behavior of biped due to

– dynamic updating of the local environment map

– situation-dependent replanning of step sequence

• Collection of invaluable experimental experience with developed concepts, techniques, and algorithms

• Important contribution to progress in autonomous biped robot walking

29

More Recent References

• Robust visual estimation using hybrid EKF technique [ICRA 2002]

• Advanced gaze control strategies for simultaneous self-localization and obstacle avoidance [ICRA 2003] , [HRC 2003]

• Vision-based step sequence execution by use of walking primitive data base [ICRA 2003] for - straight-ahead and curve walking along a given local path

- striding over or walking around obstacles

- stepping on platform or climbing stairs

• Results of experimental evaluation [IROS 2002] , [ISER 2002] , [AMS 2003]

30

Vision-Guided Humanoid Walking- Concepts and Experiments -

Günther SchmidtR. Cupec J. Denk J. F. Seara O. Lorch

ViGWaM GroupInstitute of Automatic Control Engineering

Technische Universität München

SPP 1039 Autonomes Laufen

http://www.lsr.ei.tum.de

Universität Bielefeld, 3. Juli 2003