n“PaDY,” a Robot
Co-worker Concept Kazuhiro Kosuge
Department of Bioengineering and Robotics Graduate School of Engineering
Tohoku University Sendai 980-8579, JAPAN
http://www.irs.mech.tohoku.ac.jp�
A Robot Co-worker�
http://www.dti.dk/services/robot-co-worker-for-assembly/32733�
Danish Technological Institute� Roadmap for robot helpers�http://cordis.europa.eu/ictresults/index.cfm?section=news&tpl=article&id=91400�
Frida (ABB)�http://news.cnet.com/8301-17938_105-20056649-1.html�
http://www.spacedaily.com/reports/DLR_presents_innovations_in_robotics_at_AUTOMATICA_2012_999.html�
DLR�
“Robot Co-worker” Images from Google�
A Robot Co-worker Concept�
Human-Robot Cooperation (Kosuge, 1993)
A Robot Co-worker Concept�
FLAMMARION (July 3 1997)�
A Robot Co-worker � Outline�
• Overview of Robotics (Center for Research and Development Strategies, Japan Science and Technology Agency)
• Multiple Robots Coordination • Robot Helpers • Dance Partner Robot • PaDY • Conclusions
Overview of Robotics�
http://crds.jst.go.jp/singh/wp-content/uploads/09wr041.pdf�
Past,&Today&and&&Future&of&Robo3cs�
Ill7defined&Problem�&
������ �� ����"�������#
�!����
���������
Industrial Robots�
Human Environment�
Well7defined&Problem�&Structured Environment�
Robots in Human Environments�
www.care-o-bot.de�
www.aldebaran-robotics.com�
www.paro.jp�
www.anl.gov/Science_and_Technology/History/�Master-Slave Manipulator
Robotics�
• Social Level
• Service Level�– Service enablers�
• Fundamental Technologies Level�
Social'Value�
Services�
Founda1ons�
JST, CRDS, 2009, Modified by Kosuge, August, 2011�
Social Values�
• For Individualss rQuality of Life�• For Communities rIndustrial Competitiveness�
– For Families�– For Industries�– For Local Government�– For Nations�
• For the Globe rGlobal Issues�
JST, CRDS, 2009, Modified by Kosuge, August, 2011�
Robotics Research Field�
Social Value�
Services�
Emerging Technology
Fundamental�
• Robot Kinematics and Dynamics • Manipulation • Mobility • Actuation • Physics-based Control
• Robot Systems Integration • Human Robot Interaction • Real-world Real-time Intelligence • Spatio-temporal System Design • Sensing and Machine Cognition
Global Level� Community Level� Quality of Life
• Software framework • Social Concerns • Functional Safety�• Nano-micro Robotics�• Human Modeling�• Wearable Technology�• Service Contents Design�
• Cyborg (Cybernetic organism) • Stochasticity in Robotics • Performance evaluation and Benchmarking • Ambient intelligence • Autonomous Robots�• Teleoperation�• Robotic Emotion (artificial emotion)
m E`h\da`_X`fT^>Ka`\fad\`Z�m LTfgdT^>OXeagdVXe>Eib^adTf\a`>�T`W>DXhX^ab_X`f�
m PbTVX>Eib^adTf\a`�m DXXb>R`WXdeXT>T`W>R`WXdZdag`W>�Eib^adTf\a`�
m B`f\?fXddad\e_>>>>�OXeVgX>MbXdTf\a`>>>>�m NdXhX`f\a`>aY>I`YXVf\age>D\eXTeXe�
m Rf\^\f\Xe�m OXfT\^Xd@S[a^XeT^Xd�m QdT`ebadfTf\a`�m Ca__g`\VTf\a`�m PXdh\VX>I`Wgefd\Xe�m KXW\V\`X�m EWgVTf\a`�m OXeXTdV[>T`W>�DXhX^ab_X`f�
m KXW\V\`X�m Q[XdTbj�m DT\^j>J\YX>Bee\ef�m HXT^f[VTdX�m OX[TU\^\fTf\a`�m KX`fT^>VTdX�m JXTd`\`Z�m C[\^W>VTdX�m HageX]XXb\`Z�
m PXVgd\fj�m KaU\^\fj�m P[abb\`Z�m HaUUj�m E`fXdfT\`_X`f�m Pbadfe�m Ca_Yadf>J\YX�m STfV[�m Ca__g`\VTf\a`�
m GahXd`_X`f>�PXdh\VX�m BZd\Vg^fgdX�m FadXefdj�m F\e[Xdj�m K\`\`Z�m KT`gYTVfgd\`Z�m Ca`efdgVf\a`�m STefXe>QdXTf_X`f@>�KT`TZX_X`f�
Service/A
pplication-oriented R
obotics R
obotics Foundations
JST, CRDS, 2009, Modified by Kosuge, June, 2012�
����
PXdh\VX@Bbb^\VTf\a`?�
ad\X`fXW>OaUaf\Ve�
OaUaf\Ve�
Fag`WTf\a
`e�
R`\f>QXV[`a^aZ\Xe�
OaUaf\Ve>�Fag`WTf\a`e�
Bbb^\VTf\a`e@PXdh\VXe�Da_T\`>o�E^WXd^j>CTdX�
Da_T\`>p�BZd\Vg^fgdX�
Da_T\`>q�KXW\V\`X�
CgddX`f>OaUaf>Fg`Vf\a`�
OXcg\dXW>PXdh\VXe�N
ew S
ervices�
QXV[`\VT^>IeegXe�
JST, CRDS, 2009, Modified by Kosuge, June, 2012�
OaUaf\Ve>OXeXTdV[>�
Outline�
• Overview of Robotics (Center for Research and Development Strategies, Japan Science and Technology Agency)
• Multiple Robots Coordination • Robot Helpers • Dance Partner Robot • PaDY • Conclusions
Cooperation of Humans for Handling an Object
Coordination of both arms Coordination of two arms by humans
Single-Master Multi-Slaves System (1989)
K. Kosuge, J. Ishikawa, K. Furuta, M. Sakai, ”Control of Single-Master Multi-Slave Manipulator Using VIM”, [Proceedings of the 1990 IEEE International Conference on Robotics and Automation (1990) 1172-1177]�
Cooperation of Humans for Handling an Object
Coordination of multiple humans
Multiple Mobile Manipulator Coordination (2001)
Y. Hirata, Y. Kume, Z. D. Wang, K. Kosuge, ”Decentralized Control of Multiple Mobile Manipulators Based on Virtual 3-D Caster Motion for Handling an Object in Cooperation with a Human”, [Proceedings of the 2003 IEEE International Conference on Robotics and Automation (2003) 938-943].�
Y. Kume, Y. Hirata, Z. D. Wang, K. Kosuge, ”Decentralized Control of Multiple Mobile Manipulators Handling a Single Object in Coordination”, [Proceedings of the 2002 IEEE/RSJ International Conference on Intelligent Robots and Systems (2002) 2758-2763].�
Cooperation of Mobile Dual Manipulators (2003)
Y. Hirata, Y. Kume, T. Sawada, Z. D. Wang, K. Kosuge, ”Handling of an Object by Multiple Mobile Manipulators in Coordination based on Caster-like Dynamics”, [Proceedings of the 2004 IEEE International Conference on Robotics and Automation (2004) 807-812].�
Outline�
• Overview of Robotics (Center for Research and Development Strategies, Japan Science and Technology Agency)
• Multiple Robots Coordination • Robot Helpers • Dance Partner Robot • PaDY • Conclusions
Human Power Augmentation
Human Power Augmentation (Kosuge, 1993)
K. Kosuge, Y. Fujisawa, T. Fukuda, ”Control of Robot Directly Maneuvered by Operator”, [Proceedings of the 1993 IEEE/RSJ International Conference on Intelligent Robots and Systems (1993) 49-54].�
Robot Helpers
Human-Robot Cooperation (Kosuge, 1993)
MR Helper (Mobile Robot Helper, 1997~)
Robot Helpers
K. Kosuge, M. Sato, ”Mobile Robot Helper”, [Proceedings of the 2000 IEEE International Conference on Robotics and Automation (2000) 583-588].�
�/��, <)(�, ="�*, �0�, 25�, ”�t�.�+��}yzkMR Helper”uwx'�v 3�6l, [�!$#��4�;(C-) 69 (685) (2003) 84-90].�
Robot Helpers
DR Helpers (Distributed Robot Helpers) Y. Hirata, K. Kosuge, ”Distributed Robot Helpers Handling a Single Object in Cooperation with a Human”, [Proceedings of the 2000 IEEE International Conference on Robotics and Automations (2000) 458-463].�
�)%�, :�8, �/��, &9�, ����, �,7 , ”�9t1�v���}yz|~{-tv 3uwx��'�v�6l, [�!$#��4�;(C-) 68 (668) (2002) 181-188].�
Lessons Learned
Lessons learned from Robot Helpers
• Some simple tasks, which could not be done by a human/humans, could be done with a robot helper(s).
• General tasks could not be done easily even with the assistive robot system(s), because the robot does not know how to collaborate with the human. �
Lessons Learned
• In order to collaborate with the user, the robot has to know – the task, – its user’s intention, and – how the user wants to be assisted�
Outline�
• Overview of Robotics (Center for Research and Development Strategies, Japan Science and Technology Agency)
• Multiple Robots Coordination • Robot Helpers • Dance Partner Robot • PaDY • Conclusions
Dance Partner Robot
To develop a mechanism for closer human-robot coordination/interaction
Ballroom Dances
• Consider a ball room dance performed by a dance couple, a male dancer and a female dancer.
• A dance consists of a set of steps of the dance.
Ballroom Dances
• The dance is lead by a male dancer.
Which step is coming next is controlled by the male dancer based on the rule of the dance and their surroundings.
Ballroom Dances
• The female dancer estimates the following step at each step transition through the interaction with the male dancer based on the knowledge of the dance.
Ballroom Dances
• The female dancer has to know the dance in order to dance with her partner:
– The set of the dance steps. – The dance transition rule.
– How to be lead by the male dancer or how to estimate the partner’s step.
Dance Partner Robot
" Consider a female dance robot. " The female dance robot has to
know how to dance.
MS DanceR (Mobile Smart Dance Robot)
" Omni-directional mobile base " Body Force Sensor (6-axis force/torque sensor) " Body frame " Controller " Batteries (8kg)
Omni-directional mobile base
Height 162cm Weight 43kg Ms DanceR
Body Force Sensor
Height 162cm Weight 43kg
MS DanceR (Mobile Smart Dance Robot)
Ms DanceR
" Omni-directional mobile base " Body Force Sensor (6-axis force/torque sensor) " Body frame " Controller " Batteries (8kg)
Step of Waltz
• Closed Change Left (CCL) – Consists of three steps. CCL begins with male’s left foot and ends with male’s left
foot.
Start
o o
p p
Male’s Step Female’s Step
Start
q Stop
q Stop
Left foot Left foot Right foot Right foot
A Set of Steps
• Closed Change Left (CCL) – Consists of three steps. CCL begins with
male’s left foot, and ends with male’s left foot. • Closed Change Right (CCR)
– Consists of three steps. CCR begins with male’s right foot, and ends with male’s right foot.
• Natural Turn (NT) – Consists of six steps including turning to the
right by 270 degrees. NT begins with male’s right foot, and ends with male’s left foot.
• Reverse Turn (RT) – Consists of six steps including turning to the
left by 270 degrees. RT begins with male’s left foot, and ends with male’s right foot.
• Square Turn (ST) – Consists of three steps including turning to
the right by 90 degrees. ST begins with male’s right foot, and ends with male’s right foot.
Modeling of Step Motion� Modeling of Step Motion
• We measure the motion of a dancing the waltz by using a motion capture system, and get the basic trajectories of these steps
Trajectories of NT in a plane (a) A trajectory along x-axis (b) A trajectory along y-axis (c) A trajectory about rotation
time for NT time for NT time for NT
disp
lace
men
t
disp
lace
men
t
disp
lace
men
t
Modeling of Step Transition Rule
“The Ballroom Technique”, Imperial Society of Teachers of Dancing�
A>Closed Change Left
A>Closed Impetus
A>Reverse Turn
A>Whisk
A>Hesitation Change A>Outside Change
A>Natural Spin Turn
A>Natural Turn
A>Weave from PP
A>Buck Whisk
A>Progressive Chasse to
A>Closed Change Right
A>Buck Lock
A>Reverse Corte
PCTR
CCR
BL
RC CFPP
BWH
WFPP
NT46 NT13
CI
RT13
WH
HC
OC
RT46
CCL
NST
CCL
CI
RT
WH
HC OC
NST
NT
WFPP
BWH
PCTR BL
CCR
RC
A>Chasse from PP CFPP
Right
Step Estimation�
• Design of a step estimator based on resultant force/moment applied to the upper body of the robot and detected by the body force sensor.�
Design of Estimator
Primitive Methods: Instantaneous Data at Transitions of Steps
Time
Forc
e/M
omen
t Dance Step “A” Dance Step “B”
Reference Data
effectiveT
Transition
��
Motion Generation
• Dancers continue to dance by coordinating their paces and rhythms each other.
– Motion generation of a selected step trajectory based on the modeled step motion.
– Motion coordination
Two functions are considered for the robot.
Motion Generator
+
+
Reference velocity based on Step Motion and
Corrective velocity generated based on force/moment acting on the robot
Desired velocity
where,
: force/moment applied by a human
Case1: Robot motion without motion adjustment
Case2: Robot motion with motion adjustment
: scaling factor
hx
hx
DsM +
1
α dx
fxsensorF
fx dx
α
sensorF
Control Architecture
Actuators
Step Estimator Motion Generator
Step Transition
Step Motion
Knowledge
Waltz Step Transition
Step Motion
Tango Step Transition
Step Motion
Blues
Sensors
" CAST (Control Architecture based-on Step Transition)
Robot Hardware
Sensory information Desired motion
Knowledge of ballroom dances
Sensory information
Estimated step
Basic step trajectory
Experiment Aichi Expo (March 24 ~ September 25, 2005)
Design of Estimator
Time
Forc
e/M
omen
t
Dance Step “A” Dance Step “B”
Reference Data
effectiveT
Transition
Primitive Methods: Instantaneous Data at Transitions of Steps
��
Sensory Data Used for Estimation
Time series data include uncertainty such as time-lag and variation because a dancer cannot always apply the same force/moment for each step transition.
Time Fo
rce/
Mom
ent
Dance Step “A” Dance Step “B”
Reference Data
effectiveT
Transition
Sensory Data Used for Estimation
Time
Forc
e/M
omen
t
Dance Step “A” Dance Step “B”
effectiveT
Transition
HMM-based “Step Estimator”
Next Dance Step
Design of New Robot Body
Tatsuya Okonogi, a dress designer
Partner Ballroom Dance Robot Challenges for Dance Partner Robots
• Stable physical interaction between a human and robot
• Female Dance Partner Robot – Human behavior/intention estimation
• How to read the its partner’s lead
• Male Dance Partner Robot – How to convey robot’s intention to its human partner
• Motion Entrainment – Based on Human modeling
• Emotional Behavior
Outline�
• Overview of Robotics (Center for Research and Development Strategies, Japan Science and Technology Agency)
• Multiple Robots Coordination • Robot Helpers • Dance Partner Robot • PaDY • Conclusions
Automobile Assembly Line
• If a robot could provide the worker with necessary parts and tools when he/she needs them, the worker could concentrate on the assembly tasks.
• What the worker has to do is to assemble delivered parts using delivered tools to accomplish the work.
• PaDY is a robot which delivers necessary parts and tools to a worker when he/she needs them.
– to reduce the worker’s load – to improve efficiency of the work�– to prevent mistakes of the work – etc.
Working Partner Robot “PaDY”�
“in-time Parts/tools Delivery to You” robot
PaDY
Evaluation Experiment�
Measured data�
��LRF1 �LRF2
2010/10/21�Estimated result�
Effect of PaDY�
The worker’s motion necessary for picking parts/tools has been reduced. The worker could finish his tasks earlier than the work schedule.
Outline�
• Overview of Robotics (Center for Research and Development Strategies, Japan Science and Technology Agency)
• Multiple Robots Coordination • Robot Helpers • Dance Partner Robot • PaDY • Conclusions
Conclusions�
• We introduced our research history of robot helpers and a dance partner robot as a research plat form for human-robot cooperation.
• PaDY has been proposed as a real world application of the human-robot cooperation.
����
PXdh\VX@Bbb^\VTf\a`?�
ad\X`fXW>OaUaf\Ve�
OaUaf\Ve�
Fag`WTf\a
`e�
R`\f>QXV[`a^aZ\Xe�
OaUaf\Ve>�Fag`WTf\a`e�
Bbb^\VTf\a`e@PXdh\VXe�Da_T\`>o�E^WXd^j>CTdX�
Da_T\`>p�BZd\Vg^fgdX�
Da_T\`>q�KXW\V\`X�
CgddX`f>OaUaf>Fg`Vf\a`�
OXcg\dXW>PXdh\VXe�
New
Services
�
QXV[`\VT^>IeegXe�
JST, CRDS, 2009, Modified by Kosuge, June, 2012�
OaUaf\Ve>OXeXTdV[>�
Thank you very much for your attention !�