mutual development of behavior acquisition and recognition based on value … · 2011-11-15 · g...

Mutual Development of Behavior Acquisition and Recognition based onValue System

Yasutake Takahashi, Yoshihiro Tamura, and Minoru Asada∗Dept. of Adaptive Machine Systems, Graduate School of Engineering, Osaka University,

∗JST ERATO Asada Synergistic Intelligence ProjectYamadaoka 2-1, Suita, Osaka, 565-0871, Japan

Email: {yasutake,yoshihiro.tamura,asada}@ams.eng.osaka-u.ac.jp

Abstract— Both self-learning architecture (embedded struc-ture) and explicit/implicit teaching from other agents (environ-mental design issue) are necessary not only for one behaviorlearning but more seriously for life-time behavior learning. Thispaper presents a method for a robot to understand unfamiliarbehavior shown by others through the collaboration betweenbehavior acquisition and recognition of observed behavior,where the state value has an important role not simply forbehavior acquisition (reinforcement learning) but also for be-havior recognition (observation). That is, the state value updatescan be accelerated by observation without real trial and errorwhile the learned values enrich the recognition system since it isbased on estimation of the state value of the observed behavior.The validity of the proposed method is shown by applying itto a dynamic environment where two robots play soccer.

I. INTRODUCTION

Reinforcement learning has been studied well for motorskill learning and robot behavior acquisition in both sin-gle and multi-agent environments. Especially, in the multi-agent environment, observation of others make the behaviorlearning rapid and therefore much more efficient [1], [2],[3]. Actually, it is desirable to acquire various unfamiliarbehavior with some instructions from others, for example,surrounding robots and/or humans in real environment be-cause of huge exploration space and enormous learning timeto learn. Therefore, behavior learning through observationhas been more important. Understanding observed behaviordoes not mean simply following the trajectory of an end-effector or joints of demonstrator. It means reading its/his/herintention, that is, the goal of the observed behavior andfinding a way how to achieve the goal by oneself regardlessof the difference of the trajectory. From a viewpoint ofthe reinforcement learning framework, this means readingrewards of the observed behavior and estimating sequenceof the value through the observation.

Takahashi et al.[4] proposed a method of not only tolearn and execute a variety of behavior but also to recognizebehavior of others supposing that the observer has alreadyacquired the values of all kinds of behavior the observedagent can do. The recognition means, in this paper, thatthe robot categorizes the observed behavior to a set of itsown behavior acquired beforehand. The method seamlesslycombines behavior acquisition and recognition based on“state value” in reinforcement learning scheme. Reinforce-ment learning generates not only an appropriate behavior (a

map from states to actions) to accomplish a given task butalso a utility of the behavior, an estimated discounted sumof rewards that will be received in future while the robotis taking an appropriate policy. This estimated discountedsum of reward is called “state value.” This value roughlyindicates closeness to the goal state of the given task if therobot receives a positive reward when it reaches the goal andzero else, that is, if the agent is getting closer to the goal, thevalue becomes higher. This suggests that the observer mayrecognize which goal the observed agent likes to achieve ifthe value of the corresponding task is going higher.

This paper proposes a novel method that enhances behav-ior acquisition and recognition based on interaction betweenlearning and observation of behavior. A robot learns itsbehavior through not only trial and error but also readingrewards of the observed behavior of others (including sur-rounding robots and humans). Fig.1 shows a rough idea ofour proposed method. V (s) and V̂ (s) are the state valueupdated by oneself and the state value estimated throughobservation, respectively. Takahashi et al.[4] showed the

Fig. 1. Interaction between Learning and Observation of Behavior

capability of the proposed method mainly in case that theobserver has already acquired a number of behavior to berecognized beforehand. Their case study showed how thissystem recognizes observed behavior based on the statevalue functions of self-behavior. This paper shows howthe estimated state value of observed behavior, V̂ (s), givesfeedback to learning and understanding unfamiliar observedbehavior and this feedback loop enhances the performance ofobserved behavior recognition. The validity of the proposedmethod is shown by applying it to a dynamic environmentwhere two robots play soccer.

II. OUTLINE OF THE MECHANISMS

The reinforcement learning scheme, the state/action valuefunction, the modular learning system for various behavioracquisition/emulation, and behavior recognition based on thestate value are explained, here.

A. Behavior Learning Based on Reinforcement Learning

An agent can discriminate a set S of distinct worldstates. The world is modeled as a Markov process, makingstochastic transitions based on its current state and the actiontaken by the agent based on a policy π. The agent receivesreward rt at each step t. State value V π , the discounted sumof the reward received over time under execution of policyπ, will be calculated as follows:

V π =∞∑

t=0

γtrt . (1)

In case that the agent receives a positive reward if it stays aspecified goal and zero else, then, the state value increases ifthe agent follows an appropriate policy π. The agent updatesits policy through a process of trial and error in order toreceive higher positive rewards in future. Analogously, asanimals get closer to former action sequences that led togoals, they are more likely to retry it. For further details,please refer to the textbook of Sutton and Barto[5] or asurvey of robot learning[6].

Here we introduce a model-based reinforcement learningmethod. A learning module has a forward model whichrepresents the state transition model and a behavior learnerwhich estimates the state-action value function based on theforward model in a reinforcement learning manner.

Each learning module has its own state transition model.This model estimates the state transition probability P̂a

ss′ forthe triplet of state s, action a, and next state s′:

P̂ass′ = Pr{st+1 = s′|st = s, at = a} (2)

Each module has a reward model R̂s, too:

R̂(s) = E{rt|st = s} (3)

All experiences (sequences of state-action-next state andreward) are simply stored to estimate these models. Nowwe have the estimated state transition probability P̂a

ss′ andthe expected reward R̂s, then, an approximated state-actionvalue function Q(s, a) for a state action pair s and a is givenby

Q(s, a) =∑s′

P̂ass′

[R̂(s′) + γV (s′)

](4)

V (s) = maxa

Q(s, a) , (5)

where γ is a discount factor.

B. Modular Learning System

In order to observe/learn/execute a number of behavior, amodular learning system is adopted. Many modular architec-tures have been proposed so far (for example [7], [8], [9]).Each module is responsible for learning to achieve a single

s

environment

agent

actiona

a3

BM3

V(s)

controller

a2

BM2

V(s)

controller

a1

BM1

V(s)

controller

gate

s

environment

agent

actiona

a3

BM3

V(s)

controller

a2

BM2

V(s)

controller

a1

BM1

V(s)

controller

gate

agentagent

actiona

a3

BM3

V(s)

controller

a2

BM2

V(s)

controller

a1

BM1

V(s)

controller

gate

actiona

actiona

a3

BM3

V(s)

controller

a2

BM2

V(s)

controller

a1

BM1

V(s)

controller

gate

a3

BM3

V(s)

controller

a2

BM2

V(s)

controller

a1

BM1

V(s)

controller

a3

BM3

V(s)

controllera3a3

BM3

V(s)

controller

BM3

V(s)V(s)

controllercontroller

a2

BM2

V(s)

controllera2a2

BM2

V(s)

controller

BM2

V(s)V(s)


a1

BM1

V(s)

controllera1a1

BM1

V(s)

controller

BM1

V(s)V(s)


gategategate

Fig. 2. Modular learning system

Fig. 3. Behavior inference diagram

goal. One arbiter or a gate module is responsible for merginginformation from the individual modules in order to derivea single action performed by the robot.

Fig.2 shows a sketch of such a modular learning system.We prepare a number of behavior modules (BMs in thefigure) each of which adopts the behavior learning methoddescribed in II-A. The module is assigned to one goal-oriented behavior and estimates one action value functionQ(s, a). A module receives a positive reward when it ac-complishes the assigned behavior or zero reward else. Thebehavior module has a controller that selects the action withthe maximum value based on the predictions of next statevalues. The gating module will then select one output fromthe inputs of the different behavior modules according to theplayer’s intention.

Fig.3 shows a diagram to recognize an observed behav-ior. The same behavior modules are used for the behaviorrecognition. Each behavior module estimates the state valuebased on the estimated state of the observed demonstrator1

and calculates reliability of observed behavior, that is, howlikely the demonstrator is taking the behavior of the module.The details are described in following sections.

C. Behavior Recognition based on Estimated Values

Fig. 4. Behavior recognition based on the change of state value

Fig.4 shows an example task of navigation in a grid world

1For reasons of consistency, the term ”demonstrator” is used to describeany agent from which an observer can learn, even if the demonstrator doesnot have an intention to show its behavior to the observer.

and a map of the state value of the task. There is a goalstate at the top center of the world. An agent can move oneof the neighboring square in the grids every step. It receivesa positive reward only when it stays at the goal state whilezero else. There are various optimal/suboptimal policies forthis task as shown in the figure. If one tries to match theaction that the agent took and the one based on a certainpolicy in order to recognize the observed behavior, it has tomaintain various optimal policies and evaluate all of themin the worst case. On the other hand, if the agent followsan appropriate policy, the value is going up even if it isnot exactly the optimal one. Likewise, in emulation one isnot committed with the optimal policy, as the behavior arethe ones available in the portfolio of the agent, which arenot necessarily the optimal ones, but the ones that the agentknows to lead to the goal.

While an observer watches a demonstrator’s behavior, ituses the same behavior modules for recognition of observedbehavior as shown in Fig.3. Each behavior module estimatesthe state value based on the estimated state of the observeddemonstrator and sends it to the selector. The selectorwatches the sequence of the state values and selects a set ofpossible behavior modules of which state values are goingup as a set of behavior the demonstrator is currently taking.As mentioned before, if the state value goes up during abehavior, it means that the module is valid for explainingthe behavior. The observed behavior is recognized by a setof behavior whose modules’ values are increasing.

Here we define behavior recognition reliability g that indi-cates how much the observed behavior would be reasonableto be recognized as a behavior

g =

g + β if Vt − Vt−1 > 0 and g < 1g if Vt − Vt−1 = 0g − β if Vt − Vt−1 < 0 and g > 0 ,

where β is an update parameter, and 0.1 in this paper. Thisequation indicates that the reliability g will become large ifthe estimated utility rises up and it will become low whenthe estimated utility goes down. Another condition is to keepg value from 0 to 1.

D. Learning by Observation

In the previous section, behavior recognition system basedon state value of its own behavior is described. This systemshows robust recognition of observed behavior[10] onlywhen the behavior to be recognized has been well-learnedbeforehand. If the behavior is under learning, then, therecognition system is not able to show good recognitionperformance at beginning. The trajectory of the observedbehavior can be a bias for learning behavior and mightenhance the behavior learning based on the trajectory. Theobserver cannot watch actions of observed behavior directlyand can only estimate the sequence of the state of theobserved robot. Let so

t be the estimated state of the observedrobot at time t. Then, the estimated state value V̂ o of the

observed behavior can be calculated as below:

V̂ o(s) =∑s′

P̂oss′

[R̂(s′) + γV o(s′)

](6)

where P̂oss′ is state transition probability estimated from the

behavior observation. This state value function V̂ o can beused a bias of the state value function of the learner V . Thelearner updates its state-action value function Q(s, a) duringthe process of trial and error based on the estimated statevalue of observed behavior V̂ o as below:

Q(s, a) =∑s′

P̂ass′

[R̂(s′) + γV ′(s′)

](7)

while

V ′(s) ={

V (s) if V (s) > ηnV̂ o(s)ηnV̂ o(s) else

This is a normal update equation as shown in (4) exceptusing V ′(s). The update system switches the state value ofthe next state s′ between the state value of own learningbehavior V (s′) and the one of the observed behavior V̂ o(s′)discounted by ηn. ηn is the discount factor (0 < η1) based onthe time of experiences n. This means that it receives biggerfeedback with V̂ o(s′) if the state-action transition “(s, a) →s′” is not experienced so far while smaller else. This meansthe state value update system takes V̂ o(s′) if the learnerdoes not estimate the state value V (s′) because of lack ofexperience at the state s′ from which it reaches to the goalof the behavior. V̂ o(s′) becomes a bias for reinforcing theaction a from the state s even though the state value ofits own behavior V (s′) is small so that it leads the learnerto explore the space near to the goal state of the behavioreffectively.

A demonstrator is supposed to show a number of behaviorwhich are not informed directly to the observer. The observer,who learns/recognizes the behavior, assumes that all agentsshare reward models of the behavior, that is, all agents,including human players, receive a positive reward whenthe goal of the behavior is achieved . This assumptionis very natural as we assume that we share “value” withcolleagues, friends, or our family in our daily life. In order toupdate the estimate values of the behavior the demonstratoris taking, the observer has to estimate which behavior thedemonstrator is taking correctly. If the observer waits tolearn some specific behavior by observation until it becomesable to recognize the observed behavior well, bootstrapof leaning unfamiliar behavior by observation cannot beexpected. Therefore, the observer(learner) maintains a historyof the observed trajectories and updates value function ofthe observed behavior with high reliability or high receivedreward. The observer estimates the state of the demonstratorevery step and the reward received by the demonstrator isestimated as well. If it is estimated that the demonstratorreceives a positive reward by reaching to the goal state ofthe behavior, then, the observer updates the state value of thecorresponding behavior even if it has low reliability for theobserved behavior. The update strategy enhances to estimateappropriate values of the observed behavior.

III. BEHAVIOR LEARNING BY OBSERVATION

Fig. 5. Robots with a human player in a Soccer Field

Fig.5 shows two robots, a human player and color-codedobjects, e.g., an orange ball, and a goal. The robot hasan omni-directional camera on top. A simple color imageprocessing is applied in order to detect the color-codedobjects and players in real-time. The mobile platform isbased on an omni-directional vehicle. These two robots andthe human play soccer such as dribbling a ball, kicking itto a goal, passing a ball to the other, and so on. Whileplaying with objects, they watch each other, try to understandobserved behavior of the other, and emulate them.

Fig. 6. Estimation of view of the demonstrator. Left : a captured imagethe of observer, Center : object detection and state variables for self, Right: estimation of view of the demonstrator

Each behavior module estimates a state value of observedbehavior at an arbitrary time t to accomplish the specifiedtask. An observer watches a demonstrator’s behavior andmaps the sensory information from an observer viewpointto a demonstrator’s one with a simple mapping of statevariables. Fig.6 shows a simple example of this transforma-tion. It detects color-coded objects on the omni-directionalimage, calculates distances and directions of the objects inthe world coordinate of the observer, and shifts the axesso that the position of the demonstrator comes to centerof the demonstrator’s coordinate. Then it roughly estimatesthe state information in the egocentric coordinate and thestate of the demonstrator. Every behavior module estimatesa sequence of its state value from the estimated state ofthe observed demonstrator and the system selects moduleswhich values are increasing. The learner tries to acquirea number of behavior shown in Table II. The table alsodescribes necessary state variables for each behavior. Eachstate variable is divided into 11 in order to construct aquantized state space. 4 actions are prepared to be selected bythe learning modules: Approaching the goal, approaching theteammate, going in front of the ball while watching the goal,and going in front of the ball while watching the teammate.

TABLE ILIST OF BEHAVIOR LEARNED BY SELF AND STATE VARIABLES FOR EACH

BEHAVIOR

Behavior State variablesApproaching a ball db

Approaching a goal dg

Approaching the teammate dr

Shooting a ball db, dg , θbg

Passing a ball db, dr , θbr

A. Comparison Experiment Setup

In order to validate the effect of interaction betweenacquisition and recognition of behavior through observation,two experiments are set up. One is that the learner doesnot observe the behavior of other but tries to acquire shoot-ing/passing behavior by itself. The other is that the learnerobserves the behavior of other and enhances the learningof the behavior based on the estimated state value of theobserved behavior. In former experiment, the learner followsthe learning procedure:

1) 15 episodes for behavior learning by itself2) evaluation of self-behavior performance3) evaluation of behavior recognition performance4) goto 1.

On the other hand, the later experiment, it follows :1) 5 episodes for observation of the behavior of the other

2) 10 episodes for behavior learning by self-trials withobserved experience

3) evaluation of self-behavior performance4) evaluation of behavior recognition performance5) goto 1.

Both learners attempt to acquire behavior listed in Table II.The demonstrator shows one of the behavior one by one andthe observer does not know which behavior the demonstratoris taking. In both experiments, the learner follows ε-greedymethod; it follows the greedy policy with 80% probalibityand takes a random action else. Performance of the behaviorexecution and recognition of observed behavior during thelearning time is evaluated every 15 learning episodes. Theperformance of the behavior execution is the success rate ofthe behavior while the learner, the ball, and the teammateare placed at a set of pre-defined positions. The one of thebehavior recognition is the average length of period in whichthe recognition reliability of the right behavior is larger than70% during the observation. The soccer field area is divided3 by 3 and the center of the each area is a candidate ofthe position of the ball, the learner, or the teammate. Theperformances are evaluated in all possible combinations ofthe positions.

B. Recognition of Observed Behavior

Before evaluating the performance of the behavior ex-ecution and behavior recognition of other during learningthe behavior, we briefly review how this system estimatesthe values of behavior and recognizes the observed behavior

0

2

4

6

8

10

0 20 40 60 80 100 120 140 160 180

Sta

te v

alu

e

number of frame

ball approachgoal approach

teammate approachshootpass

(a) Estimated Values

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 20 40 60 80 100 120 140 160 180

Relia

bili

ty

number of frame

ball approachgoal approach

teammate approachshootpass

(b) Reliabilities

Fig. 7. Sequence of estimated values and reliabilities during a behavior ofpushing a ball to the magenta player, red line : approaching a ball, greenline : approaching the goal, light blue line : passing, blue line : approachingthe other, magenta line : shooting

after the observer has learned behavior. When the observerwatches a behavior of the other, it recognizes the observedbehavior based on repertoire of its own behavior. Figs.7 (a)and (b) show sequences of estimated values and reliabilitiesof the behavior, respectively. The line that indicates thepassing behavior keeps tendency of increasing value duringthe behavior in this figures. This behavior is composed ofbehavior of approaching a ball and approaching the teammateagain, then, the line of approaching a ball goes up at theearlier stage and the line of approaching the teammate goesup at the later stage in Fig.7(a). All reliabilities start from0.5 and increase if the value goes up and decrease else.Even when the value stays low, if it is increasing with smallvalue, the reliability of the behavior increases rapidly. Thereliability of the behavior of pushing a ball into the teammatereaches 1.0 at middle stage of the observed behavior. “Recog-nition period rate” of observed behavior is introduced here toevaluate how long the observer can recognize the observedbehavior as a correct one. The recognition period rate is 85%here ,that means, the period in which the reliability of passingbehavior is over 70% is 85% during the observation.

C. Performance of Behavior Learning and Recognition

In this section, performances of the behavior execution andbehavior recognition during learning the behavior are shown.

Fig.8 shows the success rates of the behavior and theirvariances during learning in cases of learning with/withoutvalue update through observation. The success rates withvalue update of all kinds of behavior grows more rapidlythan the one without observation feedback. Rapid learning isone of the most important aspect for a real robot application.The success rate without value update through observationsometimes could not reach the goal of the behavior at thebeginning of the learning because there is no bias to leadthe robot to learn appropriate actions. This is the reasonwhy the variances of the rate is big. On the other hand, thesystem with value update through observation utilizes theobservation to bootstrap the learning even though it cannotread exact actions of observed behavior.

Recognition performance and recognition period rate ofobserved behavior and their variances are shown in Figs.9and 10, respectively. They indicate a similar aspect with theones of the success rates. The performance of the behavior

0

20

40

60

80

100

0 50 100 150 200

Su

cce

ss r

ate

[%]

Number of episode

through observationno observation

(a) approaching the ball

0

20

40

60

80

100

0 50 100 150 200

Su

cce

ss r

ate

[%]

Number of episode


(b) approaching the goal

0

20

40

60

80

100

0 50 100 150 200

Su

cce

ss r

ate

[%]

Number of episode


(c) approaching the teammate

0

20

40

60

80

100

0 50 100 150 200

Su

cce

ss r

ate

[%]

Number of episode


(d) shooting the ball

0

20

40

60

80

100

0 50 100 150 200

Su

cce

ss r

ate

[%]

Number of episode


(e) passing to the teammate

Fig. 8. Success rate of the behavior during learning with/without obser-vation of demonstrator’s behavior

recognition depends on the learning performance. If thelearning system has not acquired data enough to estimatestate value of the behavior, it cannot perform well. Thelearning system with value update with observed behaviorrapidly enables to recognize the behavior while the systemwithout value update based on the observation has to waitto realize a good recognition performance until it estimatesgood state value of the behavior by its own trial and error.Those figures show the importance of learning throughinteraction between behavior acquisition and recognition ofobserved behavior.

D. Experiments with a real robot and a human demonstrator

The proposed architecture was applied to the real robotsystem. The robot observed demonstrations of shooting andpassing behavior played by a human player first, then, therobot learned the behavior by itself. The human playershowed shooting and passing behavior 5 times for each. Thereal learning time was half an hour for each behavior. Figs.11 and 12 show one sciene of human demonstration of thepassing behavior and acquired passing behavior, respectively.

E. Experiment with Two Humanoid Robots

In order to show wide applicability of the proposedmethod, another experiment has been done with two sim-ulated humanoid robots. Fig. ?? shows the scenario of ourexperiment. There are two players in front of a table. Twoobjects (red and blue boxes) are on the table. A player isindependently acquiring behavior of reaching for one of the

0

20

40

60

80

100

0 50 100 150 200

Re

co

gn

ito

n s

ucce

ss r

ate

[%]

Number of episode



0

20

40

60

80

100

0 50 100 150 200

Re

co

gn

ito

n s

ucce

ss r

ate

[%]

Number of episode



0

20

40

60

80

100

0 50 100 150 200

Re

co

gn

ito

n s

ucce

ss r

ate

[%]

Number of episode



0

20

40

60

80

100

0 50 100 150 200

Re

co

gn

ito

n s

ucce

ss r

ate

[%]

Number of episode



0

20

40

60

80

100

0 50 100 150 200

Re

co

gn

ito

n s

ucce

ss r

ate

[%]

Number of episode



Fig. 9. Recognition performance of the behavior during learningwith/without observation of demonstrator’s behavior

objects and pulling it to one of the color-coded (magenta andcyan) areas on the table, and update a value system basedon reinforcement learning. While playing with objects, theywatch each other, try to understand observed behavior of theother, and emulate them.

TABLE IILIST OF BEHAVIOR LEARNED BY SELF AND STATE VARIABLES FOR EACH

BEHAVIOR

Behavior State variablesPutting red box on magenta area dr and dm

Putting red box on cyan area dr and dc

Putting blue box on magenta area db and dm

Putting blue box on cyan area db and dc

The player tries to acquire a number of behavior shown inTable II. The table also describes necessary state variablesfor each behavior. dr, db, dm, and dc indecate distancesbetween the palm of the player and the red box, the bluebox, magenta area, and cyan area, respectively. Each statevariable is divided into 30 in order to construct a quantizedstate space. 4 actions are prepared to be selected by thelearning modules: reaching the red box, reaching the bluebox, pulling the box to the magenta area, and pulling thebox to the cyan area.

The success rates o the behavior, the recognition per-formance and the recognition period rate of observed be-havior and their variances are shown in Figs.14, 15 and16, respectively. Behavior “Putting red box on magenta

0

20

40

60

80

100

0 50 100 150 200

Re

co

gn

tio

n p

erio

d r

ate

[%]

Number of episode



0

20

40

60

80

100

0 50 100 150 200

Re

co

gn

tio

n p

erio

d r

ate

[%]

Number of episode



0

20

40

60

80

100

0 50 100 150 200

Re

co

gn

tio

n p

erio

d r

ate

[%]

Number of episode



0

20

40

60

80

100

0 50 100 150 200

Re

co

gn

tio

n p

erio

d r

ate

[%]

Number of episode



0

20

40

60

80

100

0 50 100 150 200

Re

co

gn

tio

n p

erio

d r

ate

[%]

Number of episode



Fig. 10. Recognition period rate of the behavior during learningwith/without observation of demonstrator’s behavior

Fig. 11. Demonstration of apassing behavior by a human

Fig. 12. Acquired passingbehavior after the observationand learning

area” and “Putting red box on cyan area” show that theobservation feedback makes better performances on learningand recognition of the observed behavior while the case ofbehavior “Putting blue box on magenta area” and “Puttingblue box on cyan area” does not show a big advantage. Thereason is that the cyan area is too close to the red andblue boxes then the learner fails to have feedback of theobservation of behavior “Putting blue boxe on areas.” It isno problem because those behavior is easy to learn. Behavior“Putting red box on areas” needs a long period to be achivedand learned and the observation feedback works effectively.

IV. CONCLUSION

Above, values are defined as behavior, which are de-fined by the achieved goals. The observer uses its ownvalue functions to recognize what the demonstrator will do.Preliminary investigations in a similar context have beendone by Takahashi at el. [10] and they showed much better

Fig. 13. Scenario of the experiment

0

20

40

60

80

100

0 20 40 60 80 100

Su

cce

ss r

ate

[%]

Number of episode


(a) Putting red box on magentaarea

0

20

40

60

80

100

0 20 40 60 80 100

Su

cce

ss r

ate

[%]

Number of episode


(b) Putting red box on cyan area

0

20

40

60

80

100

0 20 40 60 80 100

Su

cce

ss r

ate

[%]

Number of episode


(c) Putting blue box on magentaarea

0

20

40

60

80

100

0 20 40 60 80 100

Su

cce

ss r

ate

[%]

Number of episode


(d) Putting blue box on cyan area

Fig. 14. Success rate of the behavior during learning with/withoutobservation of demonstrator’s behavior

robustness of behavior recognition than a typical method.In this paper, unknown behavior are also understood interm of one’s own value function through learning basedon the estimated values derived from the observed behavior.Furthermore, value update through the observation enhancesnot only the performance of behavior learning but also theone of recognition of the observed behavior effectively.

V. ACKNOWLEDGMENTS

This work is partially supported by Kayamori Foundationof Informational Science Advancement.

REFERENCES

[1] S. D. Whitehead, “Complexity and cooperation in q-learning,” inProceedings Eighth International Workshop on Machine Learning(ML91), 1991, pp. 363–367.

[2] B. Price and C. Boutilier, “Accelerating reinforcement learning throughimplicit imitatione,” Journal of Articial Intelligence Research, 2003.

[3] D. C. Bentivegna, C. G. Atkeson, and G. Chenga, “Learning tasksfrom observation and practice,” Robotics and Autonomous Systems,vol. 47, pp. 163–169, 2004.

[4] Y. Takahashi, T. Kawamata, M. Asada, and M. Negrello, “Emulationand behavior understanding through shared values,” in Proceedingsof the 2007 IEEE/RSJ International Conference on Intelligent Robotsand Systems, Oct 2007, pp. 3950–3955.

0

20

40

60

80

100

0 20 40 60 80 100

Re

co

gn

ito

n s

ucce

ss r

ate

[%]

Number of episode



0

20

40

60

80

100

0 20 40 60 80 100

Re

co

gn

ito

n s

ucce

ss r

ate

[%]

Number of episode



0

20

40

60

80

100

0 20 40 60 80 100

Re

co

gn

ito

n s

ucce

ss r

ate

[%]

Number of episode



0

20

40

60

80

100

0 20 40 60 80 100

Re

co

gn

ito

n s

ucce

ss r

ate

[%]

Number of episode



Fig. 15. Recognition performance of the behavior during learningwith/without observation of demonstrator’s behavior

0

20

40

60

80

100

0 20 40 60 80 100

Re

co

gn

tio

n p

erio

d r

ate

[%]

Number of episode



0

20

40

60

80

100

0 20 40 60 80 100

Re

co

gn

tio

n p

erio

d r

ate

[%]

Number of episode



0

20

40

60

80

100

0 20 40 60 80 100

Re

co

gn

tio

n p

erio

d r

ate

[%]

Number of episode



0

20

40

60

80

100

0 20 40 60 80 100

Re

co

gn

tio

n p

erio

d r

ate

[%]

Number of episode



Fig. 16. Recognition period rate of the behavior during learningwith/without observation of demonstrator’s behavior

[5] R. Sutton and A. Barto, Reinforcement Learning: An Introduction.Cambridge, MA: MIT Press, 1998. [Online]. Available:citeseer.ist.psu.edu/sutton98reinforcement.html

[6] J. H. Connell and S. Mahadevan, ROBOT LEARNING. KluwerAcademic Publishers, 1993.

[7] R. Jacobs, M. Jordan, N. S, and G. Hinton, “Adaptive mixture of localexperts,” Neural Computation, vol. 3, pp. 79–87, 1991.

[8] S. P. Singh, “Transfer of learning by composing solutions ofelemental sequential tasks.” Machine Learning, vol. 8, pp. 323–339,1992. [Online]. Available: citeseer.nj.nec.com/singh92transfer.html

[9] S. Whitehead, J. Karlsson, and J. Tenenberg, “Learning multiple goalbehavior via task decomposition and dynamic policy merging,” inROBOT LEARNING, J. H. Connell and S. Mahadevan, Eds. KluwerAcademic Publishers, 1993, ch. 3, pp. 45–78.

[10] Y. Takahashi, T. Kawamata, and M. Asada, “Learning utility forbehavior acquisition and intention inference of other agent,” in Pro-ceedings of the 2006 IEEE/RSJ IROS 2006 Workshop on Multi-objective Robotics, Oct 2006, pp. 25–31.

mutual development of behavior acquisition and recognition based on value … · 2011-11-15 · g...

Documents