Curiosity-driven Exploration for Mapless Navigationwith Deep Reinforcement Learning

Oleksii Zhelo1, Jingwei Zhang1, Lei Tai2, Ming Liu2, Wolfram Burgard1

Abstract— This paper investigates exploration strategies ofDeep Reinforcement Learning (DRL) methods to learn navi-gation policies for mobile robots. In particular, we augmentthe normal external reward for training DRL algorithms withintrinsic reward signals measured by curiosity. We test our ap-proach in a mapless navigation setting, where the autonomousagent is required to navigate without the occupancy map of theenvironment, to targets whose relative locations can be easilyacquired through low-cost solutions (e.g., visible light localiza-tion, Wi-Fi signal localization). We validate that the intrinsicmotivation is crucial to improving DRL performance in taskswith challenging exploration requirements. Our experimentalresults show that our proposed method is able to more effec-tively learn navigation policies, and has better generalizationcapabilities in previously unseen environments. A video of ourexperimental results can be found at https://goo.gl/pWbpcF.

I. INTRODUCTION

Deep Reinforcement Learning (DRL), deploying deepneural networks as function approximators for high-dimensional RL tasks, achieves state of the art performancein various fields of research [1].

DRL algorithms have been studied under the context oflearning navigation policies for mobile robots. Traditionalnavigation solutions in robotics generally require a system ofprocedures, such as Simultaneous Localization and Mapping(SLAM) [2], localization and path planning in a given map,etc. With the powerful representation learning capabilities ofdeep networks, DRL methods bring about the possibility oflearning control policies directly from raw sensory inputs,bypassing all the intermediate steps.

Eliminating the requirement for localization, mapping, orpath planning procedures, several DRL works have been pre-sented that learn successful navigation policies directly fromraw sensor inputs: target-driven navigation [3], successorfeature RL for transferring navigation policies [4], and usingauxiliary tasks to boost DRL training [5]. Many follow-upworks have also been proposed, such as embedding SLAM-like structures into DRL networks [6], or utilizing DRL formulti-robot collision avoidance [7].

In this paper, we focus specifically on mapless navigation,where the agent is expected to navigate to a designated goallocation without the knowledge of the map of its currentenvironment. We assume that the relative pose of the targetis easily acquirable for the agent via cheap localizationsolutions such as visible light localization [8] or Wi-Fi

1Department of Computer Science, Albert Ludwig University ofFreiburg. oleksii.zhelo@saturn.uni-freiburg.de,{zhang, burgard}@informatik.uni-freiburg.de

2Department of Electronic and Computer Engineering, The Hong KongUniversity of Science and Technology. {ltai, eelium}@ust.hk

start

target

robot

laser

path

Fig. 1: We study the problem of mapless navigation, wherewithout a given the map of the environment, the autonomousagent is required to navigate to a target whose relativelocation can be easily acquired by cheap localization solu-tions. Structures like long corridors and dead corners makeit challenging for DRL agents to learn optimal navigationpolicies. Trained with reward signals that are augmented byintrinsic motivation, our proposed agent efficiently tacklesthis task, and shows that its learned policy generalizes wellto previously unseen environments.

signal localization [9]. Tai et al.[10] successfully appliedDRL for mapless navigation, taking as input sparse laserrange readings, as well as the velocity of the robot and therelative target position. Trained with asynchronous DRL, thepolicy network outputs continuous control commands for anonholonomic mobile robot, and is directly deployable inreal-world indoor environments.

Most of the aforementioned methods, however, eitherrely on random exploration strategies like ε-greedy, or onstate-independent exploration by maximizing the entropy ofthe policy. As the previous works present experiments inenvironments which do not impose considerable challengesfor the exploration of DRL algorithms, we suspect that theseexploration approaches might not be sufficient for learningefficient navigation policies in more complex environments.

Proposed by Pathak et al.[11], the agent’s ability to predictthe consequences of its own actions can be used to measurethe novelty of the states, or the intrinsic curiosity. TheIntrinsic Curiosity Module (ICM) is proposed to acquirethis signal during the process of reinforcement learning,and its prediction error can serve as an intrinsic reward. Asillustrated in their experiments, the intrinsic reward, learnedcompletely in a self-supervised manner, can motivate theagent to better explore the current environment, and make

use of the structures of the environment for more efficientplanning strategies.

In this paper, we investigate exploration mechanisms foraiding DRL agents to learn successful navigation policiesin challenging environments that might contain structureslike long corridors and dead corners. We conduct a seriesof experiments in simulated environments, and show thesample-efficiency, stability, and generalization ability of ourproposed agent.

II. METHODS

A. Background

We formulate the problem of autonomous navigation as aMarkov Decision Process (MDP), where at each time step t,the agent receives an observation of its current state st, takesan action at, receives a reward Rt, and transits to the nextstate st+1 following the transition dynamics p(st+1|st,at)of the environment. For the mapless navigation task that weconsider, the state st consists of laser range readings slt andthe relative pose of the goal sgt . The task of the agent is toreach the goal position g without colliding with obstacles.

B. External Reward for Mapless Navigation

We define the extrinsic reward Re at timestep t as follows(λp and λω denote the scaling factors):

Ret = rt + λprpt + λωrωt , (1)

where rt imposes the main task (pt represents the pose ofthe agent at t):

rt =

rreach, if reaches goal,rcollision, if collides,λg(∥∥px,yt−1 − g

∥∥2− ‖px,yt − g‖2

), otherwise,

and the position- and orientation-based penalties rpt and rωtare defined as:

rpt =

{rposition, if

∥∥px,yt−1 − px,yt∥∥2= 0,

0, otherwise,

rωt = ‖atan2(pyt − gy,pxt − gx)− pωt ‖1 .

C. Intrinsic Reward for Curiosity-driven Exploration

On top of the normal external reward Re, we use intrinsicmotivation measured by curiosity, rewarding novel states toencourage exploration and consequently improve the learningand generalization performance of DRL agents in environ-ments that require more guided exploration.

Following the formulation of [11], we measure the in-trinsic reward Ri via an Intrinsic Curiosity Module (ICM)(depicted in Fig. 2). It contains several feature extractionlayers φ, a forward model (parameterized by ψf ), and aninverse model (parameterized by ψi).

First, slt and slt+1 are passed through φ, encoded into theircorresponding features φt and φt+1. Then, ψi predicts atfrom φt and φt+1 (trained through cross-entropy loss fordiscrete actions with the ground truth at), while ψf predictsφt+1 from φt and at (trained through mean squared error

at<latexit sha1_base64="Cr99y/R7FXOdf7dq/It0eHswjJw=">AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIoN6KXjxWMLbQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDVgcWhpn3eLMTplIYdN0vp7Kyura+Ud2sbW3v7O7V9w8eTJJpxn2WyER3Q2q4FIr7KFDybqo5jUPJO+HkpvA7j1wbkah7nKa8H9OREpFgFK0UBDHFcRjldDbAQb3hNt05yF/ilaQBJdqD+mcwTFgWc4VMUmN6nptiP6caBZN8Vgsyw1PKJnTEe5YqGnPTz+eZZ+TEKkMSJdo+hWSu/tzIaWzMNA7tZJHRLHuF+J/XyzC67OdCpRlyxRaHokwSTEhRABkKzRnKqSWUaWGzEjammjK0NdVsCd7yl/8S/6x51fTuzhut67KNKhzBMZyCBxfQgltogw8MUniCF3h1MufZeXPeF6MVp9w5hF9wPr4B1pWRwA==</latexit><latexit sha1_base64="Cr99y/R7FXOdf7dq/It0eHswjJw=">AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIoN6KXjxWMLbQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDVgcWhpn3eLMTplIYdN0vp7Kyura+Ud2sbW3v7O7V9w8eTJJpxn2WyER3Q2q4FIr7KFDybqo5jUPJO+HkpvA7j1wbkah7nKa8H9OREpFgFK0UBDHFcRjldDbAQb3hNt05yF/ilaQBJdqD+mcwTFgWc4VMUmN6nptiP6caBZN8Vgsyw1PKJnTEe5YqGnPTz+eZZ+TEKkMSJdo+hWSu/tzIaWzMNA7tZJHRLHuF+J/XyzC67OdCpRlyxRaHokwSTEhRABkKzRnKqSWUaWGzEjammjK0NdVsCd7yl/8S/6x51fTuzhut67KNKhzBMZyCBxfQgltogw8MUniCF3h1MufZeXPeF6MVp9w5hF9wPr4B1pWRwA==</latexit><latexit sha1_base64="Cr99y/R7FXOdf7dq/It0eHswjJw=">AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIoN6KXjxWMLbQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDVgcWhpn3eLMTplIYdN0vp7Kyura+Ud2sbW3v7O7V9w8eTJJpxn2WyER3Q2q4FIr7KFDybqo5jUPJO+HkpvA7j1wbkah7nKa8H9OREpFgFK0UBDHFcRjldDbAQb3hNt05yF/ilaQBJdqD+mcwTFgWc4VMUmN6nptiP6caBZN8Vgsyw1PKJnTEe5YqGnPTz+eZZ+TEKkMSJdo+hWSu/tzIaWzMNA7tZJHRLHuF+J/XyzC67OdCpRlyxRaHokwSTEhRABkKzRnKqSWUaWGzEjammjK0NdVsCd7yl/8S/6x51fTuzhut67KNKhzBMZyCBxfQgltogw8MUniCF3h1MufZeXPeF6MVp9w5hF9wPr4B1pWRwA==</latexit><latexit sha1_base64="Cr99y/R7FXOdf7dq/It0eHswjJw=">AAAB8nicbVBNS8NAFHypX7V+VT16WSyCp5KIoN6KXjxWMLbQlLLZbtqlm03YfRFK6N/w4kHFq7/Gm//GTZuDVgcWhpn3eLMTplIYdN0vp7Kyura+Ud2sbW3v7O7V9w8eTJJpxn2WyER3Q2q4FIr7KFDybqo5jUPJO+HkpvA7j1wbkah7nKa8H9OREpFgFK0UBDHFcRjldDbAQb3hNt05yF/ilaQBJdqD+mcwTFgWc4VMUmN6nptiP6caBZN8Vgsyw1PKJnTEe5YqGnPTz+eZZ+TEKkMSJdo+hWSu/tzIaWzMNA7tZJHRLHuF+J/XyzC67OdCpRlyxRaHokwSTEhRABkKzRnKqSWUaWGzEjammjK0NdVsCd7yl/8S/6x51fTuzhut67KNKhzBMZyCBxfQgltogw8MUniCF3h1MufZeXPeF6MVp9w5hF9wPr4B1pWRwA==</latexit>

�t<latexit sha1_base64="x3b53dPGNANEBe1aqtb91HvLaQk=">AAAB7HicbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMF0xbaUDbbTbt2swm7E6GE/gcvHlS8+oO8+W/ctjlo64OBx3szzMwLUykMuu63s7K6tr6xWdoqb+/s7u1XDg6bJsk04z5LZKLbITVcCsV9FCh5O9WcxqHkrXB0O/VbT1wbkagHHKc8iOlAiUgwilZqdtOh6GGvUnVr7gxkmXgFqUKBRq/y1e0nLIu5QiapMR3PTTHIqUbBJJ+Uu5nhKWUjOuAdSxWNuQny2bUTcmqVPokSbUshmam/J3IaGzOOQ9sZUxyaRW8q/ud1MoyuglyoNEOu2HxRlEmCCZm+TvpCc4ZybAllWthbCRtSTRnagMo2BG/x5WXin9eua979RbV+U6RRgmM4gTPw4BLqcAcN8IHBIzzDK7w5ifPivDsf89YVp5g5gj9wPn8ADTuO+g==</latexit><latexit sha1_base64="x3b53dPGNANEBe1aqtb91HvLaQk=">AAAB7HicbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMF0xbaUDbbTbt2swm7E6GE/gcvHlS8+oO8+W/ctjlo64OBx3szzMwLUykMuu63s7K6tr6xWdoqb+/s7u1XDg6bJsk04z5LZKLbITVcCsV9FCh5O9WcxqHkrXB0O/VbT1wbkagHHKc8iOlAiUgwilZqdtOh6GGvUnVr7gxkmXgFqUKBRq/y1e0nLIu5QiapMR3PTTHIqUbBJJ+Uu5nhKWUjOuAdSxWNuQny2bUTcmqVPokSbUshmam/J3IaGzOOQ9sZUxyaRW8q/ud1MoyuglyoNEOu2HxRlEmCCZm+TvpCc4ZybAllWthbCRtSTRnagMo2BG/x5WXin9eua979RbV+U6RRgmM4gTPw4BLqcAcN8IHBIzzDK7w5ifPivDsf89YVp5g5gj9wPn8ADTuO+g==</latexit><latexit sha1_base64="x3b53dPGNANEBe1aqtb91HvLaQk=">AAAB7HicbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMF0xbaUDbbTbt2swm7E6GE/gcvHlS8+oO8+W/ctjlo64OBx3szzMwLUykMuu63s7K6tr6xWdoqb+/s7u1XDg6bJsk04z5LZKLbITVcCsV9FCh5O9WcxqHkrXB0O/VbT1wbkagHHKc8iOlAiUgwilZqdtOh6GGvUnVr7gxkmXgFqUKBRq/y1e0nLIu5QiapMR3PTTHIqUbBJJ+Uu5nhKWUjOuAdSxWNuQny2bUTcmqVPokSbUshmam/J3IaGzOOQ9sZUxyaRW8q/ud1MoyuglyoNEOu2HxRlEmCCZm+TvpCc4ZybAllWthbCRtSTRnagMo2BG/x5WXin9eua979RbV+U6RRgmM4gTPw4BLqcAcN8IHBIzzDK7w5ifPivDsf89YVp5g5gj9wPn8ADTuO+g==</latexit><latexit sha1_base64="x3b53dPGNANEBe1aqtb91HvLaQk=">AAAB7HicbVBNS8NAEJ34WetX1aOXxSJ4KokI6q3oxWMF0xbaUDbbTbt2swm7E6GE/gcvHlS8+oO8+W/ctjlo64OBx3szzMwLUykMuu63s7K6tr6xWdoqb+/s7u1XDg6bJsk04z5LZKLbITVcCsV9FCh5O9WcxqHkrXB0O/VbT1wbkagHHKc8iOlAiUgwilZqdtOh6GGvUnVr7gxkmXgFqUKBRq/y1e0nLIu5QiapMR3PTTHIqUbBJJ+Uu5nhKWUjOuAdSxWNuQny2bUTcmqVPokSbUshmam/J3IaGzOOQ9sZUxyaRW8q/ud1MoyuglyoNEOu2HxRlEmCCZm+TvpCc4ZybAllWthbCRtSTRnagMo2BG/x5WXin9eua979RbV+U6RRgmM4gTPw4BLqcAcN8IHBIzzDK7w5ifPivDsf89YVp5g5gj9wPn8ADTuO+g==</latexit>

�t+1<latexit sha1_base64="PfS42iwSC6IkYBmEGyh8+B1o6hg=">AAAB8HicbVBNS8NAEJ34WetX1aOXxSIIQklEUG9FLx4rGFtsQ9lsN+3SzSbsToQS+i+8eFDx6s/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfztLyyuraemmjvLm1vbNb2dt/MEmmGfdZIhPdCqnhUijuo0DJW6nmNA4lb4bDm4nffOLaiETd4yjlQUz7SkSCUbTSYycdiG6Op964W6m6NXcKski8glShQKNb+er0EpbFXCGT1Ji256YY5FSjYJKPy53M8JSyIe3ztqWKxtwE+fTiMTm2So9EibalkEzV3xM5jY0ZxaHtjCkOzLw3Ef/z2hlGl0EuVJohV2y2KMokwYRM3ic9oTlDObKEMi3srYQNqKYMbUhlG4I3//Ii8c9qVzXv7rxavy7SKMEhHMEJeHABdbiFBvjAQMEzvMKbY5wX5935mLUuOcXMAfyB8/kDrtSQdg==</latexit><latexit sha1_base64="PfS42iwSC6IkYBmEGyh8+B1o6hg=">AAAB8HicbVBNS8NAEJ34WetX1aOXxSIIQklEUG9FLx4rGFtsQ9lsN+3SzSbsToQS+i+8eFDx6s/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfztLyyuraemmjvLm1vbNb2dt/MEmmGfdZIhPdCqnhUijuo0DJW6nmNA4lb4bDm4nffOLaiETd4yjlQUz7SkSCUbTSYycdiG6Op964W6m6NXcKski8glShQKNb+er0EpbFXCGT1Ji256YY5FSjYJKPy53M8JSyIe3ztqWKxtwE+fTiMTm2So9EibalkEzV3xM5jY0ZxaHtjCkOzLw3Ef/z2hlGl0EuVJohV2y2KMokwYRM3ic9oTlDObKEMi3srYQNqKYMbUhlG4I3//Ii8c9qVzXv7rxavy7SKMEhHMEJeHABdbiFBvjAQMEzvMKbY5wX5935mLUuOcXMAfyB8/kDrtSQdg==</latexit><latexit sha1_base64="PfS42iwSC6IkYBmEGyh8+B1o6hg=">AAAB8HicbVBNS8NAEJ34WetX1aOXxSIIQklEUG9FLx4rGFtsQ9lsN+3SzSbsToQS+i+8eFDx6s/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfztLyyuraemmjvLm1vbNb2dt/MEmmGfdZIhPdCqnhUijuo0DJW6nmNA4lb4bDm4nffOLaiETd4yjlQUz7SkSCUbTSYycdiG6Op964W6m6NXcKski8glShQKNb+er0EpbFXCGT1Ji256YY5FSjYJKPy53M8JSyIe3ztqWKxtwE+fTiMTm2So9EibalkEzV3xM5jY0ZxaHtjCkOzLw3Ef/z2hlGl0EuVJohV2y2KMokwYRM3ic9oTlDObKEMi3srYQNqKYMbUhlG4I3//Ii8c9qVzXv7rxavy7SKMEhHMEJeHABdbiFBvjAQMEzvMKbY5wX5935mLUuOcXMAfyB8/kDrtSQdg==</latexit><latexit sha1_base64="PfS42iwSC6IkYBmEGyh8+B1o6hg=">AAAB8HicbVBNS8NAEJ34WetX1aOXxSIIQklEUG9FLx4rGFtsQ9lsN+3SzSbsToQS+i+8eFDx6s/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfztLyyuraemmjvLm1vbNb2dt/MEmmGfdZIhPdCqnhUijuo0DJW6nmNA4lb4bDm4nffOLaiETd4yjlQUz7SkSCUbTSYycdiG6Op964W6m6NXcKski8glShQKNb+er0EpbFXCGT1Ji256YY5FSjYJKPy53M8JSyIe3ztqWKxtwE+fTiMTm2So9EibalkEzV3xM5jY0ZxaHtjCkOzLw3Ef/z2hlGl0EuVJohV2y2KMokwYRM3ic9oTlDObKEMi3srYQNqKYMbUhlG4I3//Ii8c9qVzXv7rxavy7SKMEhHMEJeHABdbiFBvjAQMEzvMKbY5wX5935mLUuOcXMAfyB8/kDrtSQdg==</latexit>

f<latexit sha1_base64="uIk/o9Q6WRdk1rNsfCaOulCtOw4=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG9FLx4rmLbQxrLZbtq1m92wuxFK6H/w4kHFqz/Im//GTZuDtj4YeLw3w8y8MOFMG9f9dkorq2vrG+XNytb2zu5edf+gpWWqCPWJ5FJ1QqwpZ4L6hhlOO4miOA45bYfjm9xvP1GlmRT3ZpLQIMZDwSJGsLFSq5do9hD1qzW37s6AlolXkBoUaParX72BJGlMhSEca9313MQEGVaGEU6nlV6qaYLJGA9p11KBY6qDbHbtFJ1YZYAiqWwJg2bq74kMx1pP4tB2xtiM9KKXi/953dREl0HGRJIaKsh8UZRyZCTKX0cDpigxfGIJJorZWxEZYYWJsQFVbAje4svLxD+rX9W9u/Na47pIowxHcAyn4MEFNOAWmuADgUd4hld4c6Tz4rw7H/PWklPMHMIfOJ8/B0+O9g==</latexit><latexit sha1_base64="uIk/o9Q6WRdk1rNsfCaOulCtOw4=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG9FLx4rmLbQxrLZbtq1m92wuxFK6H/w4kHFqz/Im//GTZuDtj4YeLw3w8y8MOFMG9f9dkorq2vrG+XNytb2zu5edf+gpWWqCPWJ5FJ1QqwpZ4L6hhlOO4miOA45bYfjm9xvP1GlmRT3ZpLQIMZDwSJGsLFSq5do9hD1qzW37s6AlolXkBoUaParX72BJGlMhSEca9313MQEGVaGEU6nlV6qaYLJGA9p11KBY6qDbHbtFJ1YZYAiqWwJg2bq74kMx1pP4tB2xtiM9KKXi/953dREl0HGRJIaKsh8UZRyZCTKX0cDpigxfGIJJorZWxEZYYWJsQFVbAje4svLxD+rX9W9u/Na47pIowxHcAyn4MEFNOAWmuADgUd4hld4c6Tz4rw7H/PWklPMHMIfOJ8/B0+O9g==</latexit><latexit sha1_base64="uIk/o9Q6WRdk1rNsfCaOulCtOw4=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG9FLx4rmLbQxrLZbtq1m92wuxFK6H/w4kHFqz/Im//GTZuDtj4YeLw3w8y8MOFMG9f9dkorq2vrG+XNytb2zu5edf+gpWWqCPWJ5FJ1QqwpZ4L6hhlOO4miOA45bYfjm9xvP1GlmRT3ZpLQIMZDwSJGsLFSq5do9hD1qzW37s6AlolXkBoUaParX72BJGlMhSEca9313MQEGVaGEU6nlV6qaYLJGA9p11KBY6qDbHbtFJ1YZYAiqWwJg2bq74kMx1pP4tB2xtiM9KKXi/953dREl0HGRJIaKsh8UZRyZCTKX0cDpigxfGIJJorZWxEZYYWJsQFVbAje4svLxD+rX9W9u/Na47pIowxHcAyn4MEFNOAWmuADgUd4hld4c6Tz4rw7H/PWklPMHMIfOJ8/B0+O9g==</latexit><latexit sha1_base64="uIk/o9Q6WRdk1rNsfCaOulCtOw4=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG9FLx4rmLbQxrLZbtq1m92wuxFK6H/w4kHFqz/Im//GTZuDtj4YeLw3w8y8MOFMG9f9dkorq2vrG+XNytb2zu5edf+gpWWqCPWJ5FJ1QqwpZ4L6hhlOO4miOA45bYfjm9xvP1GlmRT3ZpLQIMZDwSJGsLFSq5do9hD1qzW37s6AlolXkBoUaParX72BJGlMhSEca9313MQEGVaGEU6nlV6qaYLJGA9p11KBY6qDbHbtFJ1YZYAiqWwJg2bq74kMx1pP4tB2xtiM9KKXi/953dREl0HGRJIaKsh8UZRyZCTKX0cDpigxfGIJJorZWxEZYYWJsQFVbAje4svLxD+rX9W9u/Na47pIowxHcAyn4MEFNOAWmuADgUd4hld4c6Tz4rw7H/PWklPMHMIfOJ8/B0+O9g==</latexit>

i<latexit sha1_base64="R5fHrAGgyf1vk798jITOr1GxDvU=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG9FLx4rmLbQxrLZbtq1m92wuxFK6H/w4kHFqz/Im//GTZuDtj4YeLw3w8y8MOFMG9f9dkorq2vrG+XNytb2zu5edf+gpWWqCPWJ5FJ1QqwpZ4L6hhlOO4miOA45bYfjm9xvP1GlmRT3ZpLQIMZDwSJGsLFSq5do9sD61Zpbd2dAy8QrSA0KNPvVr95AkjSmwhCOte56bmKCDCvDCKfTSi/VNMFkjIe0a6nAMdVBNrt2ik6sMkCRVLaEQTP190SGY60ncWg7Y2xGetHLxf+8bmqiyyBjIkkNFWS+KEo5MhLlr6MBU5QYPrEEE8XsrYiMsMLE2IAqNgRv8eVl4p/Vr+re3XmtcV2kUYYjOIZT8OACGnALTfCBwCM8wyu8OdJ5cd6dj3lrySlmDuEPnM8fC9iO+Q==</latexit><latexit sha1_base64="R5fHrAGgyf1vk798jITOr1GxDvU=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG9FLx4rmLbQxrLZbtq1m92wuxFK6H/w4kHFqz/Im//GTZuDtj4YeLw3w8y8MOFMG9f9dkorq2vrG+XNytb2zu5edf+gpWWqCPWJ5FJ1QqwpZ4L6hhlOO4miOA45bYfjm9xvP1GlmRT3ZpLQIMZDwSJGsLFSq5do9sD61Zpbd2dAy8QrSA0KNPvVr95AkjSmwhCOte56bmKCDCvDCKfTSi/VNMFkjIe0a6nAMdVBNrt2ik6sMkCRVLaEQTP190SGY60ncWg7Y2xGetHLxf+8bmqiyyBjIkkNFWS+KEo5MhLlr6MBU5QYPrEEE8XsrYiMsMLE2IAqNgRv8eVl4p/Vr+re3XmtcV2kUYYjOIZT8OACGnALTfCBwCM8wyu8OdJ5cd6dj3lrySlmDuEPnM8fC9iO+Q==</latexit><latexit sha1_base64="R5fHrAGgyf1vk798jITOr1GxDvU=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG9FLx4rmLbQxrLZbtq1m92wuxFK6H/w4kHFqz/Im//GTZuDtj4YeLw3w8y8MOFMG9f9dkorq2vrG+XNytb2zu5edf+gpWWqCPWJ5FJ1QqwpZ4L6hhlOO4miOA45bYfjm9xvP1GlmRT3ZpLQIMZDwSJGsLFSq5do9sD61Zpbd2dAy8QrSA0KNPvVr95AkjSmwhCOte56bmKCDCvDCKfTSi/VNMFkjIe0a6nAMdVBNrt2ik6sMkCRVLaEQTP190SGY60ncWg7Y2xGetHLxf+8bmqiyyBjIkkNFWS+KEo5MhLlr6MBU5QYPrEEE8XsrYiMsMLE2IAqNgRv8eVl4p/Vr+re3XmtcV2kUYYjOIZT8OACGnALTfCBwCM8wyu8OdJ5cd6dj3lrySlmDuEPnM8fC9iO+Q==</latexit><latexit sha1_base64="R5fHrAGgyf1vk798jITOr1GxDvU=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEUG9FLx4rmLbQxrLZbtq1m92wuxFK6H/w4kHFqz/Im//GTZuDtj4YeLw3w8y8MOFMG9f9dkorq2vrG+XNytb2zu5edf+gpWWqCPWJ5FJ1QqwpZ4L6hhlOO4miOA45bYfjm9xvP1GlmRT3ZpLQIMZDwSJGsLFSq5do9sD61Zpbd2dAy8QrSA0KNPvVr95AkjSmwhCOte56bmKCDCvDCKfTSi/VNMFkjIe0a6nAMdVBNrt2ik6sMkCRVLaEQTP190SGY60ncWg7Y2xGetHLxf+8bmqiyyBjIkkNFWS+KEo5MhLlr6MBU5QYPrEEE8XsrYiMsMLE2IAqNgRv8eVl4p/Vr+re3XmtcV2kUYYjOIZT8OACGnALTfCBwCM8wyu8OdJ5cd6dj3lrySlmDuEPnM8fC9iO+Q==</latexit>

�t+1<latexit sha1_base64="btFPvwTuPnBe+o1whDDCf279DQs=">AAAB+HicbVBNS8NAEN34WetX1KOXxSIIQklEUG9FLx4rGFtoQthsN+3SzSbsTgol5J948aDi1Z/izX/jts1BWx8MPN6bYWZelAmuwXG+rZXVtfWNzdpWfXtnd2/fPjh80mmuKPNoKlLVjYhmgkvmAQfBupliJIkE60Sju6nfGTOleSofYZKxICEDyWNOCRgptG1/SKDwsyEvwwLO3TK0G07TmQEvE7ciDVShHdpffj+lecIkUEG07rlOBkFBFHAqWFn3c80yQkdkwHqGSpIwHRSzy0t8apQ+jlNlSgKeqb8nCpJoPUki05kQGOpFbyr+5/VyiK+DgsssBybpfFGcCwwpnsaA+1wxCmJiCKGKm1sxHRJFKJiw6iYEd/HlZeJdNG+a7sNlo3VbpVFDx+gEnSEXXaEWukdt5CGKxugZvaI3q7BerHfrY966YlUzR+gPrM8fBTeTdA==</latexit><latexit sha1_base64="btFPvwTuPnBe+o1whDDCf279DQs=">AAAB+HicbVBNS8NAEN34WetX1KOXxSIIQklEUG9FLx4rGFtoQthsN+3SzSbsTgol5J948aDi1Z/izX/jts1BWx8MPN6bYWZelAmuwXG+rZXVtfWNzdpWfXtnd2/fPjh80mmuKPNoKlLVjYhmgkvmAQfBupliJIkE60Sju6nfGTOleSofYZKxICEDyWNOCRgptG1/SKDwsyEvwwLO3TK0G07TmQEvE7ciDVShHdpffj+lecIkUEG07rlOBkFBFHAqWFn3c80yQkdkwHqGSpIwHRSzy0t8apQ+jlNlSgKeqb8nCpJoPUki05kQGOpFbyr+5/VyiK+DgsssBybpfFGcCwwpnsaA+1wxCmJiCKGKm1sxHRJFKJiw6iYEd/HlZeJdNG+a7sNlo3VbpVFDx+gEnSEXXaEWukdt5CGKxugZvaI3q7BerHfrY966YlUzR+gPrM8fBTeTdA==</latexit><latexit sha1_base64="btFPvwTuPnBe+o1whDDCf279DQs=">AAAB+HicbVBNS8NAEN34WetX1KOXxSIIQklEUG9FLx4rGFtoQthsN+3SzSbsTgol5J948aDi1Z/izX/jts1BWx8MPN6bYWZelAmuwXG+rZXVtfWNzdpWfXtnd2/fPjh80mmuKPNoKlLVjYhmgkvmAQfBupliJIkE60Sju6nfGTOleSofYZKxICEDyWNOCRgptG1/SKDwsyEvwwLO3TK0G07TmQEvE7ciDVShHdpffj+lecIkUEG07rlOBkFBFHAqWFn3c80yQkdkwHqGSpIwHRSzy0t8apQ+jlNlSgKeqb8nCpJoPUki05kQGOpFbyr+5/VyiK+DgsssBybpfFGcCwwpnsaA+1wxCmJiCKGKm1sxHRJFKJiw6iYEd/HlZeJdNG+a7sNlo3VbpVFDx+gEnSEXXaEWukdt5CGKxugZvaI3q7BerHfrY966YlUzR+gPrM8fBTeTdA==</latexit><latexit sha1_base64="btFPvwTuPnBe+o1whDDCf279DQs=">AAAB+HicbVBNS8NAEN34WetX1KOXxSIIQklEUG9FLx4rGFtoQthsN+3SzSbsTgol5J948aDi1Z/izX/jts1BWx8MPN6bYWZelAmuwXG+rZXVtfWNzdpWfXtnd2/fPjh80mmuKPNoKlLVjYhmgkvmAQfBupliJIkE60Sju6nfGTOleSofYZKxICEDyWNOCRgptG1/SKDwsyEvwwLO3TK0G07TmQEvE7ciDVShHdpffj+lecIkUEG07rlOBkFBFHAqWFn3c80yQkdkwHqGSpIwHRSzy0t8apQ+jlNlSgKeqb8nCpJoPUki05kQGOpFbyr+5/VyiK+DgsssBybpfFGcCwwpnsaA+1wxCmJiCKGKm1sxHRJFKJiw6iYEd/HlZeJdNG+a7sNlo3VbpVFDx+gEnSEXXaEWukdt5CGKxugZvaI3q7BerHfrY966YlUzR+gPrM8fBTeTdA==</latexit>

at<latexit sha1_base64="eZse1F/eK6nvthQF2QS0WceEPvk=">AAAB+nicbVBNS8NAFNzUr1q/aj16WSyCp5KIoN6KXjxWMFpoQthsN+3SzSbsvogl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoJrsO1vq7ayura+Ud9sbG3v7O4191v3OskUZS5NRKL6IdFMcMlc4CBYP1WMxKFgD+HkuvQfHpnSPJF3ME2ZH5OR5BGnBIwUNFvemEDuxQTGYZSTogggaLbtjj0DXiZORdqoQi9ofnnDhGYxk0AF0Xrg2Cn4OVHAqWBFw8s0SwmdkBEbGCpJzLSfz7IX+NgoQxwlyjwJeKb+3shJrPU0Ds1kGVIveqX4nzfIILrwcy7TDJik80NRJjAkuCwCD7liFMTUEEIVN1kxHRNFKJi6GqYEZ/HLy8Q97Vx2nNuzdveqaqOODtEROkEOOkdddIN6yEUUPaFn9IrerMJ6sd6tj/lozap2DtAfWJ8/MgyUvg==</latexit><latexit sha1_base64="eZse1F/eK6nvthQF2QS0WceEPvk=">AAAB+nicbVBNS8NAFNzUr1q/aj16WSyCp5KIoN6KXjxWMFpoQthsN+3SzSbsvogl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoJrsO1vq7ayura+Ud9sbG3v7O4191v3OskUZS5NRKL6IdFMcMlc4CBYP1WMxKFgD+HkuvQfHpnSPJF3ME2ZH5OR5BGnBIwUNFvemEDuxQTGYZSTogggaLbtjj0DXiZORdqoQi9ofnnDhGYxk0AF0Xrg2Cn4OVHAqWBFw8s0SwmdkBEbGCpJzLSfz7IX+NgoQxwlyjwJeKb+3shJrPU0Ds1kGVIveqX4nzfIILrwcy7TDJik80NRJjAkuCwCD7liFMTUEEIVN1kxHRNFKJi6GqYEZ/HLy8Q97Vx2nNuzdveqaqOODtEROkEOOkdddIN6yEUUPaFn9IrerMJ6sd6tj/lozap2DtAfWJ8/MgyUvg==</latexit><latexit sha1_base64="eZse1F/eK6nvthQF2QS0WceEPvk=">AAAB+nicbVBNS8NAFNzUr1q/aj16WSyCp5KIoN6KXjxWMFpoQthsN+3SzSbsvogl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoJrsO1vq7ayura+Ud9sbG3v7O4191v3OskUZS5NRKL6IdFMcMlc4CBYP1WMxKFgD+HkuvQfHpnSPJF3ME2ZH5OR5BGnBIwUNFvemEDuxQTGYZSTogggaLbtjj0DXiZORdqoQi9ofnnDhGYxk0AF0Xrg2Cn4OVHAqWBFw8s0SwmdkBEbGCpJzLSfz7IX+NgoQxwlyjwJeKb+3shJrPU0Ds1kGVIveqX4nzfIILrwcy7TDJik80NRJjAkuCwCD7liFMTUEEIVN1kxHRNFKJi6GqYEZ/HLy8Q97Vx2nNuzdveqaqOODtEROkEOOkdddIN6yEUUPaFn9IrerMJ6sd6tj/lozap2DtAfWJ8/MgyUvg==</latexit><latexit sha1_base64="eZse1F/eK6nvthQF2QS0WceEPvk=">AAAB+nicbVBNS8NAFNzUr1q/aj16WSyCp5KIoN6KXjxWMFpoQthsN+3SzSbsvogl5K948aDi1V/izX/jps1BWwcWhpn3eLMTpoJrsO1vq7ayura+Ud9sbG3v7O4191v3OskUZS5NRKL6IdFMcMlc4CBYP1WMxKFgD+HkuvQfHpnSPJF3ME2ZH5OR5BGnBIwUNFvemEDuxQTGYZSTogggaLbtjj0DXiZORdqoQi9ofnnDhGYxk0AF0Xrg2Cn4OVHAqWBFw8s0SwmdkBEbGCpJzLSfz7IX+NgoQxwlyjwJeKb+3shJrPU0Ds1kGVIveqX4nzfIILrwcy7TDJik80NRJjAkuCwCD7liFMTUEEIVN1kxHRNFKJi6GqYEZ/HLy8Q97Vx2nNuzdveqaqOODtEROkEOOkdddIN6yEUUPaFn9IrerMJ6sd6tj/lozap2DtAfWJ8/MgyUvg==</latexit>

�<latexit sha1_base64="LpRiA0tPxs0CLwFXsvaS+jiYeRg=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqN6KXjxWMLbQhrLZbpqlu5uwuxFK6F/w4kHFq7/Im//GTZuDtj4YeLw3w8y8MOVMG9f9dipr6xubW9Xt2s7u3v5B/fDoUSeZItQnCU9UL8Saciapb5jhtJcqikXIaTec3BZ+94kqzRL5YKYpDQQeSxYxgk0hDdKYDesNt+nOgVaJV5IGlOgM61+DUUIyQaUhHGvd99zUBDlWhhFOZ7VBpmmKyQSPad9SiQXVQT6/dYbOrDJCUaJsSYPm6u+JHAutpyK0nQKbWC97hfif189MdBXkTKaZoZIsFkUZRyZBxeNoxBQlhk8twUQxeysiMVaYGBtPzYbgLb+8SvyL5nXTu79stG/KNKpwAqdwDh60oA130AEfCMTwDK/w5gjnxXl3PhatFaecOYY/cD5/AIC3jhM=</latexit><latexit sha1_base64="LpRiA0tPxs0CLwFXsvaS+jiYeRg=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqN6KXjxWMLbQhrLZbpqlu5uwuxFK6F/w4kHFq7/Im//GTZuDtj4YeLw3w8y8MOVMG9f9dipr6xubW9Xt2s7u3v5B/fDoUSeZItQnCU9UL8Saciapb5jhtJcqikXIaTec3BZ+94kqzRL5YKYpDQQeSxYxgk0hDdKYDesNt+nOgVaJV5IGlOgM61+DUUIyQaUhHGvd99zUBDlWhhFOZ7VBpmmKyQSPad9SiQXVQT6/dYbOrDJCUaJsSYPm6u+JHAutpyK0nQKbWC97hfif189MdBXkTKaZoZIsFkUZRyZBxeNoxBQlhk8twUQxeysiMVaYGBtPzYbgLb+8SvyL5nXTu79stG/KNKpwAqdwDh60oA130AEfCMTwDK/w5gjnxXl3PhatFaecOYY/cD5/AIC3jhM=</latexit><latexit sha1_base64="LpRiA0tPxs0CLwFXsvaS+jiYeRg=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqN6KXjxWMLbQhrLZbpqlu5uwuxFK6F/w4kHFq7/Im//GTZuDtj4YeLw3w8y8MOVMG9f9dipr6xubW9Xt2s7u3v5B/fDoUSeZItQnCU9UL8Saciapb5jhtJcqikXIaTec3BZ+94kqzRL5YKYpDQQeSxYxgk0hDdKYDesNt+nOgVaJV5IGlOgM61+DUUIyQaUhHGvd99zUBDlWhhFOZ7VBpmmKyQSPad9SiQXVQT6/dYbOrDJCUaJsSYPm6u+JHAutpyK0nQKbWC97hfif189MdBXkTKaZoZIsFkUZRyZBxeNoxBQlhk8twUQxeysiMVaYGBtPzYbgLb+8SvyL5nXTu79stG/KNKpwAqdwDh60oA130AEfCMTwDK/w5gjnxXl3PhatFaecOYY/cD5/AIC3jhM=</latexit><latexit sha1_base64="LpRiA0tPxs0CLwFXsvaS+jiYeRg=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqN6KXjxWMLbQhrLZbpqlu5uwuxFK6F/w4kHFq7/Im//GTZuDtj4YeLw3w8y8MOVMG9f9dipr6xubW9Xt2s7u3v5B/fDoUSeZItQnCU9UL8Saciapb5jhtJcqikXIaTec3BZ+94kqzRL5YKYpDQQeSxYxgk0hDdKYDesNt+nOgVaJV5IGlOgM61+DUUIyQaUhHGvd99zUBDlWhhFOZ7VBpmmKyQSPad9SiQXVQT6/dYbOrDJCUaJsSYPm6u+JHAutpyK0nQKbWC97hfif189MdBXkTKaZoZIsFkUZRyZBxeNoxBQlhk8twUQxeysiMVaYGBtPzYbgLb+8SvyL5nXTu79stG/KNKpwAqdwDh60oA130AEfCMTwDK/w5gjnxXl3PhatFaecOYY/cD5/AIC3jhM=</latexit>

�<latexit sha1_base64="LpRiA0tPxs0CLwFXsvaS+jiYeRg=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqN6KXjxWMLbQhrLZbpqlu5uwuxFK6F/w4kHFq7/Im//GTZuDtj4YeLw3w8y8MOVMG9f9dipr6xubW9Xt2s7u3v5B/fDoUSeZItQnCU9UL8Saciapb5jhtJcqikXIaTec3BZ+94kqzRL5YKYpDQQeSxYxgk0hDdKYDesNt+nOgVaJV5IGlOgM61+DUUIyQaUhHGvd99zUBDlWhhFOZ7VBpmmKyQSPad9SiQXVQT6/dYbOrDJCUaJsSYPm6u+JHAutpyK0nQKbWC97hfif189MdBXkTKaZoZIsFkUZRyZBxeNoxBQlhk8twUQxeysiMVaYGBtPzYbgLb+8SvyL5nXTu79stG/KNKpwAqdwDh60oA130AEfCMTwDK/w5gjnxXl3PhatFaecOYY/cD5/AIC3jhM=</latexit><latexit sha1_base64="LpRiA0tPxs0CLwFXsvaS+jiYeRg=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqN6KXjxWMLbQhrLZbpqlu5uwuxFK6F/w4kHFq7/Im//GTZuDtj4YeLw3w8y8MOVMG9f9dipr6xubW9Xt2s7u3v5B/fDoUSeZItQnCU9UL8Saciapb5jhtJcqikXIaTec3BZ+94kqzRL5YKYpDQQeSxYxgk0hDdKYDesNt+nOgVaJV5IGlOgM61+DUUIyQaUhHGvd99zUBDlWhhFOZ7VBpmmKyQSPad9SiQXVQT6/dYbOrDJCUaJsSYPm6u+JHAutpyK0nQKbWC97hfif189MdBXkTKaZoZIsFkUZRyZBxeNoxBQlhk8twUQxeysiMVaYGBtPzYbgLb+8SvyL5nXTu79stG/KNKpwAqdwDh60oA130AEfCMTwDK/w5gjnxXl3PhatFaecOYY/cD5/AIC3jhM=</latexit><latexit sha1_base64="LpRiA0tPxs0CLwFXsvaS+jiYeRg=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqN6KXjxWMLbQhrLZbpqlu5uwuxFK6F/w4kHFq7/Im//GTZuDtj4YeLw3w8y8MOVMG9f9dipr6xubW9Xt2s7u3v5B/fDoUSeZItQnCU9UL8Saciapb5jhtJcqikXIaTec3BZ+94kqzRL5YKYpDQQeSxYxgk0hDdKYDesNt+nOgVaJV5IGlOgM61+DUUIyQaUhHGvd99zUBDlWhhFOZ7VBpmmKyQSPad9SiQXVQT6/dYbOrDJCUaJsSYPm6u+JHAutpyK0nQKbWC97hfif189MdBXkTKaZoZIsFkUZRyZBxeNoxBQlhk8twUQxeysiMVaYGBtPzYbgLb+8SvyL5nXTu79stG/KNKpwAqdwDh60oA130AEfCMTwDK/w5gjnxXl3PhatFaecOYY/cD5/AIC3jhM=</latexit><latexit sha1_base64="LpRiA0tPxs0CLwFXsvaS+jiYeRg=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqN6KXjxWMLbQhrLZbpqlu5uwuxFK6F/w4kHFq7/Im//GTZuDtj4YeLw3w8y8MOVMG9f9dipr6xubW9Xt2s7u3v5B/fDoUSeZItQnCU9UL8Saciapb5jhtJcqikXIaTec3BZ+94kqzRL5YKYpDQQeSxYxgk0hDdKYDesNt+nOgVaJV5IGlOgM61+DUUIyQaUhHGvd99zUBDlWhhFOZ7VBpmmKyQSPad9SiQXVQT6/dYbOrDJCUaJsSYPm6u+JHAutpyK0nQKbWC97hfif189MdBXkTKaZoZIsFkUZRyZBxeNoxBQlhk8twUQxeysiMVaYGBtPzYbgLb+8SvyL5nXTu79stG/KNKpwAqdwDh60oA130AEfCMTwDK/w5gjnxXl3PhatFaecOYY/cD5/AIC3jhM=</latexit>

slt

<latexit sha1_base64="agdbRQHtrrSQbCz1DGDcpK6W8HU=">AAAB9HicbVDLSsNAFL2pr1pfVZduBovgqiQiqLuiG5cVjC20aZlMJ+3QyYOZG6WE/ocbFypu/Rh3/o2TNgttPTBwOOde7pnjJ1JotO1vq7Syura+Ud6sbG3v7O5V9w8edJwqxl0Wy1i1faq5FBF3UaDk7URxGvqSt/zxTe63HrnSIo7ucZJwL6TDSASCUTRSrxtSHPlBpqc92cd+tWbX7RnIMnEKUoMCzX71qzuIWRryCJmkWnccO0EvowoFk3xa6aaaJ5SN6ZB3DI1oyLWXzVJPyYlRBiSIlXkRkpn6eyOjodaT0DeTeUq96OXif14nxeDSy0SUpMgjNj8UpJJgTPIKyEAozlBODKFMCZOVsBFVlKEpqmJKcBa/vEzcs/pV3bk7rzWuizbKcATHcAoOXEADbqEJLjBQ8Ayv8GY9WS/Wu/UxHy1Zxc4h/IH1+QN31ZKw</latexit><latexit sha1_base64="agdbRQHtrrSQbCz1DGDcpK6W8HU=">AAAB9HicbVDLSsNAFL2pr1pfVZduBovgqiQiqLuiG5cVjC20aZlMJ+3QyYOZG6WE/ocbFypu/Rh3/o2TNgttPTBwOOde7pnjJ1JotO1vq7Syura+Ud6sbG3v7O5V9w8edJwqxl0Wy1i1faq5FBF3UaDk7URxGvqSt/zxTe63HrnSIo7ucZJwL6TDSASCUTRSrxtSHPlBpqc92cd+tWbX7RnIMnEKUoMCzX71qzuIWRryCJmkWnccO0EvowoFk3xa6aaaJ5SN6ZB3DI1oyLWXzVJPyYlRBiSIlXkRkpn6eyOjodaT0DeTeUq96OXif14nxeDSy0SUpMgjNj8UpJJgTPIKyEAozlBODKFMCZOVsBFVlKEpqmJKcBa/vEzcs/pV3bk7rzWuizbKcATHcAoOXEADbqEJLjBQ8Ayv8GY9WS/Wu/UxHy1Zxc4h/IH1+QN31ZKw</latexit><latexit sha1_base64="agdbRQHtrrSQbCz1DGDcpK6W8HU=">AAAB9HicbVDLSsNAFL2pr1pfVZduBovgqiQiqLuiG5cVjC20aZlMJ+3QyYOZG6WE/ocbFypu/Rh3/o2TNgttPTBwOOde7pnjJ1JotO1vq7Syura+Ud6sbG3v7O5V9w8edJwqxl0Wy1i1faq5FBF3UaDk7URxGvqSt/zxTe63HrnSIo7ucZJwL6TDSASCUTRSrxtSHPlBpqc92cd+tWbX7RnIMnEKUoMCzX71qzuIWRryCJmkWnccO0EvowoFk3xa6aaaJ5SN6ZB3DI1oyLWXzVJPyYlRBiSIlXkRkpn6eyOjodaT0DeTeUq96OXif14nxeDSy0SUpMgjNj8UpJJgTPIKyEAozlBODKFMCZOVsBFVlKEpqmJKcBa/vEzcs/pV3bk7rzWuizbKcATHcAoOXEADbqEJLjBQ8Ayv8GY9WS/Wu/UxHy1Zxc4h/IH1+QN31ZKw</latexit><latexit sha1_base64="agdbRQHtrrSQbCz1DGDcpK6W8HU=">AAAB9HicbVDLSsNAFL2pr1pfVZduBovgqiQiqLuiG5cVjC20aZlMJ+3QyYOZG6WE/ocbFypu/Rh3/o2TNgttPTBwOOde7pnjJ1JotO1vq7Syura+Ud6sbG3v7O5V9w8edJwqxl0Wy1i1faq5FBF3UaDk7URxGvqSt/zxTe63HrnSIo7ucZJwL6TDSASCUTRSrxtSHPlBpqc92cd+tWbX7RnIMnEKUoMCzX71qzuIWRryCJmkWnccO0EvowoFk3xa6aaaJ5SN6ZB3DI1oyLWXzVJPyYlRBiSIlXkRkpn6eyOjodaT0DeTeUq96OXif14nxeDSy0SUpMgjNj8UpJJgTPIKyEAozlBODKFMCZOVsBFVlKEpqmJKcBa/vEzcs/pV3bk7rzWuizbKcATHcAoOXEADbqEJLjBQ8Ayv8GY9WS/Wu/UxHy1Zxc4h/IH1+QN31ZKw</latexit>

slt+1

<latexit sha1_base64="2SZ9Gi37o8yjqO7BQz3HsjrHqhk=">AAAB+nicbVBNS8NAFNz4WetXrUcvi0UQhJKIoN6KXjxWMLbQxrDZbtqlm03YfRFLyF/x4kHFq7/Em//GTZuDtg4sDDPv8WYnSATXYNvf1tLyyuraemWjurm1vbNb26vf6zhVlLk0FrHqBkQzwSVzgYNg3UQxEgWCdYLxdeF3HpnSPJZ3MEmYF5Gh5CGnBIzk1+r9iMAoCDOdPwg/gxMn92sNu2lPgReJU5IGKtH2a1/9QUzTiEmggmjdc+wEvIwo4FSwvNpPNUsIHZMh6xkqScS0l02z5/jIKAMcxso8CXiq/t7ISKT1JArMZJFUz3uF+J/XSyG88DIukxSYpLNDYSowxLgoAg+4YhTExBBCFTdZMR0RRSiYuqqmBGf+y4vEPW1eNp3bs0brqmyjgg7QITpGDjpHLXSD2shFFD2hZ/SK3qzcerHerY/Z6JJV7uyjP7A+fwCfSJRd</latexit><latexit sha1_base64="2SZ9Gi37o8yjqO7BQz3HsjrHqhk=">AAAB+nicbVBNS8NAFNz4WetXrUcvi0UQhJKIoN6KXjxWMLbQxrDZbtqlm03YfRFLyF/x4kHFq7/Em//GTZuDtg4sDDPv8WYnSATXYNvf1tLyyuraemWjurm1vbNb26vf6zhVlLk0FrHqBkQzwSVzgYNg3UQxEgWCdYLxdeF3HpnSPJZ3MEmYF5Gh5CGnBIzk1+r9iMAoCDOdPwg/gxMn92sNu2lPgReJU5IGKtH2a1/9QUzTiEmggmjdc+wEvIwo4FSwvNpPNUsIHZMh6xkqScS0l02z5/jIKAMcxso8CXiq/t7ISKT1JArMZJFUz3uF+J/XSyG88DIukxSYpLNDYSowxLgoAg+4YhTExBBCFTdZMR0RRSiYuqqmBGf+y4vEPW1eNp3bs0brqmyjgg7QITpGDjpHLXSD2shFFD2hZ/SK3qzcerHerY/Z6JJV7uyjP7A+fwCfSJRd</latexit><latexit sha1_base64="2SZ9Gi37o8yjqO7BQz3HsjrHqhk=">AAAB+nicbVBNS8NAFNz4WetXrUcvi0UQhJKIoN6KXjxWMLbQxrDZbtqlm03YfRFLyF/x4kHFq7/Em//GTZuDtg4sDDPv8WYnSATXYNvf1tLyyuraemWjurm1vbNb26vf6zhVlLk0FrHqBkQzwSVzgYNg3UQxEgWCdYLxdeF3HpnSPJZ3MEmYF5Gh5CGnBIzk1+r9iMAoCDOdPwg/gxMn92sNu2lPgReJU5IGKtH2a1/9QUzTiEmggmjdc+wEvIwo4FSwvNpPNUsIHZMh6xkqScS0l02z5/jIKAMcxso8CXiq/t7ISKT1JArMZJFUz3uF+J/XSyG88DIukxSYpLNDYSowxLgoAg+4YhTExBBCFTdZMR0RRSiYuqqmBGf+y4vEPW1eNp3bs0brqmyjgg7QITpGDjpHLXSD2shFFD2hZ/SK3qzcerHerY/Z6JJV7uyjP7A+fwCfSJRd</latexit><latexit sha1_base64="2SZ9Gi37o8yjqO7BQz3HsjrHqhk=">AAAB+nicbVBNS8NAFNz4WetXrUcvi0UQhJKIoN6KXjxWMLbQxrDZbtqlm03YfRFLyF/x4kHFq7/Em//GTZuDtg4sDDPv8WYnSATXYNvf1tLyyuraemWjurm1vbNb26vf6zhVlLk0FrHqBkQzwSVzgYNg3UQxEgWCdYLxdeF3HpnSPJZ3MEmYF5Gh5CGnBIzk1+r9iMAoCDOdPwg/gxMn92sNu2lPgReJU5IGKtH2a1/9QUzTiEmggmjdc+wEvIwo4FSwvNpPNUsIHZMh6xkqScS0l02z5/jIKAMcxso8CXiq/t7ISKT1JArMZJFUz3uF+J/XSyG88DIukxSYpLNDYSowxLgoAg+4YhTExBBCFTdZMR0RRSiYuqqmBGf+y4vEPW1eNp3bs0brqmyjgg7QITpGDjpHLXSD2shFFD2hZ/SK3qzcerHerY/Z6JJV7uyjP7A+fwCfSJRd</latexit>

Fig. 2: ICM architecture. slt and slt+1 are first passed throughthe feature extraction layers φ, and encoded into φt and φt+1.Then φt and φt+1 are input together into the inverse modelψi, to infer the action at. At the same time, at and φt aretogether used to predict φt+1, through the forward modelψf . The prediction error between φt+1 and φt+1 is used asthe intrinsic reward Ri.

loss with the ground truth φt+1). φ, ψf and ψi are learnedand updated together with the actor-critic parameters θπ andθv (Sec. II-D) during the training process.

The prediction error between φt+1 and φt+1 through ψf

is then used as the intrinsic reward Ri:

Ri =∣∣∣∣∣∣φt+1 − φt+1

∣∣∣∣∣∣22. (2)

Following this intrinsic reward, the agent is encouragedto visit novel states in the environment, which is crucial inguiding the agent to get out of local minimums or prema-ture convergence of sub-optimal policies. This explorationstrategy allows the agent to make better use of the learnedenvironment dynamics to accomplish the task at hand.

D. Asynchronous Deep Reinforcement Learning

For training the navigation policies, we follow the asyn-chronous advantage actor-critic (A3C) algorithm, with theweighted sum of the external reward and the intrinsic rewardas the supervision signal:

R = Re + λiRi, (3)

where λi > 0 is the scaling coefficient.A3C updates both the parameters of a policy network

θπ and a value network θv , to maximize the cumulativeexpected reward. The value estimate is bootstrapped fromn-step returns with a discount factor γ (where K representsthe maximum number of steps for a single rollout):

Gt = γK−tV (sK ; θv) +

K−1∑τ=t

γτ−tRτ . (4)

Each learning thread accumulates gradients for everyexperience contained in a K step rollout, where the gradientsare calculated according to the following equations:

dθπ = ∇θπ log π(at|st; θπ)(Gt − V (st; θv)) (5)

+ β∇θπH(π(at|st; θπ)), (6)

dθv = ∂(Gt − V (st; θv))2/∂θv, (7)

where H is the entropy of the policy, and β represents thecoefficient of the entropy regularization. This discourages

(a) Map1 (b) Map2

(c) Map3 (d) Map4

Fig. 3: Different floor plans we considered in our experi-ments. Map1,3,4 have similar wall structures, while imposinggradually more exploration challenges for DRL agents toeffectively learn navigation policies. Map2, while holding alevel of challenge for exploration similar to that of Map1,has some structural variations such as the angled walls. Weonly train agents on Map1. The performance of the trainedagents is later evaluated on Map1, and tested on Map2,3,4for their generalization capabilities. The maps are all of size5.33m× 3.76m.

premature convergence to suboptimal deterministic policies,and is the default exploration strategy that has been usedwith A3C.

III. EXPERIMENTS

A. Experimental setup

We conduct our experiments in a simulated environment,where a robot is equipped with a laser range sensor, and isnavigating in the 2D environments shown in Fig. 3. At thebeginning of each episode, the starting pose of the agent p0

and the target position g are randomly chosen such that acollision-free path is guaranteed to exist between them. Anepisode is terminated after the agent either reaches the goal,collides with an obstacle, or after a maximum of 7000 stepsduring training and 400 for testing.

The state st consists of 72-dimensional laser range read-ings slt (with a maximum range of 7m), and a 3-dimensionalrelative goal position sgt (relative distance, and sin and cos ofthe relative orientation in the agent’s local coordinate frame).At each timestep, the agent can select one of three discreteactions: {go straight for 0.06m, turn left 8°, turn right 8°}.

The hyper-parameters regarding the reward function (II-B,II-C) are the following: γ = 0.99, λp = 1, λω = 1

200π ,λg = 0.15, rreach = 1, rcollision = −5, rposition = −0.05, andthe scale for the intrinsic reward λi is set to 1.

The actor-critic network uses two convolutional layers of 8filters with stride 2, and kernel sizes of 5 and 3 respectively,each followed by ELU nonlinearities. These are followedby two fully connected layers with 64 and 16 units, also

0.0 0.5 1.0 1.5 2.0 2.5 3.0Training steps 1e6

50

40

30

20

10

0

Ave

rage

rew

ard

A3C-EntropyICMICM+Entropy

(a) Average reward.

0.0 0.5 1.0 1.5 2.0 2.5 3.0Training steps 1e6

0

200

400

600

800

1000

Ave

rage

step

s

A3C-EntropyICMICM+Entropy

(b) Average steps.

Fig. 4: Comparison of (Fig. 4a) average reward and (Fig. 4b)steps for different exploration strategies that we considered(switching on/off the entropy loss or the intrinsic reward).The mean and confidence interval for each configuration iscalculated over the statistics of 3 independent runs.

ELU nonlinearities, to transform the laser sensor readingsslt into a 16-dimensional embedding. This representationis concatenated with the relative goal position sgt , and fedinto a single layer of 16 LSTM cells. The output is thenconcatenated with sgt again, and used to produce the discreteaction probabilities by a linear layer followed by a softmax,and the value function by a linear layer.

The ICM inverse model ψi consists of three fully con-nected layers with 128, 64, and 16 units, each followed byan ELU nonlinearity, producing features φt and φt+1 fromslt and slt+1. Those features are then concatenated and putthrough a fully connected layer with 32 units and an ELUnonlinearity, the output of which predicts at by a linear layerfollowed by a softmax. The ICM forward model ψf acceptsthe true features φt and a one-hot representation of the actionat, and feeds them into two linear layers with 64 and 32 unitsrespectively, followed by ELU and a linear layer that predictsφt+1.

We train A3C with the Adam optimizer with statisticsshared across 22 learner threads, with a learning rate of1e−4. The rollout step K is set to 50. ICM is learned jointlywith shared Adam with the same learning rate. Each trainingof 3 million iterations takes approximately 15 hours usingonly CPU computation.

TABLE I: Evaluation on Map1.

Exploration Success StepsStrategy Ratio (%) (mean±std)

Map1 Entropy 34.3 313.787±135.513ICM 65.7 233.147±143.260

(w/o LSTM) ICM+Entropy 73.3 162.733±150.291

Map1 A3C- 88.3 173.063±123.277Entropy 96.7 102.220±90.230

ICM 98.7 91.230±62.511(w/ LSTM) ICM+Entropy 100 75.160±52.075

We only train agents on Map1 (Fig. 3a), varying explo-ration strategies (Sec. III-B). Map2,3,4 are used to test thegeneralization ability of the trained policies (Sec. III-C).

B. Training and Evaluation on Map1

We experiment with 4 variations of exploration strategiesby switching on and off the entropy loss and the intrinsicreward: 1) A3C-: A3C with β = 0.; 2) Entropy: A3Cwith β = 0.01; 3) ICM: A3C with β = 0., with ICM;4) ICM+Entropy: A3C with β = 0.01, with ICM.

The average reward and steps obtained in the evaluationduring training are shown in Fig. 4. We can clearly observethat without any explicit exploration mechanisms, A3C-exhibits unstable behavior during learning. ICM alone ismuch more effective in guiding the policy learning thanEntropy alone, since the exploration imposed by the latteris state-independent, while with ICM the agent is able toactively acquire new useful knowledge from the environment,and learn efficient navigation strategies for the environmentalstructures that it has encountered. ICM+Entropy stably andefficiently achieves the best performance.

We run evaluations on the same set of 300 randomepisodes on Map1 after training, and report the resultingstatistics in Table I. In addition, we conducted another set ofexperiments where we remove the LSTM layers in the actor-critic network. We report their evaluation results in Table Ias well. Although without LSTM none of the configurationsmanage to converge to a good policy (so we omit theirtraining curves in Fig. 4), we can still observe a clear trendof performance improvements for the policies trained bothwith and without LSTM, brought about by ICM exploration.

C. Generalization Tests on Map2,3,4

To test the generalization capabilities of the learned poli-cies, we deploy the networks trained on Map1, collectstatistics on a fixed set of 300 random episodes on Map2,3,4,and report the results in Table II.

As discussed earlier in Fig. 3, Map2 is relatively simplebut contains unfamiliar structures; Map3 and Map4 are morechallenging, but the structures inside these floorplans aremore similar to the training environment of Map1. With theexploratory behaviors brought by both ICM and Entropy,the agent learns well how to exploit structures similar tothose it has encountered during training. Thus from Table IIwe can observe that, although the full model ICM+Entropyis slightly worse than ICM exploration alone on Map2, it

TABLE II: Generalization tests on new maps.

Exploration Success StepsStrategy Ratio (%) (mean±std)

Map2A3C- 66.0 199.343±145.924

Entropy 85.3 115.567±107.474ICM 87.7 101.160±102.789

ICM+Entropy 82.7 109.690±107.043

Map3A3C- 51.3 197.200±141.292

Entropy 69.7 175.667±139.811ICM 63.7 152.627±135.982

ICM+Entropy 72.7 139.423±120.144

Map4A3C- 44.3 273.353±148.527

Entropy 34.3 240.197±164.565ICM 45.0 236.847±169.169

ICM+Entropy 55.0 209.500±161.767

achieves better performance in the more challenging envi-ronments of Map3 and Map4.

IV. CONCLUSIONS

We investigate exploration strategies for learning maplessnavigation policies with DRL, augmenting the reward signalswith intrinsic motivation. Our proposed method achievesbetter sample efficiency and convergence properties duringtraining than all the baseline methods, as well as bettergeneralization capabilities during tests on previously unseenmaps. Our future work includes studying how to better adaptthe weighting of the different components in the rewardfunction, as well as real-world robotics experiments.

REFERENCES

[1] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley,D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deepreinforcement learning,” in International Conference on MachineLearning, 2016, pp. 1928–1937.

[2] S. Thrun, W. Burgard, and D. Fox, Probabilistic Robotics. MIT Press,2005.

[3] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, andA. Farhadi, “Target-driven visual navigation in indoor scenes usingdeep reinforcement learning,” in Robotics and Automation (ICRA),2017 IEEE International Conference on. IEEE, 2017, pp. 3357–3364.

[4] J. Zhang, J. T. Springenberg, J. Boedecker, and W. Burgard, “Deepreinforcement learning with successor features for navigation acrosssimilar environments,” in Intelligent Robots and Systems (IROS), 2017IEEE/RSJ International Conference on. IEEE, 2017, pp. 2371–2378.

[5] P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. J. Ballard, A. Banino,M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu, et al., “Learning tonavigate in complex environments,” arXiv preprint arXiv:1611.03673,2016.

[6] J. Zhang, L. Tai, J. Boedecker, W. Burgard, and M. Liu, “NeuralSLAM,” arXiv preprint arXiv:1706.09520, 2017.

[7] P. Long, T. Fan, X. Liao, W. Liu, H. Zhang, and J. Pan, “Towardsoptimally decentralized multi-robot collision avoidance via deep rein-forcement learning,” arXiv preprint arXiv:1709.10082, 2017.

[8] Q. Liang and M. Liu, “Plugo: a VLC systematic perspective of large-scale indoor localization,” arXiv preprint arXiv:1709.06926, 2017.

[9] Y. Sun, M. Liu, and M. Q.-H. Meng, “WiFi signal strength-basedrobot indoor localization,” in Information and Automation (ICIA), 2014IEEE International Conference on. IEEE, 2014, pp. 250–256.

[10] L. Tai, G. Paolo, and M. Liu, “Virtual-to-real deep reinforcement learn-ing: Continuous control of mobile robots for mapless navigation,” inIntelligent Robots and Systems (IROS), 2017 IEEE/RSJ InternationalConference on. IEEE, 2017, pp. 31–36.

[11] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-drivenexploration by self-supervised prediction,” in International Conferenceon Machine Learning (ICML), vol. 2017, 2017.