a generalized algorithm for multi-objective reinforcement …runzhey/demo/general_exam.pdf · 2020....

May 20th, 2020

A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation

“Tony” Runzhe Yanghttps://runzhe-yang.science

Karthik Narasimhan (adviser) Sebastian Seung (adviser) Ryan Adams

https://runzhe-yang.science

https://runzhe-yang.science

“Deep reinforcement learning is one of the most eye-catching research fields in recent years.”

https://openai.com/blog/solving-rubiks-cube/

OpenAI: Solving Rubik’s Cube with a Robot Hand (2019)

https://deepmind.com/alphago-korea

DeepMind: Defeat a Go World Champion (2016)



“The goal of reinforcement learning is to find an optimal policy which maximizes the expected total

reward via trial and error.”

Reinforcement learning: single objective, scalar reward

Play Atari Games using DQNs (2013)Go Challenge

Win (+1) / Loss (-1) Score↑ / Scalar Feedback↑

Reward Signals

Limitations of Single-Objective RL: supervision is too weak

Yann LeCun’s Cake Analogy (2016)

Reinforcement Learning (cherry)

• [Sparse and delayed feedback]: machine predicts a scalar reward given once in a while.

• Only a few bits information of supervision for some samples.

“If intelligence is a cake…”

Limitations of Single-Objective RL: reward engineering is hard

A scalar reward function requires combining different objectives, which is both a tedious manual task and can lead to unintended consequences.

OpenAI: Faulty Reward Functions in the Wild

Limitations of Single-Objective RL: unadaptable to related tasks

(weather)

0.70.3

(driving)

0.50.5

successbrevity

successbrevity

Showers this evening, becoming a steady rain overnight. Low 6C. Winds S at 15 to 25 km/h. Chance of rain 100%. Rainfall near 6mm…

Turn le) at the next intersection.

success success

brev

ity

brev

ity

E.g., in task-oriented dialogue systems, users may expect either briefer or more informative dialogue.

Impossible to dynamically adapt or transfer to related tasks.

x

x

x

x

“What if reward signals are vectors encoding many competing objectives, instead of scalars?”


(weather)

0.70.3

(driving)

0.50.5

successbrevity

successbrevity



success success

brev

ity

brev

ity

Motivating Scenario: RL-based dialogue systems

x

x

x

x

)

<latexit sha1_base64="oOXeaQRXApXXBchXLoUUJCWVNT4=">AAADV3icfZJta9swEMdVZ+sy76nZXu6NWCiMEYJTyh7elT3A3oxlo2kLcSiycnFEbUlI5yzB+GPs7fa5+mk2yTFsWfEOfPx19zuddVyiM2Exiq73gs6t2/t3unfDe/cfPHx00Ht8ZlVhOEy4ypS5SJiFTEiYoMAMLrQBlicZnCdX73z+fAXGCiVPcaNhlrNUioXgDF1oGn8V6RKZMerb5UE/Gka10Zti1Ig+aWx82QtexnPFixwk8oxZOx1FGmclMyh4BlUYFxY041cshamTkuVgZ2X9zxU9dJE5XSjjPom0jv5dUbLc5gyXA+qER2yt7CZPBjTJ64PS0l3kqZ3K9bbFbn9cvJ6VQuoCQfJt+0WRUVTUj4XOhQGO2YaG8XtwrzHwyV37WYNhqMyLMmYmzdm6cq9LaTygXv8PFfIP6nQbumKmKr1rA7haVaV3bYBNXSfv2gAwroV3bQC6/Gl7Wrt07GeMWI5N1d5nraEhk6T84MDw0I2VcSPcPlC+ZIZxdKsYuk0b/btXN8XZ0XB0PHzz5bh/8rbZuS55Sp6R52REXpET8pGMyYRwosh38oP8DK6DX539TneLBntNzROyY53eb2VXGDQ=</latexit>


(weather)

0.70.3

(driving)

0.50.5

successbrevity

successbrevity



success success

brev

ity

brev

ity

Motivating Scenario: advantages of multi-objective settings

x

x

• Provides more training signals.

• Reduces the dependence on reward design.

• Allows dynamic adaptation or transfer to related tasks through inferring their underlying preferences.

x

x

“The goal of multi-objective reinforcement learning is to find an (many?) optimal policy-ies which maximizes (?) the expected total vectorial reward via trial and error.”

AB

C

D

E

F

G

H

K

L

QUANTITY OF OBJECTIVE 1

QU

ANTI

TY O

F O

BJEC

TIVE

2 CCS

Non-optimal

Pareto Frontier

F


QU

ANTI

TY O

F O

BJEC

TIVE

2 F Non-preferred

Preference

D Preferred Solution

! 2 ⌦<latexit sha1_base64="ePxP6q1byPeQR8z8PEXhpWI4T3M=">AAACWnicfZFdSxwxFIaz40d1tLpqvfImdBFK2S4zRdBL0V54U7TQVcEsy5ns2TGYjyHJiMuwP8Zb/UVCf4yZcaFaoQcCT07ec07yJiukcD5JnlrR3PzC4oel5Xhl9ePaentj89yZ0nLscyONvczAoRQa+154iZeFRVCZxIvs5rg+v7hF64TRv/2kwIGCXIux4OBDatjeZpmizCjMgTKhKTutcdjuJL2kCfoe0hl0yCzOhhutb2xkeKlQey7Buas0KfygAusFlziNWemwAH4DOV4F1KDQDarm/lO6GzIjOjY2LO1pk31dUYFyCvx1lwaoJa4hN1FZl2aq2ZhCh0a16u0sPz4YVEIXpUfNX0aNS0m9obUddCQsci8nNGY/MNzc4s/Q4rRAC97YrxUDmyu4m4aX5JR1ac3/kwr9Vxo43g0TgFsRbKD8GixwH34jDgan/9r5Hs6/99Kkl/7a6xwezaxeIjvkM/lCUrJPDskJOSN9wklF7skDeWz9iaJoOVp5kUatWc0WeRPRp2fCfrJd</latexit><latexit sha1_base64="ePxP6q1byPeQR8z8PEXhpWI4T3M=">AAACWnicfZFdSxwxFIaz40d1tLpqvfImdBFK2S4zRdBL0V54U7TQVcEsy5ns2TGYjyHJiMuwP8Zb/UVCf4yZcaFaoQcCT07ec07yJiukcD5JnlrR3PzC4oel5Xhl9ePaentj89yZ0nLscyONvczAoRQa+154iZeFRVCZxIvs5rg+v7hF64TRv/2kwIGCXIux4OBDatjeZpmizCjMgTKhKTutcdjuJL2kCfoe0hl0yCzOhhutb2xkeKlQey7Buas0KfygAusFlziNWemwAH4DOV4F1KDQDarm/lO6GzIjOjY2LO1pk31dUYFyCvx1lwaoJa4hN1FZl2aq2ZhCh0a16u0sPz4YVEIXpUfNX0aNS0m9obUddCQsci8nNGY/MNzc4s/Q4rRAC97YrxUDmyu4m4aX5JR1ac3/kwr9Vxo43g0TgFsRbKD8GixwH34jDgan/9r5Hs6/99Kkl/7a6xwezaxeIjvkM/lCUrJPDskJOSN9wklF7skDeWz9iaJoOVp5kUatWc0WeRPRp2fCfrJd</latexit><latexit sha1_base64="ePxP6q1byPeQR8z8PEXhpWI4T3M=">AAACWnicfZFdSxwxFIaz40d1tLpqvfImdBFK2S4zRdBL0V54U7TQVcEsy5ns2TGYjyHJiMuwP8Zb/UVCf4yZcaFaoQcCT07ec07yJiukcD5JnlrR3PzC4oel5Xhl9ePaentj89yZ0nLscyONvczAoRQa+154iZeFRVCZxIvs5rg+v7hF64TRv/2kwIGCXIux4OBDatjeZpmizCjMgTKhKTutcdjuJL2kCfoe0hl0yCzOhhutb2xkeKlQey7Buas0KfygAusFlziNWemwAH4DOV4F1KDQDarm/lO6GzIjOjY2LO1pk31dUYFyCvx1lwaoJa4hN1FZl2aq2ZhCh0a16u0sPz4YVEIXpUfNX0aNS0m9obUddCQsci8nNGY/MNzc4s/Q4rRAC97YrxUDmyu4m4aX5JR1ac3/kwr9Vxo43g0TgFsRbKD8GixwH34jDgan/9r5Hs6/99Kkl/7a6xwezaxeIjvkM/lCUrJPDskJOSN9wklF7skDeWz9iaJoOVp5kUatWc0WeRPRp2fCfrJd</latexit><latexit sha1_base64="ePxP6q1byPeQR8z8PEXhpWI4T3M=">AAACWnicfZFdSxwxFIaz40d1tLpqvfImdBFK2S4zRdBL0V54U7TQVcEsy5ns2TGYjyHJiMuwP8Zb/UVCf4yZcaFaoQcCT07ec07yJiukcD5JnlrR3PzC4oel5Xhl9ePaentj89yZ0nLscyONvczAoRQa+154iZeFRVCZxIvs5rg+v7hF64TRv/2kwIGCXIux4OBDatjeZpmizCjMgTKhKTutcdjuJL2kCfoe0hl0yCzOhhutb2xkeKlQey7Buas0KfygAusFlziNWemwAH4DOV4F1KDQDarm/lO6GzIjOjY2LO1pk31dUYFyCvx1lwaoJa4hN1FZl2aq2ZhCh0a16u0sPz4YVEIXpUfNX0aNS0m9obUddCQsci8nNGY/MNzc4s/Q4rRAC97YrxUDmyu4m4aX5JR1ac3/kwr9Vxo43g0TgFsRbKD8GixwH34jDgan/9r5Hs6/99Kkl/7a6xwezaxeIjvkM/lCUrJPDskJOSN9wklF7skDeWz9iaJoOVp5kUatWc0WeRPRp2fCfrJd</latexit>

!|r̂D > !|r̂F<latexit sha1_base64="v+HK6S9OOx7MNeHr24Cpmm81jK4=">AAACkHicjZBtSxwxEMdz2yfdPnjqy74JPYRSrseuCPVVa1VELKUWeiqY6zGbm9sL5mFJsqXHsh/KT1N8Z79Js+tBa4XSgcAvk//MZP5ZIYXzSXLVie7df/Dw0dJy/PjJ02cr3dW1E2dKy3HIjTT2LAOHUmgceuElnhUWQWUST7OLveb99BtaJ4z+4ucFjhTkWkwFBx9S4+4HlinKjMIcvlZMaI+Wg6wpm4Gvmjdbj/fpW/ofsoNxt5cMkjboXUgX0COLOB6vdl6zieGlQu25BOfO06TwowqsF1xiHbPSYQH8AnI8D6hBoRtV7dY13QiZCZ0aG472tM3+WVGBcgr8rE8DNBLXkpurrE8z1V5MoUOjRnV7lp9ujyqhi9Kj5jejpqWk3tDGRDoRFrmXcxqzfQw/t/gxtPhUoAVv7KuKgc0VfK/DJjllfdrwv6RC/5YGjjfCBOBWBBson4EFHgx3cTA4/dvOu3CyOUiTQfp5q7ezu7B6iTwnL8hLkpI3ZIcckmMyJJxckh/kmvyM1qLt6F30/kYadRY16+RWREe/AAi9yBw=</latexit><latexit sha1_base64="v+HK6S9OOx7MNeHr24Cpmm81jK4=">AAACkHicjZBtSxwxEMdz2yfdPnjqy74JPYRSrseuCPVVa1VELKUWeiqY6zGbm9sL5mFJsqXHsh/KT1N8Z79Js+tBa4XSgcAvk//MZP5ZIYXzSXLVie7df/Dw0dJy/PjJ02cr3dW1E2dKy3HIjTT2LAOHUmgceuElnhUWQWUST7OLveb99BtaJ4z+4ucFjhTkWkwFBx9S4+4HlinKjMIcvlZMaI+Wg6wpm4Gvmjdbj/fpW/ofsoNxt5cMkjboXUgX0COLOB6vdl6zieGlQu25BOfO06TwowqsF1xiHbPSYQH8AnI8D6hBoRtV7dY13QiZCZ0aG472tM3+WVGBcgr8rE8DNBLXkpurrE8z1V5MoUOjRnV7lp9ujyqhi9Kj5jejpqWk3tDGRDoRFrmXcxqzfQw/t/gxtPhUoAVv7KuKgc0VfK/DJjllfdrwv6RC/5YGjjfCBOBWBBson4EFHgx3cTA4/dvOu3CyOUiTQfp5q7ezu7B6iTwnL8hLkpI3ZIcckmMyJJxckh/kmvyM1qLt6F30/kYadRY16+RWREe/AAi9yBw=</latexit><latexit sha1_base64="v+HK6S9OOx7MNeHr24Cpmm81jK4=">AAACkHicjZBtSxwxEMdz2yfdPnjqy74JPYRSrseuCPVVa1VELKUWeiqY6zGbm9sL5mFJsqXHsh/KT1N8Z79Js+tBa4XSgcAvk//MZP5ZIYXzSXLVie7df/Dw0dJy/PjJ02cr3dW1E2dKy3HIjTT2LAOHUmgceuElnhUWQWUST7OLveb99BtaJ4z+4ucFjhTkWkwFBx9S4+4HlinKjMIcvlZMaI+Wg6wpm4Gvmjdbj/fpW/ofsoNxt5cMkjboXUgX0COLOB6vdl6zieGlQu25BOfO06TwowqsF1xiHbPSYQH8AnI8D6hBoRtV7dY13QiZCZ0aG472tM3+WVGBcgr8rE8DNBLXkpurrE8z1V5MoUOjRnV7lp9ujyqhi9Kj5jejpqWk3tDGRDoRFrmXcxqzfQw/t/gxtPhUoAVv7KuKgc0VfK/DJjllfdrwv6RC/5YGjjfCBOBWBBson4EFHgx3cTA4/dvOu3CyOUiTQfp5q7ezu7B6iTwnL8hLkpI3ZIcckmMyJJxckh/kmvyM1qLt6F30/kYadRY16+RWREe/AAi9yBw=</latexit><latexit sha1_base64="v+HK6S9OOx7MNeHr24Cpmm81jK4=">AAACkHicjZBtSxwxEMdz2yfdPnjqy74JPYRSrseuCPVVa1VELKUWeiqY6zGbm9sL5mFJsqXHsh/KT1N8Z79Js+tBa4XSgcAvk//MZP5ZIYXzSXLVie7df/Dw0dJy/PjJ02cr3dW1E2dKy3HIjTT2LAOHUmgceuElnhUWQWUST7OLveb99BtaJ4z+4ucFjhTkWkwFBx9S4+4HlinKjMIcvlZMaI+Wg6wpm4Gvmjdbj/fpW/ofsoNxt5cMkjboXUgX0COLOB6vdl6zieGlQu25BOfO06TwowqsF1xiHbPSYQH8AnI8D6hBoRtV7dY13QiZCZ0aG472tM3+WVGBcgr8rE8DNBLXkpurrE8z1V5MoUOjRnV7lp9ujyqhi9Kj5jejpqWk3tDGRDoRFrmXcxqzfQw/t/gxtPhUoAVv7KuKgc0VfK/DJjllfdrwv6RC/5YGjjfCBOBWBBson4EFHgx3cTA4/dvOu3CyOUiTQfp5q7ezu7B6iTwnL8hLkpI3ZIcckmMyJJxckh/kmvyM1qLt6F30/kYadRY16+RWREe/AAi9yBw=</latexit>

a. b.

DUtility Projection


QU

ANTI

TY O

F O

BJEC

TIVE

2

D

F

!2<latexit sha1_base64="+UouUJfOmsYL+mxiFW/Pw7MTXn0=">AAACT3icfZFNSxxBEIZ7NsaPMfHzmEvjIoSwWWYkEI+iHryEGHBVcJalprd2bLa/6O4Rl2H/htf4lzz6S3IL6ZkMJCpY0PB09dtV1W/nRnDnk+Qx6rxZeLu4tLwSr757v7a+sbl17nRpGQ6YFtpe5uBQcIUDz73AS2MRZC7wIp8e1ecXN2gd1+rMzwwOJRSKTzgDH1JZlkuaaYkFjPZGG92knzRBX0LaQpe0cTrajD5nY81KicozAc5dpYnxwwqs50zgPM5KhwbYFAq8CqhAohtWzdBzuhsyYzrRNizlaZP9/0YF0knw1z0aoJa4htxM5j2ay2ajjQqFatXTXn6yP6y4MqVHxf62mpSCek1rD+iYW2RezGicHWOY3OK3UOK7QQte209VBraQcDsPLylo1qM1vybl6p80cLwbOgCzPNhA2TVYYD58QRwMTp/b+RLO9/pp0k9/fOkeHLZWL5MPZId8JCn5Sg7ICTklA8KIIXfkJ7mPHqJf0e9OK+1ELWyTJ9FZ+QMa87Cp</latexit><latexit sha1_base64="+UouUJfOmsYL+mxiFW/Pw7MTXn0=">AAACT3icfZFNSxxBEIZ7NsaPMfHzmEvjIoSwWWYkEI+iHryEGHBVcJalprd2bLa/6O4Rl2H/htf4lzz6S3IL6ZkMJCpY0PB09dtV1W/nRnDnk+Qx6rxZeLu4tLwSr757v7a+sbl17nRpGQ6YFtpe5uBQcIUDz73AS2MRZC7wIp8e1ecXN2gd1+rMzwwOJRSKTzgDH1JZlkuaaYkFjPZGG92knzRBX0LaQpe0cTrajD5nY81KicozAc5dpYnxwwqs50zgPM5KhwbYFAq8CqhAohtWzdBzuhsyYzrRNizlaZP9/0YF0knw1z0aoJa4htxM5j2ay2ajjQqFatXTXn6yP6y4MqVHxf62mpSCek1rD+iYW2RezGicHWOY3OK3UOK7QQte209VBraQcDsPLylo1qM1vybl6p80cLwbOgCzPNhA2TVYYD58QRwMTp/b+RLO9/pp0k9/fOkeHLZWL5MPZId8JCn5Sg7ICTklA8KIIXfkJ7mPHqJf0e9OK+1ELWyTJ9FZ+QMa87Cp</latexit><latexit sha1_base64="+UouUJfOmsYL+mxiFW/Pw7MTXn0=">AAACT3icfZFNSxxBEIZ7NsaPMfHzmEvjIoSwWWYkEI+iHryEGHBVcJalprd2bLa/6O4Rl2H/htf4lzz6S3IL6ZkMJCpY0PB09dtV1W/nRnDnk+Qx6rxZeLu4tLwSr757v7a+sbl17nRpGQ6YFtpe5uBQcIUDz73AS2MRZC7wIp8e1ecXN2gd1+rMzwwOJRSKTzgDH1JZlkuaaYkFjPZGG92knzRBX0LaQpe0cTrajD5nY81KicozAc5dpYnxwwqs50zgPM5KhwbYFAq8CqhAohtWzdBzuhsyYzrRNizlaZP9/0YF0knw1z0aoJa4htxM5j2ay2ajjQqFatXTXn6yP6y4MqVHxf62mpSCek1rD+iYW2RezGicHWOY3OK3UOK7QQte209VBraQcDsPLylo1qM1vybl6p80cLwbOgCzPNhA2TVYYD58QRwMTp/b+RLO9/pp0k9/fOkeHLZWL5MPZId8JCn5Sg7ICTklA8KIIXfkJ7mPHqJf0e9OK+1ELWyTJ9FZ+QMa87Cp</latexit><latexit sha1_base64="+UouUJfOmsYL+mxiFW/Pw7MTXn0=">AAACT3icfZFNSxxBEIZ7NsaPMfHzmEvjIoSwWWYkEI+iHryEGHBVcJalprd2bLa/6O4Rl2H/htf4lzz6S3IL6ZkMJCpY0PB09dtV1W/nRnDnk+Qx6rxZeLu4tLwSr757v7a+sbl17nRpGQ6YFtpe5uBQcIUDz73AS2MRZC7wIp8e1ecXN2gd1+rMzwwOJRSKTzgDH1JZlkuaaYkFjPZGG92knzRBX0LaQpe0cTrajD5nY81KicozAc5dpYnxwwqs50zgPM5KhwbYFAq8CqhAohtWzdBzuhsyYzrRNizlaZP9/0YF0knw1z0aoJa4htxM5j2ay2ajjQqFatXTXn6yP6y4MqVHxf62mpSCek1rD+iYW2RezGicHWOY3OK3UOK7QQte209VBraQcDsPLylo1qM1vybl6p80cLwbOgCzPNhA2TVYYD58QRwMTp/b+RLO9/pp0k9/fOkeHLZWL5MPZId8JCn5Sg7ICTklA8KIIXfkJ7mPHqJf0e9OK+1ELWyTJ9FZ+QMa87Cp</latexit>

!1<latexit sha1_base64="9Bu7KHoXXragPIuL8XXDhUOAfck=">AAACT3icfZFNSxxBEIZ71sSPMRo1Ry9NFkFkXWaCoEfRHHIJGnBVcJalprd2bLa/6O4JWYb9G17NX/LoL8ktpGcykKhgQcPT1W9XVb+dG8GdT5LHqLPw5u3i0vJKvPpubf39xubWpdOlZThgWmh7nYNDwRUOPPcCr41FkLnAq3x6Wp9ffUfruFYXfmZwKKFQfMIZ+JDKslzSTEssYJSONrpJP2mCvoS0hS5p43y0Ge1nY81KicozAc7dpInxwwqs50zgPM5KhwbYFAq8CahAohtWzdBzuhMyYzrRNizlaZP9/0YF0knwtz0aoJa4htxM5j2ay2ajjQqFatXTXn5yNKy4MqVHxf62mpSCek1rD+iYW2RezGicfcYwucWvocSZQQte270qA1tI+DEPLylo1qM1vybl6p80cLwTOgCzPNhA2S1YYD58QRwMTp/b+RIuP/XTpJ9+O+gen7RWL5Nt8pHskpQckmPyhZyTAWHEkDtyT35GD9Gv6HenlXaiFj6QJ9FZ+QMZE7Co</latexit><latexit sha1_base64="9Bu7KHoXXragPIuL8XXDhUOAfck=">AAACT3icfZFNSxxBEIZ71sSPMRo1Ry9NFkFkXWaCoEfRHHIJGnBVcJalprd2bLa/6O4JWYb9G17NX/LoL8ktpGcykKhgQcPT1W9XVb+dG8GdT5LHqLPw5u3i0vJKvPpubf39xubWpdOlZThgWmh7nYNDwRUOPPcCr41FkLnAq3x6Wp9ffUfruFYXfmZwKKFQfMIZ+JDKslzSTEssYJSONrpJP2mCvoS0hS5p43y0Ge1nY81KicozAc7dpInxwwqs50zgPM5KhwbYFAq8CahAohtWzdBzuhMyYzrRNizlaZP9/0YF0knwtz0aoJa4htxM5j2ay2ajjQqFatXTXn5yNKy4MqVHxf62mpSCek1rD+iYW2RezGicfcYwucWvocSZQQte270qA1tI+DEPLylo1qM1vybl6p80cLwTOgCzPNhA2S1YYD58QRwMTp/b+RIuP/XTpJ9+O+gen7RWL5Nt8pHskpQckmPyhZyTAWHEkDtyT35GD9Gv6HenlXaiFj6QJ9FZ+QMZE7Co</latexit><latexit sha1_base64="9Bu7KHoXXragPIuL8XXDhUOAfck=">AAACT3icfZFNSxxBEIZ71sSPMRo1Ry9NFkFkXWaCoEfRHHIJGnBVcJalprd2bLa/6O4JWYb9G17NX/LoL8ktpGcykKhgQcPT1W9XVb+dG8GdT5LHqLPw5u3i0vJKvPpubf39xubWpdOlZThgWmh7nYNDwRUOPPcCr41FkLnAq3x6Wp9ffUfruFYXfmZwKKFQfMIZ+JDKslzSTEssYJSONrpJP2mCvoS0hS5p43y0Ge1nY81KicozAc7dpInxwwqs50zgPM5KhwbYFAq8CahAohtWzdBzuhMyYzrRNizlaZP9/0YF0knwtz0aoJa4htxM5j2ay2ajjQqFatXTXn5yNKy4MqVHxf62mpSCek1rD+iYW2RezGicfcYwucWvocSZQQte270qA1tI+DEPLylo1qM1vybl6p80cLwTOgCzPNhA2S1YYD58QRwMTp/b+RIuP/XTpJ9+O+gen7RWL5Nt8pHskpQckmPyhZyTAWHEkDtyT35GD9Gv6HenlXaiFj6QJ9FZ+QMZE7Co</latexit><latexit sha1_base64="9Bu7KHoXXragPIuL8XXDhUOAfck=">AAACT3icfZFNSxxBEIZ71sSPMRo1Ry9NFkFkXWaCoEfRHHIJGnBVcJalprd2bLa/6O4JWYb9G17NX/LoL8ktpGcykKhgQcPT1W9XVb+dG8GdT5LHqLPw5u3i0vJKvPpubf39xubWpdOlZThgWmh7nYNDwRUOPPcCr41FkLnAq3x6Wp9ffUfruFYXfmZwKKFQfMIZ+JDKslzSTEssYJSONrpJP2mCvoS0hS5p43y0Ge1nY81KicozAc7dpInxwwqs50zgPM5KhwbYFAq8CahAohtWzdBzuhMyYzrRNizlaZP9/0YF0knwtz0aoJa4htxM5j2ay2ajjQqFatXTXn5yNKy4MqVHxf62mpSCek1rD+iYW2RezGicfcYwucWvocSZQQte270qA1tI+DEPLylo1qM1vybl6p80cLwTOgCzPNhA2S1YYD58QRwMTp/b+RIuP/XTpJ9+O+gen7RWL5Nt8pHskpQckmPyhZyTAWHEkDtyT35GD9Gv6HenlXaiFj6QJ9FZ+QMZE7Co</latexit>

A Snapshot of Deep MORL Algorithm

D F

Optimal Solutions

Sampled Preferences

c.

Optimality Concept: Pareto frontier

V ⇡(s) := E⌧⇠(P,⇡) [r̂⌧ ]

:= E⌧⇠(P,⇡)

" 1X

t=0

�tr(st, at)

#

<latexit sha1_base64="wfPymSPLnWfC3bP30auDOCfB+9Q=">AAADC3icnVJdi9NAFJ3Er7V+bFcffblYVlqpJRFBERYWP8AXsYrtLnS64WY6aYfNJGHmRiwhP8Ff45v4qv/Bf+Mkrei64IMXBs7cOXPOnZPERaosBcEPz79w8dLlKztXO9eu37i52927NbV5aYSciDzNzXGMVqYqkxNSlMrjwkjUcSqP4tPnzfnRB2msyrP3tC7kXOMyU4kSSK4Vdb/zWMP0pOKFqvt2APeeHgDXSKs4hpdRxQlL4FZp6LddgSmMh+DYg5qnMqEZ8BVS1aiYesOvgRu1XNGc8/+Rs6WOKjoIajeUyhJaO70lao0nFTnYGPVtREPAiAa/rKJuLxgFbcF5EG5Bj21rHO15D/giF6WWGYkUrZ2FQUHzCg0pkcq6w0srCxSnuJQzBzPU0s6rNvAa9l1nAUlu3MoI2u6fNyrUtnmfG1LbhmJbZNc6HkKs201eZE6oYZ31ouTJvFJZUZLMxMYqKVOgHJrvBwtlpKB0DR3+QrrJjXztJN4U0iDl5n7F0Sw1fqzdS5bAh9Dgf1FV9pvqcGffOaAwysUAYoUGBbnfp+MCDv+O8zyYPhyFwSh8+6h3+Gwb9Q67w+6yPgvZY3bIXrExmzDhDb133szj/if/s//F/7qh+t72zm12pvxvPwFZ2/KX</latexit><latexit sha1_base64="wfPymSPLnWfC3bP30auDOCfB+9Q=">AAADC3icnVJdi9NAFJ3Er7V+bFcffblYVlqpJRFBERYWP8AXsYrtLnS64WY6aYfNJGHmRiwhP8Ff45v4qv/Bf+Mkrei64IMXBs7cOXPOnZPERaosBcEPz79w8dLlKztXO9eu37i52927NbV5aYSciDzNzXGMVqYqkxNSlMrjwkjUcSqP4tPnzfnRB2msyrP3tC7kXOMyU4kSSK4Vdb/zWMP0pOKFqvt2APeeHgDXSKs4hpdRxQlL4FZp6LddgSmMh+DYg5qnMqEZ8BVS1aiYesOvgRu1XNGc8/+Rs6WOKjoIajeUyhJaO70lao0nFTnYGPVtREPAiAa/rKJuLxgFbcF5EG5Bj21rHO15D/giF6WWGYkUrZ2FQUHzCg0pkcq6w0srCxSnuJQzBzPU0s6rNvAa9l1nAUlu3MoI2u6fNyrUtnmfG1LbhmJbZNc6HkKs201eZE6oYZ31ouTJvFJZUZLMxMYqKVOgHJrvBwtlpKB0DR3+QrrJjXztJN4U0iDl5n7F0Sw1fqzdS5bAh9Dgf1FV9pvqcGffOaAwysUAYoUGBbnfp+MCDv+O8zyYPhyFwSh8+6h3+Gwb9Q67w+6yPgvZY3bIXrExmzDhDb133szj/if/s//F/7qh+t72zm12pvxvPwFZ2/KX</latexit><latexit sha1_base64="wfPymSPLnWfC3bP30auDOCfB+9Q=">AAADC3icnVJdi9NAFJ3Er7V+bFcffblYVlqpJRFBERYWP8AXsYrtLnS64WY6aYfNJGHmRiwhP8Ff45v4qv/Bf+Mkrei64IMXBs7cOXPOnZPERaosBcEPz79w8dLlKztXO9eu37i52927NbV5aYSciDzNzXGMVqYqkxNSlMrjwkjUcSqP4tPnzfnRB2msyrP3tC7kXOMyU4kSSK4Vdb/zWMP0pOKFqvt2APeeHgDXSKs4hpdRxQlL4FZp6LddgSmMh+DYg5qnMqEZ8BVS1aiYesOvgRu1XNGc8/+Rs6WOKjoIajeUyhJaO70lao0nFTnYGPVtREPAiAa/rKJuLxgFbcF5EG5Bj21rHO15D/giF6WWGYkUrZ2FQUHzCg0pkcq6w0srCxSnuJQzBzPU0s6rNvAa9l1nAUlu3MoI2u6fNyrUtnmfG1LbhmJbZNc6HkKs201eZE6oYZ31ouTJvFJZUZLMxMYqKVOgHJrvBwtlpKB0DR3+QrrJjXztJN4U0iDl5n7F0Sw1fqzdS5bAh9Dgf1FV9pvqcGffOaAwysUAYoUGBbnfp+MCDv+O8zyYPhyFwSh8+6h3+Gwb9Q67w+6yPgvZY3bIXrExmzDhDb133szj/if/s//F/7qh+t72zm12pvxvPwFZ2/KX</latexit><latexit sha1_base64="wfPymSPLnWfC3bP30auDOCfB+9Q=">AAADC3icnVJdi9NAFJ3Er7V+bFcffblYVlqpJRFBERYWP8AXsYrtLnS64WY6aYfNJGHmRiwhP8Ff45v4qv/Bf+Mkrei64IMXBs7cOXPOnZPERaosBcEPz79w8dLlKztXO9eu37i52927NbV5aYSciDzNzXGMVqYqkxNSlMrjwkjUcSqP4tPnzfnRB2msyrP3tC7kXOMyU4kSSK4Vdb/zWMP0pOKFqvt2APeeHgDXSKs4hpdRxQlL4FZp6LddgSmMh+DYg5qnMqEZ8BVS1aiYesOvgRu1XNGc8/+Rs6WOKjoIajeUyhJaO70lao0nFTnYGPVtREPAiAa/rKJuLxgFbcF5EG5Bj21rHO15D/giF6WWGYkUrZ2FQUHzCg0pkcq6w0srCxSnuJQzBzPU0s6rNvAa9l1nAUlu3MoI2u6fNyrUtnmfG1LbhmJbZNc6HkKs201eZE6oYZ31ouTJvFJZUZLMxMYqKVOgHJrvBwtlpKB0DR3+QrrJjXztJN4U0iDl5n7F0Sw1fqzdS5bAh9Dgf1FV9pvqcGffOaAwysUAYoUGBbnfp+MCDv+O8zyYPhyFwSh8+6h3+Gwb9Q67w+6yPgvZY3bIXrExmzDhDb133szj/if/s//F/7qh+t72zm12pvxvPwFZ2/KX</latexit>

The expected total rewards are vectors

Optimality Concept: Pareto frontier

AB

C

D

E

F

G

H

K

L


QU

ANTI

TY O

F O

BJEC

TIVE

2 CCS

Non-optimal

Pareto Frontier

F


QU

ANTI

TY O

F O

BJEC

TIVE

2 F Non-preferred

Preference




a. b.

DUtility Projection


QU

ANTI

TY O

F O

BJEC

TIVE

2

D

F




D F

Optimal Solutions

Sampled Preferences

c.

⇧⇤ := {⇡ | @⇡0 2 ⇧,⇡0 � ⇡}<latexit sha1_base64="xhPnEM4hzpDYMmEZonqz5J4Y/78=">AAACf3icfZHbahsxEIblTduk20Oc9rI3Q01ICa7ZLYUcIBDSXvSmxIU6CUSu0cpjR0QnJG2JWfwufZrcJrd9m2o3hpygA4JvRr80ml+FlcKHLPvbSpaePH22vPI8ffHy1evV9tqbI29Kx3HAjTTupGAepdA4CCJIPLEOmSokHhfnX+r949/ovDD6Z5hZHCo21WIiOAuxNGrv0r74tQm7e0AragVQJcZANV7E1h5iZYMKDVHUbRKgvuS8RjoftTtZL2sCHkO+gA5ZRH+01vpIx4aXCnXgknl/mmc2DCvmguAS5yktPVrGz9kUTyNqptAPq2bIOazHyhgmxsWlAzTVuycqprxi4awLEWqJb8jPVNGFQjWJsTpeVKvu9wqT7WEltC0Dan7TalJKCAZqz2AsHPIgZ5DSrxhf7vB7vOLQomPBuM2KMjdV7GIeJ5kC7ULN/5MKfSuNnK7HDow7EW0AfsYc4yF+WRoNzh/a+RiOPvXyrJf/+NzZP1hYvULekffkA8nJFtkn30ifDAgnf8gluSLXSSvZSHpJdiNNWoszb8m9SHb+AZn+vuw=</latexit><latexit sha1_base64="xhPnEM4hzpDYMmEZonqz5J4Y/78=">AAACf3icfZHbahsxEIblTduk20Oc9rI3Q01ICa7ZLYUcIBDSXvSmxIU6CUSu0cpjR0QnJG2JWfwufZrcJrd9m2o3hpygA4JvRr80ml+FlcKHLPvbSpaePH22vPI8ffHy1evV9tqbI29Kx3HAjTTupGAepdA4CCJIPLEOmSokHhfnX+r949/ovDD6Z5hZHCo21WIiOAuxNGrv0r74tQm7e0AragVQJcZANV7E1h5iZYMKDVHUbRKgvuS8RjoftTtZL2sCHkO+gA5ZRH+01vpIx4aXCnXgknl/mmc2DCvmguAS5yktPVrGz9kUTyNqptAPq2bIOazHyhgmxsWlAzTVuycqprxi4awLEWqJb8jPVNGFQjWJsTpeVKvu9wqT7WEltC0Dan7TalJKCAZqz2AsHPIgZ5DSrxhf7vB7vOLQomPBuM2KMjdV7GIeJ5kC7ULN/5MKfSuNnK7HDow7EW0AfsYc4yF+WRoNzh/a+RiOPvXyrJf/+NzZP1hYvULekffkA8nJFtkn30ifDAgnf8gluSLXSSvZSHpJdiNNWoszb8m9SHb+AZn+vuw=</latexit><latexit sha1_base64="xhPnEM4hzpDYMmEZonqz5J4Y/78=">AAACf3icfZHbahsxEIblTduk20Oc9rI3Q01ICa7ZLYUcIBDSXvSmxIU6CUSu0cpjR0QnJG2JWfwufZrcJrd9m2o3hpygA4JvRr80ml+FlcKHLPvbSpaePH22vPI8ffHy1evV9tqbI29Kx3HAjTTupGAepdA4CCJIPLEOmSokHhfnX+r949/ovDD6Z5hZHCo21WIiOAuxNGrv0r74tQm7e0AragVQJcZANV7E1h5iZYMKDVHUbRKgvuS8RjoftTtZL2sCHkO+gA5ZRH+01vpIx4aXCnXgknl/mmc2DCvmguAS5yktPVrGz9kUTyNqptAPq2bIOazHyhgmxsWlAzTVuycqprxi4awLEWqJb8jPVNGFQjWJsTpeVKvu9wqT7WEltC0Dan7TalJKCAZqz2AsHPIgZ5DSrxhf7vB7vOLQomPBuM2KMjdV7GIeJ5kC7ULN/5MKfSuNnK7HDow7EW0AfsYc4yF+WRoNzh/a+RiOPvXyrJf/+NzZP1hYvULekffkA8nJFtkn30ifDAgnf8gluSLXSSvZSHpJdiNNWoszb8m9SHb+AZn+vuw=</latexit><latexit sha1_base64="xhPnEM4hzpDYMmEZonqz5J4Y/78=">AAACf3icfZHbahsxEIblTduk20Oc9rI3Q01ICa7ZLYUcIBDSXvSmxIU6CUSu0cpjR0QnJG2JWfwufZrcJrd9m2o3hpygA4JvRr80ml+FlcKHLPvbSpaePH22vPI8ffHy1evV9tqbI29Kx3HAjTTupGAepdA4CCJIPLEOmSokHhfnX+r949/ovDD6Z5hZHCo21WIiOAuxNGrv0r74tQm7e0AragVQJcZANV7E1h5iZYMKDVHUbRKgvuS8RjoftTtZL2sCHkO+gA5ZRH+01vpIx4aXCnXgknl/mmc2DCvmguAS5yktPVrGz9kUTyNqptAPq2bIOazHyhgmxsWlAzTVuycqprxi4awLEWqJb8jPVNGFQjWJsTpeVKvu9wqT7WEltC0Dan7TalJKCAZqz2AsHPIgZ5DSrxhf7vB7vOLQomPBuM2KMjdV7GIeJ5kC7ULN/5MKfSuNnK7HDow7EW0AfsYc4yF+WRoNzh/a+RiOPvXyrJf/+NzZP1hYvULekffkA8nJFtkn30ifDAgnf8gluSLXSSvZSHpJdiNNWoszb8m9SHb+AZn+vuw=</latexit>

Pareto optimal policies:

⇡0 � ⇡ , 8i 2 [m], V ⇡0

i (s0) > V ⇡i (s0)

<latexit sha1_base64="q6BP35vsthSblSNJAF5bQr4CNDY=">AAACl3icfZFda9RAFIZn41eNX1u9Em8OLsUq65JIoV5pUZFeKLbibgubGCazJ9mh8xFmJuoS9n/5V7zxVn+GkzSiteCBwDPvvGfOzJu8Ety6KPo2CC5cvHT5ysbV8Nr1GzdvDTdvz6yuDcMp00Kb45xaFFzh1HEn8LgySGUu8Cg/ednuH31CY7lWH9yqwlTSUvGCM+q8lA3fJxV/AImtGQOPkLzBwhleLh01Rn+GpNCGCgF+hyuYy3QMs49N27TO+LbNoofwrFd+C9lwFE2iruA8xD2MSF8H2ebgcbLQrJaoHBPU2nkcVS5tqHGcCVyHSW2xouyEljj3qKhEmzbd49ew5ZUF+Gv6Tzno1L87GiqtpG45Bg+txXZkVzIfQy67ha6UP6h1nZ3liqdpw1VVO1TsdFRRC3Aa2ixhwQ0yJ1YQJq/Q39zgW3/EuwoNddo8ahJqSkm/rP1LSkjG0PL/rFz9sXoOt/wEygz3MQBbUkOZ878y9AHH/8Z5HmZPJnE0iQ93Rnsv+qg3yD1yn2yTmOySPbJPDsiUMPKVfCc/yM/gbvA8eB3sn1qDQd9zh5yp4PAX4LbIiQ==</latexit><latexit sha1_base64="q6BP35vsthSblSNJAF5bQr4CNDY=">AAACl3icfZFda9RAFIZn41eNX1u9Em8OLsUq65JIoV5pUZFeKLbibgubGCazJ9mh8xFmJuoS9n/5V7zxVn+GkzSiteCBwDPvvGfOzJu8Ety6KPo2CC5cvHT5ysbV8Nr1GzdvDTdvz6yuDcMp00Kb45xaFFzh1HEn8LgySGUu8Cg/ednuH31CY7lWH9yqwlTSUvGCM+q8lA3fJxV/AImtGQOPkLzBwhleLh01Rn+GpNCGCgF+hyuYy3QMs49N27TO+LbNoofwrFd+C9lwFE2iruA8xD2MSF8H2ebgcbLQrJaoHBPU2nkcVS5tqHGcCVyHSW2xouyEljj3qKhEmzbd49ew5ZUF+Gv6Tzno1L87GiqtpG45Bg+txXZkVzIfQy67ha6UP6h1nZ3liqdpw1VVO1TsdFRRC3Aa2ixhwQ0yJ1YQJq/Q39zgW3/EuwoNddo8ahJqSkm/rP1LSkjG0PL/rFz9sXoOt/wEygz3MQBbUkOZ878y9AHH/8Z5HmZPJnE0iQ93Rnsv+qg3yD1yn2yTmOySPbJPDsiUMPKVfCc/yM/gbvA8eB3sn1qDQd9zh5yp4PAX4LbIiQ==</latexit><latexit sha1_base64="q6BP35vsthSblSNJAF5bQr4CNDY=">AAACl3icfZFda9RAFIZn41eNX1u9Em8OLsUq65JIoV5pUZFeKLbibgubGCazJ9mh8xFmJuoS9n/5V7zxVn+GkzSiteCBwDPvvGfOzJu8Ety6KPo2CC5cvHT5ysbV8Nr1GzdvDTdvz6yuDcMp00Kb45xaFFzh1HEn8LgySGUu8Cg/ednuH31CY7lWH9yqwlTSUvGCM+q8lA3fJxV/AImtGQOPkLzBwhleLh01Rn+GpNCGCgF+hyuYy3QMs49N27TO+LbNoofwrFd+C9lwFE2iruA8xD2MSF8H2ebgcbLQrJaoHBPU2nkcVS5tqHGcCVyHSW2xouyEljj3qKhEmzbd49ew5ZUF+Gv6Tzno1L87GiqtpG45Bg+txXZkVzIfQy67ha6UP6h1nZ3liqdpw1VVO1TsdFRRC3Aa2ixhwQ0yJ1YQJq/Q39zgW3/EuwoNddo8ahJqSkm/rP1LSkjG0PL/rFz9sXoOt/wEygz3MQBbUkOZ878y9AHH/8Z5HmZPJnE0iQ93Rnsv+qg3yD1yn2yTmOySPbJPDsiUMPKVfCc/yM/gbvA8eB3sn1qDQd9zh5yp4PAX4LbIiQ==</latexit><latexit sha1_base64="q6BP35vsthSblSNJAF5bQr4CNDY=">AAACl3icfZFda9RAFIZn41eNX1u9Em8OLsUq65JIoV5pUZFeKLbibgubGCazJ9mh8xFmJuoS9n/5V7zxVn+GkzSiteCBwDPvvGfOzJu8Ety6KPo2CC5cvHT5ysbV8Nr1GzdvDTdvz6yuDcMp00Kb45xaFFzh1HEn8LgySGUu8Cg/ednuH31CY7lWH9yqwlTSUvGCM+q8lA3fJxV/AImtGQOPkLzBwhleLh01Rn+GpNCGCgF+hyuYy3QMs49N27TO+LbNoofwrFd+C9lwFE2iruA8xD2MSF8H2ebgcbLQrJaoHBPU2nkcVS5tqHGcCVyHSW2xouyEljj3qKhEmzbd49ew5ZUF+Gv6Tzno1L87GiqtpG45Bg+txXZkVzIfQy67ha6UP6h1nZ3liqdpw1VVO1TsdFRRC3Aa2ixhwQ0yJ1YQJq/Q39zgW3/EuwoNddo8ahJqSkm/rP1LSkjG0PL/rFz9sXoOt/wEygz3MQBbUkOZ878y9AHH/8Z5HmZPJnE0iQ93Rnsv+qg3yD1yn2yTmOySPbJPDsiUMPKVfCc/yM/gbvA8eB3sn1qDQd9zh5yp4PAX4LbIiQ==</latexit>

Policy dominance:

F⇤ := {V ⇡(s0) | ⇡ 2 ⇧⇤}<latexit sha1_base64="6c8+yxeJK0sBKXjZoEtqqPwGkxs=">AAACfnicfZFdaxQxFIaz41c7fm31sjcHl0ot2+2MCFahUFTEG3EFd1totksme3Ybmi+SjLgM81v6a7zVa/+NmemC1oIHAk9O3pOT86awUviQZb86yY2bt27fWVtP7967/+Bhd+PR2JvScRxxI407LphHKTSOgggSj61DpgqJR8X52+b86Cs6L4z+EpYWJ4ottJgLzkJMTbuvqGLhjDMJ70934PUB0IoWCsanFbWihm0/zZ4BVWIGcQ9UaKBDEZW0nnZ72SBrA65DvoIeWcVwutHZpTPDS4U6cMm8P8kzGyYVc0FwiXVKS4+W8XO2wJOImin0k6qdsYatmJnB3Li4dIA2+3dFxZRvRulDhEbiW/JLVfShUO3GWB0valRXe4X5/qQS2pYBNb9sNS8lBAONZTATDnmQS0jpO4wvd/gxXvHJomPBuJ2KMrdQ7FsdJ1kA7UPD/5MK/UcaOd2KHRh3ItoA/Iw5xkP8sTQanP9r53UYPx/k2SD//KJ3+GZl9RrZJE/INsnJS3JIPpAhGRFOLsh38oP8TEjyNNlN9i6lSWdV85hciWT/N0LxvcM=</latexit><latexit sha1_base64="6c8+yxeJK0sBKXjZoEtqqPwGkxs=">AAACfnicfZFdaxQxFIaz41c7fm31sjcHl0ot2+2MCFahUFTEG3EFd1totksme3Ybmi+SjLgM81v6a7zVa/+NmemC1oIHAk9O3pOT86awUviQZb86yY2bt27fWVtP7967/+Bhd+PR2JvScRxxI407LphHKTSOgggSj61DpgqJR8X52+b86Cs6L4z+EpYWJ4ottJgLzkJMTbuvqGLhjDMJ70934PUB0IoWCsanFbWihm0/zZ4BVWIGcQ9UaKBDEZW0nnZ72SBrA65DvoIeWcVwutHZpTPDS4U6cMm8P8kzGyYVc0FwiXVKS4+W8XO2wJOImin0k6qdsYatmJnB3Li4dIA2+3dFxZRvRulDhEbiW/JLVfShUO3GWB0valRXe4X5/qQS2pYBNb9sNS8lBAONZTATDnmQS0jpO4wvd/gxXvHJomPBuJ2KMrdQ7FsdJ1kA7UPD/5MK/UcaOd2KHRh3ItoA/Iw5xkP8sTQanP9r53UYPx/k2SD//KJ3+GZl9RrZJE/INsnJS3JIPpAhGRFOLsh38oP8TEjyNNlN9i6lSWdV85hciWT/N0LxvcM=</latexit><latexit sha1_base64="6c8+yxeJK0sBKXjZoEtqqPwGkxs=">AAACfnicfZFdaxQxFIaz41c7fm31sjcHl0ot2+2MCFahUFTEG3EFd1totksme3Ybmi+SjLgM81v6a7zVa/+NmemC1oIHAk9O3pOT86awUviQZb86yY2bt27fWVtP7967/+Bhd+PR2JvScRxxI407LphHKTSOgggSj61DpgqJR8X52+b86Cs6L4z+EpYWJ4ottJgLzkJMTbuvqGLhjDMJ70934PUB0IoWCsanFbWihm0/zZ4BVWIGcQ9UaKBDEZW0nnZ72SBrA65DvoIeWcVwutHZpTPDS4U6cMm8P8kzGyYVc0FwiXVKS4+W8XO2wJOImin0k6qdsYatmJnB3Li4dIA2+3dFxZRvRulDhEbiW/JLVfShUO3GWB0valRXe4X5/qQS2pYBNb9sNS8lBAONZTATDnmQS0jpO4wvd/gxXvHJomPBuJ2KMrdQ7FsdJ1kA7UPD/5MK/UcaOd2KHRh3ItoA/Iw5xkP8sTQanP9r53UYPx/k2SD//KJ3+GZl9RrZJE/INsnJS3JIPpAhGRFOLsh38oP8TEjyNNlN9i6lSWdV85hciWT/N0LxvcM=</latexit><latexit sha1_base64="6c8+yxeJK0sBKXjZoEtqqPwGkxs=">AAACfnicfZFdaxQxFIaz41c7fm31sjcHl0ot2+2MCFahUFTEG3EFd1totksme3Ybmi+SjLgM81v6a7zVa/+NmemC1oIHAk9O3pOT86awUviQZb86yY2bt27fWVtP7967/+Bhd+PR2JvScRxxI407LphHKTSOgggSj61DpgqJR8X52+b86Cs6L4z+EpYWJ4ottJgLzkJMTbuvqGLhjDMJ70934PUB0IoWCsanFbWihm0/zZ4BVWIGcQ9UaKBDEZW0nnZ72SBrA65DvoIeWcVwutHZpTPDS4U6cMm8P8kzGyYVc0FwiXVKS4+W8XO2wJOImin0k6qdsYatmJnB3Li4dIA2+3dFxZRvRulDhEbiW/JLVfShUO3GWB0valRXe4X5/qQS2pYBNb9sNS8lBAONZTATDnmQS0jpO4wvd/gxXvHJomPBuJ2KMrdQ7FsdJ1kA7UPD/5MK/UcaOd2KHRh3ItoA/Iw5xkP8sTQanP9r53UYPx/k2SD//KJ3+GZl9RrZJE/INsnJS3JIPpAhGRFOLsh38oP8TEjyNNlN9i6lSWdV85hciWT/N0LxvcM=</latexit>

Pareto (optimal returns) frontier:

Preference Functions: selection of desired policies

AB

C

D

E

F

G

H

K

L


QU

ANTI

TY O

F O

BJEC

TIVE

2 CCS

Non-optimal

Pareto Frontier

F

QUANTITY OF OBJECTIVE 1Q

UAN

TITY

OF

OBJ

ECTI

VE 2 F Non-preferred

Preference




a. b.

DUtility Projection


QU

ANTI

TY O

F O

BJEC

TIVE

2

D

F




D F

Optimal Solutions

Sampled Preferences

c.Preference function f : Rm ! R

<latexit sha1_base64="PBQ8SUD2z0V3yFgYqj2xDtOir1U=">AAACaXicfVHLahRBFK1pX7F9zegm6KZwDIhMhm4RFFfBuHAjRnGSQHocbtfc7ilSj6bqtjo08wP5mmz1T/wGf8LqzkCMAS8UnDp17utUXinpKUl+9aIrV69dv7FxM751+87de/3B/X1vaydwIqyy7jAHj0oanJAkhYeVQ9C5woP8eLd9P/iKzktrPtOywqmG0shCCqBAzfpPitc800CLPOefvmieOVkuCJyz3875WX+YjJMu+GWQrsGQrWNvNuhtZ3Mrao2GhALvj9KkomkDjqRQuIqz2mMF4hhKPArQgEY/bbp1VnwrMHNeWBeOId6xf2c0oH072ogH0Ep8h/xS5yOe6+5iKxMKtaqLvah4NW2kqWpCI85aFbXiZHnrDp9Lh4LUksfZWwyTO3wfSnyo0AFZ96zJwJUavq/CJiXPRrzF/5NKcy4NON4KHUA4GWzgYgEOBIXPiYPB6b92Xgb7z8dpMk4/vhjuvFlbvcEescfsKUvZS7bD3rE9NmGCnbBT9oP97P2OBtFm9PBMGvXWOQ/YhYiGfwDbebhz</latexit><latexit sha1_base64="PBQ8SUD2z0V3yFgYqj2xDtOir1U=">AAACaXicfVHLahRBFK1pX7F9zegm6KZwDIhMhm4RFFfBuHAjRnGSQHocbtfc7ilSj6bqtjo08wP5mmz1T/wGf8LqzkCMAS8UnDp17utUXinpKUl+9aIrV69dv7FxM751+87de/3B/X1vaydwIqyy7jAHj0oanJAkhYeVQ9C5woP8eLd9P/iKzktrPtOywqmG0shCCqBAzfpPitc800CLPOefvmieOVkuCJyz3875WX+YjJMu+GWQrsGQrWNvNuhtZ3Mrao2GhALvj9KkomkDjqRQuIqz2mMF4hhKPArQgEY/bbp1VnwrMHNeWBeOId6xf2c0oH072ogH0Ep8h/xS5yOe6+5iKxMKtaqLvah4NW2kqWpCI85aFbXiZHnrDp9Lh4LUksfZWwyTO3wfSnyo0AFZ96zJwJUavq/CJiXPRrzF/5NKcy4NON4KHUA4GWzgYgEOBIXPiYPB6b92Xgb7z8dpMk4/vhjuvFlbvcEescfsKUvZS7bD3rE9NmGCnbBT9oP97P2OBtFm9PBMGvXWOQ/YhYiGfwDbebhz</latexit><latexit sha1_base64="PBQ8SUD2z0V3yFgYqj2xDtOir1U=">AAACaXicfVHLahRBFK1pX7F9zegm6KZwDIhMhm4RFFfBuHAjRnGSQHocbtfc7ilSj6bqtjo08wP5mmz1T/wGf8LqzkCMAS8UnDp17utUXinpKUl+9aIrV69dv7FxM751+87de/3B/X1vaydwIqyy7jAHj0oanJAkhYeVQ9C5woP8eLd9P/iKzktrPtOywqmG0shCCqBAzfpPitc800CLPOefvmieOVkuCJyz3875WX+YjJMu+GWQrsGQrWNvNuhtZ3Mrao2GhALvj9KkomkDjqRQuIqz2mMF4hhKPArQgEY/bbp1VnwrMHNeWBeOId6xf2c0oH072ogH0Ep8h/xS5yOe6+5iKxMKtaqLvah4NW2kqWpCI85aFbXiZHnrDp9Lh4LUksfZWwyTO3wfSnyo0AFZ96zJwJUavq/CJiXPRrzF/5NKcy4NON4KHUA4GWzgYgEOBIXPiYPB6b92Xgb7z8dpMk4/vhjuvFlbvcEescfsKUvZS7bD3rE9NmGCnbBT9oP97P2OBtFm9PBMGvXWOQ/YhYiGfwDbebhz</latexit><latexit sha1_base64="PBQ8SUD2z0V3yFgYqj2xDtOir1U=">AAACaXicfVHLahRBFK1pX7F9zegm6KZwDIhMhm4RFFfBuHAjRnGSQHocbtfc7ilSj6bqtjo08wP5mmz1T/wGf8LqzkCMAS8UnDp17utUXinpKUl+9aIrV69dv7FxM751+87de/3B/X1vaydwIqyy7jAHj0oanJAkhYeVQ9C5woP8eLd9P/iKzktrPtOywqmG0shCCqBAzfpPitc800CLPOefvmieOVkuCJyz3875WX+YjJMu+GWQrsGQrWNvNuhtZ3Mrao2GhALvj9KkomkDjqRQuIqz2mMF4hhKPArQgEY/bbp1VnwrMHNeWBeOId6xf2c0oH072ogH0Ep8h/xS5yOe6+5iKxMKtaqLvah4NW2kqWpCI85aFbXiZHnrDp9Lh4LUksfZWwyTO3wfSnyo0AFZ96zJwJUavq/CJiXPRrzF/5NKcy4NON4KHUA4GWzgYgEOBIXPiYPB6b92Xgb7z8dpMk4/vhjuvFlbvcEescfsKUvZS7bD3rE9NmGCnbBT9oP97P2OBtFm9PBMGvXWOQ/YhYiGfwDbebhz</latexit>

f � V ⇡(s)<latexit sha1_base64="Joam1jC8wPven1HPPmWdLHyoGTA=">AAACWHicfZFNaxRBEIZrJ2qS8SOb5OilcQlEWZcZEfQYogcvwQjuJpBZl5remk2T/qK7R1yG/S1e9Sfpr0nPZEFjwIKGp6vfqup+u7RS+JBlv3rJxr37Dza3ttOHjx4/2env7k28qR2nMTfSuPMSPUmhaRxEkHRuHaEqJZ2VV+/a87Ov5Lww+nNYWpoqXGhRCY4hpmb9/argwvGiVGzypbCCHfrns/4gG2VdsLuQr2EA6zid7fZeFnPDa0U6cIneX+SZDdMGXRBc0iotak8W+RUu6CKiRkV+2nS3X7GDmJmzyri4dGBd9u+KBpVXGC6HLEIr8R35pSqHrFTdxlgdG7Wq27NC9XbaCG3rQJrfjKpqyYJhrRlsLhzxIJcsLd5TvLmjk9jioyWHwbgXTYFuofDbKr5kwYoha/l/UqH/SCOnB3ECcieiDYxfokMe4l+k0eD8XzvvwuTVKM9G+afXg6PjtdVb8BSewSHk8AaO4AOcwhg4LOE7/ICfvd8JJJvJ9o006a1r9uFWJHvXsIaxXA==</latexit><latexit sha1_base64="Joam1jC8wPven1HPPmWdLHyoGTA=">AAACWHicfZFNaxRBEIZrJ2qS8SOb5OilcQlEWZcZEfQYogcvwQjuJpBZl5remk2T/qK7R1yG/S1e9Sfpr0nPZEFjwIKGp6vfqup+u7RS+JBlv3rJxr37Dza3ttOHjx4/2env7k28qR2nMTfSuPMSPUmhaRxEkHRuHaEqJZ2VV+/a87Ov5Lww+nNYWpoqXGhRCY4hpmb9/argwvGiVGzypbCCHfrns/4gG2VdsLuQr2EA6zid7fZeFnPDa0U6cIneX+SZDdMGXRBc0iotak8W+RUu6CKiRkV+2nS3X7GDmJmzyri4dGBd9u+KBpVXGC6HLEIr8R35pSqHrFTdxlgdG7Wq27NC9XbaCG3rQJrfjKpqyYJhrRlsLhzxIJcsLd5TvLmjk9jioyWHwbgXTYFuofDbKr5kwYoha/l/UqH/SCOnB3ECcieiDYxfokMe4l+k0eD8XzvvwuTVKM9G+afXg6PjtdVb8BSewSHk8AaO4AOcwhg4LOE7/ICfvd8JJJvJ9o006a1r9uFWJHvXsIaxXA==</latexit><latexit sha1_base64="Joam1jC8wPven1HPPmWdLHyoGTA=">AAACWHicfZFNaxRBEIZrJ2qS8SOb5OilcQlEWZcZEfQYogcvwQjuJpBZl5remk2T/qK7R1yG/S1e9Sfpr0nPZEFjwIKGp6vfqup+u7RS+JBlv3rJxr37Dza3ttOHjx4/2env7k28qR2nMTfSuPMSPUmhaRxEkHRuHaEqJZ2VV+/a87Ov5Lww+nNYWpoqXGhRCY4hpmb9/argwvGiVGzypbCCHfrns/4gG2VdsLuQr2EA6zid7fZeFnPDa0U6cIneX+SZDdMGXRBc0iotak8W+RUu6CKiRkV+2nS3X7GDmJmzyri4dGBd9u+KBpVXGC6HLEIr8R35pSqHrFTdxlgdG7Wq27NC9XbaCG3rQJrfjKpqyYJhrRlsLhzxIJcsLd5TvLmjk9jioyWHwbgXTYFuofDbKr5kwYoha/l/UqH/SCOnB3ECcieiDYxfokMe4l+k0eD8XzvvwuTVKM9G+afXg6PjtdVb8BSewSHk8AaO4AOcwhg4LOE7/ICfvd8JJJvJ9o006a1r9uFWJHvXsIaxXA==</latexit><latexit sha1_base64="9VF5nnSTe/psDq3XUJsbUwGqaRc=">AAACNXicfZHLSgMxFIYz3h3vazdBEURqmXGjS0EXbkQF2wqdImfS0xrMZUgyYhn6Am59JV/CV3AnvoCZaUFrwQOBLyd/zjXNBLcuit6Dmdm5+YXFpeVwZTVcW9/YXG1anRuGDaaFNncpWBRcYcNxJ/AuMwgyFdhKH8/K99YTGsu1unWDDDsS+or3OAPnXdf3m7tRPaqMTkM8hl0ytvut4DDpapZLVI4JsLYdR5nrFGAcZwKHYZJbzIA9Qh/bHhVItJ2iqnNI97ynS3va+KMcrby/fxQgrQT3UKMeSomtyA5kWqOprC46Uz5QqZrM5XonnYKrLHeo2ChVLxfUaVq2TbvcIHNiQMPkHH3lBi99iKsMDThtDooETF/C89B30qdJjZb8n5SrH6nncM9nAGa4HwNlD2CAOT/10M83/jvNaWge1eOoHt9EZIlskx2yT2JyTE7JBbkmDcJIl7yQ1+At+Ag+R3uYCcYL2SITFnx9A81eq1A=</latexit><latexit sha1_base64="XXXJq1DxLhoTLXosRJjoDaq7OYI=">AAACTXicfZFNaxsxEIbH248k27Rx2mMvIiGQFtfs9tIeA+2hl5IUYieQdc2sPOuI6AtJW2IW/5Ze25/U/ppoN4Z8QQYEj0avZqR3SiuFD1n2r5c8efrs+dr6Rvpi8+Wrrf725tib2nEacSONOy3RkxSaRkEESafWEapS0kl58aU9P/lFzgujj8PC0kThXItKcAwxNe2/qQouHC9KxcY/CyvYvn837e9mw6wL9hDyFezCKo6m270PxczwWpEOXKL3Z3lmw6RBFwSXtEyL2pNFfoFzOouoUZGfNN3rl2wvZmasMi4uHViXvX2jQeUVhvMBi9BKfEd+ocoBK1W3MVbHQq3qbq9QfZ40Qts6kObXrapasmBYawabCUc8yAVLi68UX+7oeyxxaMlhMO59U6CbK7xcxp/MWTFgLT8mFfpGGjndix2QOxFtYPwcHfIQZ5FGg/P7dj6E8cdhng3zHxmsw1vYgX3I4RMcwDc4ghFwWMBv+AN/e/8TSNauR5H0VjN5DXci2bgCC+ewmA==</latexit><latexit sha1_base64="XXXJq1DxLhoTLXosRJjoDaq7OYI=">AAACTXicfZFNaxsxEIbH248k27Rx2mMvIiGQFtfs9tIeA+2hl5IUYieQdc2sPOuI6AtJW2IW/5Ze25/U/ppoN4Z8QQYEj0avZqR3SiuFD1n2r5c8efrs+dr6Rvpi8+Wrrf725tib2nEacSONOy3RkxSaRkEESafWEapS0kl58aU9P/lFzgujj8PC0kThXItKcAwxNe2/qQouHC9KxcY/CyvYvn837e9mw6wL9hDyFezCKo6m270PxczwWpEOXKL3Z3lmw6RBFwSXtEyL2pNFfoFzOouoUZGfNN3rl2wvZmasMi4uHViXvX2jQeUVhvMBi9BKfEd+ocoBK1W3MVbHQq3qbq9QfZ40Qts6kObXrapasmBYawabCUc8yAVLi68UX+7oeyxxaMlhMO59U6CbK7xcxp/MWTFgLT8mFfpGGjndix2QOxFtYPwcHfIQZ5FGg/P7dj6E8cdhng3zHxmsw1vYgX3I4RMcwDc4ghFwWMBv+AN/e/8TSNauR5H0VjN5DXci2bgCC+ewmA==</latexit><latexit sha1_base64="05/Qnfrap/jjpaEdDO/QCc7Mzxw=">AAACWHicfZFNbxMxEIYnW6Dt8pW2Ry4WUaWCQrTLBY4VcOCCKFKTVuqGaNaZTa36S7a3arTKb+kVfhL8GrzbSLRUYiRLj8fvzNivSyuFD1n2q5dsPHj4aHNrO3385Omz5/2d3Yk3teM05kYad1qiJyk0jYMIkk6tI1SlpJPy4mN7fnJJzgujj8PS0lThQotKcAwxNevvVQUXjhelYpPvhRXswL+a9QfZKOuC3Yd8DQNYx9Fsp/emmBteK9KBS/T+LM9smDboguCSVmlRe7LIL3BBZxE1KvLTprv9iu3HzJxVxsWlA+uytysaVF5hOB+yCK3Ed+SXqhyyUnUbY3Vs1KruzgrV+2kjtK0DaX4zqqolC4a1ZrC5cMSDXLK0+ETx5o6+xBZfLTkMxr1uCnQLhVer+JIFK4as5f9Jhf4rjZzuxwnInYg2MH6ODnmIf5FGg/N/7bwPk7ejPBvl37LB4Ye11VvwAl7CAeTwDg7hMxzBGDgs4Rp+wM/e7wSSzWT7Rpr01jV7cCeS3T+vRrFY</latexit><latexit sha1_base64="Joam1jC8wPven1HPPmWdLHyoGTA=">AAACWHicfZFNaxRBEIZrJ2qS8SOb5OilcQlEWZcZEfQYogcvwQjuJpBZl5remk2T/qK7R1yG/S1e9Sfpr0nPZEFjwIKGp6vfqup+u7RS+JBlv3rJxr37Dza3ttOHjx4/2env7k28qR2nMTfSuPMSPUmhaRxEkHRuHaEqJZ2VV+/a87Ov5Lww+nNYWpoqXGhRCY4hpmb9/argwvGiVGzypbCCHfrns/4gG2VdsLuQr2EA6zid7fZeFnPDa0U6cIneX+SZDdMGXRBc0iotak8W+RUu6CKiRkV+2nS3X7GDmJmzyri4dGBd9u+KBpVXGC6HLEIr8R35pSqHrFTdxlgdG7Wq27NC9XbaCG3rQJrfjKpqyYJhrRlsLhzxIJcsLd5TvLmjk9jioyWHwbgXTYFuofDbKr5kwYoha/l/UqH/SCOnB3ECcieiDYxfokMe4l+k0eD8XzvvwuTVKM9G+afXg6PjtdVb8BSewSHk8AaO4AOcwhg4LOE7/ICfvd8JJJvJ9o006a1r9uFWJHvXsIaxXA==</latexit><latexit sha1_base64="Joam1jC8wPven1HPPmWdLHyoGTA=">AAACWHicfZFNaxRBEIZrJ2qS8SOb5OilcQlEWZcZEfQYogcvwQjuJpBZl5remk2T/qK7R1yG/S1e9Sfpr0nPZEFjwIKGp6vfqup+u7RS+JBlv3rJxr37Dza3ttOHjx4/2env7k28qR2nMTfSuPMSPUmhaRxEkHRuHaEqJZ2VV+/a87Ov5Lww+nNYWpoqXGhRCY4hpmb9/argwvGiVGzypbCCHfrns/4gG2VdsLuQr2EA6zid7fZeFnPDa0U6cIneX+SZDdMGXRBc0iotak8W+RUu6CKiRkV+2nS3X7GDmJmzyri4dGBd9u+KBpVXGC6HLEIr8R35pSqHrFTdxlgdG7Wq27NC9XbaCG3rQJrfjKpqyYJhrRlsLhzxIJcsLd5TvLmjk9jioyWHwbgXTYFuofDbKr5kwYoha/l/UqH/SCOnB3ECcieiDYxfokMe4l+k0eD8XzvvwuTVKM9G+afXg6PjtdVb8BSewSHk8AaO4AOcwhg4LOE7/ICfvd8JJJvJ9o006a1r9uFWJHvXsIaxXA==</latexit><latexit sha1_base64="Joam1jC8wPven1HPPmWdLHyoGTA=">AAACWHicfZFNaxRBEIZrJ2qS8SOb5OilcQlEWZcZEfQYogcvwQjuJpBZl5remk2T/qK7R1yG/S1e9Sfpr0nPZEFjwIKGp6vfqup+u7RS+JBlv3rJxr37Dza3ttOHjx4/2env7k28qR2nMTfSuPMSPUmhaRxEkHRuHaEqJZ2VV+/a87Ov5Lww+nNYWpoqXGhRCY4hpmb9/argwvGiVGzypbCCHfrns/4gG2VdsLuQr2EA6zid7fZeFnPDa0U6cIneX+SZDdMGXRBc0iotak8W+RUu6CKiRkV+2nS3X7GDmJmzyri4dGBd9u+KBpVXGC6HLEIr8R35pSqHrFTdxlgdG7Wq27NC9XbaCG3rQJrfjKpqyYJhrRlsLhzxIJcsLd5TvLmjk9jioyWHwbgXTYFuofDbKr5kwYoha/l/UqH/SCOnB3ECcieiDYxfokMe4l+k0eD8XzvvwuTVKM9G+afXg6PjtdVb8BSewSHk8AaO4AOcwhg4LOE7/ICfvd8JJJvJ9o006a1r9uFWJHvXsIaxXA==</latexit><latexit sha1_base64="Joam1jC8wPven1HPPmWdLHyoGTA=">AAACWHicfZFNaxRBEIZrJ2qS8SOb5OilcQlEWZcZEfQYogcvwQjuJpBZl5remk2T/qK7R1yG/S1e9Sfpr0nPZEFjwIKGp6vfqup+u7RS+JBlv3rJxr37Dza3ttOHjx4/2env7k28qR2nMTfSuPMSPUmhaRxEkHRuHaEqJZ2VV+/a87Ov5Lww+nNYWpoqXGhRCY4hpmb9/argwvGiVGzypbCCHfrns/4gG2VdsLuQr2EA6zid7fZeFnPDa0U6cIneX+SZDdMGXRBc0iotak8W+RUu6CKiRkV+2nS3X7GDmJmzyri4dGBd9u+KBpVXGC6HLEIr8R35pSqHrFTdxlgdG7Wq27NC9XbaCG3rQJrfjKpqyYJhrRlsLhzxIJcsLd5TvLmjk9jioyWHwbgXTYFuofDbKr5kwYoha/l/UqH/SCOnB3ECcieiDYxfokMe4l+k0eD8XzvvwuTVKM9G+afXg6PjtdVb8BSewSHk8AaO4AOcwhg4LOE7/ICfvd8JJJvJ9o006a1r9uFWJHvXsIaxXA==</latexit><latexit sha1_base64="Joam1jC8wPven1HPPmWdLHyoGTA=">AAACWHicfZFNaxRBEIZrJ2qS8SOb5OilcQlEWZcZEfQYogcvwQjuJpBZl5remk2T/qK7R1yG/S1e9Sfpr0nPZEFjwIKGp6vfqup+u7RS+JBlv3rJxr37Dza3ttOHjx4/2env7k28qR2nMTfSuPMSPUmhaRxEkHRuHaEqJZ2VV+/a87Ov5Lww+nNYWpoqXGhRCY4hpmb9/argwvGiVGzypbCCHfrns/4gG2VdsLuQr2EA6zid7fZeFnPDa0U6cIneX+SZDdMGXRBc0iotak8W+RUu6CKiRkV+2nS3X7GDmJmzyri4dGBd9u+KBpVXGC6HLEIr8R35pSqHrFTdxlgdG7Wq27NC9XbaCG3rQJrfjKpqyYJhrRlsLhzxIJcsLd5TvLmjk9jioyWHwbgXTYFuofDbKr5kwYoha/l/UqH/SCOnB3ECcieiDYxfokMe4l+k0eD8XzvvwuTVKM9G+afXg6PjtdVb8BSewSHk8AaO4AOcwhg4LOE7/ICfvd8JJJvJ9o006a1r9uFWJHvXsIaxXA==</latexit><latexit sha1_base64="Joam1jC8wPven1HPPmWdLHyoGTA=">AAACWHicfZFNaxRBEIZrJ2qS8SOb5OilcQlEWZcZEfQYogcvwQjuJpBZl5remk2T/qK7R1yG/S1e9Sfpr0nPZEFjwIKGp6vfqup+u7RS+JBlv3rJxr37Dza3ttOHjx4/2env7k28qR2nMTfSuPMSPUmhaRxEkHRuHaEqJZ2VV+/a87Ov5Lww+nNYWpoqXGhRCY4hpmb9/argwvGiVGzypbCCHfrns/4gG2VdsLuQr2EA6zid7fZeFnPDa0U6cIneX+SZDdMGXRBc0iotak8W+RUu6CKiRkV+2nS3X7GDmJmzyri4dGBd9u+KBpVXGC6HLEIr8R35pSqHrFTdxlgdG7Wq27NC9XbaCG3rQJrfjKpqyYJhrRlsLhzxIJcsLd5TvLmjk9jioyWHwbgXTYFuofDbKr5kwYoha/l/UqH/SCOnB3ECcieiDYxfokMe4l+k0eD8XzvvwuTVKM9G+afXg6PjtdVb8BSewSHk8AaO4AOcwhg4LOE7/ICfvd8JJJvJ9o006a1r9uFWJHvXsIaxXA==</latexit>

f(r̂⌧ )<latexit sha1_base64="gk3DKCmbQm9+A+eLCHa2CKYysD8=">AAACWnicfZFNaxRBEIZ7x6jJxI+NMadcGpdAlHWZkYAeg3rwIongJoHMstT01uw26Y+huya4DPNjvJpfJPhj7JksJDFgQcPT1W9Vdb+dl0p6SpLfvejB2sNHj9c34s0nT58972+9OPG2cgLHwirrznLwqKTBMUlSeFY6BJ0rPM0vPrXnp5fovLTmOy1LnGiYG1lIARRS0/5OsZ8tgOos19w10zojqJrX0/4gGSVd8PuQrmDAVnE83eq9zWZWVBoNCQXen6dJSZMaHEmhsImzymMJ4gLmeB7QgEY/qbv7N3wvZGa8sC4sQ7zL3q6oQXsNtBjyAK3Ed+SXOh/yXHcbW5rQqFXdnUXFh0ktTVkRGnE9qqgUJ8tbO/hMOhSkljzOPmO4ucOvocVRiQ7Iujd1Bm6u4UcTXjLn2ZC3/D+pNDfSwPFemADCyWADFwtwICj8RhwMTv+18z6cvBulySj9djA4/Liyep3tsldsn6XsPTtkX9gxGzPBavaT/WJXvT9RFG1Em9fSqLeq2WZ3Inr5F+hUsvc=</latexit><latexit sha1_base64="gk3DKCmbQm9+A+eLCHa2CKYysD8=">AAACWnicfZFNaxRBEIZ7x6jJxI+NMadcGpdAlHWZkYAeg3rwIongJoHMstT01uw26Y+huya4DPNjvJpfJPhj7JksJDFgQcPT1W9Vdb+dl0p6SpLfvejB2sNHj9c34s0nT58972+9OPG2cgLHwirrznLwqKTBMUlSeFY6BJ0rPM0vPrXnp5fovLTmOy1LnGiYG1lIARRS0/5OsZ8tgOos19w10zojqJrX0/4gGSVd8PuQrmDAVnE83eq9zWZWVBoNCQXen6dJSZMaHEmhsImzymMJ4gLmeB7QgEY/qbv7N3wvZGa8sC4sQ7zL3q6oQXsNtBjyAK3Ed+SXOh/yXHcbW5rQqFXdnUXFh0ktTVkRGnE9qqgUJ8tbO/hMOhSkljzOPmO4ucOvocVRiQ7Iujd1Bm6u4UcTXjLn2ZC3/D+pNDfSwPFemADCyWADFwtwICj8RhwMTv+18z6cvBulySj9djA4/Liyep3tsldsn6XsPTtkX9gxGzPBavaT/WJXvT9RFG1Em9fSqLeq2WZ3Inr5F+hUsvc=</latexit><latexit sha1_base64="gk3DKCmbQm9+A+eLCHa2CKYysD8=">AAACWnicfZFNaxRBEIZ7x6jJxI+NMadcGpdAlHWZkYAeg3rwIongJoHMstT01uw26Y+huya4DPNjvJpfJPhj7JksJDFgQcPT1W9Vdb+dl0p6SpLfvejB2sNHj9c34s0nT58972+9OPG2cgLHwirrznLwqKTBMUlSeFY6BJ0rPM0vPrXnp5fovLTmOy1LnGiYG1lIARRS0/5OsZ8tgOos19w10zojqJrX0/4gGSVd8PuQrmDAVnE83eq9zWZWVBoNCQXen6dJSZMaHEmhsImzymMJ4gLmeB7QgEY/qbv7N3wvZGa8sC4sQ7zL3q6oQXsNtBjyAK3Ed+SXOh/yXHcbW5rQqFXdnUXFh0ktTVkRGnE9qqgUJ8tbO/hMOhSkljzOPmO4ucOvocVRiQ7Iujd1Bm6u4UcTXjLn2ZC3/D+pNDfSwPFemADCyWADFwtwICj8RhwMTv+18z6cvBulySj9djA4/Liyep3tsldsn6XsPTtkX9gxGzPBavaT/WJXvT9RFG1Em9fSqLeq2WZ3Inr5F+hUsvc=</latexit><latexit sha1_base64="gk3DKCmbQm9+A+eLCHa2CKYysD8=">AAACWnicfZFNaxRBEIZ7x6jJxI+NMadcGpdAlHWZkYAeg3rwIongJoHMstT01uw26Y+huya4DPNjvJpfJPhj7JksJDFgQcPT1W9Vdb+dl0p6SpLfvejB2sNHj9c34s0nT58972+9OPG2cgLHwirrznLwqKTBMUlSeFY6BJ0rPM0vPrXnp5fovLTmOy1LnGiYG1lIARRS0/5OsZ8tgOos19w10zojqJrX0/4gGSVd8PuQrmDAVnE83eq9zWZWVBoNCQXen6dJSZMaHEmhsImzymMJ4gLmeB7QgEY/qbv7N3wvZGa8sC4sQ7zL3q6oQXsNtBjyAK3Ed+SXOh/yXHcbW5rQqFXdnUXFh0ktTVkRGnE9qqgUJ8tbO/hMOhSkljzOPmO4ucOvocVRiQ7Iujd1Bm6u4UcTXjLn2ZC3/D+pNDfSwPFemADCyWADFwtwICj8RhwMTv+18z6cvBulySj9djA4/Liyep3tsldsn6XsPTtkX9gxGzPBavaT/WJXvT9RFG1Em9fSqLeq2WZ3Inr5F+hUsvc=</latexit>

orPolicy's utility under f

<latexit sha1_base64="YRBvEFd0dLSOpctC2IWIXNVXms4=">AAACRHicfZDLShxBFIarjRrteI1LN4XDgMg4dIsQlxJdZBNUcJyB6UFO15weC+vSVFVLhmaewG3ySr5D3sFdcCtWtwPeIAcKvjr1n0v9aS64dVH0N5j5NDs3/3lhMfyytLyyurb+9cLqwjDsMC206aVgUXCFHcedwF5uEGQqsJteH1Xv3Rs0lmt17sY5DiSMFM84A+dTZ9nlWiNqR3XQjxBPoUGmcXq5HuwmQ80KicoxAdb24yh3gxKM40zgJEwKizmwaxhh36MCiXZQ1ptOaNNnhjTTxh/laJ19XVGCtBLcVYt6qCS2JjuWaYumsr7oXPlGlertLJcdDEqu8sKhYs+jskJQp2n1cTrkBpkTYxomx+g3N/jTtzjJ0YDTZqdMwIwk/Jr4n4xo0qIV/0/K1YvUc9j0E4AZ7m2g7AoMMOd9D73B8Xs7P8LFXjuO2vHZfuPw+9TqBbJJtsg2ick3ckh+kFPSIYwguSW/yZ/gLrgP/gUPz9KZYFqzQd5E8PgET7SuXg==</latexit><latexit sha1_base64="YRBvEFd0dLSOpctC2IWIXNVXms4=">AAACRHicfZDLShxBFIarjRrteI1LN4XDgMg4dIsQlxJdZBNUcJyB6UFO15weC+vSVFVLhmaewG3ySr5D3sFdcCtWtwPeIAcKvjr1n0v9aS64dVH0N5j5NDs3/3lhMfyytLyyurb+9cLqwjDsMC206aVgUXCFHcedwF5uEGQqsJteH1Xv3Rs0lmt17sY5DiSMFM84A+dTZ9nlWiNqR3XQjxBPoUGmcXq5HuwmQ80KicoxAdb24yh3gxKM40zgJEwKizmwaxhh36MCiXZQ1ptOaNNnhjTTxh/laJ19XVGCtBLcVYt6qCS2JjuWaYumsr7oXPlGlertLJcdDEqu8sKhYs+jskJQp2n1cTrkBpkTYxomx+g3N/jTtzjJ0YDTZqdMwIwk/Jr4n4xo0qIV/0/K1YvUc9j0E4AZ7m2g7AoMMOd9D73B8Xs7P8LFXjuO2vHZfuPw+9TqBbJJtsg2ick3ckh+kFPSIYwguSW/yZ/gLrgP/gUPz9KZYFqzQd5E8PgET7SuXg==</latexit><latexit sha1_base64="YRBvEFd0dLSOpctC2IWIXNVXms4=">AAACRHicfZDLShxBFIarjRrteI1LN4XDgMg4dIsQlxJdZBNUcJyB6UFO15weC+vSVFVLhmaewG3ySr5D3sFdcCtWtwPeIAcKvjr1n0v9aS64dVH0N5j5NDs3/3lhMfyytLyyurb+9cLqwjDsMC206aVgUXCFHcedwF5uEGQqsJteH1Xv3Rs0lmt17sY5DiSMFM84A+dTZ9nlWiNqR3XQjxBPoUGmcXq5HuwmQ80KicoxAdb24yh3gxKM40zgJEwKizmwaxhh36MCiXZQ1ptOaNNnhjTTxh/laJ19XVGCtBLcVYt6qCS2JjuWaYumsr7oXPlGlertLJcdDEqu8sKhYs+jskJQp2n1cTrkBpkTYxomx+g3N/jTtzjJ0YDTZqdMwIwk/Jr4n4xo0qIV/0/K1YvUc9j0E4AZ7m2g7AoMMOd9D73B8Xs7P8LFXjuO2vHZfuPw+9TqBbJJtsg2ick3ckh+kFPSIYwguSW/yZ/gLrgP/gUPz9KZYFqzQd5E8PgET7SuXg==</latexit><latexit sha1_base64="YRBvEFd0dLSOpctC2IWIXNVXms4=">AAACRHicfZDLShxBFIarjRrteI1LN4XDgMg4dIsQlxJdZBNUcJyB6UFO15weC+vSVFVLhmaewG3ySr5D3sFdcCtWtwPeIAcKvjr1n0v9aS64dVH0N5j5NDs3/3lhMfyytLyyurb+9cLqwjDsMC206aVgUXCFHcedwF5uEGQqsJteH1Xv3Rs0lmt17sY5DiSMFM84A+dTZ9nlWiNqR3XQjxBPoUGmcXq5HuwmQ80KicoxAdb24yh3gxKM40zgJEwKizmwaxhh36MCiXZQ1ptOaNNnhjTTxh/laJ19XVGCtBLcVYt6qCS2JjuWaYumsr7oXPlGlertLJcdDEqu8sKhYs+jskJQp2n1cTrkBpkTYxomx+g3N/jTtzjJ0YDTZqdMwIwk/Jr4n4xo0qIV/0/K1YvUc9j0E4AZ7m2g7AoMMOd9D73B8Xs7P8LFXjuO2vHZfuPw+9TqBbJJtsg2ick3ckh+kFPSIYwguSW/yZ/gLrgP/gUPz9KZYFqzQd5E8PgET7SuXg==</latexit>

f!(r) = !|r<latexit sha1_base64="PbJfbRKiac9yxbS4aVzyuO7ikI4=">AAACdXicfZDfShwxFMaz0z/q2NbVXkohdGuxZV1nRGhvCtJ60ZtSBVcFZ7ucyZ4Zg/kzJJnSZZjX6NN4a9/BJ/HWzOxCawUPBH758p2c5EsLwa2LoutO8Ojxk6cLi0vh8rPnL1a6q2vHVpeG4ZBpoc1pChYFVzh03Ak8LQyCTAWepBdfmvOTn2gs1+rITQscScgVzzgD56VxN8rGVaIl5lBvJqmk5h39RBuYiT+qhCuHhoGoW9mMu71oELVF70M8hx6Z18F4tbOVTDQrJSrHBFh7FkeFG1VgHGcC6zApLRbALiDHM48KJNpR1X6tphtemdBMG7+Uo636b0cF0kpw533qobHYluxUpn2aynajC+Uvalx3Z7ns46jiqigdKjYblZWCOk2bpOiEG2ROTGmY7KN/ucFv/orvBRpw2ryvEjC5hF+1/0lOkz5t+CErV3+tnsMNPwGY4T4Gys7BAPNB29AHHP8f53043hnE0SA+3O3tfZ5HvUjWyWuySWLygeyRr+SADAkjv8kluSJ/OjfBq+BN8HZmDTrznpfkTgXbt3bVvUE=</latexit><latexit sha1_base64="PbJfbRKiac9yxbS4aVzyuO7ikI4=">AAACdXicfZDfShwxFMaz0z/q2NbVXkohdGuxZV1nRGhvCtJ60ZtSBVcFZ7ucyZ4Zg/kzJJnSZZjX6NN4a9/BJ/HWzOxCawUPBH758p2c5EsLwa2LoutO8Ojxk6cLi0vh8rPnL1a6q2vHVpeG4ZBpoc1pChYFVzh03Ak8LQyCTAWepBdfmvOTn2gs1+rITQscScgVzzgD56VxN8rGVaIl5lBvJqmk5h39RBuYiT+qhCuHhoGoW9mMu71oELVF70M8hx6Z18F4tbOVTDQrJSrHBFh7FkeFG1VgHGcC6zApLRbALiDHM48KJNpR1X6tphtemdBMG7+Uo636b0cF0kpw533qobHYluxUpn2aynajC+Uvalx3Z7ns46jiqigdKjYblZWCOk2bpOiEG2ROTGmY7KN/ucFv/orvBRpw2ryvEjC5hF+1/0lOkz5t+CErV3+tnsMNPwGY4T4Gys7BAPNB29AHHP8f53043hnE0SA+3O3tfZ5HvUjWyWuySWLygeyRr+SADAkjv8kluSJ/OjfBq+BN8HZmDTrznpfkTgXbt3bVvUE=</latexit><latexit sha1_base64="PbJfbRKiac9yxbS4aVzyuO7ikI4=">AAACdXicfZDfShwxFMaz0z/q2NbVXkohdGuxZV1nRGhvCtJ60ZtSBVcFZ7ucyZ4Zg/kzJJnSZZjX6NN4a9/BJ/HWzOxCawUPBH758p2c5EsLwa2LoutO8Ojxk6cLi0vh8rPnL1a6q2vHVpeG4ZBpoc1pChYFVzh03Ak8LQyCTAWepBdfmvOTn2gs1+rITQscScgVzzgD56VxN8rGVaIl5lBvJqmk5h39RBuYiT+qhCuHhoGoW9mMu71oELVF70M8hx6Z18F4tbOVTDQrJSrHBFh7FkeFG1VgHGcC6zApLRbALiDHM48KJNpR1X6tphtemdBMG7+Uo636b0cF0kpw533qobHYluxUpn2aynajC+Uvalx3Z7ns46jiqigdKjYblZWCOk2bpOiEG2ROTGmY7KN/ucFv/orvBRpw2ryvEjC5hF+1/0lOkz5t+CErV3+tnsMNPwGY4T4Gys7BAPNB29AHHP8f53043hnE0SA+3O3tfZ5HvUjWyWuySWLygeyRr+SADAkjv8kluSJ/OjfBq+BN8HZmDTrznpfkTgXbt3bVvUE=</latexit><latexit sha1_base64="PbJfbRKiac9yxbS4aVzyuO7ikI4=">AAACdXicfZDfShwxFMaz0z/q2NbVXkohdGuxZV1nRGhvCtJ60ZtSBVcFZ7ucyZ4Zg/kzJJnSZZjX6NN4a9/BJ/HWzOxCawUPBH758p2c5EsLwa2LoutO8Ojxk6cLi0vh8rPnL1a6q2vHVpeG4ZBpoc1pChYFVzh03Ak8LQyCTAWepBdfmvOTn2gs1+rITQscScgVzzgD56VxN8rGVaIl5lBvJqmk5h39RBuYiT+qhCuHhoGoW9mMu71oELVF70M8hx6Z18F4tbOVTDQrJSrHBFh7FkeFG1VgHGcC6zApLRbALiDHM48KJNpR1X6tphtemdBMG7+Uo636b0cF0kpw533qobHYluxUpn2aynajC+Uvalx3Z7ns46jiqigdKjYblZWCOk2bpOiEG2ROTGmY7KN/ucFv/orvBRpw2ryvEjC5hF+1/0lOkz5t+CErV3+tnsMNPwGY4T4Gys7BAPNB29AHHP8f53043hnE0SA+3O3tfZ5HvUjWyWuySWLygeyRr+SADAkjv8kluSJ/OjfBq+BN8HZmDTrznpfkTgXbt3bVvUE=</latexit>

Linear preferences

Relative importances / Weights

Optimality Concept under Linear Preferences

AB

C

D

E

F

G

H

K

L


QU

ANTI

TY O

F O

BJEC

TIVE

2 CCS

Non-optimal

Pareto Frontier

F


QU

ANTI

TY O

F O

BJEC

TIVE

2 F Non-preferred

Preference




a. b.

DUtility Projection


QU

ANTI

TY O

F O

BJEC

TIVE

2

D

F




D F

Optimal Solutions

Sampled Preferences

c.

CCS := {r̂ 2 F⇤ | 9! 2 ⌦ s.t. !|r̂ � !|r̂0, 8r̂0 2 F⇤}

f!(r) = !|r<latexit sha1_base64="PbJfbRKiac9yxbS4aVzyuO7ikI4=">AAACdXicfZDfShwxFMaz0z/q2NbVXkohdGuxZV1nRGhvCtJ60ZtSBVcFZ7ucyZ4Zg/kzJJnSZZjX6NN4a9/BJ/HWzOxCawUPBH758p2c5EsLwa2LoutO8Ojxk6cLi0vh8rPnL1a6q2vHVpeG4ZBpoc1pChYFVzh03Ak8LQyCTAWepBdfmvOTn2gs1+rITQscScgVzzgD56VxN8rGVaIl5lBvJqmk5h39RBuYiT+qhCuHhoGoW9mMu71oELVF70M8hx6Z18F4tbOVTDQrJSrHBFh7FkeFG1VgHGcC6zApLRbALiDHM48KJNpR1X6tphtemdBMG7+Uo636b0cF0kpw533qobHYluxUpn2aynajC+Uvalx3Z7ns46jiqigdKjYblZWCOk2bpOiEG2ROTGmY7KN/ucFv/orvBRpw2ryvEjC5hF+1/0lOkz5t+CErV3+tnsMNPwGY4T4Gys7BAPNB29AHHP8f53043hnE0SA+3O3tfZ5HvUjWyWuySWLygeyRr+SADAkjv8kluSJ/OjfBq+BN8HZmDTrznpfkTgXbt3bVvUE=</latexit><latexit sha1_base64="PbJfbRKiac9yxbS4aVzyuO7ikI4=">AAACdXicfZDfShwxFMaz0z/q2NbVXkohdGuxZV1nRGhvCtJ60ZtSBVcFZ7ucyZ4Zg/kzJJnSZZjX6NN4a9/BJ/HWzOxCawUPBH758p2c5EsLwa2LoutO8Ojxk6cLi0vh8rPnL1a6q2vHVpeG4ZBpoc1pChYFVzh03Ak8LQyCTAWepBdfmvOTn2gs1+rITQscScgVzzgD56VxN8rGVaIl5lBvJqmk5h39RBuYiT+qhCuHhoGoW9mMu71oELVF70M8hx6Z18F4tbOVTDQrJSrHBFh7FkeFG1VgHGcC6zApLRbALiDHM48KJNpR1X6tphtemdBMG7+Uo636b0cF0kpw533qobHYluxUpn2aynajC+Uvalx3Z7ns46jiqigdKjYblZWCOk2bpOiEG2ROTGmY7KN/ucFv/orvBRpw2ryvEjC5hF+1/0lOkz5t+CErV3+tnsMNPwGY4T4Gys7BAPNB29AHHP8f53043hnE0SA+3O3tfZ5HvUjWyWuySWLygeyRr+SADAkjv8kluSJ/OjfBq+BN8HZmDTrznpfkTgXbt3bVvUE=</latexit><latexit sha1_base64="PbJfbRKiac9yxbS4aVzyuO7ikI4=">AAACdXicfZDfShwxFMaz0z/q2NbVXkohdGuxZV1nRGhvCtJ60ZtSBVcFZ7ucyZ4Zg/kzJJnSZZjX6NN4a9/BJ/HWzOxCawUPBH758p2c5EsLwa2LoutO8Ojxk6cLi0vh8rPnL1a6q2vHVpeG4ZBpoc1pChYFVzh03Ak8LQyCTAWepBdfmvOTn2gs1+rITQscScgVzzgD56VxN8rGVaIl5lBvJqmk5h39RBuYiT+qhCuHhoGoW9mMu71oELVF70M8hx6Z18F4tbOVTDQrJSrHBFh7FkeFG1VgHGcC6zApLRbALiDHM48KJNpR1X6tphtemdBMG7+Uo636b0cF0kpw533qobHYluxUpn2aynajC+Uvalx3Z7ns46jiqigdKjYblZWCOk2bpOiEG2ROTGmY7KN/ucFv/orvBRpw2ryvEjC5hF+1/0lOkz5t+CErV3+tnsMNPwGY4T4Gys7BAPNB29AHHP8f53043hnE0SA+3O3tfZ5HvUjWyWuySWLygeyRr+SADAkjv8kluSJ/OjfBq+BN8HZmDTrznpfkTgXbt3bVvUE=</latexit><latexit sha1_base64="PbJfbRKiac9yxbS4aVzyuO7ikI4=">AAACdXicfZDfShwxFMaz0z/q2NbVXkohdGuxZV1nRGhvCtJ60ZtSBVcFZ7ucyZ4Zg/kzJJnSZZjX6NN4a9/BJ/HWzOxCawUPBH758p2c5EsLwa2LoutO8Ojxk6cLi0vh8rPnL1a6q2vHVpeG4ZBpoc1pChYFVzh03Ak8LQyCTAWepBdfmvOTn2gs1+rITQscScgVzzgD56VxN8rGVaIl5lBvJqmk5h39RBuYiT+qhCuHhoGoW9mMu71oELVF70M8hx6Z18F4tbOVTDQrJSrHBFh7FkeFG1VgHGcC6zApLRbALiDHM48KJNpR1X6tphtemdBMG7+Uo636b0cF0kpw533qobHYluxUpn2aynajC+Uvalx3Z7ns46jiqigdKjYblZWCOk2bpOiEG2ROTGmY7KN/ucFv/orvBRpw2ryvEjC5hF+1/0lOkz5t+CErV3+tnsMNPwGY4T4Gys7BAPNB29AHHP8f53043hnE0SA+3O3tfZ5HvUjWyWuySWLygeyRr+SADAkjv8kluSJ/OjfBq+BN8HZmDTrznpfkTgXbt3bVvUE=</latexit>

Linear preferences:

Convex Coverage Set:

Challenges of Learning all Optimal Policies

Single Policy

Roijers et al. (2013)

Multiple Policies

• Known preference.

• Convert the multi-objective problem into a single-objective one through various techniques.

• Not adaptable — learn an “average” policy over the space of preferences and cannot be tailored to be optimal for specific preferences.

• Unknown preference.

• Compute a set of optimal policies that encompass the entire space of possible preferences.

• Lack of scalability – represent a Pareto front (or its CCS) by learning several individual policies, which grow with the domain size.

“Our goal: to learn a single model to produce all optimal policies under different preference

condition under the linear preference scenario.”

Two Phases: learning and adaptation

Learning Phase To learn all optimal policies corresponding to the entire convex coverage set, without being given any specific preference (assume abundant computational resources).

Adaptation Phase To (i) perform the optimal policy according to a given preference, or (ii) infer the underlying preference of a new task in few-shot, and perform the optimal policy (assume limited computational resources).

⇡ 2 ⇧L ) 9 ! 2 ⌦, s.t. 8⇡0 2 ⇧,!|v⇡(s0) � !|v⇡0(s0)

<latexit sha1_base64="CnPQ18DLqaJp+3eKBJX25OXn4ZE=">AAADBnicjVHLbhMxFPUMrxIeTWHJ5oqoSkEhmkFI7bICFiyoGhBJKtUh8jg3E6v2eLCdkmg0WfM17BBb+Az+Bs8kAkolxJUsHR+fc699nORSWBdFP4LwytVr129s3Wzcun3n7nZz597A6rnh2OdaanOSMItSZNh3wkk8yQ0ylUgcJmcvqvPhORordPbOLXMcKZZmYio4c54aN7/TXAAVGdCeGBdUMTfjTMLrEuhbkc4cM0Z/BIoLfxULK5oooFphymrPcYU6QFWiF4Xtum65WgGdasOk9I3bm8Ze8cv3vvCkQ+OnlOCrOjn3ZC7KPTuOHgFN8cN/6dtrw7jZirpRXXAZxBvQIpvqjXeCJ3Si+Vxh5rhk1p7GUe5GBTNOcIllg84t5oyfsRRPPcyYQjsq6qxL2PXMBPwT/coc1OyfjoIpW6XYAQ8qia2RXaqkA4mqNzrPfKNKdXGWmx6MCpHlc4cZX4+aziU4DdXXwUQY5E4uoUFfor+5wSPf4jhHw5w2jwvKTKrYovQvSYF2oML/korst9Tjxq6fwLgRPgbgM2YY97nbhg84/jvOy2DwtBtH3fjNs9bh803UW+QBeUj2SEz2ySF5RXqkT3jQDo6CQTAMP4Wfwy/h17U0DDae++RChd9+Al0g8VY=</latexit><latexit sha1_base64="CnPQ18DLqaJp+3eKBJX25OXn4ZE=">AAADBnicjVHLbhMxFPUMrxIeTWHJ5oqoSkEhmkFI7bICFiyoGhBJKtUh8jg3E6v2eLCdkmg0WfM17BBb+Az+Bs8kAkolxJUsHR+fc699nORSWBdFP4LwytVr129s3Wzcun3n7nZz597A6rnh2OdaanOSMItSZNh3wkk8yQ0ylUgcJmcvqvPhORordPbOLXMcKZZmYio4c54aN7/TXAAVGdCeGBdUMTfjTMLrEuhbkc4cM0Z/BIoLfxULK5oooFphymrPcYU6QFWiF4Xtum65WgGdasOk9I3bm8Ze8cv3vvCkQ+OnlOCrOjn3ZC7KPTuOHgFN8cN/6dtrw7jZirpRXXAZxBvQIpvqjXeCJ3Si+Vxh5rhk1p7GUe5GBTNOcIllg84t5oyfsRRPPcyYQjsq6qxL2PXMBPwT/coc1OyfjoIpW6XYAQ8qia2RXaqkA4mqNzrPfKNKdXGWmx6MCpHlc4cZX4+aziU4DdXXwUQY5E4uoUFfor+5wSPf4jhHw5w2jwvKTKrYovQvSYF2oML/korst9Tjxq6fwLgRPgbgM2YY97nbhg84/jvOy2DwtBtH3fjNs9bh803UW+QBeUj2SEz2ySF5RXqkT3jQDo6CQTAMP4Wfwy/h17U0DDae++RChd9+Al0g8VY=</latexit><latexit sha1_base64="CnPQ18DLqaJp+3eKBJX25OXn4ZE=">AAADBnicjVHLbhMxFPUMrxIeTWHJ5oqoSkEhmkFI7bICFiyoGhBJKtUh8jg3E6v2eLCdkmg0WfM17BBb+Az+Bs8kAkolxJUsHR+fc699nORSWBdFP4LwytVr129s3Wzcun3n7nZz597A6rnh2OdaanOSMItSZNh3wkk8yQ0ylUgcJmcvqvPhORordPbOLXMcKZZmYio4c54aN7/TXAAVGdCeGBdUMTfjTMLrEuhbkc4cM0Z/BIoLfxULK5oooFphymrPcYU6QFWiF4Xtum65WgGdasOk9I3bm8Ze8cv3vvCkQ+OnlOCrOjn3ZC7KPTuOHgFN8cN/6dtrw7jZirpRXXAZxBvQIpvqjXeCJ3Si+Vxh5rhk1p7GUe5GBTNOcIllg84t5oyfsRRPPcyYQjsq6qxL2PXMBPwT/coc1OyfjoIpW6XYAQ8qia2RXaqkA4mqNzrPfKNKdXGWmx6MCpHlc4cZX4+aziU4DdXXwUQY5E4uoUFfor+5wSPf4jhHw5w2jwvKTKrYovQvSYF2oML/korst9Tjxq6fwLgRPgbgM2YY97nbhg84/jvOy2DwtBtH3fjNs9bh803UW+QBeUj2SEz2ySF5RXqkT3jQDo6CQTAMP4Wfwy/h17U0DDae++RChd9+Al0g8VY=</latexit><latexit sha1_base64="CnPQ18DLqaJp+3eKBJX25OXn4ZE=">AAADBnicjVHLbhMxFPUMrxIeTWHJ5oqoSkEhmkFI7bICFiyoGhBJKtUh8jg3E6v2eLCdkmg0WfM17BBb+Az+Bs8kAkolxJUsHR+fc699nORSWBdFP4LwytVr129s3Wzcun3n7nZz597A6rnh2OdaanOSMItSZNh3wkk8yQ0ylUgcJmcvqvPhORordPbOLXMcKZZmYio4c54aN7/TXAAVGdCeGBdUMTfjTMLrEuhbkc4cM0Z/BIoLfxULK5oooFphymrPcYU6QFWiF4Xtum65WgGdasOk9I3bm8Ze8cv3vvCkQ+OnlOCrOjn3ZC7KPTuOHgFN8cN/6dtrw7jZirpRXXAZxBvQIpvqjXeCJ3Si+Vxh5rhk1p7GUe5GBTNOcIllg84t5oyfsRRPPcyYQjsq6qxL2PXMBPwT/coc1OyfjoIpW6XYAQ8qia2RXaqkA4mqNzrPfKNKdXGWmx6MCpHlc4cZX4+aziU4DdXXwUQY5E4uoUFfor+5wSPf4jhHw5w2jwvKTKrYovQvSYF2oML/korst9Tjxq6fwLgRPgbgM2YY97nbhg84/jvOy2DwtBtH3fjNs9bh803UW+QBeUj2SEz2ySF5RXqkT3jQDo6CQTAMP4Wfwy/h17U0DDae++RChd9+Al0g8VY=</latexit>

Linear Preference Scenario: RL-based dialogue systems

(weather)

0.70.3

(driving)

0.50.5

successbrevity

successbrevity



success success

brev

ity

brev

ity

Adaptation Phase: choose the best dialogue policy according to a given preference.

x

x

x

x

Learning Phase: train a bot without specific preferences.

“Learning Phase Algorithms: How to efficiently learn all optimal policies with a single model?”

General Framework: Banach’s fixed point theorem

Contraction Mapping in a Metric Space

T<latexit sha1_base64="3uHIbaQHjXM//QrN9dNTqNvIZ5Q=">AAADVnicfZJda9swFIZVZ1077yvdLncjFgpjhGB3n5dlH7CbsQyStqwORVZOXFFZEpIcEoT/xW6337X9mTHJMWxZ8Q748FrnOXolcXLFmbFJ8nMn6t3Yvbm3fyu+fefuvfv9gwcnRlaawpRKLvVZTgxwJmBqmeVwpjSQMudwml+9DfXTJWjDpJjYtYJZSQrBFowS65e+ZCWxl5RwPLnoD5JR0gS+LtJWDFAb44uD6GU2l7QqQVjKiTHnaaLszBFtGeVQx1llQBF6RQo491KQEszMNUeu8aFfmeOF1P4TFjerf3c4UppwtiH2IiCmUWZd5kOcl82PVMJvFKitztXGYtvfLl7PHBOqsiDoxn5RcWwlDq+C50wDtXyN4+wd+Nto+Oi3/aRAEyv1U5cRXZRkVfvbFTgb4qD/hzLxB/W6C10SXbuQugAql7ULqQswhXcKqQsA7S1C6gKsr0+6y8qXmymx1o113e2zUtCSee7eezA+9M9KqGZ+HjC9JJpQ6ycx9pOW/jtX18XJ0Sh9Nnrx+fng+E07c/voEXqMnqAUvULH6AMaoymiSKCv6Bv6Hv2IfvV2e3sbNNppex6irej1fwO8HhdK</latexit>

T (x)<latexit sha1_base64="WYUtXN0DbYcIEAExglszjCS3QO4=">AAADWXicfZJda9swFIaVeB+Z99Wul7sRC4VuhGB367bLsg/YzVgGSVuoTZGVE1dUllRJDgnGv2O3288a+zOTHMOWFe+AD691nqNXEidTnBkbRT97/eDW7Tt3B/fC+w8ePnq8s/vkxMhSU5hRyaU+y4gBzgTMLLMczpQGUmQcTrOr975+ugRtmBRTu1aQFiQXbMEosW4pTQpiLynheHqwen6xM4zGURP4pohbMURtTC52+6+TuaRlAcJSTow5jyNl04poyyiHOkxKA4rQK5LDuZOCFGDSqjl1jffdyhwvpHafsLhZ/bujIoXxxxthJzxiGmXWRTbCWdH8SCXcRp7a6lxtLLb97eJtWjGhSguCbuwXJcdWYv8weM40UMvXOEw+gLuNhs9u2y8KNLFSv6gSovOCrGp3uxwnI+z1/1Am/qBOd6FLouvKpy6AymVd+dQFmNw5+dQFgHYWPnUB1tWn3WXlys2gWFtNdN3ts1LQkllWfXRguO+elVDN3Dxgekk0odYNY+gmLf53rm6Kk8Nx/HJ89PXV8PhdO3MD9BQ9QwcoRm/QMfqEJmiGKLpG39B39KP/K+gFgyDcoP1e27OHtiLY+w1fPxcy</latexit>

T (x0)<latexit sha1_base64="LqhKGG4/bklNlFLOoy5jUja0af0=">AAADWnicfZJdb9MwFIa9BNiW8bEN7rixqCYGqqqEweBy4kPiBlGkdpu0lMlxTzNrjm3ZTtUqyv/gFv4VEj8GO40EZQpHytEbn+f4ta2TKc6MjeOfG0F46/adza3taOfuvfsPdvf2T40sNYUxlVzq84wY4EzA2DLL4VxpIEXG4Sy7fufrZ3PQhkkxsksFk4Lkgs0YJdYtfU0LYq8o4Xh0uHj67HK3Fw/iJvBNkbSih9oYXu4Fx+lU0rIAYSknxlwksbKTimjLKIc6SksDitBrksOFk4IUYCZVc+waH7iVKZ5J7T5hcbP6d0dFCuPP18dOeMQ0yiyLrI+zovmRSriNPLXWuVhZrPvb2ZtJxYQqLQi6sp+VHFuJ/cvgKdNALV/iKH0P7jYaPrltPyvQxEr9vEqJzguyqN3tcpz2sdf/Q5n4gzrdhc6JriufugAq53XlUxdgcufkUxcA2ln41AVYVx91l5UrN5NibTXUdbfPQkFLZln1wYHRgXtWQjVz84DpFdGEWjeNkZu05N+5uilOXwySo8GrLy97J2/bmdtCj9ETdIgS9BqdoI9oiMaIIo2+oe/oR/ArDMLtcGeFBhttz0O0FuGj3/uQF2M=</latexit>

x<latexit sha1_base64="Rm+hr7LA0Iwpisrx+PSe4Hupq0I=">AAADTXicfZLfa9RAEMe3qdoaf/SHj74sHgWR40i02j6WquCLeIVeW2iOstmbS5cmu8vu5Lgj5C/wVf8un/1DfBNxNxfQs8SBDN+d+cxOdphU58JiFH1fC9bv3L23sXk/fPDw0eOt7Z3dM6tKw2HEVa7MRcos5ELCCAXmcKENsCLN4Ty9eevz5zMwVih5igsN44JlUkwFZ+hCJ/Or7V40iBqjt0Xcih5pbXi1E7xJJoqXBUjkObP2Mo40jitmUPAc6jApLWjGb1gGl05KVoAdV82f1nTPRSZ0qoz7JNIm+ndFxQpbMLzuUyc8YhtlF0Xap2nRHJSW7iJPrVTOly1W++P0cFwJqUsEyZftp2VOUVE/DDoRBjjmCxom78C9xsBHd+0nDYahMi+qhJmsYPPavS6jSZ96/T9UyD+o013ojJm68q4L4GpWV951ATZznbzrAsC4Ft51Aejyp91p7dKJnzFiNTR1d5+5hpZM0+q9A8M9N1bGjXD7QPk1M4yjW8DQbVr8717dFmcvB/GrweuT/d7Rcbtzm+QpeUaek5gckCPygQzJiHAC5DP5Qr4G34Ifwc/g1xIN1tqaJ2TF1jd+A0g1Fbw=</latexit>

x0<latexit sha1_base64="c+tfc8oa69+EC1TCxf0UeuVSh1o=">AAADTnicfZJLb9QwEMfdlMcSXi0cuVisKhBarRLex4qHxAWxoG5bqVlVjneSWk1sy3ZWu4ryDbjC5+LKF+GGYJyNBEsVRsro75nfeOLRpLoQ1kXR961g+9LlK1cH18LrN27eur2ze+fQqspwmHJVKHOcMguFkDB1whVwrA2wMi3gKD1/7fNHCzBWKHngVhpmJculyARnDkOflg9Od4bROGqNXhRxJ4aks8npbvA8mStelSAdL5i1J3Gk3axmxgleQBMmlQXN+DnL4QSlZCXYWd3+akP3MDKnmTL4SUfb6N8VNSttydzZiKLwiG2VXZXpiKZle1Ba4kWe2qhcrlts9nfZy1ktpK4cSL5un1UFdYr6adC5MMBdsaJh8gbwNQbe47UfNBjmlHlUJ8zkJVs2+LqcJiPq9f9QIf+gqPvQBTNN7V0fwNWiqb3rA2yOnbzrA8BgC+/6AIf5g/60xnTiZ+xcPTFNf5+lho5M0/otguEejpVxI3AfKD9jhnGHGxjipsX/7tVFcfh4HD8ZP/v4dLj/qtu5AblH7pOHJCYvyD55RyZkSjjJyGfyhXwNvgU/gp/BrzUabHU1d8mGbQ9+A+B1Fe0=</latexit>

8>>>>>

>>><>>>>

>>>>:

<latexit sha1_base64="HG5QBo4408rB2HIFaiwPD+jO/LI=">AAADeXicfVLbattAEN1YvaTqJU772JdtTSCkxlht6OUt9AJ9KXUhTgKRMav1WFki7S67K2Mj9BH9mr62n9Fv6UtnZUHrGnVYDUczZ+YwwyQ6E9YNhz93OsGNm7du794J7967/2Cvu//wzKrCcBhzlSlzkTALmZAwdsJlcKENsDzJ4Dy5fufz5wswVih56lYaJjlLpZgLzhyGpt1ncQKpkCXHHrYK43jrgZw12Wm3NxwMa6PbIGpAjzQ2mu53XsYzxYscpOMZs/YyGmo3KZlxgmeAcoUFzfg1S+ESoWQ52ElZT1XRA4zM6FwZ/KSjdfTvipLlNmfuqk8ReIqtkV3lSZ8mef2jtMRGnrVRuVxLbOq7+etJKaQuHEi+lp8XGXWK+sXRmTDAXbaiYfwecBoDn7DtZw2GOWWOypiZNGfLCqdLadynHv+PKuQfKuI26oKZqvSujcDVoiq9ayPYFJW8ayOAQQnv2ggO86ftaY3p2O/YuXJkqnadpYaGmSTlBySGB7hWxo3Ae6D8ihnGHR5riJcW/XtX2+Ds+SA6Hrz5ctw7edvc3C55TJ6SQxKRV+SEfCQjMiacfCXfyHfyo/MreBIcBkdramenqXlENix48RsntCK5</latexit>

8>>>>>><>>>>>>:

<latexit sha1_base64="jT2fIV260Lz2a0+oe+jd7YejiF4=">AAADdnicfVLbattAEN1YvaTuJU77WChLjWkoxkglJO1b6AX6UupCnAQiY1brsbJE2l12V8ZG6Bf6NX1t/6N/0sfOyoLWDeqwGo5mzsxhhkl0JqwLw587neDW7Tt3d+917z94+Givt//4zKrCcJhwlSlzkTALmZAwccJlcKENsDzJ4Dy5fufz50swVih56tYapjlLpVgIzhyGZr2DOIFUyJJjD1t143j7gZw3qVmvH47C2uhNEDWgTxobz/Y7R/Fc8SIH6XjGrL2MQu2mJTNO8AxQq7CgGb9mKVwilCwHOy3rkSo6wMicLpTBTzpaR/+uKFluc+auhhSBp9ga2XWeDGmS1z9KS2zkWVuVq43Etr5bvJ6WQurCgeQb+UWRUaeo3xqdCwPcZWvajd8DTmPgE7b9rMEwp8zLMmYmzdmqwulSGg+px/+jCvmHiriNumSmKr1rI3C1rErv2gg2RSXv2ghgUMK7NoLD/Gl7WmM69jt2rhybql1npaFhJkn5AYndAa6VcSPwHii/YoZxh5faxUuL/r2rm+Ds1Sg6HL35ctg/edvc3C55Sp6TAxKRY3JCPpIxmRBOvpJv5Dv50fkVPAsGwYsNtbPT1DwhWxaEvwFZryHZ</latexit>

d(T(x), T

(x 0))

<latexit sha1_base64="/nKd8St5bhlWU5J1Wa95k5vqtj0=">AAADbnicfVJda9swFFXjfXTeV9rBXsaYWChLRgj2KN32VvYBexnLIGkLdQiycuOKypKQ5JBg/Lpfs9ftv+xf7CdMcgJrVrwLvhzde66Orzip4szYKPq10wpu3Lx1e/dOePfe/QcP23v7J0YWmsKYSi71WUoMcCZgbJnlcKY0kDzlcJpevvf90wVow6QY2ZWCSU4yweaMEutK0zaedZOc2AtKOB51l70+vnp80etN251oENWBr4N4AzpoE8PpXusomUla5CAs5cSY8zhSdlISbRnlUIVJYUARekkyOHdQkBzMpKxXqfCBq8zwXGr3CYvr6tWJkuTG/2AfO+AppkZmlad9nOb1QSrhLvKsrcnlWmJb387fTEomVGFB0LX8vODYSuxfC8+YBmr5CofJB3DbaPjsrv2iQBMr9csyITrLybJy22U46WOP/0dl4i/V4Sbqguiq9KmJQOWiKn1qIpjMKfnURADtJHxqIljXHzW3lWvXVrG2HOqqWWepYMNM0/KjI4YH7lkJ1cz5AdMLogm1zqGhc1r8r6+ug5NXg/hw8PbrYef43cZzu+gJeo66KEav0TH6hIZojCj6hr6jH+hn63fwOHgaPFtTWzubmUdoK4LuH7k0HcE=</latexit>

d(x, x 0)

<latexit sha1_base64="SZXdHMS3IsX3oZRBg4OHVtRfZJ4=">AAADVHicfZLfi9QwEMdzradn1fuhj74El8NTlqWVwx9vhz/AF3GF27uF63Kk2dleuDQJSbp0Kf0nfNW/S/B/8cGkW9D1qAMdvpn5TKYZJlOcGRvHP7eC8Nb27Ts7d6N79x/s7u0fPDwzstQUJlRyqacZMcCZgIlllsNUaSBFxuE8u37n8+dL0IZJcWpXCmYFyQVbMEqsC03nR9UQV0+fXe4P4lHcGr4pkk4MUGfjy4PgZTqXtCxAWMqJMRdJrOysJtoyyqGJ0tKAIvSa5HDhpCAFmFnd/nCDD11kjhdSu09Y3Eb/rqhJYQpir4bYCY+YVplVkQ1xVrQHqYS7yFMbldW6xWZ/u3g9q5lQpQVB1+0XJcdWYj8TPGcaqOUrHKXvwb1Gwyd37WcFmlipn9cp0XlBqsa9LsfpEHv9P5SJP6jTfeiS6Kb2rg+gctnU3vUBJnedvOsDQLsW3vUB1uVP+9PKpVM/Y2vrsW76+1QKOjLL6g8OjA7dWAnVzO0DpldEE2rdHkZu05J/9+qmOHsxSo5Hb74cD07edju3gx6jJ+gIJegVOkEf0RhNEEUcfUXf0PfgR/ArDMPtNRpsdTWP0IaFu78B1VYVqQ==</latexit>

General Framework: Banach’s fixed point theorem

…T

<latexit sha1_base64="3uHIbaQHjXM//QrN9dNTqNvIZ5Q=">AAADVnicfZJda9swFIZVZ1077yvdLncjFgpjhGB3n5dlH7CbsQyStqwORVZOXFFZEpIcEoT/xW6337X9mTHJMWxZ8Q748FrnOXolcXLFmbFJ8nMn6t3Yvbm3fyu+fefuvfv9gwcnRlaawpRKLvVZTgxwJmBqmeVwpjSQMudwml+9DfXTJWjDpJjYtYJZSQrBFowS65e+ZCWxl5RwPLnoD5JR0gS+LtJWDFAb44uD6GU2l7QqQVjKiTHnaaLszBFtGeVQx1llQBF6RQo491KQEszMNUeu8aFfmeOF1P4TFjerf3c4UppwtiH2IiCmUWZd5kOcl82PVMJvFKitztXGYtvfLl7PHBOqsiDoxn5RcWwlDq+C50wDtXyN4+wd+Nto+Oi3/aRAEyv1U5cRXZRkVfvbFTgb4qD/hzLxB/W6C10SXbuQugAql7ULqQswhXcKqQsA7S1C6gKsr0+6y8qXmymx1o113e2zUtCSee7eezA+9M9KqGZ+HjC9JJpQ6ycx9pOW/jtX18XJ0Sh9Nnrx+fng+E07c/voEXqMnqAUvULH6AMaoymiSKCv6Bv6Hv2IfvV2e3sbNNppex6irej1fwO8HhdK</latexit>

T<latexit sha1_base64="3uHIbaQHjXM//QrN9dNTqNvIZ5Q=">AAADVnicfZJda9swFIZVZ1077yvdLncjFgpjhGB3n5dlH7CbsQyStqwORVZOXFFZEpIcEoT/xW6337X9mTHJMWxZ8Q748FrnOXolcXLFmbFJ8nMn6t3Yvbm3fyu+fefuvfv9gwcnRlaawpRKLvVZTgxwJmBqmeVwpjSQMudwml+9DfXTJWjDpJjYtYJZSQrBFowS65e+ZCWxl5RwPLnoD5JR0gS+LtJWDFAb44uD6GU2l7QqQVjKiTHnaaLszBFtGeVQx1llQBF6RQo491KQEszMNUeu8aFfmeOF1P4TFjerf3c4UppwtiH2IiCmUWZd5kOcl82PVMJvFKitztXGYtvfLl7PHBOqsiDoxn5RcWwlDq+C50wDtXyN4+wd+Nto+Oi3/aRAEyv1U5cRXZRkVfvbFTgb4qD/hzLxB/W6C10SXbuQugAql7ULqQswhXcKqQsA7S1C6gKsr0+6y8qXmymx1o113e2zUtCSee7eezA+9M9KqGZ+HjC9JJpQ6ycx9pOW/jtX18XJ0Sh9Nnrx+fng+E07c/voEXqMnqAUvULH6AMaoymiSKCv6Bv6Hv2IfvV2e3sbNNppex6irej1fwO8HhdK</latexit> T (x⇤) = x⇤

<latexit sha1_base64="op8lYYjhQi9fTTprhuGD/D6KkqE=">AAADYXicfZJda9swFIbVeB+t95W2l70RC4WuhGDv+2ZQ1g12M5ZB0hbqLMiK4orKkpDkkGD8W3a7/aRe74/syDFsWfEO+PBa5zl6JXFSLbh1UXSz1Qnu3L13f3snfPDw0eMn3d29M6sKQ9mYKqHMRUosE1yyseNOsAttGMlTwc7T61NfP18wY7mSI7fSbJKTTPI5p8TB0rS7n+TEXVEi8Oho+e342TtI024vGkR14NsibkQPNTGc7nZeJzNFi5xJRwWx9jKOtJuUxDhOBavCpLBME3pNMnYJUpKc2UlZn77Ch7Ayw3Nl4JMO16t/d5Qkt/6QfQzCI7ZWdpWnfZzm9Y/SEjby1Ebncm2x6e/mbycll7pwTNK1/bwQ2CnsHwjPuGHUiRUOkw8MbmPYZ9j2i2aGOGWOy4SYLCfLCm6X4aSPvf4fyuUfFHQbuiCmKn1qA6haVKVPbYDNwMmnNoAZsPCpDXBQH7WXNZTrcXGuHJqq3WepWUOmafkRwPAQnpVQw2EeML0ihlAHQxnCpMX/ztVtcfZ8EL8YvPr6snfyvpm5bXSAnqIjFKM36AR9QkM0RhSt0Hf0A/3s/Ap2gm6wt0Y7W03PPtqI4OA3LzAZZA==</latexit>






Deep Q-Learning: contraction in single-objective Q-function space

…T









Q = RS⇥A<latexit sha1_base64="rywZpjUaC4Qvm1OU752aBQcQJsc=">AAACcHicfVHdThNBFJ4uKrD+FbwyXni0ITGkNrvERLwwweAFNwRQKyRsbc5OT8uE+dnMzBqbTV/Cp+EW38Ln4AWYXRoUSTzJJN/55js/801eSOF8kvxuRQt37t5bXFqO7z94+Ohxe2X1qzOl5dTnRhp7lKMjKTT1vfCSjgpLqHJJh/npdn1/+J2sE0Z/8dOCBgonWowFRx+oYbubKfQnHCUcwHtokjyHT9+qa/4zZF4ocvBhNmx3kl7SBNwG6Rx02Dz2hyut19nI8FKR9lyic8dpUvhBhdYLLmkWZ6WjAvkpTug4QI1h0KBqnjWDtcCMYGxsONpDw/5dUaFy9ZZdCKCWuAa5qcq7kKsmMYUOjWrVzVl+vDmohC5KT5pfjRqXEryB2iUYCUvcyynE2UcKm1vaDS32CrLojV2vMrQThT9m4SUTyLpQ4/9Jhf4jDTheCxOQWxFsAH6CFrkPnxQHg9N/7bwN+hu9d7304E1na3Pu9BJ7xl6yVyxlb9kW22H7rM84+8nO2Dn71bqInkbPoxdX0qg1r3nCbkS0fglOa7sD</latexit><latexit sha1_base64="rywZpjUaC4Qvm1OU752aBQcQJsc=">AAACcHicfVHdThNBFJ4uKrD+FbwyXni0ITGkNrvERLwwweAFNwRQKyRsbc5OT8uE+dnMzBqbTV/Cp+EW38Ln4AWYXRoUSTzJJN/55js/801eSOF8kvxuRQt37t5bXFqO7z94+Ohxe2X1qzOl5dTnRhp7lKMjKTT1vfCSjgpLqHJJh/npdn1/+J2sE0Z/8dOCBgonWowFRx+oYbubKfQnHCUcwHtokjyHT9+qa/4zZF4ocvBhNmx3kl7SBNwG6Rx02Dz2hyut19nI8FKR9lyic8dpUvhBhdYLLmkWZ6WjAvkpTug4QI1h0KBqnjWDtcCMYGxsONpDw/5dUaFy9ZZdCKCWuAa5qcq7kKsmMYUOjWrVzVl+vDmohC5KT5pfjRqXEryB2iUYCUvcyynE2UcKm1vaDS32CrLojV2vMrQThT9m4SUTyLpQ4/9Jhf4jDTheCxOQWxFsAH6CFrkPnxQHg9N/7bwN+hu9d7304E1na3Pu9BJ7xl6yVyxlb9kW22H7rM84+8nO2Dn71bqInkbPoxdX0qg1r3nCbkS0fglOa7sD</latexit><latexit sha1_base64="rywZpjUaC4Qvm1OU752aBQcQJsc=">AAACcHicfVHdThNBFJ4uKrD+FbwyXni0ITGkNrvERLwwweAFNwRQKyRsbc5OT8uE+dnMzBqbTV/Cp+EW38Ln4AWYXRoUSTzJJN/55js/801eSOF8kvxuRQt37t5bXFqO7z94+Ohxe2X1qzOl5dTnRhp7lKMjKTT1vfCSjgpLqHJJh/npdn1/+J2sE0Z/8dOCBgonWowFRx+oYbubKfQnHCUcwHtokjyHT9+qa/4zZF4ocvBhNmx3kl7SBNwG6Rx02Dz2hyut19nI8FKR9lyic8dpUvhBhdYLLmkWZ6WjAvkpTug4QI1h0KBqnjWDtcCMYGxsONpDw/5dUaFy9ZZdCKCWuAa5qcq7kKsmMYUOjWrVzVl+vDmohC5KT5pfjRqXEryB2iUYCUvcyynE2UcKm1vaDS32CrLojV2vMrQThT9m4SUTyLpQ4/9Jhf4jDTheCxOQWxFsAH6CFrkPnxQHg9N/7bwN+hu9d7304E1na3Pu9BJ7xl6yVyxlb9kW22H7rM84+8nO2Dn71bqInkbPoxdX0qg1r3nCbkS0fglOa7sD</latexit><latexit sha1_base64="rywZpjUaC4Qvm1OU752aBQcQJsc=">AAACcHicfVHdThNBFJ4uKrD+FbwyXni0ITGkNrvERLwwweAFNwRQKyRsbc5OT8uE+dnMzBqbTV/Cp+EW38Ln4AWYXRoUSTzJJN/55js/801eSOF8kvxuRQt37t5bXFqO7z94+Ohxe2X1qzOl5dTnRhp7lKMjKTT1vfCSjgpLqHJJh/npdn1/+J2sE0Z/8dOCBgonWowFRx+oYbubKfQnHCUcwHtokjyHT9+qa/4zZF4ocvBhNmx3kl7SBNwG6Rx02Dz2hyut19nI8FKR9lyic8dpUvhBhdYLLmkWZ6WjAvkpTug4QI1h0KBqnjWDtcCMYGxsONpDw/5dUaFy9ZZdCKCWuAa5qcq7kKsmMYUOjWrVzVl+vDmohC5KT5pfjRqXEryB2iUYCUvcyynE2UcKm1vaDS32CrLojV2vMrQThT9m4SUTyLpQ4/9Jhf4jDTheCxOQWxFsAH6CFrkPnxQHg9N/7bwN+hu9d7304E1na3Pu9BJ7xl6yVyxlb9kW22H7rM84+8nO2Dn71bqInkbPoxdX0qg1r3nCbkS0fglOa7sD</latexit>

Value Space: all the bounded functions in

Value Metric: d(Q,Q0) = sups,a

|Q(s, a)�Q0(s, a)|<latexit sha1_base64="VqdwjUH4UDTQ1f95jV6LsCAHgl4=">AAACaHicfZHdahQxFMez40fr+NFdvRDxJnQt3ZbpMiOC7YVQ0AtvxA64ttBZljPZs9vQfJFkxGW6L9Cn8VbfxGfwJcxMF7QWPBDyy8n/5CT/lEZw59P0Zye6dfvO3bX1e/H9Bw8fbXR7jz87XVmGI6aFticlOBRc4chzL/DEWARZCjwuz982+8df0Dqu1Se/MDiWMFd8xhn4kJp0X0wHeZJv79A3tHCVmdQugeVFPgjTzl6+3c4Xk24/HaZt0JuQraBPVnE06XX2iqlmlUTlmQDnTrPU+HEN1nMmcBkXlUMD7BzmeBpQgUQ3rtvnLOlWyEzpTNswlKdt9u+KGqST4M8SGqCRuJbcQpYJLWW70EaFgxrV9V5+tj+uuTKVR8WuWs0qQb2mjTt0yi0yLxY0Lt5huLnFD+GIjwYteG136wLsXMLXZXjJnBYJbfh/Uq7+SAPHW6EDMMuDDZSdgQXmw+fEweDsXztvwujl8GCY5a/6h/srp9fJc7JJBiQjr8kheU+OyIgwckm+ke/kR+dX1IueRs+upFFnVfOEXIto8zfwH7YX</latexit><latexit sha1_base64="VqdwjUH4UDTQ1f95jV6LsCAHgl4=">AAACaHicfZHdahQxFMez40fr+NFdvRDxJnQt3ZbpMiOC7YVQ0AtvxA64ttBZljPZs9vQfJFkxGW6L9Cn8VbfxGfwJcxMF7QWPBDyy8n/5CT/lEZw59P0Zye6dfvO3bX1e/H9Bw8fbXR7jz87XVmGI6aFticlOBRc4chzL/DEWARZCjwuz982+8df0Dqu1Se/MDiWMFd8xhn4kJp0X0wHeZJv79A3tHCVmdQugeVFPgjTzl6+3c4Xk24/HaZt0JuQraBPVnE06XX2iqlmlUTlmQDnTrPU+HEN1nMmcBkXlUMD7BzmeBpQgUQ3rtvnLOlWyEzpTNswlKdt9u+KGqST4M8SGqCRuJbcQpYJLWW70EaFgxrV9V5+tj+uuTKVR8WuWs0qQb2mjTt0yi0yLxY0Lt5huLnFD+GIjwYteG136wLsXMLXZXjJnBYJbfh/Uq7+SAPHW6EDMMuDDZSdgQXmw+fEweDsXztvwujl8GCY5a/6h/srp9fJc7JJBiQjr8kheU+OyIgwckm+ke/kR+dX1IueRs+upFFnVfOEXIto8zfwH7YX</latexit><latexit sha1_base64="VqdwjUH4UDTQ1f95jV6LsCAHgl4=">AAACaHicfZHdahQxFMez40fr+NFdvRDxJnQt3ZbpMiOC7YVQ0AtvxA64ttBZljPZs9vQfJFkxGW6L9Cn8VbfxGfwJcxMF7QWPBDyy8n/5CT/lEZw59P0Zye6dfvO3bX1e/H9Bw8fbXR7jz87XVmGI6aFticlOBRc4chzL/DEWARZCjwuz982+8df0Dqu1Se/MDiWMFd8xhn4kJp0X0wHeZJv79A3tHCVmdQugeVFPgjTzl6+3c4Xk24/HaZt0JuQraBPVnE06XX2iqlmlUTlmQDnTrPU+HEN1nMmcBkXlUMD7BzmeBpQgUQ3rtvnLOlWyEzpTNswlKdt9u+KGqST4M8SGqCRuJbcQpYJLWW70EaFgxrV9V5+tj+uuTKVR8WuWs0qQb2mjTt0yi0yLxY0Lt5huLnFD+GIjwYteG136wLsXMLXZXjJnBYJbfh/Uq7+SAPHW6EDMMuDDZSdgQXmw+fEweDsXztvwujl8GCY5a/6h/srp9fJc7JJBiQjr8kheU+OyIgwckm+ke/kR+dX1IueRs+upFFnVfOEXIto8zfwH7YX</latexit><latexit sha1_base64="ZMQji/t3LeKjelD70Y2jPy9GPA4=">AAACNHicfZHLSgMxFIYzXut4a9dugiKI1DLjRt0JunAjKlgrdIqcSU9rMJchyYhl6Au49ZV8CV/BlfgEZsaCN/BA4MvJn3NNM8Gti6KXYGp6ZnZuvrYQLi6Fyyur9aUrq3PDsM200OY6BYuCK2w77gReZwZBpgI76d1R+d65R2O5VpdulGFPwlDxAWfgvOv8pr4RtaLK6F+IJ7BBJnbTCHaSvma5ROWYAGu7cZS5XgHGcSZwHCa5xQzYHQyx61GBRNsrqjrHdNN7+nSgjT/K0cr7/UcB0kpwt03qoZTYiuxIpk2ayuqiM+UDlaqfudxgv1dwleUOFftMNcgFdZqWbdM+N8icGNEwOUZfucFTH+IsQwNOm+0iATOU8DD2nQxp0qQl/yfl6kvqOdz0GYAZ7sdA2S0YYM5PPfTzjX9P8y+0d1sHrfgiIjWyRtbJFonJHjkkJ+SctAkjffJInoLn4DV4+1zDVDDZR4P8sOD9Ax8qqyU=</latexit><latexit sha1_base64="+IT+olQogzhite9SvSOj+fQHhVc=">AAACXXicfZHNbhMxEMed5aNlKTThglAvFqVqirbRbi+FAxISHHqp6EqEVupG0awzSa36S7a3ItrmBXgarvAmPAMvUe82UlsqMZLln8d/z3hmSiO482n6pxM9ePjo8crqk/jp2rPn693e2jenK8twyLTQ9qQEh4IrHHruBZ4YiyBLgcfl+afm/vgCreNaffVzgyMJM8WnnIEPrnH3zaSfJ/n2Dv1AC1eZce0SWFzm/bDt7Obb7X457m6mg7Q1eh+yJWySpR2Ne53dYqJZJVF5JsC50yw1flSD9ZwJXMRF5dAAO4cZngZUINGN6racBd0KngmdahuW8rT13n5Rg3QS/FlCAzQS15KbyzKhpWwP2qgQqFHdzeWn70Y1V6byqNh1qmklqNe06Q6dcIvMizmNi88Yfm7xMIT4YtCC1/ZtXYCdSfi+CJXMaJHQhv8n5epGGjjeChmAWR7aQNkZWGA+DCcODc7+bed9GO4N3g+yPCWrZIO8Jn2SkX3ykRyQIzIkjPwgP8kv8rvzN+pFL68nEXWWI3lB7lj06godIbVN</latexit><latexit sha1_base64="+IT+olQogzhite9SvSOj+fQHhVc=">AAACXXicfZHNbhMxEMed5aNlKTThglAvFqVqirbRbi+FAxISHHqp6EqEVupG0awzSa36S7a3ItrmBXgarvAmPAMvUe82UlsqMZLln8d/z3hmSiO482n6pxM9ePjo8crqk/jp2rPn693e2jenK8twyLTQ9qQEh4IrHHruBZ4YiyBLgcfl+afm/vgCreNaffVzgyMJM8WnnIEPrnH3zaSfJ/n2Dv1AC1eZce0SWFzm/bDt7Obb7X457m6mg7Q1eh+yJWySpR2Ne53dYqJZJVF5JsC50yw1flSD9ZwJXMRF5dAAO4cZngZUINGN6racBd0KngmdahuW8rT13n5Rg3QS/FlCAzQS15KbyzKhpWwP2qgQqFHdzeWn70Y1V6byqNh1qmklqNe06Q6dcIvMizmNi88Yfm7xMIT4YtCC1/ZtXYCdSfi+CJXMaJHQhv8n5epGGjjeChmAWR7aQNkZWGA+DCcODc7+bed9GO4N3g+yPCWrZIO8Jn2SkX3ykRyQIzIkjPwgP8kv8rvzN+pFL68nEXWWI3lB7lj06godIbVN</latexit><latexit sha1_base64="kfg9pqd5rtW9Vr9us99lV+z2w4A=">AAACaHicfZHNbhMxEMedLdCyfCXlgBAXq6FqirbRbi8tB6RKcOCC6EqEVupG0awzSa36S7a3ItrmBXgarvAmPAMvgXcbCUolRrL88/g/Hvvv0gjufJr+7ERrd+7eW9+4Hz94+Ojxk25v87PTlWU4Ylpoe1qCQ8EVjjz3Ak+NRZClwJPy4m2zf3KJ1nGtPvmFwbGEueIzzsCH1KT7cjrIk3xnl76hhavMpHYJLK/yQZh29/Kddr6adPvpMG2D3oZsBX2yiuNJr7NXTDWrJCrPBDh3lqXGj2uwnjOBy7ioHBpgFzDHs4AKJLpx3T5nSbdDZkpn2oahPG2zf1fUIJ0Ef57QAI3EteQWskxoKduFNioc1Khu9vKzw3HNlak8KnbdalYJ6jVt3KFTbpF5saBx8Q7DzS1+CEd8NGjBa/uqLsDOJXxZhpfMaZHQhv8n5eqPNHC8HToAszzYQNk5WGA+fE4cDM7+tfM2jPaHr4dZnvaPDldOb5AXZIsMSEYOyBF5T47JiDDylXwj38mPzq+oFz2Lnl9Lo86q5im5EdHWb+7fthM=</latexit><latexit sha1_base64="VqdwjUH4UDTQ1f95jV6LsCAHgl4=">AAACaHicfZHdahQxFMez40fr+NFdvRDxJnQt3ZbpMiOC7YVQ0AtvxA64ttBZljPZs9vQfJFkxGW6L9Cn8VbfxGfwJcxMF7QWPBDyy8n/5CT/lEZw59P0Zye6dfvO3bX1e/H9Bw8fbXR7jz87XVmGI6aFticlOBRc4chzL/DEWARZCjwuz982+8df0Dqu1Se/MDiWMFd8xhn4kJp0X0wHeZJv79A3tHCVmdQugeVFPgjTzl6+3c4Xk24/HaZt0JuQraBPVnE06XX2iqlmlUTlmQDnTrPU+HEN1nMmcBkXlUMD7BzmeBpQgUQ3rtvnLOlWyEzpTNswlKdt9u+KGqST4M8SGqCRuJbcQpYJLWW70EaFgxrV9V5+tj+uuTKVR8WuWs0qQb2mjTt0yi0yLxY0Lt5huLnFD+GIjwYteG136wLsXMLXZXjJnBYJbfh/Uq7+SAPHW6EDMMuDDZSdgQXmw+fEweDsXztvwujl8GCY5a/6h/srp9fJc7JJBiQjr8kheU+OyIgwckm+ke/kR+dX1IueRs+upFFnVfOEXIto8zfwH7YX</latexit><latexit sha1_base64="VqdwjUH4UDTQ1f95jV6LsCAHgl4=">AAACaHicfZHdahQxFMez40fr+NFdvRDxJnQt3ZbpMiOC7YVQ0AtvxA64ttBZljPZs9vQfJFkxGW6L9Cn8VbfxGfwJcxMF7QWPBDyy8n/5CT/lEZw59P0Zye6dfvO3bX1e/H9Bw8fbXR7jz87XVmGI6aFticlOBRc4chzL/DEWARZCjwuz982+8df0Dqu1Se/MDiWMFd8xhn4kJp0X0wHeZJv79A3tHCVmdQugeVFPgjTzl6+3c4Xk24/HaZt0JuQraBPVnE06XX2iqlmlUTlmQDnTrPU+HEN1nMmcBkXlUMD7BzmeBpQgUQ3rtvnLOlWyEzpTNswlKdt9u+KGqST4M8SGqCRuJbcQpYJLWW70EaFgxrV9V5+tj+uuTKVR8WuWs0qQb2mjTt0yi0yLxY0Lt5huLnFD+GIjwYteG136wLsXMLXZXjJnBYJbfh/Uq7+SAPHW6EDMMuDDZSdgQXmw+fEweDsXztvwujl8GCY5a/6h/srp9fJc7JJBiQjr8kheU+OyIgwckm+ke/kR+dX1IueRs+upFFnVfOEXIto8zfwH7YX</latexit><latexit sha1_base64="VqdwjUH4UDTQ1f95jV6LsCAHgl4=">AAACaHicfZHdahQxFMez40fr+NFdvRDxJnQt3ZbpMiOC7YVQ0AtvxA64ttBZljPZs9vQfJFkxGW6L9Cn8VbfxGfwJcxMF7QWPBDyy8n/5CT/lEZw59P0Zye6dfvO3bX1e/H9Bw8fbXR7jz87XVmGI6aFticlOBRc4chzL/DEWARZCjwuz982+8df0Dqu1Se/MDiWMFd8xhn4kJp0X0wHeZJv79A3tHCVmdQugeVFPgjTzl6+3c4Xk24/HaZt0JuQraBPVnE06XX2iqlmlUTlmQDnTrPU+HEN1nMmcBkXlUMD7BzmeBpQgUQ3rtvnLOlWyEzpTNswlKdt9u+KGqST4M8SGqCRuJbcQpYJLWW70EaFgxrV9V5+tj+uuTKVR8WuWs0qQb2mjTt0yi0yLxY0Lt5huLnFD+GIjwYteG136wLsXMLXZXjJnBYJbfh/Uq7+SAPHW6EDMMuDDZSdgQXmw+fEweDsXztvwujl8GCY5a/6h/srp9fJc7JJBiQjr8kheU+OyIgwckm+ke/kR+dX1IueRs+upFFnVfOEXIto8zfwH7YX</latexit><latexit sha1_base64="VqdwjUH4UDTQ1f95jV6LsCAHgl4=">AAACaHicfZHdahQxFMez40fr+NFdvRDxJnQt3ZbpMiOC7YVQ0AtvxA64ttBZljPZs9vQfJFkxGW6L9Cn8VbfxGfwJcxMF7QWPBDyy8n/5CT/lEZw59P0Zye6dfvO3bX1e/H9Bw8fbXR7jz87XVmGI6aFticlOBRc4chzL/DEWARZCjwuz982+8df0Dqu1Se/MDiWMFd8xhn4kJp0X0wHeZJv79A3tHCVmdQugeVFPgjTzl6+3c4Xk24/HaZt0JuQraBPVnE06XX2iqlmlUTlmQDnTrPU+HEN1nMmcBkXlUMD7BzmeBpQgUQ3rtvnLOlWyEzpTNswlKdt9u+KGqST4M8SGqCRuJbcQpYJLWW70EaFgxrV9V5+tj+uuTKVR8WuWs0qQb2mjTt0yi0yLxY0Lt5huLnFD+GIjwYteG136wLsXMLXZXjJnBYJbfh/Uq7+SAPHW6EDMMuDDZSdgQXmw+fEweDsXztvwujl8GCY5a/6h/srp9fJc7JJBiQjr8kheU+OyIgwckm+ke/kR+dX1IueRs+upFFnVfOEXIto8zfwH7YX</latexit><latexit sha1_base64="VqdwjUH4UDTQ1f95jV6LsCAHgl4=">AAACaHicfZHdahQxFMez40fr+NFdvRDxJnQt3ZbpMiOC7YVQ0AtvxA64ttBZljPZs9vQfJFkxGW6L9Cn8VbfxGfwJcxMF7QWPBDyy8n/5CT/lEZw59P0Zye6dfvO3bX1e/H9Bw8fbXR7jz87XVmGI6aFticlOBRc4chzL/DEWARZCjwuz982+8df0Dqu1Se/MDiWMFd8xhn4kJp0X0wHeZJv79A3tHCVmdQugeVFPgjTzl6+3c4Xk24/HaZt0JuQraBPVnE06XX2iqlmlUTlmQDnTrPU+HEN1nMmcBkXlUMD7BzmeBpQgUQ3rtvnLOlWyEzpTNswlKdt9u+KGqST4M8SGqCRuJbcQpYJLWW70EaFgxrV9V5+tj+uuTKVR8WuWs0qQb2mjTt0yi0yLxY0Lt5huLnFD+GIjwYteG136wLsXMLXZXjJnBYJbfh/Uq7+SAPHW6EDMMuDDZSdgQXmw+fEweDsXztvwujl8GCY5a/6h/srp9fJc7JJBiQjr8kheU+OyIgwckm+ke/kR+dX1IueRs+upFFnVfOEXIto8zfwH7YX</latexit><latexit sha1_base64="VqdwjUH4UDTQ1f95jV6LsCAHgl4=">AAACaHicfZHdahQxFMez40fr+NFdvRDxJnQt3ZbpMiOC7YVQ0AtvxA64ttBZljPZs9vQfJFkxGW6L9Cn8VbfxGfwJcxMF7QWPBDyy8n/5CT/lEZw59P0Zye6dfvO3bX1e/H9Bw8fbXR7jz87XVmGI6aFticlOBRc4chzL/DEWARZCjwuz982+8df0Dqu1Se/MDiWMFd8xhn4kJp0X0wHeZJv79A3tHCVmdQugeVFPgjTzl6+3c4Xk24/HaZt0JuQraBPVnE06XX2iqlmlUTlmQDnTrPU+HEN1nMmcBkXlUMD7BzmeBpQgUQ3rtvnLOlWyEzpTNswlKdt9u+KGqST4M8SGqCRuJbcQpYJLWW70EaFgxrV9V5+tj+uuTKVR8WuWs0qQb2mjTt0yi0yLxY0Lt5huLnFD+GIjwYteG136wLsXMLXZXjJnBYJbfh/Uq7+SAPHW6EDMMuDDZSdgQXmw+fEweDsXztvwujl8GCY5a/6h/srp9fJc7JJBiQjr8kheU+OyIgwckm+ke/kR+dX1IueRs+upFFnVfOEXIto8zfwH7YX</latexit>

Optimality Operator:

Optimality Filtera contraction with the fixed-point Q⇤

<latexit sha1_base64="QYeqB4qibX68TNZ3vh9ZERq/sv0=">AAACRXicfZDdahNBFMdno7Zxq7Wxl94MhoCUGHal0PQuYC+8ERM0JtCN5ezkZDN0PpaZ2dKw5BG8ra/UV/AlvBJvdXYb0FrogYHfnPmfj/mnueDWRdH3oPHg4aOt7ebjcOfJ091ne63nn60uDMMx00KbaQoWBVc4dtwJnOYGQaYCJ+n52+p9coHGcq0+uVWOMwmZ4gvOwPnUx9GXg7O9dtSL6qB3Id5Am2xieNYKXidzzQqJyjEB1p7GUe5mJRjHmcB1mBQWc2DnkOGpRwUS7aysd13Tjs/M6UIbf5SjdfbfihKkleCWXeqhktia7EqmXZrK+qJz5RtVqtuz3KI/K7nKC4eK3YxaFII6Tauv0zk3yJxY0TA5Qb+5wfe+xYccDThtDsoETCbhcu1/ktGkSyu+T8rVX6nnsOMnADPc20DZEgww550PvcHx/3behfGb3nEvHh22B/2N003ygrwkr0hMjsiAvCNDMiaMZOQruSLfguvgR/Az+HUjbQSbmn1yK4LffwC+L66v</latexit><latexit sha1_base64="QYeqB4qibX68TNZ3vh9ZERq/sv0=">AAACRXicfZDdahNBFMdno7Zxq7Wxl94MhoCUGHal0PQuYC+8ERM0JtCN5ezkZDN0PpaZ2dKw5BG8ra/UV/AlvBJvdXYb0FrogYHfnPmfj/mnueDWRdH3oPHg4aOt7ebjcOfJ091ne63nn60uDMMx00KbaQoWBVc4dtwJnOYGQaYCJ+n52+p9coHGcq0+uVWOMwmZ4gvOwPnUx9GXg7O9dtSL6qB3Id5Am2xieNYKXidzzQqJyjEB1p7GUe5mJRjHmcB1mBQWc2DnkOGpRwUS7aysd13Tjs/M6UIbf5SjdfbfihKkleCWXeqhktia7EqmXZrK+qJz5RtVqtuz3KI/K7nKC4eK3YxaFII6Tauv0zk3yJxY0TA5Qb+5wfe+xYccDThtDsoETCbhcu1/ktGkSyu+T8rVX6nnsOMnADPc20DZEgww550PvcHx/3behfGb3nEvHh22B/2N003ygrwkr0hMjsiAvCNDMiaMZOQruSLfguvgR/Az+HUjbQSbmn1yK4LffwC+L66v</latexit><latexit sha1_base64="QYeqB4qibX68TNZ3vh9ZERq/sv0=">AAACRXicfZDdahNBFMdno7Zxq7Wxl94MhoCUGHal0PQuYC+8ERM0JtCN5ezkZDN0PpaZ2dKw5BG8ra/UV/AlvBJvdXYb0FrogYHfnPmfj/mnueDWRdH3oPHg4aOt7ebjcOfJ091ne63nn60uDMMx00KbaQoWBVc4dtwJnOYGQaYCJ+n52+p9coHGcq0+uVWOMwmZ4gvOwPnUx9GXg7O9dtSL6qB3Id5Am2xieNYKXidzzQqJyjEB1p7GUe5mJRjHmcB1mBQWc2DnkOGpRwUS7aysd13Tjs/M6UIbf5SjdfbfihKkleCWXeqhktia7EqmXZrK+qJz5RtVqtuz3KI/K7nKC4eK3YxaFII6Tauv0zk3yJxY0TA5Qb+5wfe+xYccDThtDsoETCbhcu1/ktGkSyu+T8rVX6nnsOMnADPc20DZEgww550PvcHx/3behfGb3nEvHh22B/2N003ygrwkr0hMjsiAvCNDMiaMZOQruSLfguvgR/Az+HUjbQSbmn1yK4LffwC+L66v</latexit><latexit sha1_base64="QYeqB4qibX68TNZ3vh9ZERq/sv0=">AAACRXicfZDdahNBFMdno7Zxq7Wxl94MhoCUGHal0PQuYC+8ERM0JtCN5ezkZDN0PpaZ2dKw5BG8ra/UV/AlvBJvdXYb0FrogYHfnPmfj/mnueDWRdH3oPHg4aOt7ebjcOfJ091ne63nn60uDMMx00KbaQoWBVc4dtwJnOYGQaYCJ+n52+p9coHGcq0+uVWOMwmZ4gvOwPnUx9GXg7O9dtSL6qB3Id5Am2xieNYKXidzzQqJyjEB1p7GUe5mJRjHmcB1mBQWc2DnkOGpRwUS7aysd13Tjs/M6UIbf5SjdfbfihKkleCWXeqhktia7EqmXZrK+qJz5RtVqtuz3KI/K7nKC4eK3YxaFII6Tauv0zk3yJxY0TA5Qb+5wfe+xYccDThtDsoETCbhcu1/ktGkSyu+T8rVX6nnsOMnADPc20DZEgww550PvcHx/3behfGb3nEvHh22B/2N003ygrwkr0hMjsiAvCNDMiaMZOQruSLfguvgR/Az+HUjbQSbmn1yK4LffwC+L66v</latexit>

(T Q)(s, a) := r(s, a) + �Es0⇠P(·|s,a) supa02A

Q(s0, a0)<latexit sha1_base64="Gz1PSbWJXtK4Ul/Xo08OSwyQtbc=">AAACs3icfZFdb9MwFIadbMAIXx1ccmNRTU0hVAnaBEJCGl8SN4hWWrdJdRWdOG5mLbYj20FUIf+SG/4KVzhZxRiTOJKlx8fv8Tl+nVUlNzaOf3r+1vaNm7d2bgd37t67/2Cw+/DYqFpTNqeqVPo0A8NKLtnccluy00ozEFnJTrLz9935yVemDVfyyK4rthRQSL7iFKxLpQMZEgH2jEKJj/BsHJoIxq/fYN3DM1KAENArsgx/TBszwsRwgf8UTUNCc2W/mwjDuHWHdZU2MCJcXmretngWmlEEo3E6GMaTuA98HZINDNEmpumu95zkitaCSUtLMGaRxJVdNqAtpyVrA1IbVgE9h4ItHEoQzCyb3pgW77lMjldKuyUt7rN/VzQgTDelG16YTmJ6MmuRRTgT/UZV0l3Uqa72sqtXy4bLqrZM0otWq7rEVuHOZ5xzzagt1zggH5ibXLPP7oovFdNglX7aENCFgG+te0mBSYQ7/p+Uy0up42DPdQCqubMB0zPQQK375sAZnPxr53U4fjFJ4kky2x8evttYvYMeoycoRAl6iQ7RJzRFc0TRD/TL2/K2/QN/4Wd+fiH1vU3NI3QlfPEbQbvPXg==</latexit><latexit sha1_base64="Gz1PSbWJXtK4Ul/Xo08OSwyQtbc=">AAACs3icfZFdb9MwFIadbMAIXx1ccmNRTU0hVAnaBEJCGl8SN4hWWrdJdRWdOG5mLbYj20FUIf+SG/4KVzhZxRiTOJKlx8fv8Tl+nVUlNzaOf3r+1vaNm7d2bgd37t67/2Cw+/DYqFpTNqeqVPo0A8NKLtnccluy00ozEFnJTrLz9935yVemDVfyyK4rthRQSL7iFKxLpQMZEgH2jEKJj/BsHJoIxq/fYN3DM1KAENArsgx/TBszwsRwgf8UTUNCc2W/mwjDuHWHdZU2MCJcXmretngWmlEEo3E6GMaTuA98HZINDNEmpumu95zkitaCSUtLMGaRxJVdNqAtpyVrA1IbVgE9h4ItHEoQzCyb3pgW77lMjldKuyUt7rN/VzQgTDelG16YTmJ6MmuRRTgT/UZV0l3Uqa72sqtXy4bLqrZM0otWq7rEVuHOZ5xzzagt1zggH5ibXLPP7oovFdNglX7aENCFgG+te0mBSYQ7/p+Uy0up42DPdQCqubMB0zPQQK375sAZnPxr53U4fjFJ4kky2x8evttYvYMeoycoRAl6iQ7RJzRFc0TRD/TL2/K2/QN/4Wd+fiH1vU3NI3QlfPEbQbvPXg==</latexit><latexit sha1_base64="Gz1PSbWJXtK4Ul/Xo08OSwyQtbc=">AAACs3icfZFdb9MwFIadbMAIXx1ccmNRTU0hVAnaBEJCGl8SN4hWWrdJdRWdOG5mLbYj20FUIf+SG/4KVzhZxRiTOJKlx8fv8Tl+nVUlNzaOf3r+1vaNm7d2bgd37t67/2Cw+/DYqFpTNqeqVPo0A8NKLtnccluy00ozEFnJTrLz9935yVemDVfyyK4rthRQSL7iFKxLpQMZEgH2jEKJj/BsHJoIxq/fYN3DM1KAENArsgx/TBszwsRwgf8UTUNCc2W/mwjDuHWHdZU2MCJcXmretngWmlEEo3E6GMaTuA98HZINDNEmpumu95zkitaCSUtLMGaRxJVdNqAtpyVrA1IbVgE9h4ItHEoQzCyb3pgW77lMjldKuyUt7rN/VzQgTDelG16YTmJ6MmuRRTgT/UZV0l3Uqa72sqtXy4bLqrZM0otWq7rEVuHOZ5xzzagt1zggH5ibXLPP7oovFdNglX7aENCFgG+te0mBSYQ7/p+Uy0up42DPdQCqubMB0zPQQK375sAZnPxr53U4fjFJ4kky2x8evttYvYMeoycoRAl6iQ7RJzRFc0TRD/TL2/K2/QN/4Wd+fiH1vU3NI3QlfPEbQbvPXg==</latexit><latexit sha1_base64="Gz1PSbWJXtK4Ul/Xo08OSwyQtbc=">AAACs3icfZFdb9MwFIadbMAIXx1ccmNRTU0hVAnaBEJCGl8SN4hWWrdJdRWdOG5mLbYj20FUIf+SG/4KVzhZxRiTOJKlx8fv8Tl+nVUlNzaOf3r+1vaNm7d2bgd37t67/2Cw+/DYqFpTNqeqVPo0A8NKLtnccluy00ozEFnJTrLz9935yVemDVfyyK4rthRQSL7iFKxLpQMZEgH2jEKJj/BsHJoIxq/fYN3DM1KAENArsgx/TBszwsRwgf8UTUNCc2W/mwjDuHWHdZU2MCJcXmretngWmlEEo3E6GMaTuA98HZINDNEmpumu95zkitaCSUtLMGaRxJVdNqAtpyVrA1IbVgE9h4ItHEoQzCyb3pgW77lMjldKuyUt7rN/VzQgTDelG16YTmJ6MmuRRTgT/UZV0l3Uqa72sqtXy4bLqrZM0otWq7rEVuHOZ5xzzagt1zggH5ibXLPP7oovFdNglX7aENCFgG+te0mBSYQ7/p+Uy0up42DPdQCqubMB0zPQQK375sAZnPxr53U4fjFJ4kky2x8evttYvYMeoycoRAl6iQ7RJzRFc0TRD/TL2/K2/QN/4Wd+fiH1vU3NI3QlfPEbQbvPXg==</latexit>

Update Scheme: asynchronous value iteration

Scalarized MOQ-Learning

…T










Value Metric:

Q = (⌦ ! R)S⇥A<latexit sha1_base64="LU42Dlx2GUr+voz6fHK4fGdRNG4=">AAACjnicfVFbaxNBFJ6st7pemuijLwdDoUoMuyK0PhQr+lAEaavGFroxnJ2cbIbOZZmZVcOS/+SvEXzSn+LsNlRrwQMD33zznds3eSmF80nyoxNduXrt+o21m/Gt23furnd79z46U1lOI26kscc5OpJC08gLL+m4tIQql3SUn75q3o8+k3XC6A9+UdJYYaHFTHD0gZp032QK/ZyjhEPYgc1sX1GBkFlRzD1aa75AK8hzePfoU30ufg+ZF4ocnDMvl5NuPxkmbcBlkK5An63iYNLrPMmmhleKtOcSnTtJk9KPa7RecEnLOKsclchPsaCTADWGjuO6XXoJG4GZwszYcLSHlv07o0blmuEGEEAjcS1yC5UPIFftxZQ6FGpUF3v52fa4FrqsPGl+1mpWSfAGGg9hKixxLxcQZ68pTG7pbSixX5JFb+zjOkNbKPy6DJsUkA2gwf+TCv1HGnC8ETogtyLYAHyOFrkPXxgHg9N/7bwMRk+Hz4fp4bP+7vbK6TX2gD1kmyxlW2yX7bEDNmKcfWPf2U/2K+pFW9FO9OJMGnVWOffZhYj2fgM7WMYB</latexit><latexit sha1_base64="LU42Dlx2GUr+voz6fHK4fGdRNG4=">AAACjnicfVFbaxNBFJ6st7pemuijLwdDoUoMuyK0PhQr+lAEaavGFroxnJ2cbIbOZZmZVcOS/+SvEXzSn+LsNlRrwQMD33zznds3eSmF80nyoxNduXrt+o21m/Gt23furnd79z46U1lOI26kscc5OpJC08gLL+m4tIQql3SUn75q3o8+k3XC6A9+UdJYYaHFTHD0gZp032QK/ZyjhEPYgc1sX1GBkFlRzD1aa75AK8hzePfoU30ufg+ZF4ocnDMvl5NuPxkmbcBlkK5An63iYNLrPMmmhleKtOcSnTtJk9KPa7RecEnLOKsclchPsaCTADWGjuO6XXoJG4GZwszYcLSHlv07o0blmuEGEEAjcS1yC5UPIFftxZQ6FGpUF3v52fa4FrqsPGl+1mpWSfAGGg9hKixxLxcQZ68pTG7pbSixX5JFb+zjOkNbKPy6DJsUkA2gwf+TCv1HGnC8ETogtyLYAHyOFrkPXxgHg9N/7bwMRk+Hz4fp4bP+7vbK6TX2gD1kmyxlW2yX7bEDNmKcfWPf2U/2K+pFW9FO9OJMGnVWOffZhYj2fgM7WMYB</latexit><latexit sha1_base64="LU42Dlx2GUr+voz6fHK4fGdRNG4=">AAACjnicfVFbaxNBFJ6st7pemuijLwdDoUoMuyK0PhQr+lAEaavGFroxnJ2cbIbOZZmZVcOS/+SvEXzSn+LsNlRrwQMD33zznds3eSmF80nyoxNduXrt+o21m/Gt23furnd79z46U1lOI26kscc5OpJC08gLL+m4tIQql3SUn75q3o8+k3XC6A9+UdJYYaHFTHD0gZp032QK/ZyjhEPYgc1sX1GBkFlRzD1aa75AK8hzePfoU30ufg+ZF4ocnDMvl5NuPxkmbcBlkK5An63iYNLrPMmmhleKtOcSnTtJk9KPa7RecEnLOKsclchPsaCTADWGjuO6XXoJG4GZwszYcLSHlv07o0blmuEGEEAjcS1yC5UPIFftxZQ6FGpUF3v52fa4FrqsPGl+1mpWSfAGGg9hKixxLxcQZ68pTG7pbSixX5JFb+zjOkNbKPy6DJsUkA2gwf+TCv1HGnC8ETogtyLYAHyOFrkPXxgHg9N/7bwMRk+Hz4fp4bP+7vbK6TX2gD1kmyxlW2yX7bEDNmKcfWPf2U/2K+pFW9FO9OJMGnVWOffZhYj2fgM7WMYB</latexit><latexit sha1_base64="LU42Dlx2GUr+voz6fHK4fGdRNG4=">AAACjnicfVFbaxNBFJ6st7pemuijLwdDoUoMuyK0PhQr+lAEaavGFroxnJ2cbIbOZZmZVcOS/+SvEXzSn+LsNlRrwQMD33zznds3eSmF80nyoxNduXrt+o21m/Gt23furnd79z46U1lOI26kscc5OpJC08gLL+m4tIQql3SUn75q3o8+k3XC6A9+UdJYYaHFTHD0gZp032QK/ZyjhEPYgc1sX1GBkFlRzD1aa75AK8hzePfoU30ufg+ZF4ocnDMvl5NuPxkmbcBlkK5An63iYNLrPMmmhleKtOcSnTtJk9KPa7RecEnLOKsclchPsaCTADWGjuO6XXoJG4GZwszYcLSHlv07o0blmuEGEEAjcS1yC5UPIFftxZQ6FGpUF3v52fa4FrqsPGl+1mpWSfAGGg9hKixxLxcQZ68pTG7pbSixX5JFb+zjOkNbKPy6DJsUkA2gwf+TCv1HGnC8ETogtyLYAHyOFrkPXxgHg9N/7bwMRk+Hz4fp4bP+7vbK6TX2gD1kmyxlW2yX7bEDNmKcfWPf2U/2K+pFW9FO9OJMGnVWOffZhYj2fgM7WMYB</latexit>

d(Q,Q0) = sups,a

sup!

|Q(s, a,!)�Q0(s, a,!)|<latexit sha1_base64="c9z3UFIpSR/B0yQovj3ZZLzIyFA=">AAACg3icfZFNa9wwEIa1Tpum7tcmOfYiuoRsirPYIdD0EBJoD72UxtBtAvGyjLWzjoj1gSSXLo5/Tn9Nr+2h/6ayd6FJAx0QPDN6RyO9ynXJrYvj371g7cHD9Ucbj8MnT589f9Hf3PpiVWUYjpkqlbnIwWLJJY4ddyVeaIMg8hLP8+t37f75VzSWK/nZLTROBBSSzzkD50vT/slsmEbp7h49ppmt9LS2ETRLypTAApqbdOhr0TLb2093b6c30/4gHsVd0PuQrGBAVnE23eztZzPFKoHSsRKsvUxi7SY1GMdZiU2YVRY1sGso8NKjBIF2UncvbeiOr8zoXBm/pKNd9XZHDcIKcFcR9dBKbEd2IfKI5qJLlJb+oFZ1d5abH01qLnXlULLlqHlVUqdoaxydcYPMlQsaZu/R39zgR3/EJ40GnDKv6wxMIeBb419S0CyiLf9PyuVfqedwx08AZri3gbIrMMCc/7fQG5z8a+d9GB+M3o6S9HBwerRyeoO8JK/IkCTkDTklH8gZGRNGvpMf5Cf5FawHUXAQHC6lQW/Vs03uRHD8B1/bwLg=</latexit><latexit sha1_base64="c9z3UFIpSR/B0yQovj3ZZLzIyFA=">AAACg3icfZFNa9wwEIa1Tpum7tcmOfYiuoRsirPYIdD0EBJoD72UxtBtAvGyjLWzjoj1gSSXLo5/Tn9Nr+2h/6ayd6FJAx0QPDN6RyO9ynXJrYvj371g7cHD9Ucbj8MnT589f9Hf3PpiVWUYjpkqlbnIwWLJJY4ddyVeaIMg8hLP8+t37f75VzSWK/nZLTROBBSSzzkD50vT/slsmEbp7h49ppmt9LS2ETRLypTAApqbdOhr0TLb2093b6c30/4gHsVd0PuQrGBAVnE23eztZzPFKoHSsRKsvUxi7SY1GMdZiU2YVRY1sGso8NKjBIF2UncvbeiOr8zoXBm/pKNd9XZHDcIKcFcR9dBKbEd2IfKI5qJLlJb+oFZ1d5abH01qLnXlULLlqHlVUqdoaxydcYPMlQsaZu/R39zgR3/EJ40GnDKv6wxMIeBb419S0CyiLf9PyuVfqedwx08AZri3gbIrMMCc/7fQG5z8a+d9GB+M3o6S9HBwerRyeoO8JK/IkCTkDTklH8gZGRNGvpMf5Cf5FawHUXAQHC6lQW/Vs03uRHD8B1/bwLg=</latexit><latexit sha1_base64="c9z3UFIpSR/B0yQovj3ZZLzIyFA=">AAACg3icfZFNa9wwEIa1Tpum7tcmOfYiuoRsirPYIdD0EBJoD72UxtBtAvGyjLWzjoj1gSSXLo5/Tn9Nr+2h/6ayd6FJAx0QPDN6RyO9ynXJrYvj371g7cHD9Ucbj8MnT589f9Hf3PpiVWUYjpkqlbnIwWLJJY4ddyVeaIMg8hLP8+t37f75VzSWK/nZLTROBBSSzzkD50vT/slsmEbp7h49ppmt9LS2ETRLypTAApqbdOhr0TLb2093b6c30/4gHsVd0PuQrGBAVnE23eztZzPFKoHSsRKsvUxi7SY1GMdZiU2YVRY1sGso8NKjBIF2UncvbeiOr8zoXBm/pKNd9XZHDcIKcFcR9dBKbEd2IfKI5qJLlJb+oFZ1d5abH01qLnXlULLlqHlVUqdoaxydcYPMlQsaZu/R39zgR3/EJ40GnDKv6wxMIeBb419S0CyiLf9PyuVfqedwx08AZri3gbIrMMCc/7fQG5z8a+d9GB+M3o6S9HBwerRyeoO8JK/IkCTkDTklH8gZGRNGvpMf5Cf5FawHUXAQHC6lQW/Vs03uRHD8B1/bwLg=</latexit><latexit sha1_base64="c9z3UFIpSR/B0yQovj3ZZLzIyFA=">AAACg3icfZFNa9wwEIa1Tpum7tcmOfYiuoRsirPYIdD0EBJoD72UxtBtAvGyjLWzjoj1gSSXLo5/Tn9Nr+2h/6ayd6FJAx0QPDN6RyO9ynXJrYvj371g7cHD9Ucbj8MnT589f9Hf3PpiVWUYjpkqlbnIwWLJJY4ddyVeaIMg8hLP8+t37f75VzSWK/nZLTROBBSSzzkD50vT/slsmEbp7h49ppmt9LS2ETRLypTAApqbdOhr0TLb2093b6c30/4gHsVd0PuQrGBAVnE23eztZzPFKoHSsRKsvUxi7SY1GMdZiU2YVRY1sGso8NKjBIF2UncvbeiOr8zoXBm/pKNd9XZHDcIKcFcR9dBKbEd2IfKI5qJLlJb+oFZ1d5abH01qLnXlULLlqHlVUqdoaxydcYPMlQsaZu/R39zgR3/EJ40GnDKv6wxMIeBb419S0CyiLf9PyuVfqedwx08AZri3gbIrMMCc/7fQG5z8a+d9GB+M3o6S9HBwerRyeoO8JK/IkCTkDTklH8gZGRNGvpMf5Cf5FawHUXAQHC6lQW/Vs03uRHD8B1/bwLg=</latexit>

hQ, di<latexit sha1_base64="FiJBvVYx/Glek8KTGvWpQ1+k+iY=">AAACYHicfVFdSxtBFJ2sVuO2amLfqg9Dg1Akhl0pVN8C9aEvRYWmCm4Idyc36+B8LDOzYljy4q/xVf+Nz/6Rzq4Bv6AXBs6cOffrTJoLbl0UPTSChcUPS8vNlfDjp9W19VZ746/VhWE4YFpoc5aCRcEVDhx3As9ygyBTgafp5c/q/fQKjeVa/XHTHIcSMsUnnIHz1Ki1lQhQmUCaSHAXDAQ96dJxYmpy1OpEvagO+h7Ec9Ah8zgetRu7yVizQqJyTIC153GUu2EJxnEmcBYmhcUc2CVkeO6hAol2WNZrzOi2Z8Z0oo0/ytGafZlRgrTVlF3qQSWxNbJTmXZpKuuLzpUvVKle93KT/WHJVV44VOyp1aQQ1GlauULH3CBzYkrD5BD95AZ/+xJHORpw2uyUCZhMwvXMb5LRpEsr/D8pV89Sj8Nt3wGY4d4Gyi7AAHP+U0JvcPzWzvdgsNc76MUn3zv9/bnTTbJJvpJvJCY/SJ/8IsdkQBi5Ibfkjtw3HoOVYD1oP0mDxjznM3kVwZd/W0a1IA==</latexit><latexit sha1_base64="FiJBvVYx/Glek8KTGvWpQ1+k+iY=">AAACYHicfVFdSxtBFJ2sVuO2amLfqg9Dg1Akhl0pVN8C9aEvRYWmCm4Idyc36+B8LDOzYljy4q/xVf+Nz/6Rzq4Bv6AXBs6cOffrTJoLbl0UPTSChcUPS8vNlfDjp9W19VZ746/VhWE4YFpoc5aCRcEVDhx3As9ygyBTgafp5c/q/fQKjeVa/XHTHIcSMsUnnIHz1Ki1lQhQmUCaSHAXDAQ96dJxYmpy1OpEvagO+h7Ec9Ah8zgetRu7yVizQqJyTIC153GUu2EJxnEmcBYmhcUc2CVkeO6hAol2WNZrzOi2Z8Z0oo0/ytGafZlRgrTVlF3qQSWxNbJTmXZpKuuLzpUvVKle93KT/WHJVV44VOyp1aQQ1GlauULH3CBzYkrD5BD95AZ/+xJHORpw2uyUCZhMwvXMb5LRpEsr/D8pV89Sj8Nt3wGY4d4Gyi7AAHP+U0JvcPzWzvdgsNc76MUn3zv9/bnTTbJJvpJvJCY/SJ/8IsdkQBi5Ibfkjtw3HoOVYD1oP0mDxjznM3kVwZd/W0a1IA==</latexit><latexit sha1_base64="FiJBvVYx/Glek8KTGvWpQ1+k+iY=">AAACYHicfVFdSxtBFJ2sVuO2amLfqg9Dg1Akhl0pVN8C9aEvRYWmCm4Idyc36+B8LDOzYljy4q/xVf+Nz/6Rzq4Bv6AXBs6cOffrTJoLbl0UPTSChcUPS8vNlfDjp9W19VZ746/VhWE4YFpoc5aCRcEVDhx3As9ygyBTgafp5c/q/fQKjeVa/XHTHIcSMsUnnIHz1Ki1lQhQmUCaSHAXDAQ96dJxYmpy1OpEvagO+h7Ec9Ah8zgetRu7yVizQqJyTIC153GUu2EJxnEmcBYmhcUc2CVkeO6hAol2WNZrzOi2Z8Z0oo0/ytGafZlRgrTVlF3qQSWxNbJTmXZpKuuLzpUvVKle93KT/WHJVV44VOyp1aQQ1GlauULH3CBzYkrD5BD95AZ/+xJHORpw2uyUCZhMwvXMb5LRpEsr/D8pV89Sj8Nt3wGY4d4Gyi7AAHP+U0JvcPzWzvdgsNc76MUn3zv9/bnTTbJJvpJvJCY/SJ/8IsdkQBi5Ibfkjtw3HoOVYD1oP0mDxjznM3kVwZd/W0a1IA==</latexit><latexit sha1_base64="FiJBvVYx/Glek8KTGvWpQ1+k+iY=">AAACYHicfVFdSxtBFJ2sVuO2amLfqg9Dg1Akhl0pVN8C9aEvRYWmCm4Idyc36+B8LDOzYljy4q/xVf+Nz/6Rzq4Bv6AXBs6cOffrTJoLbl0UPTSChcUPS8vNlfDjp9W19VZ746/VhWE4YFpoc5aCRcEVDhx3As9ygyBTgafp5c/q/fQKjeVa/XHTHIcSMsUnnIHz1Ki1lQhQmUCaSHAXDAQ96dJxYmpy1OpEvagO+h7Ec9Ah8zgetRu7yVizQqJyTIC153GUu2EJxnEmcBYmhcUc2CVkeO6hAol2WNZrzOi2Z8Z0oo0/ytGafZlRgrTVlF3qQSWxNbJTmXZpKuuLzpUvVKle93KT/WHJVV44VOyp1aQQ1GlauULH3CBzYkrD5BD95AZ/+xJHORpw2uyUCZhMwvXMb5LRpEsr/D8pV89Sj8Nt3wGY4d4Gyi7AAHP+U0JvcPzWzvdgsNc76MUn3zv9/bnTTbJJvpJvJCY/SJ/8IsdkQBi5Ibfkjtw3HoOVYD1oP0mDxjznM3kVwZd/W0a1IA==</latexit>

is still complete.

Utility-Based Multi-Objective Q-Network

s<latexit sha1_base64="WeT1t+HIs6bnr9Bzl+1QZ8jeIJs=">AAACRHicfZDLahsxFIY1Sdqm00vsZJmNqAmU4pqZEGiWoe2imxIb4gtkjDkjH9siugySJtQMfoJuk1fqO/QdsivZhmrGhtQx9IDg09F/LvrTTHDrouh3sLW98+z5i92X4avXb97u1er7Patzw7DLtNBmkIJFwRV2HXcCB5lBkKnAfnr1pXzvX6OxXKsLN89wKGGq+IQzcD7VsaNaI2pFVdBNiFfQIKtoj+rBx2SsWS5ROSbA2ss4ytywAOM4E7gIk9xiBuwKpnjpUYFEOyyqTRf0yGfGdKKNP8rRKvtvRQHSSnCzJvVQSmxFdi7TJk1lddGZ8o1K1fosNzkdFlxluUPFlqMmuaBO0/LjdMwNMifmNEy+ot/c4Hff4jxDA06bD0UCZirhx8L/ZEqTJi35f1KuHqWewyM/AZjh3gbKZmCAOe976A2On9q5Cb3jVhy14s5J4+zzyupdckjekfckJp/IGflG2qRLGEHyk9yQ2+BXcBf8Ce6X0q1gVXNA1iJ4+AtoFK5r</latexit><latexit sha1_base64="WeT1t+HIs6bnr9Bzl+1QZ8jeIJs=">AAACRHicfZDLahsxFIY1Sdqm00vsZJmNqAmU4pqZEGiWoe2imxIb4gtkjDkjH9siugySJtQMfoJuk1fqO/QdsivZhmrGhtQx9IDg09F/LvrTTHDrouh3sLW98+z5i92X4avXb97u1er7Patzw7DLtNBmkIJFwRV2HXcCB5lBkKnAfnr1pXzvX6OxXKsLN89wKGGq+IQzcD7VsaNaI2pFVdBNiFfQIKtoj+rBx2SsWS5ROSbA2ss4ytywAOM4E7gIk9xiBuwKpnjpUYFEOyyqTRf0yGfGdKKNP8rRKvtvRQHSSnCzJvVQSmxFdi7TJk1lddGZ8o1K1fosNzkdFlxluUPFlqMmuaBO0/LjdMwNMifmNEy+ot/c4Hff4jxDA06bD0UCZirhx8L/ZEqTJi35f1KuHqWewyM/AZjh3gbKZmCAOe976A2On9q5Cb3jVhy14s5J4+zzyupdckjekfckJp/IGflG2qRLGEHyk9yQ2+BXcBf8Ce6X0q1gVXNA1iJ4+AtoFK5r</latexit><latexit sha1_base64="WeT1t+HIs6bnr9Bzl+1QZ8jeIJs=">AAACRHicfZDLahsxFIY1Sdqm00vsZJmNqAmU4pqZEGiWoe2imxIb4gtkjDkjH9siugySJtQMfoJuk1fqO/QdsivZhmrGhtQx9IDg09F/LvrTTHDrouh3sLW98+z5i92X4avXb97u1er7Patzw7DLtNBmkIJFwRV2HXcCB5lBkKnAfnr1pXzvX6OxXKsLN89wKGGq+IQzcD7VsaNaI2pFVdBNiFfQIKtoj+rBx2SsWS5ROSbA2ss4ytywAOM4E7gIk9xiBuwKpnjpUYFEOyyqTRf0yGfGdKKNP8rRKvtvRQHSSnCzJvVQSmxFdi7TJk1lddGZ8o1K1fosNzkdFlxluUPFlqMmuaBO0/LjdMwNMifmNEy+ot/c4Hff4jxDA06bD0UCZirhx8L/ZEqTJi35f1KuHqWewyM/AZjh3gbKZmCAOe976A2On9q5Cb3jVhy14s5J4+zzyupdckjekfckJp/IGflG2qRLGEHyk9yQ2+BXcBf8Ce6X0q1gVXNA1iJ4+AtoFK5r</latexit><latexit sha1_base64="WeT1t+HIs6bnr9Bzl+1QZ8jeIJs=">AAACRHicfZDLahsxFIY1Sdqm00vsZJmNqAmU4pqZEGiWoe2imxIb4gtkjDkjH9siugySJtQMfoJuk1fqO/QdsivZhmrGhtQx9IDg09F/LvrTTHDrouh3sLW98+z5i92X4avXb97u1er7Patzw7DLtNBmkIJFwRV2HXcCB5lBkKnAfnr1pXzvX6OxXKsLN89wKGGq+IQzcD7VsaNaI2pFVdBNiFfQIKtoj+rBx2SsWS5ROSbA2ss4ytywAOM4E7gIk9xiBuwKpnjpUYFEOyyqTRf0yGfGdKKNP8rRKvtvRQHSSnCzJvVQSmxFdi7TJk1lddGZ8o1K1fosNzkdFlxluUPFlqMmuaBO0/LjdMwNMifmNEy+ot/c4Hff4jxDA06bD0UCZirhx8L/ZEqTJi35f1KuHqWewyM/AZjh3gbKZmCAOe976A2On9q5Cb3jVhy14s5J4+zzyupdckjekfckJp/IGflG2qRLGEHyk9yQ2+BXcBf8Ce6X0q1gVXNA1iJ4+AtoFK5r</latexit>

!<latexit sha1_base64="12odp0flAtwjr8U+b9A0Xo1vN+U=">AAACTXicfZFLSyQxEMfT47vX9Xn0EhwEkdmhW4Tdo6gHL6KCo7L2INWZmjaYR5Oklx2a+RZe9St59oN4EzHdDvgCCwK/VP5VlfyT5oJbF0UPQWNsfGJyanom/DH7c25+YXHp1OrCMOwwLbQ5T8Gi4Ao7jjuB57lBkKnAs/R6tzo/+4fGcq1O3CDHroRM8T5n4Hzqb5JKmmiJGVwuNKN2VAf9CvEImmQUR5eLwa+kp1khUTkmwNqLOMpdtwTjOBM4DJPCYg7sGjK88KhAou2W9ZWHdM1nerSvjV/K0Tr7vqIEaSW4qxb1UElsTXYg0xZNZb3RufKNKtXHWa7/p1tylRcOFXsd1S8EdZpWDtAeN8icGNAw2UN/c4MHvsVhjgacNhtlAiaT8H/oX5LRpEUr/k7K1ZvUc7jmJwAz3NtA2RUYYM5/QOgNjj/b+RVON9tx1I6Pt5rbOyOrp8kKWSXrJCa/yTbZJ0ekQxhR5IbckrvgPngMnoLnV2kjGNUskw/RmHoB2W6xAw==</latexit><latexit sha1_base64="12odp0flAtwjr8U+b9A0Xo1vN+U=">AAACTXicfZFLSyQxEMfT47vX9Xn0EhwEkdmhW4Tdo6gHL6KCo7L2INWZmjaYR5Oklx2a+RZe9St59oN4EzHdDvgCCwK/VP5VlfyT5oJbF0UPQWNsfGJyanom/DH7c25+YXHp1OrCMOwwLbQ5T8Gi4Ao7jjuB57lBkKnAs/R6tzo/+4fGcq1O3CDHroRM8T5n4Hzqb5JKmmiJGVwuNKN2VAf9CvEImmQUR5eLwa+kp1khUTkmwNqLOMpdtwTjOBM4DJPCYg7sGjK88KhAou2W9ZWHdM1nerSvjV/K0Tr7vqIEaSW4qxb1UElsTXYg0xZNZb3RufKNKtXHWa7/p1tylRcOFXsd1S8EdZpWDtAeN8icGNAw2UN/c4MHvsVhjgacNhtlAiaT8H/oX5LRpEUr/k7K1ZvUc7jmJwAz3NtA2RUYYM5/QOgNjj/b+RVON9tx1I6Pt5rbOyOrp8kKWSXrJCa/yTbZJ0ekQxhR5IbckrvgPngMnoLnV2kjGNUskw/RmHoB2W6xAw==</latexit><latexit sha1_base64="12odp0flAtwjr8U+b9A0Xo1vN+U=">AAACTXicfZFLSyQxEMfT47vX9Xn0EhwEkdmhW4Tdo6gHL6KCo7L2INWZmjaYR5Oklx2a+RZe9St59oN4EzHdDvgCCwK/VP5VlfyT5oJbF0UPQWNsfGJyanom/DH7c25+YXHp1OrCMOwwLbQ5T8Gi4Ao7jjuB57lBkKnAs/R6tzo/+4fGcq1O3CDHroRM8T5n4Hzqb5JKmmiJGVwuNKN2VAf9CvEImmQUR5eLwa+kp1khUTkmwNqLOMpdtwTjOBM4DJPCYg7sGjK88KhAou2W9ZWHdM1nerSvjV/K0Tr7vqIEaSW4qxb1UElsTXYg0xZNZb3RufKNKtXHWa7/p1tylRcOFXsd1S8EdZpWDtAeN8icGNAw2UN/c4MHvsVhjgacNhtlAiaT8H/oX5LRpEUr/k7K1ZvUc7jmJwAz3NtA2RUYYM5/QOgNjj/b+RVON9tx1I6Pt5rbOyOrp8kKWSXrJCa/yTbZJ0ekQxhR5IbckrvgPngMnoLnV2kjGNUskw/RmHoB2W6xAw==</latexit><latexit sha1_base64="12odp0flAtwjr8U+b9A0Xo1vN+U=">AAACTXicfZFLSyQxEMfT47vX9Xn0EhwEkdmhW4Tdo6gHL6KCo7L2INWZmjaYR5Oklx2a+RZe9St59oN4EzHdDvgCCwK/VP5VlfyT5oJbF0UPQWNsfGJyanom/DH7c25+YXHp1OrCMOwwLbQ5T8Gi4Ao7jjuB57lBkKnAs/R6tzo/+4fGcq1O3CDHroRM8T5n4Hzqb5JKmmiJGVwuNKN2VAf9CvEImmQUR5eLwa+kp1khUTkmwNqLOMpdtwTjOBM4DJPCYg7sGjK88KhAou2W9ZWHdM1nerSvjV/K0Tr7vqIEaSW4qxb1UElsTXYg0xZNZb3RufKNKtXHWa7/p1tylRcOFXsd1S8EdZpWDtAeN8icGNAw2UN/c4MHvsVhjgacNhtlAiaT8H/oX5LRpEUr/k7K1ZvUc7jmJwAz3NtA2RUYYM5/QOgNjj/b+RVON9tx1I6Pt5rbOyOrp8kKWSXrJCa/yTbZJ0ekQxhR5IbckrvgPngMnoLnV2kjGNUskw/RmHoB2W6xAw==</latexit>

Q(s, a1,!)<latexit sha1_base64="VH6tYVjfpyM1XEoHElU/3akcPUo=">AAACWnicfZHbSiQxEIYzre5q6+6OpytvgoOgyzh0i6CXol54I6uwo4I9DNWZmjaYQ5OkxaGZh/FWn0jwYUy3A+sBtiDwpfJXqvInzQW3LoqeG8HU9My377Nz4fzCj5+/motLF1YXhmGXaaHNVQoWBVfYddwJvMoNgkwFXqa3R9X55R0ay7X660Y59iRkig85A+dT/ebq+aZtU+jHbZqkkiZaYgZb/WYr6kR10K8QT6BFJnHWX2xsJwPNConKMQHWXsdR7nolGMeZwHGYFBZzYLeQ4bVHBRJtr6znH9MNnxnQoTZ+KUfr7PuKEqSV4G78oNJWEluTHcm0TVNZb3Su/EWV6mMvN9zvlVzlhUPF3loNC0GdppUddMANMidGNEyO0U9u8NRf8SdHA06b32UCJpNwP/YvyWjSphX/T8rVP6nncMN3AGa4t4GyGzDAnP+N0Bscf7bzK1zsdOKoE5/vtg4OJ1bPkjWyTjZJTPbIATkhZ6RLGCnJA3kkT42XIAjmgvk3adCY1CyTDxGsvAKi1rFD</latexit><latexit sha1_base64="VH6tYVjfpyM1XEoHElU/3akcPUo=">AAACWnicfZHbSiQxEIYzre5q6+6OpytvgoOgyzh0i6CXol54I6uwo4I9DNWZmjaYQ5OkxaGZh/FWn0jwYUy3A+sBtiDwpfJXqvInzQW3LoqeG8HU9My377Nz4fzCj5+/motLF1YXhmGXaaHNVQoWBVfYddwJvMoNgkwFXqa3R9X55R0ay7X660Y59iRkig85A+dT/ebq+aZtU+jHbZqkkiZaYgZb/WYr6kR10K8QT6BFJnHWX2xsJwPNConKMQHWXsdR7nolGMeZwHGYFBZzYLeQ4bVHBRJtr6znH9MNnxnQoTZ+KUfr7PuKEqSV4G78oNJWEluTHcm0TVNZb3Su/EWV6mMvN9zvlVzlhUPF3loNC0GdppUddMANMidGNEyO0U9u8NRf8SdHA06b32UCJpNwP/YvyWjSphX/T8rVP6nncMN3AGa4t4GyGzDAnP+N0Bscf7bzK1zsdOKoE5/vtg4OJ1bPkjWyTjZJTPbIATkhZ6RLGCnJA3kkT42XIAjmgvk3adCY1CyTDxGsvAKi1rFD</latexit><latexit sha1_base64="VH6tYVjfpyM1XEoHElU/3akcPUo=">AAACWnicfZHbSiQxEIYzre5q6+6OpytvgoOgyzh0i6CXol54I6uwo4I9DNWZmjaYQ5OkxaGZh/FWn0jwYUy3A+sBtiDwpfJXqvInzQW3LoqeG8HU9My377Nz4fzCj5+/motLF1YXhmGXaaHNVQoWBVfYddwJvMoNgkwFXqa3R9X55R0ay7X660Y59iRkig85A+dT/ebq+aZtU+jHbZqkkiZaYgZb/WYr6kR10K8QT6BFJnHWX2xsJwPNConKMQHWXsdR7nolGMeZwHGYFBZzYLeQ4bVHBRJtr6znH9MNnxnQoTZ+KUfr7PuKEqSV4G78oNJWEluTHcm0TVNZb3Su/EWV6mMvN9zvlVzlhUPF3loNC0GdppUddMANMidGNEyO0U9u8NRf8SdHA06b32UCJpNwP/YvyWjSphX/T8rVP6nncMN3AGa4t4GyGzDAnP+N0Bscf7bzK1zsdOKoE5/vtg4OJ1bPkjWyTjZJTPbIATkhZ6RLGCnJA3kkT42XIAjmgvk3adCY1CyTDxGsvAKi1rFD</latexit><latexit sha1_base64="VH6tYVjfpyM1XEoHElU/3akcPUo=">AAACWnicfZHbSiQxEIYzre5q6+6OpytvgoOgyzh0i6CXol54I6uwo4I9DNWZmjaYQ5OkxaGZh/FWn0jwYUy3A+sBtiDwpfJXqvInzQW3LoqeG8HU9My377Nz4fzCj5+/motLF1YXhmGXaaHNVQoWBVfYddwJvMoNgkwFXqa3R9X55R0ay7X660Y59iRkig85A+dT/ebq+aZtU+jHbZqkkiZaYgZb/WYr6kR10K8QT6BFJnHWX2xsJwPNConKMQHWXsdR7nolGMeZwHGYFBZzYLeQ4bVHBRJtr6znH9MNnxnQoTZ+KUfr7PuKEqSV4G78oNJWEluTHcm0TVNZb3Su/EWV6mMvN9zvlVzlhUPF3loNC0GdppUddMANMidGNEyO0U9u8NRf8SdHA06b32UCJpNwP/YvyWjSphX/T8rVP6nncMN3AGa4t4GyGzDAnP+N0Bscf7bzK1zsdOKoE5/vtg4OJ1bPkjWyTjZJTPbIATkhZ6RLGCnJA3kkT42XIAjmgvk3adCY1CyTDxGsvAKi1rFD</latexit>

Q(s, a2,!)<latexit sha1_base64="kgKqOccALm4Kpvhk6rvqBnRyrOw=">AAACWnicfZHfSiMxFMbTcV3rqLutq1d7EyyCSrfMiKCXsu6FN6LCVgWnlDPp6RjMnyHJiGXow3irTyT4MJsZC64r7IHALyffyTn5kuaCWxdFz41g7tP854XmYri0vPLla6u9emF1YRj2mRbaXKVgUXCFfcedwKvcIMhU4GV6e1SdX96hsVyr326S40BCpviYM3A+NWytn2/ZLoXhbpcmqaSJlpjB9rDViXpRHfQjxDPokFmcDduNH8lIs0KickyAtddxlLtBCcZxJnAaJoXFHNgtZHjtUYFEOyjr+ad002dGdKyNX8rROvt3RQnSSnA3flBpK4mtyU5k2qWprDc6V/6iSvW+lxsfDEqu8sKhYq+txoWgTtPKDjriBpkTExomv9BPbvDEX3GaowGnzU6ZgMkk3E/9SzKadGnF/5Ny9Sb1HG76DsAM9zZQdgMGmPO/EXqD43/t/AgXu7046sXne53DnzOrm+Q72SBbJCb75JAckzPSJ4yU5IE8kqfGSxAEi8HSqzRozGq+kXcRrP0BpMOxRA==</latexit><latexit sha1_base64="kgKqOccALm4Kpvhk6rvqBnRyrOw=">AAACWnicfZHfSiMxFMbTcV3rqLutq1d7EyyCSrfMiKCXsu6FN6LCVgWnlDPp6RjMnyHJiGXow3irTyT4MJsZC64r7IHALyffyTn5kuaCWxdFz41g7tP854XmYri0vPLla6u9emF1YRj2mRbaXKVgUXCFfcedwKvcIMhU4GV6e1SdX96hsVyr326S40BCpviYM3A+NWytn2/ZLoXhbpcmqaSJlpjB9rDViXpRHfQjxDPokFmcDduNH8lIs0KickyAtddxlLtBCcZxJnAaJoXFHNgtZHjtUYFEOyjr+ad002dGdKyNX8rROvt3RQnSSnA3flBpK4mtyU5k2qWprDc6V/6iSvW+lxsfDEqu8sKhYq+txoWgTtPKDjriBpkTExomv9BPbvDEX3GaowGnzU6ZgMkk3E/9SzKadGnF/5Ny9Sb1HG76DsAM9zZQdgMGmPO/EXqD43/t/AgXu7046sXne53DnzOrm+Q72SBbJCb75JAckzPSJ4yU5IE8kqfGSxAEi8HSqzRozGq+kXcRrP0BpMOxRA==</latexit><latexit sha1_base64="kgKqOccALm4Kpvhk6rvqBnRyrOw=">AAACWnicfZHfSiMxFMbTcV3rqLutq1d7EyyCSrfMiKCXsu6FN6LCVgWnlDPp6RjMnyHJiGXow3irTyT4MJsZC64r7IHALyffyTn5kuaCWxdFz41g7tP854XmYri0vPLla6u9emF1YRj2mRbaXKVgUXCFfcedwKvcIMhU4GV6e1SdX96hsVyr326S40BCpviYM3A+NWytn2/ZLoXhbpcmqaSJlpjB9rDViXpRHfQjxDPokFmcDduNH8lIs0KickyAtddxlLtBCcZxJnAaJoXFHNgtZHjtUYFEOyjr+ad002dGdKyNX8rROvt3RQnSSnA3flBpK4mtyU5k2qWprDc6V/6iSvW+lxsfDEqu8sKhYq+txoWgTtPKDjriBpkTExomv9BPbvDEX3GaowGnzU6ZgMkk3E/9SzKadGnF/5Ny9Sb1HG76DsAM9zZQdgMGmPO/EXqD43/t/AgXu7046sXne53DnzOrm+Q72SBbJCb75JAckzPSJ4yU5IE8kqfGSxAEi8HSqzRozGq+kXcRrP0BpMOxRA==</latexit><latexit sha1_base64="kgKqOccALm4Kpvhk6rvqBnRyrOw=">AAACWnicfZHfSiMxFMbTcV3rqLutq1d7EyyCSrfMiKCXsu6FN6LCVgWnlDPp6RjMnyHJiGXow3irTyT4MJsZC64r7IHALyffyTn5kuaCWxdFz41g7tP854XmYri0vPLla6u9emF1YRj2mRbaXKVgUXCFfcedwKvcIMhU4GV6e1SdX96hsVyr326S40BCpviYM3A+NWytn2/ZLoXhbpcmqaSJlpjB9rDViXpRHfQjxDPokFmcDduNH8lIs0KickyAtddxlLtBCcZxJnAaJoXFHNgtZHjtUYFEOyjr+ad002dGdKyNX8rROvt3RQnSSnA3flBpK4mtyU5k2qWprDc6V/6iSvW+lxsfDEqu8sKhYq+txoWgTtPKDjriBpkTExomv9BPbvDEX3GaowGnzU6ZgMkk3E/9SzKadGnF/5Ny9Sb1HG76DsAM9zZQdgMGmPO/EXqD43/t/AgXu7046sXne53DnzOrm+Q72SBbJCb75JAckzPSJ4yU5IE8kqfGSxAEi8HSqzRozGq+kXcRrP0BpMOxRA==</latexit>

Q(s, a3,!)<latexit sha1_base64="hAoV54eUHhmCvEt1hi/xfdnkB3I=">AAACWnicfZFdSxwxFIazY6vuWO360StvQhfBynaZUUEvF+uFN0WFrgrOspzJnh2D+RiSjLgM+2O8bX+R0B9jZlxQK/RA4MnJe3JO3qS54NZF0WMjmPvwcX5hsRkufVpe+dxaXbuwujAM+0wLba5SsCi4wr7jTuBVbhBkKvAyvf1RnV/eobFcq19ukuNAQqb4mDNwPjVsfTnfth0Kw70OTVJJEy0xg2/DVjvqRnXQ9xDPoE1mcTZcbXxPRpoVEpVjAqy9jqPcDUowjjOB0zApLObAbiHDa48KJNpBWc8/pVs+M6JjbfxSjtbZ1xUlSCvB3fhBpa0ktiY7kWmHprLe6Fz5iyrV215ufDgoucoLh4o9txoXgjpNKzvoiBtkTkxomByjn9zgT3/FaY4GnDY7ZQImk3A/9S/JaNKhFf9PytWL1HO45TsAM9zbQNkNGGDO/0boDY7/tfM9XOx246gbn++3e0czqxfJJvlKtklMDkiPnJAz0ieMlOSB/CZ/Gn+DIGgGS8/SoDGrWSdvIth4AqawsUU=</latexit><latexit sha1_base64="hAoV54eUHhmCvEt1hi/xfdnkB3I=">AAACWnicfZFdSxwxFIazY6vuWO360StvQhfBynaZUUEvF+uFN0WFrgrOspzJnh2D+RiSjLgM+2O8bX+R0B9jZlxQK/RA4MnJe3JO3qS54NZF0WMjmPvwcX5hsRkufVpe+dxaXbuwujAM+0wLba5SsCi4wr7jTuBVbhBkKvAyvf1RnV/eobFcq19ukuNAQqb4mDNwPjVsfTnfth0Kw70OTVJJEy0xg2/DVjvqRnXQ9xDPoE1mcTZcbXxPRpoVEpVjAqy9jqPcDUowjjOB0zApLObAbiHDa48KJNpBWc8/pVs+M6JjbfxSjtbZ1xUlSCvB3fhBpa0ktiY7kWmHprLe6Fz5iyrV215ufDgoucoLh4o9txoXgjpNKzvoiBtkTkxomByjn9zgT3/FaY4GnDY7ZQImk3A/9S/JaNKhFf9PytWL1HO45TsAM9zbQNkNGGDO/0boDY7/tfM9XOx246gbn++3e0czqxfJJvlKtklMDkiPnJAz0ieMlOSB/CZ/Gn+DIGgGS8/SoDGrWSdvIth4AqawsUU=</latexit><latexit sha1_base64="hAoV54eUHhmCvEt1hi/xfdnkB3I=">AAACWnicfZFdSxwxFIazY6vuWO360StvQhfBynaZUUEvF+uFN0WFrgrOspzJnh2D+RiSjLgM+2O8bX+R0B9jZlxQK/RA4MnJe3JO3qS54NZF0WMjmPvwcX5hsRkufVpe+dxaXbuwujAM+0wLba5SsCi4wr7jTuBVbhBkKvAyvf1RnV/eobFcq19ukuNAQqb4mDNwPjVsfTnfth0Kw70OTVJJEy0xg2/DVjvqRnXQ9xDPoE1mcTZcbXxPRpoVEpVjAqy9jqPcDUowjjOB0zApLObAbiHDa48KJNpBWc8/pVs+M6JjbfxSjtbZ1xUlSCvB3fhBpa0ktiY7kWmHprLe6Fz5iyrV215ufDgoucoLh4o9txoXgjpNKzvoiBtkTkxomByjn9zgT3/FaY4GnDY7ZQImk3A/9S/JaNKhFf9PytWL1HO45TsAM9zbQNkNGGDO/0boDY7/tfM9XOx246gbn++3e0czqxfJJvlKtklMDkiPnJAz0ieMlOSB/CZ/Gn+DIGgGS8/SoDGrWSdvIth4AqawsUU=</latexit><latexit sha1_base64="hAoV54eUHhmCvEt1hi/xfdnkB3I=">AAACWnicfZFdSxwxFIazY6vuWO360StvQhfBynaZUUEvF+uFN0WFrgrOspzJnh2D+RiSjLgM+2O8bX+R0B9jZlxQK/RA4MnJe3JO3qS54NZF0WMjmPvwcX5hsRkufVpe+dxaXbuwujAM+0wLba5SsCi4wr7jTuBVbhBkKvAyvf1RnV/eobFcq19ukuNAQqb4mDNwPjVsfTnfth0Kw70OTVJJEy0xg2/DVjvqRnXQ9xDPoE1mcTZcbXxPRpoVEpVjAqy9jqPcDUowjjOB0zApLObAbiHDa48KJNpBWc8/pVs+M6JjbfxSjtbZ1xUlSCvB3fhBpa0ktiY7kWmHprLe6Fz5iyrV215ufDgoucoLh4o9txoXgjpNKzvoiBtkTkxomByjn9zgT3/FaY4GnDY7ZQImk3A/9S/JaNKhFf9PytWL1HO45TsAM9zbQNkNGGDO/0boDY7/tfM9XOx246gbn++3e0czqxfJJvlKtklMDkiPnJAz0ieMlOSB/CZ/Gn+DIGgGS8/SoDGrWSdvIth4AqawsUU=</latexit>

Q(s, a4,!)<latexit sha1_base64="s5yT09j5RoHRH37yShLPIrL4Aek=">AAACWnicfZHfSiMxFMbTcV3rqLvV1au9CRZBpVtmpKCXsu6FN6LCVgWnlDPp6RjMnyHJiGXow3irTyT4MJsZC64r7IHALyffyTn5kuaCWxdFz41g7tP854XmYri0vPLla2t17cLqwjDsMy20uUrBouAK+447gVe5QZCpwMv09qg6v7xDY7lWv90kx4GETPExZ+B8atjaON+2HQrDXocmqaSJlpjBzrDVjrpRHfQjxDNok1mcDVcbP5KRZoVE5ZgAa6/jKHeDEozjTOA0TAqLObBbyPDaowKJdlDW80/pls+M6Fgbv5SjdfbvihKkleBu/KDSVhJbk53ItENTWW90rvxFlep9Lzc+GJRc5YVDxV5bjQtBnaaVHXTEDTInJjRMfqGf3OCJv+I0RwNOm90yAZNJuJ/6l2Q06dCK/yfl6k3qOdzyHYAZ7m2g7AYMMOd/I/QGx//a+REu9rpx1I3Pe+3DnzOrm+Q72STbJCb75JAckzPSJ4yU5IE8kqfGSxAEi8HSqzRozGq+kXcRrP8BqJ2xRg==</latexit><latexit sha1_base64="s5yT09j5RoHRH37yShLPIrL4Aek=">AAACWnicfZHfSiMxFMbTcV3rqLvV1au9CRZBpVtmpKCXsu6FN6LCVgWnlDPp6RjMnyHJiGXow3irTyT4MJsZC64r7IHALyffyTn5kuaCWxdFz41g7tP854XmYri0vPLla2t17cLqwjDsMy20uUrBouAK+447gVe5QZCpwMv09qg6v7xDY7lWv90kx4GETPExZ+B8atjaON+2HQrDXocmqaSJlpjBzrDVjrpRHfQjxDNok1mcDVcbP5KRZoVE5ZgAa6/jKHeDEozjTOA0TAqLObBbyPDaowKJdlDW80/pls+M6Fgbv5SjdfbvihKkleBu/KDSVhJbk53ItENTWW90rvxFlep9Lzc+GJRc5YVDxV5bjQtBnaaVHXTEDTInJjRMfqGf3OCJv+I0RwNOm90yAZNJuJ/6l2Q06dCK/yfl6k3qOdzyHYAZ7m2g7AYMMOd/I/QGx//a+REu9rpx1I3Pe+3DnzOrm+Q72STbJCb75JAckzPSJ4yU5IE8kqfGSxAEi8HSqzRozGq+kXcRrP8BqJ2xRg==</latexit><latexit sha1_base64="s5yT09j5RoHRH37yShLPIrL4Aek=">AAACWnicfZHfSiMxFMbTcV3rqLvV1au9CRZBpVtmpKCXsu6FN6LCVgWnlDPp6RjMnyHJiGXow3irTyT4MJsZC64r7IHALyffyTn5kuaCWxdFz41g7tP854XmYri0vPLla2t17cLqwjDsMy20uUrBouAK+447gVe5QZCpwMv09qg6v7xDY7lWv90kx4GETPExZ+B8atjaON+2HQrDXocmqaSJlpjBzrDVjrpRHfQjxDNok1mcDVcbP5KRZoVE5ZgAa6/jKHeDEozjTOA0TAqLObBbyPDaowKJdlDW80/pls+M6Fgbv5SjdfbvihKkleBu/KDSVhJbk53ItENTWW90rvxFlep9Lzc+GJRc5YVDxV5bjQtBnaaVHXTEDTInJjRMfqGf3OCJv+I0RwNOm90yAZNJuJ/6l2Q06dCK/yfl6k3qOdzyHYAZ7m2g7AYMMOd/I/QGx//a+REu9rpx1I3Pe+3DnzOrm+Q72STbJCb75JAckzPSJ4yU5IE8kqfGSxAEi8HSqzRozGq+kXcRrP8BqJ2xRg==</latexit><latexit sha1_base64="s5yT09j5RoHRH37yShLPIrL4Aek=">AAACWnicfZHfSiMxFMbTcV3rqLvV1au9CRZBpVtmpKCXsu6FN6LCVgWnlDPp6RjMnyHJiGXow3irTyT4MJsZC64r7IHALyffyTn5kuaCWxdFz41g7tP854XmYri0vPLla2t17cLqwjDsMy20uUrBouAK+447gVe5QZCpwMv09qg6v7xDY7lWv90kx4GETPExZ+B8atjaON+2HQrDXocmqaSJlpjBzrDVjrpRHfQjxDNok1mcDVcbP5KRZoVE5ZgAa6/jKHeDEozjTOA0TAqLObBbyPDaowKJdlDW80/pls+M6Fgbv5SjdfbvihKkleBu/KDSVhJbk53ItENTWW90rvxFlep9Lzc+GJRc5YVDxV5bjQtBnaaVHXTEDTInJjRMfqGf3OCJv+I0RwNOm90yAZNJuJ/6l2Q06dCK/yfl6k3qOdzyHYAZ7m2g7AYMMOd/I/QGx//a+REu9rpx1I3Pe+3DnzOrm+Q72STbJCb75JAckzPSJ4yU5IE8kqfGSxAEi8HSqzRozGq+kXcRrP8BqJ2xRg==</latexit>

Scalarized MOQ-Learning

…T










Value Metric:


Update Scheme: hindsight experience reply [OpenAI, NIPS2017]

Q = (⌦ ! R)S⇥A<latexit sha1_base64="LU42Dlx2GUr+voz6fHK4fGdRNG4=">AAACjnicfVFbaxNBFJ6st7pemuijLwdDoUoMuyK0PhQr+lAEaavGFroxnJ2cbIbOZZmZVcOS/+SvEXzSn+LsNlRrwQMD33zznds3eSmF80nyoxNduXrt+o21m/Gt23furnd79z46U1lOI26kscc5OpJC08gLL+m4tIQql3SUn75q3o8+k3XC6A9+UdJYYaHFTHD0gZp032QK/ZyjhEPYgc1sX1GBkFlRzD1aa75AK8hzePfoU30ufg+ZF4ocnDMvl5NuPxkmbcBlkK5An63iYNLrPMmmhleKtOcSnTtJk9KPa7RecEnLOKsclchPsaCTADWGjuO6XXoJG4GZwszYcLSHlv07o0blmuEGEEAjcS1yC5UPIFftxZQ6FGpUF3v52fa4FrqsPGl+1mpWSfAGGg9hKixxLxcQZ68pTG7pbSixX5JFb+zjOkNbKPy6DJsUkA2gwf+TCv1HGnC8ETogtyLYAHyOFrkPXxgHg9N/7bwMRk+Hz4fp4bP+7vbK6TX2gD1kmyxlW2yX7bEDNmKcfWPf2U/2K+pFW9FO9OJMGnVWOffZhYj2fgM7WMYB</latexit><latexit sha1_base64="LU42Dlx2GUr+voz6fHK4fGdRNG4=">AAACjnicfVFbaxNBFJ6st7pemuijLwdDoUoMuyK0PhQr+lAEaavGFroxnJ2cbIbOZZmZVcOS/+SvEXzSn+LsNlRrwQMD33zznds3eSmF80nyoxNduXrt+o21m/Gt23furnd79z46U1lOI26kscc5OpJC08gLL+m4tIQql3SUn75q3o8+k3XC6A9+UdJYYaHFTHD0gZp032QK/ZyjhEPYgc1sX1GBkFlRzD1aa75AK8hzePfoU30ufg+ZF4ocnDMvl5NuPxkmbcBlkK5An63iYNLrPMmmhleKtOcSnTtJk9KPa7RecEnLOKsclchPsaCTADWGjuO6XXoJG4GZwszYcLSHlv07o0blmuEGEEAjcS1yC5UPIFftxZQ6FGpUF3v52fa4FrqsPGl+1mpWSfAGGg9hKixxLxcQZ68pTG7pbSixX5JFb+zjOkNbKPy6DJsUkA2gwf+TCv1HGnC8ETogtyLYAHyOFrkPXxgHg9N/7bwMRk+Hz4fp4bP+7vbK6TX2gD1kmyxlW2yX7bEDNmKcfWPf2U/2K+pFW9FO9OJMGnVWOffZhYj2fgM7WMYB</latexit><latexit sha1_base64="LU42Dlx2GUr+voz6fHK4fGdRNG4=">AAACjnicfVFbaxNBFJ6st7pemuijLwdDoUoMuyK0PhQr+lAEaavGFroxnJ2cbIbOZZmZVcOS/+SvEXzSn+LsNlRrwQMD33zznds3eSmF80nyoxNduXrt+o21m/Gt23furnd79z46U1lOI26kscc5OpJC08gLL+m4tIQql3SUn75q3o8+k3XC6A9+UdJYYaHFTHD0gZp032QK/ZyjhEPYgc1sX1GBkFlRzD1aa75AK8hzePfoU30ufg+ZF4ocnDMvl5NuPxkmbcBlkK5An63iYNLrPMmmhleKtOcSnTtJk9KPa7RecEnLOKsclchPsaCTADWGjuO6XXoJG4GZwszYcLSHlv07o0blmuEGEEAjcS1yC5UPIFftxZQ6FGpUF3v52fa4FrqsPGl+1mpWSfAGGg9hKixxLxcQZ68pTG7pbSixX5JFb+zjOkNbKPy6DJsUkA2gwf+TCv1HGnC8ETogtyLYAHyOFrkPXxgHg9N/7bwMRk+Hz4fp4bP+7vbK6TX2gD1kmyxlW2yX7bEDNmKcfWPf2U/2K+pFW9FO9OJMGnVWOffZhYj2fgM7WMYB</latexit><latexit sha1_base64="LU42Dlx2GUr+voz6fHK4fGdRNG4=">AAACjnicfVFbaxNBFJ6st7pemuijLwdDoUoMuyK0PhQr+lAEaavGFroxnJ2cbIbOZZmZVcOS/+SvEXzSn+LsNlRrwQMD33zznds3eSmF80nyoxNduXrt+o21m/Gt23furnd79z46U1lOI26kscc5OpJC08gLL+m4tIQql3SUn75q3o8+k3XC6A9+UdJYYaHFTHD0gZp032QK/ZyjhEPYgc1sX1GBkFlRzD1aa75AK8hzePfoU30ufg+ZF4ocnDMvl5NuPxkmbcBlkK5An63iYNLrPMmmhleKtOcSnTtJk9KPa7RecEnLOKsclchPsaCTADWGjuO6XXoJG4GZwszYcLSHlv07o0blmuEGEEAjcS1yC5UPIFftxZQ6FGpUF3v52fa4FrqsPGl+1mpWSfAGGg9hKixxLxcQZ68pTG7pbSixX5JFb+zjOkNbKPy6DJsUkA2gwf+TCv1HGnC8ETogtyLYAHyOFrkPXxgHg9N/7bwMRk+Hz4fp4bP+7vbK6TX2gD1kmyxlW2yX7bEDNmKcfWPf2U/2K+pFW9FO9OJMGnVWOffZhYj2fgM7WMYB</latexit>

d(Q,Q0) = sups,a

sup!

|Q(s, a,!)�Q0(s, a,!)|<latexit sha1_base64="c9z3UFIpSR/B0yQovj3ZZLzIyFA=">AAACg3icfZFNa9wwEIa1Tpum7tcmOfYiuoRsirPYIdD0EBJoD72UxtBtAvGyjLWzjoj1gSSXLo5/Tn9Nr+2h/6ayd6FJAx0QPDN6RyO9ynXJrYvj371g7cHD9Ucbj8MnT589f9Hf3PpiVWUYjpkqlbnIwWLJJY4ddyVeaIMg8hLP8+t37f75VzSWK/nZLTROBBSSzzkD50vT/slsmEbp7h49ppmt9LS2ETRLypTAApqbdOhr0TLb2093b6c30/4gHsVd0PuQrGBAVnE23eztZzPFKoHSsRKsvUxi7SY1GMdZiU2YVRY1sGso8NKjBIF2UncvbeiOr8zoXBm/pKNd9XZHDcIKcFcR9dBKbEd2IfKI5qJLlJb+oFZ1d5abH01qLnXlULLlqHlVUqdoaxydcYPMlQsaZu/R39zgR3/EJ40GnDKv6wxMIeBb419S0CyiLf9PyuVfqedwx08AZri3gbIrMMCc/7fQG5z8a+d9GB+M3o6S9HBwerRyeoO8JK/IkCTkDTklH8gZGRNGvpMf5Cf5FawHUXAQHC6lQW/Vs03uRHD8B1/bwLg=</latexit><latexit sha1_base64="c9z3UFIpSR/B0yQovj3ZZLzIyFA=">AAACg3icfZFNa9wwEIa1Tpum7tcmOfYiuoRsirPYIdD0EBJoD72UxtBtAvGyjLWzjoj1gSSXLo5/Tn9Nr+2h/6ayd6FJAx0QPDN6RyO9ynXJrYvj371g7cHD9Ucbj8MnT589f9Hf3PpiVWUYjpkqlbnIwWLJJY4ddyVeaIMg8hLP8+t37f75VzSWK/nZLTROBBSSzzkD50vT/slsmEbp7h49ppmt9LS2ETRLypTAApqbdOhr0TLb2093b6c30/4gHsVd0PuQrGBAVnE23eztZzPFKoHSsRKsvUxi7SY1GMdZiU2YVRY1sGso8NKjBIF2UncvbeiOr8zoXBm/pKNd9XZHDcIKcFcR9dBKbEd2IfKI5qJLlJb+oFZ1d5abH01qLnXlULLlqHlVUqdoaxydcYPMlQsaZu/R39zgR3/EJ40GnDKv6wxMIeBb419S0CyiLf9PyuVfqedwx08AZri3gbIrMMCc/7fQG5z8a+d9GB+M3o6S9HBwerRyeoO8JK/IkCTkDTklH8gZGRNGvpMf5Cf5FawHUXAQHC6lQW/Vs03uRHD8B1/bwLg=</latexit><latexit sha1_base64="c9z3UFIpSR/B0yQovj3ZZLzIyFA=">AAACg3icfZFNa9wwEIa1Tpum7tcmOfYiuoRsirPYIdD0EBJoD72UxtBtAvGyjLWzjoj1gSSXLo5/Tn9Nr+2h/6ayd6FJAx0QPDN6RyO9ynXJrYvj371g7cHD9Ucbj8MnT589f9Hf3PpiVWUYjpkqlbnIwWLJJY4ddyVeaIMg8hLP8+t37f75VzSWK/nZLTROBBSSzzkD50vT/slsmEbp7h49ppmt9LS2ETRLypTAApqbdOhr0TLb2093b6c30/4gHsVd0PuQrGBAVnE23eztZzPFKoHSsRKsvUxi7SY1GMdZiU2YVRY1sGso8NKjBIF2UncvbeiOr8zoXBm/pKNd9XZHDcIKcFcR9dBKbEd2IfKI5qJLlJb+oFZ1d5abH01qLnXlULLlqHlVUqdoaxydcYPMlQsaZu/R39zgR3/EJ40GnDKv6wxMIeBb419S0CyiLf9PyuVfqedwx08AZri3gbIrMMCc/7fQG5z8a+d9GB+M3o6S9HBwerRyeoO8JK/IkCTkDTklH8gZGRNGvpMf5Cf5FawHUXAQHC6lQW/Vs03uRHD8B1/bwLg=</latexit><latexit sha1_base64="c9z3UFIpSR/B0yQovj3ZZLzIyFA=">AAACg3icfZFNa9wwEIa1Tpum7tcmOfYiuoRsirPYIdD0EBJoD72UxtBtAvGyjLWzjoj1gSSXLo5/Tn9Nr+2h/6ayd6FJAx0QPDN6RyO9ynXJrYvj371g7cHD9Ucbj8MnT589f9Hf3PpiVWUYjpkqlbnIwWLJJY4ddyVeaIMg8hLP8+t37f75VzSWK/nZLTROBBSSzzkD50vT/slsmEbp7h49ppmt9LS2ETRLypTAApqbdOhr0TLb2093b6c30/4gHsVd0PuQrGBAVnE23eztZzPFKoHSsRKsvUxi7SY1GMdZiU2YVRY1sGso8NKjBIF2UncvbeiOr8zoXBm/pKNd9XZHDcIKcFcR9dBKbEd2IfKI5qJLlJb+oFZ1d5abH01qLnXlULLlqHlVUqdoaxydcYPMlQsaZu/R39zgR3/EJ40GnDKv6wxMIeBb419S0CyiLf9PyuVfqedwx08AZri3gbIrMMCc/7fQG5z8a+d9GB+M3o6S9HBwerRyeoO8JK/IkCTkDTklH8gZGRNGvpMf5Cf5FawHUXAQHC6lQW/Vs03uRHD8B1/bwLg=</latexit>

hQ, di<latexit sha1_base64="FiJBvVYx/Glek8KTGvWpQ1+k+iY=">AAACYHicfVFdSxtBFJ2sVuO2amLfqg9Dg1Akhl0pVN8C9aEvRYWmCm4Idyc36+B8LDOzYljy4q/xVf+Nz/6Rzq4Bv6AXBs6cOffrTJoLbl0UPTSChcUPS8vNlfDjp9W19VZ746/VhWE4YFpoc5aCRcEVDhx3As9ygyBTgafp5c/q/fQKjeVa/XHTHIcSMsUnnIHz1Ki1lQhQmUCaSHAXDAQ96dJxYmpy1OpEvagO+h7Ec9Ah8zgetRu7yVizQqJyTIC153GUu2EJxnEmcBYmhcUc2CVkeO6hAol2WNZrzOi2Z8Z0oo0/ytGafZlRgrTVlF3qQSWxNbJTmXZpKuuLzpUvVKle93KT/WHJVV44VOyp1aQQ1GlauULH3CBzYkrD5BD95AZ/+xJHORpw2uyUCZhMwvXMb5LRpEsr/D8pV89Sj8Nt3wGY4d4Gyi7AAHP+U0JvcPzWzvdgsNc76MUn3zv9/bnTTbJJvpJvJCY/SJ/8IsdkQBi5Ibfkjtw3HoOVYD1oP0mDxjznM3kVwZd/W0a1IA==</latexit><latexit sha1_base64="FiJBvVYx/Glek8KTGvWpQ1+k+iY=">AAACYHicfVFdSxtBFJ2sVuO2amLfqg9Dg1Akhl0pVN8C9aEvRYWmCm4Idyc36+B8LDOzYljy4q/xVf+Nz/6Rzq4Bv6AXBs6cOffrTJoLbl0UPTSChcUPS8vNlfDjp9W19VZ746/VhWE4YFpoc5aCRcEVDhx3As9ygyBTgafp5c/q/fQKjeVa/XHTHIcSMsUnnIHz1Ki1lQhQmUCaSHAXDAQ96dJxYmpy1OpEvagO+h7Ec9Ah8zgetRu7yVizQqJyTIC153GUu2EJxnEmcBYmhcUc2CVkeO6hAol2WNZrzOi2Z8Z0oo0/ytGafZlRgrTVlF3qQSWxNbJTmXZpKuuLzpUvVKle93KT/WHJVV44VOyp1aQQ1GlauULH3CBzYkrD5BD95AZ/+xJHORpw2uyUCZhMwvXMb5LRpEsr/D8pV89Sj8Nt3wGY4d4Gyi7AAHP+U0JvcPzWzvdgsNc76MUn3zv9/bnTTbJJvpJvJCY/SJ/8IsdkQBi5Ibfkjtw3HoOVYD1oP0mDxjznM3kVwZd/W0a1IA==</latexit><latexit sha1_base64="FiJBvVYx/Glek8KTGvWpQ1+k+iY=">AAACYHicfVFdSxtBFJ2sVuO2amLfqg9Dg1Akhl0pVN8C9aEvRYWmCm4Idyc36+B8LDOzYljy4q/xVf+Nz/6Rzq4Bv6AXBs6cOffrTJoLbl0UPTSChcUPS8vNlfDjp9W19VZ746/VhWE4YFpoc5aCRcEVDhx3As9ygyBTgafp5c/q/fQKjeVa/XHTHIcSMsUnnIHz1Ki1lQhQmUCaSHAXDAQ96dJxYmpy1OpEvagO+h7Ec9Ah8zgetRu7yVizQqJyTIC153GUu2EJxnEmcBYmhcUc2CVkeO6hAol2WNZrzOi2Z8Z0oo0/ytGafZlRgrTVlF3qQSWxNbJTmXZpKuuLzpUvVKle93KT/WHJVV44VOyp1aQQ1GlauULH3CBzYkrD5BD95AZ/+xJHORpw2uyUCZhMwvXMb5LRpEsr/D8pV89Sj8Nt3wGY4d4Gyi7AAHP+U0JvcPzWzvdgsNc76MUn3zv9/bnTTbJJvpJvJCY/SJ/8IsdkQBi5Ibfkjtw3HoOVYD1oP0mDxjznM3kVwZd/W0a1IA==</latexit><latexit sha1_base64="FiJBvVYx/Glek8KTGvWpQ1+k+iY=">AAACYHicfVFdSxtBFJ2sVuO2amLfqg9Dg1Akhl0pVN8C9aEvRYWmCm4Idyc36+B8LDOzYljy4q/xVf+Nz/6Rzq4Bv6AXBs6cOffrTJoLbl0UPTSChcUPS8vNlfDjp9W19VZ746/VhWE4YFpoc5aCRcEVDhx3As9ygyBTgafp5c/q/fQKjeVa/XHTHIcSMsUnnIHz1Ki1lQhQmUCaSHAXDAQ96dJxYmpy1OpEvagO+h7Ec9Ah8zgetRu7yVizQqJyTIC153GUu2EJxnEmcBYmhcUc2CVkeO6hAol2WNZrzOi2Z8Z0oo0/ytGafZlRgrTVlF3qQSWxNbJTmXZpKuuLzpUvVKle93KT/WHJVV44VOyp1aQQ1GlauULH3CBzYkrD5BD95AZ/+xJHORpw2uyUCZhMwvXMb5LRpEsr/D8pV89Sj8Nt3wGY4d4Gyi7AAHP+U0JvcPzWzvdgsNc76MUn3zv9/bnTTbJJvpJvJCY/SJ/8IsdkQBi5Ibfkjtw3HoOVYD1oP0mDxjznM3kVwZd/W0a1IA==</latexit>

is still complete.

(T Q)(s, a,!) := !|r(s, a) + �Es0⇠P(·|s,a)(HQ)(s0,!).<latexit sha1_base64="X/QmAz/6BMS53czGBCPdD4+w+gE=">AAAC2nicfVFdaxQxFM2MX3X92uqjL1eX0t12XGakUBGE4gf0RdxCty0063Inm52GTpIhyYrLOC++ia/64/wtvpiZDq214IXAyc3JOTcnaZEL6+L4VxBeu37j5q2V2507d+/df9BdfXhg9cIwPmY61+YoRctzofjYCZfzo8JwlGnOD9PTN/X54SdurNBq3y0LPpGYKTEXDJ1vTbs/+1SiO2GYwz7sDfo2woimkmrJMxzAy1dwvvtYUqEcN55b1V0wNXsAm0AzlBKhUUpTeDct7Tq1QsK59qhP2Uw7+AI2AhxUF667jet61Ci2tsNptxcP46bgKkha0CNtjaarwTM602whuXIsR2uPk7hwkxKNEyznVYcuLC+QnWLGjz1UKLmdlE1+Faz5zgzm2vilHDTdv2+UKG09rx9d2ppiG2SXMo0glc1GF8oL1azLXm7+YlIKVSwcV+zMar7IwWmovwNmwnDm8iV06FvuJzf8vZf4UHCDTpuNkqLJJH6u/EsyoBHU+H9UoS6oHnfWvAMyI3wMwE7QIPM/aDs+4OTfOK+Cg+fDJB4me1u9nddt1CvkMXlK+iQh22SH7JIRGRNGfgdPgo1gM6Th1/Bb+P2MGgbtnUfkUoU//gBZdt0I</latexit><latexit sha1_base64="X/QmAz/6BMS53czGBCPdD4+w+gE=">AAAC2nicfVFdaxQxFM2MX3X92uqjL1eX0t12XGakUBGE4gf0RdxCty0063Inm52GTpIhyYrLOC++ia/64/wtvpiZDq214IXAyc3JOTcnaZEL6+L4VxBeu37j5q2V2507d+/df9BdfXhg9cIwPmY61+YoRctzofjYCZfzo8JwlGnOD9PTN/X54SdurNBq3y0LPpGYKTEXDJ1vTbs/+1SiO2GYwz7sDfo2woimkmrJMxzAy1dwvvtYUqEcN55b1V0wNXsAm0AzlBKhUUpTeDct7Tq1QsK59qhP2Uw7+AI2AhxUF667jet61Ci2tsNptxcP46bgKkha0CNtjaarwTM602whuXIsR2uPk7hwkxKNEyznVYcuLC+QnWLGjz1UKLmdlE1+Faz5zgzm2vilHDTdv2+UKG09rx9d2ppiG2SXMo0glc1GF8oL1azLXm7+YlIKVSwcV+zMar7IwWmovwNmwnDm8iV06FvuJzf8vZf4UHCDTpuNkqLJJH6u/EsyoBHU+H9UoS6oHnfWvAMyI3wMwE7QIPM/aDs+4OTfOK+Cg+fDJB4me1u9nddt1CvkMXlK+iQh22SH7JIRGRNGfgdPgo1gM6Th1/Bb+P2MGgbtnUfkUoU//gBZdt0I</latexit><latexit sha1_base64="X/QmAz/6BMS53czGBCPdD4+w+gE=">AAAC2nicfVFdaxQxFM2MX3X92uqjL1eX0t12XGakUBGE4gf0RdxCty0063Inm52GTpIhyYrLOC++ia/64/wtvpiZDq214IXAyc3JOTcnaZEL6+L4VxBeu37j5q2V2507d+/df9BdfXhg9cIwPmY61+YoRctzofjYCZfzo8JwlGnOD9PTN/X54SdurNBq3y0LPpGYKTEXDJ1vTbs/+1SiO2GYwz7sDfo2woimkmrJMxzAy1dwvvtYUqEcN55b1V0wNXsAm0AzlBKhUUpTeDct7Tq1QsK59qhP2Uw7+AI2AhxUF667jet61Ci2tsNptxcP46bgKkha0CNtjaarwTM602whuXIsR2uPk7hwkxKNEyznVYcuLC+QnWLGjz1UKLmdlE1+Faz5zgzm2vilHDTdv2+UKG09rx9d2ppiG2SXMo0glc1GF8oL1azLXm7+YlIKVSwcV+zMar7IwWmovwNmwnDm8iV06FvuJzf8vZf4UHCDTpuNkqLJJH6u/EsyoBHU+H9UoS6oHnfWvAMyI3wMwE7QIPM/aDs+4OTfOK+Cg+fDJB4me1u9nddt1CvkMXlK+iQh22SH7JIRGRNGfgdPgo1gM6Th1/Bb+P2MGgbtnUfkUoU//gBZdt0I</latexit><latexit sha1_base64="X/QmAz/6BMS53czGBCPdD4+w+gE=">AAAC2nicfVFdaxQxFM2MX3X92uqjL1eX0t12XGakUBGE4gf0RdxCty0063Inm52GTpIhyYrLOC++ia/64/wtvpiZDq214IXAyc3JOTcnaZEL6+L4VxBeu37j5q2V2507d+/df9BdfXhg9cIwPmY61+YoRctzofjYCZfzo8JwlGnOD9PTN/X54SdurNBq3y0LPpGYKTEXDJ1vTbs/+1SiO2GYwz7sDfo2woimkmrJMxzAy1dwvvtYUqEcN55b1V0wNXsAm0AzlBKhUUpTeDct7Tq1QsK59qhP2Uw7+AI2AhxUF667jet61Ci2tsNptxcP46bgKkha0CNtjaarwTM602whuXIsR2uPk7hwkxKNEyznVYcuLC+QnWLGjz1UKLmdlE1+Faz5zgzm2vilHDTdv2+UKG09rx9d2ppiG2SXMo0glc1GF8oL1azLXm7+YlIKVSwcV+zMar7IwWmovwNmwnDm8iV06FvuJzf8vZf4UHCDTpuNkqLJJH6u/EsyoBHU+H9UoS6oHnfWvAMyI3wMwE7QIPM/aDs+4OTfOK+Cg+fDJB4me1u9nddt1CvkMXlK+iQh22SH7JIRGRNGfgdPgo1gM6Th1/Bb+P2MGgbtnUfkUoU//gBZdt0I</latexit>

a contraction with the fixed-point Q⇤

<latexit sha1_base64="QYeqB4qibX68TNZ3vh9ZERq/sv0=">AAACRXicfZDdahNBFMdno7Zxq7Wxl94MhoCUGHal0PQuYC+8ERM0JtCN5ezkZDN0PpaZ2dKw5BG8ra/UV/AlvBJvdXYb0FrogYHfnPmfj/mnueDWRdH3oPHg4aOt7ebjcOfJ091ne63nn60uDMMx00KbaQoWBVc4dtwJnOYGQaYCJ+n52+p9coHGcq0+uVWOMwmZ4gvOwPnUx9GXg7O9dtSL6qB3Id5Am2xieNYKXidzzQqJyjEB1p7GUe5mJRjHmcB1mBQWc2DnkOGpRwUS7aysd13Tjs/M6UIbf5SjdfbfihKkleCWXeqhktia7EqmXZrK+qJz5RtVqtuz3KI/K7nKC4eK3YxaFII6Tauv0zk3yJxY0TA5Qb+5wfe+xYccDThtDsoETCbhcu1/ktGkSyu+T8rVX6nnsOMnADPc20DZEgww550PvcHx/3behfGb3nEvHh22B/2N003ygrwkr0hMjsiAvCNDMiaMZOQruSLfguvgR/Az+HUjbQSbmn1yK4LffwC+L66v</latexit><latexit sha1_base64="QYeqB4qibX68TNZ3vh9ZERq/sv0=">AAACRXicfZDdahNBFMdno7Zxq7Wxl94MhoCUGHal0PQuYC+8ERM0JtCN5ezkZDN0PpaZ2dKw5BG8ra/UV/AlvBJvdXYb0FrogYHfnPmfj/mnueDWRdH3oPHg4aOt7ebjcOfJ091ne63nn60uDMMx00KbaQoWBVc4dtwJnOYGQaYCJ+n52+p9coHGcq0+uVWOMwmZ4gvOwPnUx9GXg7O9dtSL6qB3Id5Am2xieNYKXidzzQqJyjEB1p7GUe5mJRjHmcB1mBQWc2DnkOGpRwUS7aysd13Tjs/M6UIbf5SjdfbfihKkleCWXeqhktia7EqmXZrK+qJz5RtVqtuz3KI/K7nKC4eK3YxaFII6Tauv0zk3yJxY0TA5Qb+5wfe+xYccDThtDsoETCbhcu1/ktGkSyu+T8rVX6nnsOMnADPc20DZEgww550PvcHx/3behfGb3nEvHh22B/2N003ygrwkr0hMjsiAvCNDMiaMZOQruSLfguvgR/Az+HUjbQSbmn1yK4LffwC+L66v</latexit><latexit sha1_base64="QYeqB4qibX68TNZ3vh9ZERq/sv0=">AAACRXicfZDdahNBFMdno7Zxq7Wxl94MhoCUGHal0PQuYC+8ERM0JtCN5ezkZDN0PpaZ2dKw5BG8ra/UV/AlvBJvdXYb0FrogYHfnPmfj/mnueDWRdH3oPHg4aOt7ebjcOfJ091ne63nn60uDMMx00KbaQoWBVc4dtwJnOYGQaYCJ+n52+p9coHGcq0+uVWOMwmZ4gvOwPnUx9GXg7O9dtSL6qB3Id5Am2xieNYKXidzzQqJyjEB1p7GUe5mJRjHmcB1mBQWc2DnkOGpRwUS7aysd13Tjs/M6UIbf5SjdfbfihKkleCWXeqhktia7EqmXZrK+qJz5RtVqtuz3KI/K7nKC4eK3YxaFII6Tauv0zk3yJxY0TA5Qb+5wfe+xYccDThtDsoETCbhcu1/ktGkSyu+T8rVX6nnsOMnADPc20DZEgww550PvcHx/3behfGb3nEvHh22B/2N003ygrwkr0hMjsiAvCNDMiaMZOQruSLfguvgR/Az+HUjbQSbmn1yK4LffwC+L66v</latexit><latexit sha1_base64="QYeqB4qibX68TNZ3vh9ZERq/sv0=">AAACRXicfZDdahNBFMdno7Zxq7Wxl94MhoCUGHal0PQuYC+8ERM0JtCN5ezkZDN0PpaZ2dKw5BG8ra/UV/AlvBJvdXYb0FrogYHfnPmfj/mnueDWRdH3oPHg4aOt7ebjcOfJ091ne63nn60uDMMx00KbaQoWBVc4dtwJnOYGQaYCJ+n52+p9coHGcq0+uVWOMwmZ4gvOwPnUx9GXg7O9dtSL6qB3Id5Am2xieNYKXidzzQqJyjEB1p7GUe5mJRjHmcB1mBQWc2DnkOGpRwUS7aysd13Tjs/M6UIbf5SjdfbfihKkleCWXeqhktia7EqmXZrK+qJz5RtVqtuz3KI/K7nKC4eK3YxaFII6Tauv0zk3yJxY0TA5Qb+5wfe+xYccDThtDsoETCbhcu1/ktGkSyu+T8rVX6nnsOMnADPc20DZEgww550PvcHx/3behfGb3nEvHh22B/2N003ygrwkr0hMjsiAvCNDMiaMZOQruSLfguvgR/Az+HUjbQSbmn1yK4LffwC+L66v</latexit>

Optimality Filter

(HQ)(s,!) := supa0

Q(s, a0,!)<latexit sha1_base64="Eo/hrDn/vOeKD+7qnaPujkDz/RU=">AAACdXicfVFdaxNBFJ2sVev6lbaPggzG2rTEuFuEVqFQaB/6IjZgbKEbwt3JzXbofDEzK4Ylv8Nf46v+hv4THzu7jWgteGHgzLnn3jv3TG4Edz5JLlvRnaW79+4vP4gfPnr85Gl7ZfWz06VlOGRaaHuag0PBFQ499wJPjUWQucCT/OKgzp98Qeu4Vp/8zOBIQqH4lDPwgRq3024mwZ8zEPSIDja7rpdpiQVsvt/LXGnGFWzMB4GFjd+JcbuT9JMm6G2QLkCHLOJ4vNJ6nU00KyUqzwQ4d5Ymxo8qsJ4zgfM4Kx0aYBdQ4FmACiS6UdXsNqfrgZnQqbbhKE8b9u+KCqSrN+jRAGqJa5CbybxHc9lctFGhUa26OctPd0cVV6b0qNj1qGkpqNe0topOuEXmxYzG2SGGl1v8EFp8NGjBa7tVZWALCV/nYZOCZj1a4/9JufojDTheDxOAWR5soOwcLDAffioOBqf/2nkbDLf77/rp4G1nf3fh9DJ5Rl6QLknJDtknR+SYDAkj38h38oP8bP2Knkcvo1fX0qi1qFkjNyJ6cwWgSbvw</latexit><latexit sha1_base64="Eo/hrDn/vOeKD+7qnaPujkDz/RU=">AAACdXicfVFdaxNBFJ2sVev6lbaPggzG2rTEuFuEVqFQaB/6IjZgbKEbwt3JzXbofDEzK4Ylv8Nf46v+hv4THzu7jWgteGHgzLnn3jv3TG4Edz5JLlvRnaW79+4vP4gfPnr85Gl7ZfWz06VlOGRaaHuag0PBFQ499wJPjUWQucCT/OKgzp98Qeu4Vp/8zOBIQqH4lDPwgRq3024mwZ8zEPSIDja7rpdpiQVsvt/LXGnGFWzMB4GFjd+JcbuT9JMm6G2QLkCHLOJ4vNJ6nU00KyUqzwQ4d5Ymxo8qsJ4zgfM4Kx0aYBdQ4FmACiS6UdXsNqfrgZnQqbbhKE8b9u+KCqSrN+jRAGqJa5CbybxHc9lctFGhUa26OctPd0cVV6b0qNj1qGkpqNe0topOuEXmxYzG2SGGl1v8EFp8NGjBa7tVZWALCV/nYZOCZj1a4/9JufojDTheDxOAWR5soOwcLDAffioOBqf/2nkbDLf77/rp4G1nf3fh9DJ5Rl6QLknJDtknR+SYDAkj38h38oP8bP2Knkcvo1fX0qi1qFkjNyJ6cwWgSbvw</latexit><latexit sha1_base64="Eo/hrDn/vOeKD+7qnaPujkDz/RU=">AAACdXicfVFdaxNBFJ2sVev6lbaPggzG2rTEuFuEVqFQaB/6IjZgbKEbwt3JzXbofDEzK4Ylv8Nf46v+hv4THzu7jWgteGHgzLnn3jv3TG4Edz5JLlvRnaW79+4vP4gfPnr85Gl7ZfWz06VlOGRaaHuag0PBFQ499wJPjUWQucCT/OKgzp98Qeu4Vp/8zOBIQqH4lDPwgRq3024mwZ8zEPSIDja7rpdpiQVsvt/LXGnGFWzMB4GFjd+JcbuT9JMm6G2QLkCHLOJ4vNJ6nU00KyUqzwQ4d5Ymxo8qsJ4zgfM4Kx0aYBdQ4FmACiS6UdXsNqfrgZnQqbbhKE8b9u+KCqSrN+jRAGqJa5CbybxHc9lctFGhUa26OctPd0cVV6b0qNj1qGkpqNe0topOuEXmxYzG2SGGl1v8EFp8NGjBa7tVZWALCV/nYZOCZj1a4/9JufojDTheDxOAWR5soOwcLDAffioOBqf/2nkbDLf77/rp4G1nf3fh9DJ5Rl6QLknJDtknR+SYDAkj38h38oP8bP2Knkcvo1fX0qi1qFkjNyJ6cwWgSbvw</latexit><latexit sha1_base64="Eo/hrDn/vOeKD+7qnaPujkDz/RU=">AAACdXicfVFdaxNBFJ2sVev6lbaPggzG2rTEuFuEVqFQaB/6IjZgbKEbwt3JzXbofDEzK4Ylv8Nf46v+hv4THzu7jWgteGHgzLnn3jv3TG4Edz5JLlvRnaW79+4vP4gfPnr85Gl7ZfWz06VlOGRaaHuag0PBFQ499wJPjUWQucCT/OKgzp98Qeu4Vp/8zOBIQqH4lDPwgRq3024mwZ8zEPSIDja7rpdpiQVsvt/LXGnGFWzMB4GFjd+JcbuT9JMm6G2QLkCHLOJ4vNJ6nU00KyUqzwQ4d5Ymxo8qsJ4zgfM4Kx0aYBdQ4FmACiS6UdXsNqfrgZnQqbbhKE8b9u+KCqSrN+jRAGqJa5CbybxHc9lctFGhUa26OctPd0cVV6b0qNj1qGkpqNe0topOuEXmxYzG2SGGl1v8EFp8NGjBa7tVZWALCV/nYZOCZj1a4/9JufojDTheDxOAWR5soOwcLDAffioOBqf/2nkbDLf77/rp4G1nf3fh9DJ5Rl6QLknJDtknR+SYDAkj38h38oP8bP2Knkcvo1fX0qi1qFkjNyJ6cwWgSbvw</latexit>

Two Problems of Scalarized MORL


QU

ANTI

TY O

F O

BJEC

TIVE

2Preference

Optimal Solutions


a.

Utility Control Frontier

D’

D

Pseudo Solutions

F

F’Non-Optimal Solutions


QU

ANTI

TY O

F O

BJEC

TIVE

2

D

F!2

<latexit sha1_base64="+UouUJfOmsYL+mxiFW/Pw7MTXn0=">AAACT3icfZFNSxxBEIZ7NsaPMfHzmEvjIoSwWWYkEI+iHryEGHBVcJalprd2bLa/6O4Rl2H/htf4lzz6S3IL6ZkMJCpY0PB09dtV1W/nRnDnk+Qx6rxZeLu4tLwSr757v7a+sbl17nRpGQ6YFtpe5uBQcIUDz73AS2MRZC7wIp8e1ecXN2gd1+rMzwwOJRSKTzgDH1JZlkuaaYkFjPZGG92knzRBX0LaQpe0cTrajD5nY81KicozAc5dpYnxwwqs50zgPM5KhwbYFAq8CqhAohtWzdBzuhsyYzrRNizlaZP9/0YF0knw1z0aoJa4htxM5j2ay2ajjQqFatXTXn6yP6y4MqVHxf62mpSCek1rD+iYW2RezGicHWOY3OK3UOK7QQte209VBraQcDsPLylo1qM1vybl6p80cLwbOgCzPNhA2TVYYD58QRwMTp/b+RLO9/pp0k9/fOkeHLZWL5MPZId8JCn5Sg7ICTklA8KIIXfkJ7mPHqJf0e9OK+1ELWyTJ9FZ+QMa87Cp</latexit><latexit sha1_base64="+UouUJfOmsYL+mxiFW/Pw7MTXn0=">AAACT3icfZFNSxxBEIZ7NsaPMfHzmEvjIoSwWWYkEI+iHryEGHBVcJalprd2bLa/6O4Rl2H/htf4lzz6S3IL6ZkMJCpY0PB09dtV1W/nRnDnk+Qx6rxZeLu4tLwSr757v7a+sbl17nRpGQ6YFtpe5uBQcIUDz73AS2MRZC7wIp8e1ecXN2gd1+rMzwwOJRSKTzgDH1JZlkuaaYkFjPZGG92knzRBX0LaQpe0cTrajD5nY81KicozAc5dpYnxwwqs50zgPM5KhwbYFAq8CqhAohtWzdBzuhsyYzrRNizlaZP9/0YF0knw1z0aoJa4htxM5j2ay2ajjQqFatXTXn6yP6y4MqVHxf62mpSCek1rD+iYW2RezGicHWOY3OK3UOK7QQte209VBraQcDsPLylo1qM1vybl6p80cLwbOgCzPNhA2TVYYD58QRwMTp/b+RLO9/pp0k9/fOkeHLZWL5MPZId8JCn5Sg7ICTklA8KIIXfkJ7mPHqJf0e9OK+1ELWyTJ9FZ+QMa87Cp</latexit><latexit sha1_base64="+UouUJfOmsYL+mxiFW/Pw7MTXn0=">AAACT3icfZFNSxxBEIZ7NsaPMfHzmEvjIoSwWWYkEI+iHryEGHBVcJalprd2bLa/6O4Rl2H/htf4lzz6S3IL6ZkMJCpY0PB09dtV1W/nRnDnk+Qx6rxZeLu4tLwSr757v7a+sbl17nRpGQ6YFtpe5uBQcIUDz73AS2MRZC7wIp8e1ecXN2gd1+rMzwwOJRSKTzgDH1JZlkuaaYkFjPZGG92knzRBX0LaQpe0cTrajD5nY81KicozAc5dpYnxwwqs50zgPM5KhwbYFAq8CqhAohtWzdBzuhsyYzrRNizlaZP9/0YF0knw1z0aoJa4htxM5j2ay2ajjQqFatXTXn6yP6y4MqVHxf62mpSCek1rD+iYW2RezGicHWOY3OK3UOK7QQte209VBraQcDsPLylo1qM1vybl6p80cLwbOgCzPNhA2TVYYD58QRwMTp/b+RLO9/pp0k9/fOkeHLZWL5MPZId8JCn5Sg7ICTklA8KIIXfkJ7mPHqJf0e9OK+1ELWyTJ9FZ+QMa87Cp</latexit><latexit sha1_base64="+UouUJfOmsYL+mxiFW/Pw7MTXn0=">AAACT3icfZFNSxxBEIZ7NsaPMfHzmEvjIoSwWWYkEI+iHryEGHBVcJalprd2bLa/6O4Rl2H/htf4lzz6S3IL6ZkMJCpY0PB09dtV1W/nRnDnk+Qx6rxZeLu4tLwSr757v7a+sbl17nRpGQ6YFtpe5uBQcIUDz73AS2MRZC7wIp8e1ecXN2gd1+rMzwwOJRSKTzgDH1JZlkuaaYkFjPZGG92knzRBX0LaQpe0cTrajD5nY81KicozAc5dpYnxwwqs50zgPM5KhwbYFAq8CqhAohtWzdBzuhsyYzrRNizlaZP9/0YF0knw1z0aoJa4htxM5j2ay2ajjQqFatXTXn6yP6y4MqVHxf62mpSCek1rD+iYW2RezGicHWOY3OK3UOK7QQte209VBraQcDsPLylo1qM1vybl6p80cLwbOgCzPNhA2TVYYD58QRwMTp/b+RLO9/pp0k9/fOkeHLZWL5MPZId8JCn5Sg7ICTklA8KIIXfkJ7mPHqJf0e9OK+1ELWyTJ9FZ+QMa87Cp</latexit>



D F

Optimal Solutions

Sampled Preferences

b.


QU

ANTI

TY O

F O

BJEC

TIVE

2

Preference

Optimal Solutions


a.

Utility Control Frontier

D’

D

Pseudo Solutions

F

F’Non-Optimal Solutions


QU

ANTI

TY O

F O

BJEC

TIVE

2

D

F!2

<latexit sha1_base64="+UouUJfOmsYL+mxiFW/Pw7MTXn0=">AAACT3icfZFNSxxBEIZ7NsaPMfHzmEvjIoSwWWYkEI+iHryEGHBVcJalprd2bLa/6O4Rl2H/htf4lzz6S3IL6ZkMJCpY0PB09dtV1W/nRnDnk+Qx6rxZeLu4tLwSr757v7a+sbl17nRpGQ6YFtpe5uBQcIUDz73AS2MRZC7wIp8e1ecXN2gd1+rMzwwOJRSKTzgDH1JZlkuaaYkFjPZGG92knzRBX0LaQpe0cTrajD5nY81KicozAc5dpYnxwwqs50zgPM5KhwbYFAq8CqhAohtWzdBzuhsyYzrRNizlaZP9/0YF0knw1z0aoJa4htxM5j2ay2ajjQqFatXTXn6yP6y4MqVHxf62mpSCek1rD+iYW2RezGicHWOY3OK3UOK7QQte209VBraQcDsPLylo1qM1vybl6p80cLwbOgCzPNhA2TVYYD58QRwMTp/b+RLO9/pp0k9/fOkeHLZWL5MPZId8JCn5Sg7ICTklA8KIIXfkJ7mPHqJf0e9OK+1ELWyTJ9FZ+QMa87Cp</latexit><latexit sha1_base64="+UouUJfOmsYL+mxiFW/Pw7MTXn0=">AAACT3icfZFNSxxBEIZ7NsaPMfHzmEvjIoSwWWYkEI+iHryEGHBVcJalprd2bLa/6O4Rl2H/htf4lzz6S3IL6ZkMJCpY0PB09dtV1W/nRnDnk+Qx6rxZeLu4tLwSr757v7a+sbl17nRpGQ6YFtpe5uBQcIUDz73AS2MRZC7wIp8e1ecXN2gd1+rMzwwOJRSKTzgDH1JZlkuaaYkFjPZGG92knzRBX0LaQpe0cTrajD5nY81KicozAc5dpYnxwwqs50zgPM5KhwbYFAq8CqhAohtWzdBzuhsyYzrRNizlaZP9/0YF0knw1z0aoJa4htxM5j2ay2ajjQqFatXTXn6yP6y4MqVHxf62mpSCek1rD+iYW2RezGicHWOY3OK3UOK7QQte209VBraQcDsPLylo1qM1vybl6p80cLwbOgCzPNhA2TVYYD58QRwMTp/b+RLO9/pp0k9/fOkeHLZWL5MPZId8JCn5Sg7ICTklA8KIIXfkJ7mPHqJf0e9OK+1ELWyTJ9FZ+QMa87Cp</latexit><latexit sha1_base64="+UouUJfOmsYL+mxiFW/Pw7MTXn0=">AAACT3icfZFNSxxBEIZ7NsaPMfHzmEvjIoSwWWYkEI+iHryEGHBVcJalprd2bLa/6O4Rl2H/htf4lzz6S3IL6ZkMJCpY0PB09dtV1W/nRnDnk+Qx6rxZeLu4tLwSr757v7a+sbl17nRpGQ6YFtpe5uBQcIUDz73AS2MRZC7wIp8e1ecXN2gd1+rMzwwOJRSKTzgDH1JZlkuaaYkFjPZGG92knzRBX0LaQpe0cTrajD5nY81KicozAc5dpYnxwwqs50zgPM5KhwbYFAq8CqhAohtWzdBzuhsyYzrRNizlaZP9/0YF0knw1z0aoJa4htxM5j2ay2ajjQqFatXTXn6yP6y4MqVHxf62mpSCek1rD+iYW2RezGicHWOY3OK3UOK7QQte209VBraQcDsPLylo1qM1vybl6p80cLwbOgCzPNhA2TVYYD58QRwMTp/b+RLO9/pp0k9/fOkeHLZWL5MPZId8JCn5Sg7ICTklA8KIIXfkJ7mPHqJf0e9OK+1ELWyTJ9FZ+QMa87Cp</latexit><latexit sha1_base64="+UouUJfOmsYL+mxiFW/Pw7MTXn0=">AAACT3icfZFNSxxBEIZ7NsaPMfHzmEvjIoSwWWYkEI+iHryEGHBVcJalprd2bLa/6O4Rl2H/htf4lzz6S3IL6ZkMJCpY0PB09dtV1W/nRnDnk+Qx6rxZeLu4tLwSr757v7a+sbl17nRpGQ6YFtpe5uBQcIUDz73AS2MRZC7wIp8e1ecXN2gd1+rMzwwOJRSKTzgDH1JZlkuaaYkFjPZGG92knzRBX0LaQpe0cTrajD5nY81KicozAc5dpYnxwwqs50zgPM5KhwbYFAq8CqhAohtWzdBzuhsyYzrRNizlaZP9/0YF0knw1z0aoJa4htxM5j2ay2ajjQqFatXTXn6yP6y4MqVHxf62mpSCek1rD+iYW2RezGicHWOY3OK3UOK7QQte209VBraQcDsPLylo1qM1vybl6p80cLwbOgCzPNhA2TVYYD58QRwMTp/b+RLO9/pp0k9/fOkeHLZWL5MPZId8JCn5Sg7ICTklA8KIIXfkJ7mPHqJf0e9OK+1ELWyTJ9FZ+QMa87Cp</latexit>



D F

Optimal Solutions

Sampled Preferences

b.

AB

C

D

E

F

G

H

K

L


QU

ANTI

TY O

F O

BJEC

TIVE

2 CCS

Non-optimal

Pareto Frontier

F


QU

ANTI

TY O

F O

BJEC

TIVE

2 F Non-preferred

Preference




a. b.

DUtility Projection


QU

ANTI

TY O

F O

BJEC

TIVE

2

D

F




D F

Optimal Solutions

Sampled Preferences

c.A

BC

D

E

F

G

H

K

L


QU

ANTI

TY O

F O

BJEC

TIVE

2 CCS

Non-optimal

Pareto Frontier

F


QU

ANTI

TY O

F O

BJEC

TIVE

2 F Non-preferred

Preference




a. b.

DUtility Projection


QU

ANTI

TY O

F O

BJEC

TIVE

2

D

F




D F

Optimal Solutions

Sampled Preferences

c.

Degeneracy problem: unable to recover reward from utility

Two Problems of Scalarized MORL

Efficiency problem: misalignment between preferences and rewards

AB

C

D

E

F

G

H

K

L


QU

ANTI

TY O

F O

BJEC

TIVE

2 CCS

Non-optimal

Pareto Frontier

F


QU

ANTI

TY O

F O

BJEC

TIVE

2 F Non-preferred

Preference




a. b.

DUtility Projection L


QU

ANTI

TY O

F O

BJEC

TIVE

2

D

F




D F

Optimal Solutions

Sampled Preferences

c.

Optimality Filter

(HQ)(s,!) := supa0

Q(s, a0,!)<latexit sha1_base64="Eo/hrDn/vOeKD+7qnaPujkDz/RU=">AAACdXicfVFdaxNBFJ2sVev6lbaPggzG2rTEuFuEVqFQaB/6IjZgbKEbwt3JzXbofDEzK4Ylv8Nf46v+hv4THzu7jWgteGHgzLnn3jv3TG4Edz5JLlvRnaW79+4vP4gfPnr85Gl7ZfWz06VlOGRaaHuag0PBFQ499wJPjUWQucCT/OKgzp98Qeu4Vp/8zOBIQqH4lDPwgRq3024mwZ8zEPSIDja7rpdpiQVsvt/LXGnGFWzMB4GFjd+JcbuT9JMm6G2QLkCHLOJ4vNJ6nU00KyUqzwQ4d5Ymxo8qsJ4zgfM4Kx0aYBdQ4FmACiS6UdXsNqfrgZnQqbbhKE8b9u+KCqSrN+jRAGqJa5CbybxHc9lctFGhUa26OctPd0cVV6b0qNj1qGkpqNe0topOuEXmxYzG2SGGl1v8EFp8NGjBa7tVZWALCV/nYZOCZj1a4/9JufojDTheDxOAWR5soOwcLDAffioOBqf/2nkbDLf77/rp4G1nf3fh9DJ5Rl6QLknJDtknR+SYDAkj38h38oP8bP2Knkcvo1fX0qi1qFkjNyJ6cwWgSbvw</latexit><latexit sha1_base64="Eo/hrDn/vOeKD+7qnaPujkDz/RU=">AAACdXicfVFdaxNBFJ2sVev6lbaPggzG2rTEuFuEVqFQaB/6IjZgbKEbwt3JzXbofDEzK4Ylv8Nf46v+hv4THzu7jWgteGHgzLnn3jv3TG4Edz5JLlvRnaW79+4vP4gfPnr85Gl7ZfWz06VlOGRaaHuag0PBFQ499wJPjUWQucCT/OKgzp98Qeu4Vp/8zOBIQqH4lDPwgRq3024mwZ8zEPSIDja7rpdpiQVsvt/LXGnGFWzMB4GFjd+JcbuT9JMm6G2QLkCHLOJ4vNJ6nU00KyUqzwQ4d5Ymxo8qsJ4zgfM4Kx0aYBdQ4FmACiS6UdXsNqfrgZnQqbbhKE8b9u+KCqSrN+jRAGqJa5CbybxHc9lctFGhUa26OctPd0cVV6b0qNj1qGkpqNe0topOuEXmxYzG2SGGl1v8EFp8NGjBa7tVZWALCV/nYZOCZj1a4/9JufojDTheDxOAWR5soOwcLDAffioOBqf/2nkbDLf77/rp4G1nf3fh9DJ5Rl6QLknJDtknR+SYDAkj38h38oP8bP2Knkcvo1fX0qi1qFkjNyJ6cwWgSbvw</latexit><latexit sha1_base64="Eo/hrDn/vOeKD+7qnaPujkDz/RU=">AAACdXicfVFdaxNBFJ2sVev6lbaPggzG2rTEuFuEVqFQaB/6IjZgbKEbwt3JzXbofDEzK4Ylv8Nf46v+hv4THzu7jWgteGHgzLnn3jv3TG4Edz5JLlvRnaW79+4vP4gfPnr85Gl7ZfWz06VlOGRaaHuag0PBFQ499wJPjUWQucCT/OKgzp98Qeu4Vp/8zOBIQqH4lDPwgRq3024mwZ8zEPSIDja7rpdpiQVsvt/LXGnGFWzMB4GFjd+JcbuT9JMm6G2QLkCHLOJ4vNJ6nU00KyUqzwQ4d5Ymxo8qsJ4zgfM4Kx0aYBdQ4FmACiS6UdXsNqfrgZnQqbbhKE8b9u+KCqSrN+jRAGqJa5CbybxHc9lctFGhUa26OctPd0cVV6b0qNj1qGkpqNe0topOuEXmxYzG2SGGl1v8EFp8NGjBa7tVZWALCV/nYZOCZj1a4/9JufojDTheDxOAWR5soOwcLDAffioOBqf/2nkbDLf77/rp4G1nf3fh9DJ5Rl6QLknJDtknR+SYDAkj38h38oP8bP2Knkcvo1fX0qi1qFkjNyJ6cwWgSbvw</latexit><latexit sha1_base64="Eo/hrDn/vOeKD+7qnaPujkDz/RU=">AAACdXicfVFdaxNBFJ2sVev6lbaPggzG2rTEuFuEVqFQaB/6IjZgbKEbwt3JzXbofDEzK4Ylv8Nf46v+hv4THzu7jWgteGHgzLnn3jv3TG4Edz5JLlvRnaW79+4vP4gfPnr85Gl7ZfWz06VlOGRaaHuag0PBFQ499wJPjUWQucCT/OKgzp98Qeu4Vp/8zOBIQqH4lDPwgRq3024mwZ8zEPSIDja7rpdpiQVsvt/LXGnGFWzMB4GFjd+JcbuT9JMm6G2QLkCHLOJ4vNJ6nU00KyUqzwQ4d5Ymxo8qsJ4zgfM4Kx0aYBdQ4FmACiS6UdXsNqfrgZnQqbbhKE8b9u+KCqSrN+jRAGqJa5CbybxHc9lctFGhUa26OctPd0cVV6b0qNj1qGkpqNe0topOuEXmxYzG2SGGl1v8EFp8NGjBa7tVZWALCV/nYZOCZj1a4/9JufojDTheDxOAWR5soOwcLDAffioOBqf/2nkbDLf77/rp4G1nf3fh9DJ5Rl6QLknJDtknR+SYDAkj38h38oP8bP2Knkcvo1fX0qi1qFkjNyJ6cwWgSbvw</latexit>

Envelope MOQ-Learning

…T








T (x⇤) 2 X⇤<latexit sha1_base64="5cn5svyxEGnOKqiUulmXO4q/nEg=">AAADZXicfVJdb9MwFPUaPkZgrAMEDzxgUU0aVVUlfD9OfEi8IIrUbpWWrnLc28yaY1u2W7WK8mt4hR/EL+BvYKeRoEzhSrk6vvdcn1zrpIozY6Po504ruHb9xs3dW+HtO3t399sH906MXGgKIyq51OOUGOBMwMgyy2GsNJA85XCaXr73/dMlaMOkGNq1gklOMsHmjBLrStP2oyQn9oISjodHq/PuM5wwgcfn3Wm7E/WjKvBVENegg+oYTA9ar5OZpIschKWcGHMWR8pOCqItoxzKMFkYUIRekgzOHBQkBzMpqg1KfOgqMzyX2n3C4qr690RBcuN/tIcd8BRTIbPO0x5O8+oglXAXedbW5Gojsa1v528nBRNqYUHQjfx8wbGV2D8SnjEN1PI1DpMP4LbR8Nld+0WBJlbqbpEQneVkVbrtMpz0sMf/ozLxh+pwE3VJdFn41ESgclkWPjURTOaUfGoigHYSPjURrOsPm9vKtSvLWFsMdNmss1JQM9O0+OiI4aF7VkI1c37A9IJoQq0zZuicFv/rq6vg5Hk/ftF/9fVl5/hd7bld9Bg9RUcoRm/QMfqEBmiEKCrRN/Qd/Wj9CvaCB8HDDbW1U8/cR1sRPPkNDNAaog==</latexit>

x⇤ 2 X⇤<latexit sha1_base64="6YrKkUFt8Z/j6x/7c+UY54vE+rs=">AAADWHicfZLfa9swEMfVeFtb71fbPe5FLBRGCMHptnaPZT9gL2MZNG2gTousXFxRWxLSOSQY/xt73f6t7a+Z5Bi2rHgHPr66+5zOOi7RmbAYRT+3OsG9+w+2d3bDh48eP3m6t39wblVhOIy5ypSZJMxCJiSMUWAGE22A5UkGF8nte5+/WICxQskzXGmY5iyVYi44QxeKl1c9GgtJJ1e98HqvGw2i2uhdMWxElzQ2ut7vHMczxYscJPKMWXs5jDROS2ZQ8AyqMC4saMZvWQqXTkqWg52W9U9X9NBFZnSujPsk0jr6d0XJcpszvOlTJzxia2VXedKnSV4flJbuIk9tVC7XLTb74/zttBRSFwiSr9vPi4yion4udCYMcMxWNIw/gHuNgc/u2i8aDENlemXMTJqzZeVel9K4T73+HyrkH9TpNnTBTFV61wZwtahK79oAm7pO3rUBYFwL79oAdPmz9rR26djPGLEcmaq9z1JDQyZJ+dGB4aEbK+NGuH2g/IYZxtHtot+04b97dVecHw2GrwZvvr7unr5rdm6HPCcvyEsyJCfklHwiIzImnGjyjXwnPzq/AhJsB7trtLPV1DwjGxYc/AYHERYS</latexit>


Value Metric:

Q = (⌦ ! Rm)S⇥A<latexit sha1_base64="ZEl+y2mCj4CIcSZ0UNpvtt49GQw=">AAACkHicfVFbaxNBFJ6st7pemtpHXw6GQpUYdkWwfRBb9EFEaavGFrppODs52QydyzIzq4Ylf8pfo4/6S5zdhmoteGDgm2++c/smL6VwPkl+dKIrV69dv7FyM751+87d1e7avU/OVJbTkBtp7FGOjqTQNPTCSzoqLaHKJR3mpy+b98PPZJ0w+qOflzRSWGgxFRx9oMbdt5lCP+Mo4QCew2a2p6hAyKwoZh6tNV+gFeQ5vD9RD0/qc/kHyLxQ5OCc2V2Mu71kkLQBl0G6BD22jP3xWudxNjG8UqQ9l+jccZqUflSj9YJLWsRZ5ahEfooFHQeoMXQc1e3aC9gIzASmxoajPbTs3xk1KtcM14cAGolrkZurvA+5ai+m1KFQo7rYy0+3RrXQZeVJ87NW00qCN9C4CBNhiXs5hzh7RWFyS+9Cib2SLHpjH9UZ2kLh10XYpICsDw3+n1ToP9KA443QAbkVwQbgM7TIffjEOBic/mvnZTB8MtgepAdPeztbS6dX2H32gG2ylD1jO+w122dDxtk39p39ZL+i9Wg7ehHtnkmjzjJnnV2I6M1vOk/G4A==</latexit><latexit sha1_base64="ZEl+y2mCj4CIcSZ0UNpvtt49GQw=">AAACkHicfVFbaxNBFJ6st7pemtpHXw6GQpUYdkWwfRBb9EFEaavGFrppODs52QydyzIzq4Ylf8pfo4/6S5zdhmoteGDgm2++c/smL6VwPkl+dKIrV69dv7FyM751+87d1e7avU/OVJbTkBtp7FGOjqTQNPTCSzoqLaHKJR3mpy+b98PPZJ0w+qOflzRSWGgxFRx9oMbdt5lCP+Mo4QCew2a2p6hAyKwoZh6tNV+gFeQ5vD9RD0/qc/kHyLxQ5OCc2V2Mu71kkLQBl0G6BD22jP3xWudxNjG8UqQ9l+jccZqUflSj9YJLWsRZ5ahEfooFHQeoMXQc1e3aC9gIzASmxoajPbTs3xk1KtcM14cAGolrkZurvA+5ai+m1KFQo7rYy0+3RrXQZeVJ87NW00qCN9C4CBNhiXs5hzh7RWFyS+9Cib2SLHpjH9UZ2kLh10XYpICsDw3+n1ToP9KA443QAbkVwQbgM7TIffjEOBic/mvnZTB8MtgepAdPeztbS6dX2H32gG2ylD1jO+w122dDxtk39p39ZL+i9Wg7ehHtnkmjzjJnnV2I6M1vOk/G4A==</latexit><latexit sha1_base64="ZEl+y2mCj4CIcSZ0UNpvtt49GQw=">AAACkHicfVFbaxNBFJ6st7pemtpHXw6GQpUYdkWwfRBb9EFEaavGFrppODs52QydyzIzq4Ylf8pfo4/6S5zdhmoteGDgm2++c/smL6VwPkl+dKIrV69dv7FyM751+87d1e7avU/OVJbTkBtp7FGOjqTQNPTCSzoqLaHKJR3mpy+b98PPZJ0w+qOflzRSWGgxFRx9oMbdt5lCP+Mo4QCew2a2p6hAyKwoZh6tNV+gFeQ5vD9RD0/qc/kHyLxQ5OCc2V2Mu71kkLQBl0G6BD22jP3xWudxNjG8UqQ9l+jccZqUflSj9YJLWsRZ5ahEfooFHQeoMXQc1e3aC9gIzASmxoajPbTs3xk1KtcM14cAGolrkZurvA+5ai+m1KFQo7rYy0+3RrXQZeVJ87NW00qCN9C4CBNhiXs5hzh7RWFyS+9Cib2SLHpjH9UZ2kLh10XYpICsDw3+n1ToP9KA443QAbkVwQbgM7TIffjEOBic/mvnZTB8MtgepAdPeztbS6dX2H32gG2ylD1jO+w122dDxtk39p39ZL+i9Wg7ehHtnkmjzjJnnV2I6M1vOk/G4A==</latexit><latexit sha1_base64="ZEl+y2mCj4CIcSZ0UNpvtt49GQw=">AAACkHicfVFbaxNBFJ6st7pemtpHXw6GQpUYdkWwfRBb9EFEaavGFrppODs52QydyzIzq4Ylf8pfo4/6S5zdhmoteGDgm2++c/smL6VwPkl+dKIrV69dv7FyM751+87d1e7avU/OVJbTkBtp7FGOjqTQNPTCSzoqLaHKJR3mpy+b98PPZJ0w+qOflzRSWGgxFRx9oMbdt5lCP+Mo4QCew2a2p6hAyKwoZh6tNV+gFeQ5vD9RD0/qc/kHyLxQ5OCc2V2Mu71kkLQBl0G6BD22jP3xWudxNjG8UqQ9l+jccZqUflSj9YJLWsRZ5ahEfooFHQeoMXQc1e3aC9gIzASmxoajPbTs3xk1KtcM14cAGolrkZurvA+5ai+m1KFQo7rYy0+3RrXQZeVJ87NW00qCN9C4CBNhiXs5hzh7RWFyS+9Cib2SLHpjH9UZ2kLh10XYpICsDw3+n1ToP9KA443QAbkVwQbgM7TIffjEOBic/mvnZTB8MtgepAdPeztbS6dX2H32gG2ylD1jO+w122dDxtk39p39ZL+i9Wg7ehHtnkmjzjJnnV2I6M1vOk/G4A==</latexit>

d(Q,Q0) := sups2S,a2A

!2⌦

|!|(Q(s, a,!)�Q0(s, a,!))|<latexit sha1_base64="gUGzftr0VYUvpdgwENC6j0q3CcA=">AAAC7XicfVFbb9MwFHbCbRTYuvHIyxHVRIuyKkGTQJOGxuWBF7RN0G3SXCrHdTNrvkS2g6iy/AzeEK/wl/g3OGmkdVTiSLY/n/Odiz+nueDWxfGfILx1+87de2v3Ow8ePlrf6G5unVhdGMpGVAttzlJimeCKjRx3gp3lhhGZCnaaXr6r46dfmbFcq89unrOxJJniM06J865J9/e0j1MJxxE0x7MB7O0DtkU+Kf2eWkfoZWkxV1gSd0GJgE8RkOX7G4ybXKwly5rIYQ2qCq6u3V9KH3DM+IQKFh371heKllIHsNMOsRoaXE26vXgYNwarIGlBD7V2NNkMdvBU00Iy5agg1p4nce7GJTGOU8GqDi4sy/3zSMbOPVREMjsuG0kr2PaeKcy08Us5aLzLGSWRtlbADyptTbENsnOZRpDK5qJz5QvVrJu93OzVuOQqLxxTdNFqVghwGuofgik3jDoxhw5+z/zkhn30JQ5zZojT5nmJickk+Vb5l2SAI6jx/6hcXVM97mz7DoQa7mUAekEMof5jbMcLnPwr5yo4eTFM4mFyvNs7eNtKvYaeoKeojxL0Eh2gD+gIjRAN1oPdYD94Herwe/gj/LmghkGb8xjdsPDXX2EV5Dc=</latexit><latexit sha1_base64="gUGzftr0VYUvpdgwENC6j0q3CcA=">AAAC7XicfVFbb9MwFHbCbRTYuvHIyxHVRIuyKkGTQJOGxuWBF7RN0G3SXCrHdTNrvkS2g6iy/AzeEK/wl/g3OGmkdVTiSLY/n/Odiz+nueDWxfGfILx1+87de2v3Ow8ePlrf6G5unVhdGMpGVAttzlJimeCKjRx3gp3lhhGZCnaaXr6r46dfmbFcq89unrOxJJniM06J865J9/e0j1MJxxE0x7MB7O0DtkU+Kf2eWkfoZWkxV1gSd0GJgE8RkOX7G4ybXKwly5rIYQ2qCq6u3V9KH3DM+IQKFh371heKllIHsNMOsRoaXE26vXgYNwarIGlBD7V2NNkMdvBU00Iy5agg1p4nce7GJTGOU8GqDi4sy/3zSMbOPVREMjsuG0kr2PaeKcy08Us5aLzLGSWRtlbADyptTbENsnOZRpDK5qJz5QvVrJu93OzVuOQqLxxTdNFqVghwGuofgik3jDoxhw5+z/zkhn30JQ5zZojT5nmJickk+Vb5l2SAI6jx/6hcXVM97mz7DoQa7mUAekEMof5jbMcLnPwr5yo4eTFM4mFyvNs7eNtKvYaeoKeojxL0Eh2gD+gIjRAN1oPdYD94Herwe/gj/LmghkGb8xjdsPDXX2EV5Dc=</latexit><latexit sha1_base64="gUGzftr0VYUvpdgwENC6j0q3CcA=">AAAC7XicfVFbb9MwFHbCbRTYuvHIyxHVRIuyKkGTQJOGxuWBF7RN0G3SXCrHdTNrvkS2g6iy/AzeEK/wl/g3OGmkdVTiSLY/n/Odiz+nueDWxfGfILx1+87de2v3Ow8ePlrf6G5unVhdGMpGVAttzlJimeCKjRx3gp3lhhGZCnaaXr6r46dfmbFcq89unrOxJJniM06J865J9/e0j1MJxxE0x7MB7O0DtkU+Kf2eWkfoZWkxV1gSd0GJgE8RkOX7G4ybXKwly5rIYQ2qCq6u3V9KH3DM+IQKFh371heKllIHsNMOsRoaXE26vXgYNwarIGlBD7V2NNkMdvBU00Iy5agg1p4nce7GJTGOU8GqDi4sy/3zSMbOPVREMjsuG0kr2PaeKcy08Us5aLzLGSWRtlbADyptTbENsnOZRpDK5qJz5QvVrJu93OzVuOQqLxxTdNFqVghwGuofgik3jDoxhw5+z/zkhn30JQ5zZojT5nmJickk+Vb5l2SAI6jx/6hcXVM97mz7DoQa7mUAekEMof5jbMcLnPwr5yo4eTFM4mFyvNs7eNtKvYaeoKeojxL0Eh2gD+gIjRAN1oPdYD94Herwe/gj/LmghkGb8xjdsPDXX2EV5Dc=</latexit><latexit sha1_base64="gUGzftr0VYUvpdgwENC6j0q3CcA=">AAAC7XicfVFbb9MwFHbCbRTYuvHIyxHVRIuyKkGTQJOGxuWBF7RN0G3SXCrHdTNrvkS2g6iy/AzeEK/wl/g3OGmkdVTiSLY/n/Odiz+nueDWxfGfILx1+87de2v3Ow8ePlrf6G5unVhdGMpGVAttzlJimeCKjRx3gp3lhhGZCnaaXr6r46dfmbFcq89unrOxJJniM06J865J9/e0j1MJxxE0x7MB7O0DtkU+Kf2eWkfoZWkxV1gSd0GJgE8RkOX7G4ybXKwly5rIYQ2qCq6u3V9KH3DM+IQKFh371heKllIHsNMOsRoaXE26vXgYNwarIGlBD7V2NNkMdvBU00Iy5agg1p4nce7GJTGOU8GqDi4sy/3zSMbOPVREMjsuG0kr2PaeKcy08Us5aLzLGSWRtlbADyptTbENsnOZRpDK5qJz5QvVrJu93OzVuOQqLxxTdNFqVghwGuofgik3jDoxhw5+z/zkhn30JQ5zZojT5nmJickk+Vb5l2SAI6jx/6hcXVM97mz7DoQa7mUAekEMof5jbMcLnPwr5yo4eTFM4mFyvNs7eNtKvYaeoKeojxL0Eh2gD+gIjRAN1oPdYD94Herwe/gj/LmghkGb8xjdsPDXX2EV5Dc=</latexit>

(Pseudo-metric)







Q(s, a5,!)<latexit sha1_base64="mUw9hAGbN9KmJ5/c4aBiz3nxck8=">AAACWnicfZFdSxwxFIazY6vuWO360StvQhfBynaZEUUvF+uFN0WFrgrOspzJnh2D+RiSjLgM+2O8bX+R0B9jZlxQK/RA4MnJe3JO3qS54NZF0WMjmPvwcX5hsRkufVpe+dxaXbuwujAM+0wLba5SsCi4wr7jTuBVbhBkKvAyvf1RnV/eobFcq19ukuNAQqb4mDNwPjVsfTnfth0Kw/0OTVJJEy0xg2/DVjvqRnXQ9xDPoE1mcTZcbXxPRpoVEpVjAqy9jqPcDUowjjOB0zApLObAbiHDa48KJNpBWc8/pVs+M6JjbfxSjtbZ1xUlSCvB3fhBpa0ktiY7kWmHprLe6Fz5iyrV215ufDgoucoLh4o9txoXgjpNKzvoiBtkTkxomByjn9zgT3/FaY4GnDY7ZQImk3A/9S/JaNKhFf9PytWL1HO45TsAM9zbQNkNGGDO/0boDY7/tfM9XOx246gbn++1e0czqxfJJvlKtklMDkiPnJAz0ieMlOSB/CZ/Gn+DIGgGS8/SoDGrWSdvIth4AqqKsUc=</latexit><latexit sha1_base64="mUw9hAGbN9KmJ5/c4aBiz3nxck8=">AAACWnicfZFdSxwxFIazY6vuWO360StvQhfBynaZEUUvF+uFN0WFrgrOspzJnh2D+RiSjLgM+2O8bX+R0B9jZlxQK/RA4MnJe3JO3qS54NZF0WMjmPvwcX5hsRkufVpe+dxaXbuwujAM+0wLba5SsCi4wr7jTuBVbhBkKvAyvf1RnV/eobFcq19ukuNAQqb4mDNwPjVsfTnfth0Kw/0OTVJJEy0xg2/DVjvqRnXQ9xDPoE1mcTZcbXxPRpoVEpVjAqy9jqPcDUowjjOB0zApLObAbiHDa48KJNpBWc8/pVs+M6JjbfxSjtbZ1xUlSCvB3fhBpa0ktiY7kWmHprLe6Fz5iyrV215ufDgoucoLh4o9txoXgjpNKzvoiBtkTkxomByjn9zgT3/FaY4GnDY7ZQImk3A/9S/JaNKhFf9PytWL1HO45TsAM9zbQNkNGGDO/0boDY7/tfM9XOx246gbn++1e0czqxfJJvlKtklMDkiPnJAz0ieMlOSB/CZ/Gn+DIGgGS8/SoDGrWSdvIth4AqqKsUc=</latexit><latexit sha1_base64="mUw9hAGbN9KmJ5/c4aBiz3nxck8=">AAACWnicfZFdSxwxFIazY6vuWO360StvQhfBynaZEUUvF+uFN0WFrgrOspzJnh2D+RiSjLgM+2O8bX+R0B9jZlxQK/RA4MnJe3JO3qS54NZF0WMjmPvwcX5hsRkufVpe+dxaXbuwujAM+0wLba5SsCi4wr7jTuBVbhBkKvAyvf1RnV/eobFcq19ukuNAQqb4mDNwPjVsfTnfth0Kw/0OTVJJEy0xg2/DVjvqRnXQ9xDPoE1mcTZcbXxPRpoVEpVjAqy9jqPcDUowjjOB0zApLObAbiHDa48KJNpBWc8/pVs+M6JjbfxSjtbZ1xUlSCvB3fhBpa0ktiY7kWmHprLe6Fz5iyrV215ufDgoucoLh4o9txoXgjpNKzvoiBtkTkxomByjn9zgT3/FaY4GnDY7ZQImk3A/9S/JaNKhFf9PytWL1HO45TsAM9zbQNkNGGDO/0boDY7/tfM9XOx246gbn++1e0czqxfJJvlKtklMDkiPnJAz0ieMlOSB/CZ/Gn+DIGgGS8/SoDGrWSdvIth4AqqKsUc=</latexit><latexit sha1_base64="mUw9hAGbN9KmJ5/c4aBiz3nxck8=">AAACWnicfZFdSxwxFIazY6vuWO360StvQhfBynaZEUUvF+uFN0WFrgrOspzJnh2D+RiSjLgM+2O8bX+R0B9jZlxQK/RA4MnJe3JO3qS54NZF0WMjmPvwcX5hsRkufVpe+dxaXbuwujAM+0wLba5SsCi4wr7jTuBVbhBkKvAyvf1RnV/eobFcq19ukuNAQqb4mDNwPjVsfTnfth0Kw/0OTVJJEy0xg2/DVjvqRnXQ9xDPoE1mcTZcbXxPRpoVEpVjAqy9jqPcDUowjjOB0zApLObAbiHDa48KJNpBWc8/pVs+M6JjbfxSjtbZ1xUlSCvB3fhBpa0ktiY7kWmHprLe6Fz5iyrV215ufDgoucoLh4o9txoXgjpNKzvoiBtkTkxomByjn9zgT3/FaY4GnDY7ZQImk3A/9S/JaNKhFf9PytWL1HO45TsAM9zbQNkNGGDO/0boDY7/tfM9XOx246gbn++1e0czqxfJJvlKtklMDkiPnJAz0ieMlOSB/CZ/Gn+DIGgGS8/SoDGrWSdvIth4AqqKsUc=</latexit>

Q<latexit sha1_base64="R587TJV12btdDxF7LQiGhMR6c1M=">AAACSHicfZDLSiNBFIaro46xdWa8LN0UBkEkE7oHQZdBXbgRFYwKdpDTlZNYpC5NVbUYmjyDW30l38C3mN3gzuo24A08UPDVqf9c6k8zwa2LoqegNjU982O2PhfOL/z89XtxafnM6tww7DAttLlIwaLgCjuOO4EXmUGQqcDzdLhXvp/foLFcq1M3yrArYaB4nzNwPtVJUklPrhYbUSuqgn6FeAINMonjq6XgT9LTLJeoHBNg7WUcZa5bgHGcCRyHSW4xAzaEAV56VCDRdotq2zFd95ke7Wvjj3K0yr6vKEBaCe66ST2UEluRHcm0SVNZXXSmfKNS9XGW6+90C66y3KFir6P6uaBO0/LztMcNMidGNEz20W9u8NC3OMrQgNNms0jADCTcjv1PBjRp0pK/k3L1JvUcrvsJwAz3NlB2DQaY896H3uD4s51f4exvK45a8clWo707sbpOVska2SAx2SZtckCOSYcwwskduScPwWPwL/gfPL9Ka8GkZoV8iFrtBSy/rr0=</latexit><latexit sha1_base64="R587TJV12btdDxF7LQiGhMR6c1M=">AAACSHicfZDLSiNBFIaro46xdWa8LN0UBkEkE7oHQZdBXbgRFYwKdpDTlZNYpC5NVbUYmjyDW30l38C3mN3gzuo24A08UPDVqf9c6k8zwa2LoqegNjU982O2PhfOL/z89XtxafnM6tww7DAttLlIwaLgCjuOO4EXmUGQqcDzdLhXvp/foLFcq1M3yrArYaB4nzNwPtVJUklPrhYbUSuqgn6FeAINMonjq6XgT9LTLJeoHBNg7WUcZa5bgHGcCRyHSW4xAzaEAV56VCDRdotq2zFd95ke7Wvjj3K0yr6vKEBaCe66ST2UEluRHcm0SVNZXXSmfKNS9XGW6+90C66y3KFir6P6uaBO0/LztMcNMidGNEz20W9u8NC3OMrQgNNms0jADCTcjv1PBjRp0pK/k3L1JvUcrvsJwAz3NlB2DQaY896H3uD4s51f4exvK45a8clWo707sbpOVska2SAx2SZtckCOSYcwwskduScPwWPwL/gfPL9Ka8GkZoV8iFrtBSy/rr0=</latexit><latexit sha1_base64="R587TJV12btdDxF7LQiGhMR6c1M=">AAACSHicfZDLSiNBFIaro46xdWa8LN0UBkEkE7oHQZdBXbgRFYwKdpDTlZNYpC5NVbUYmjyDW30l38C3mN3gzuo24A08UPDVqf9c6k8zwa2LoqegNjU982O2PhfOL/z89XtxafnM6tww7DAttLlIwaLgCjuOO4EXmUGQqcDzdLhXvp/foLFcq1M3yrArYaB4nzNwPtVJUklPrhYbUSuqgn6FeAINMonjq6XgT9LTLJeoHBNg7WUcZa5bgHGcCRyHSW4xAzaEAV56VCDRdotq2zFd95ke7Wvjj3K0yr6vKEBaCe66ST2UEluRHcm0SVNZXXSmfKNS9XGW6+90C66y3KFir6P6uaBO0/LztMcNMidGNEz20W9u8NC3OMrQgNNms0jADCTcjv1PBjRp0pK/k3L1JvUcrvsJwAz3NlB2DQaY896H3uD4s51f4exvK45a8clWo707sbpOVska2SAx2SZtckCOSYcwwskduScPwWPwL/gfPL9Ka8GkZoV8iFrtBSy/rr0=</latexit><latexit sha1_base64="R587TJV12btdDxF7LQiGhMR6c1M=">AAACSHicfZDLSiNBFIaro46xdWa8LN0UBkEkE7oHQZdBXbgRFYwKdpDTlZNYpC5NVbUYmjyDW30l38C3mN3gzuo24A08UPDVqf9c6k8zwa2LoqegNjU982O2PhfOL/z89XtxafnM6tww7DAttLlIwaLgCjuOO4EXmUGQqcDzdLhXvp/foLFcq1M3yrArYaB4nzNwPtVJUklPrhYbUSuqgn6FeAINMonjq6XgT9LTLJeoHBNg7WUcZa5bgHGcCRyHSW4xAzaEAV56VCDRdotq2zFd95ke7Wvjj3K0yr6vKEBaCe66ST2UEluRHcm0SVNZXXSmfKNS9XGW6+90C66y3KFir6P6uaBO0/LztMcNMidGNEz20W9u8NC3OMrQgNNms0jADCTcjv1PBjRp0pK/k3L1JvUcrvsJwAz3NlB2DQaY896H3uD4s51f4exvK45a8clWo707sbpOVska2SAx2SZtckCOSYcwwskduScPwWPwL/gfPL9Ka8GkZoV8iFrtBSy/rr0=</latexit>





Envelope MOQ-Network

Envelope MOQ-Learning

…T








T (x⇤) 2 X⇤<latexit sha1_base64="5cn5svyxEGnOKqiUulmXO4q/nEg=">AAADZXicfVJdb9MwFPUaPkZgrAMEDzxgUU0aVVUlfD9OfEi8IIrUbpWWrnLc28yaY1u2W7WK8mt4hR/EL+BvYKeRoEzhSrk6vvdcn1zrpIozY6Po504ruHb9xs3dW+HtO3t399sH906MXGgKIyq51OOUGOBMwMgyy2GsNJA85XCaXr73/dMlaMOkGNq1gklOMsHmjBLrStP2oyQn9oISjodHq/PuM5wwgcfn3Wm7E/WjKvBVENegg+oYTA9ar5OZpIschKWcGHMWR8pOCqItoxzKMFkYUIRekgzOHBQkBzMpqg1KfOgqMzyX2n3C4qr690RBcuN/tIcd8BRTIbPO0x5O8+oglXAXedbW5Gojsa1v528nBRNqYUHQjfx8wbGV2D8SnjEN1PI1DpMP4LbR8Nld+0WBJlbqbpEQneVkVbrtMpz0sMf/ozLxh+pwE3VJdFn41ESgclkWPjURTOaUfGoigHYSPjURrOsPm9vKtSvLWFsMdNmss1JQM9O0+OiI4aF7VkI1c37A9IJoQq0zZuicFv/rq6vg5Hk/ftF/9fVl5/hd7bld9Bg9RUcoRm/QMfqEBmiEKCrRN/Qd/Wj9CvaCB8HDDbW1U8/cR1sRPPkNDNAaog==</latexit>

x⇤ 2 X⇤<latexit sha1_base64="6YrKkUFt8Z/j6x/7c+UY54vE+rs=">AAADWHicfZLfa9swEMfVeFtb71fbPe5FLBRGCMHptnaPZT9gL2MZNG2gTousXFxRWxLSOSQY/xt73f6t7a+Z5Bi2rHgHPr66+5zOOi7RmbAYRT+3OsG9+w+2d3bDh48eP3m6t39wblVhOIy5ypSZJMxCJiSMUWAGE22A5UkGF8nte5+/WICxQskzXGmY5iyVYi44QxeKl1c9GgtJJ1e98HqvGw2i2uhdMWxElzQ2ut7vHMczxYscJPKMWXs5jDROS2ZQ8AyqMC4saMZvWQqXTkqWg52W9U9X9NBFZnSujPsk0jr6d0XJcpszvOlTJzxia2VXedKnSV4flJbuIk9tVC7XLTb74/zttBRSFwiSr9vPi4yion4udCYMcMxWNIw/gHuNgc/u2i8aDENlemXMTJqzZeVel9K4T73+HyrkH9TpNnTBTFV61wZwtahK79oAm7pO3rUBYFwL79oAdPmz9rR26djPGLEcmaq9z1JDQyZJ+dGB4aEbK+NGuH2g/IYZxtHtot+04b97dVecHw2GrwZvvr7unr5rdm6HPCcvyEsyJCfklHwiIzImnGjyjXwnPzq/AhJsB7trtLPV1DwjGxYc/AYHERYS</latexit>


Value Metric:


Q = (⌦ ! Rm)S⇥A<latexit sha1_base64="ZEl+y2mCj4CIcSZ0UNpvtt49GQw=">AAACkHicfVFbaxNBFJ6st7pemtpHXw6GQpUYdkWwfRBb9EFEaavGFrppODs52QydyzIzq4Ylf8pfo4/6S5zdhmoteGDgm2++c/smL6VwPkl+dKIrV69dv7FyM751+87d1e7avU/OVJbTkBtp7FGOjqTQNPTCSzoqLaHKJR3mpy+b98PPZJ0w+qOflzRSWGgxFRx9oMbdt5lCP+Mo4QCew2a2p6hAyKwoZh6tNV+gFeQ5vD9RD0/qc/kHyLxQ5OCc2V2Mu71kkLQBl0G6BD22jP3xWudxNjG8UqQ9l+jccZqUflSj9YJLWsRZ5ahEfooFHQeoMXQc1e3aC9gIzASmxoajPbTs3xk1KtcM14cAGolrkZurvA+5ai+m1KFQo7rYy0+3RrXQZeVJ87NW00qCN9C4CBNhiXs5hzh7RWFyS+9Cib2SLHpjH9UZ2kLh10XYpICsDw3+n1ToP9KA443QAbkVwQbgM7TIffjEOBic/mvnZTB8MtgepAdPeztbS6dX2H32gG2ylD1jO+w122dDxtk39p39ZL+i9Wg7ehHtnkmjzjJnnV2I6M1vOk/G4A==</latexit><latexit sha1_base64="ZEl+y2mCj4CIcSZ0UNpvtt49GQw=">AAACkHicfVFbaxNBFJ6st7pemtpHXw6GQpUYdkWwfRBb9EFEaavGFrppODs52QydyzIzq4Ylf8pfo4/6S5zdhmoteGDgm2++c/smL6VwPkl+dKIrV69dv7FyM751+87d1e7avU/OVJbTkBtp7FGOjqTQNPTCSzoqLaHKJR3mpy+b98PPZJ0w+qOflzRSWGgxFRx9oMbdt5lCP+Mo4QCew2a2p6hAyKwoZh6tNV+gFeQ5vD9RD0/qc/kHyLxQ5OCc2V2Mu71kkLQBl0G6BD22jP3xWudxNjG8UqQ9l+jccZqUflSj9YJLWsRZ5ahEfooFHQeoMXQc1e3aC9gIzASmxoajPbTs3xk1KtcM14cAGolrkZurvA+5ai+m1KFQo7rYy0+3RrXQZeVJ87NW00qCN9C4CBNhiXs5hzh7RWFyS+9Cib2SLHpjH9UZ2kLh10XYpICsDw3+n1ToP9KA443QAbkVwQbgM7TIffjEOBic/mvnZTB8MtgepAdPeztbS6dX2H32gG2ylD1jO+w122dDxtk39p39ZL+i9Wg7ehHtnkmjzjJnnV2I6M1vOk/G4A==</latexit><latexit sha1_base64="ZEl+y2mCj4CIcSZ0UNpvtt49GQw=">AAACkHicfVFbaxNBFJ6st7pemtpHXw6GQpUYdkWwfRBb9EFEaavGFrppODs52QydyzIzq4Ylf8pfo4/6S5zdhmoteGDgm2++c/smL6VwPkl+dKIrV69dv7FyM751+87d1e7avU/OVJbTkBtp7FGOjqTQNPTCSzoqLaHKJR3mpy+b98PPZJ0w+qOflzRSWGgxFRx9oMbdt5lCP+Mo4QCew2a2p6hAyKwoZh6tNV+gFeQ5vD9RD0/qc/kHyLxQ5OCc2V2Mu71kkLQBl0G6BD22jP3xWudxNjG8UqQ9l+jccZqUflSj9YJLWsRZ5ahEfooFHQeoMXQc1e3aC9gIzASmxoajPbTs3xk1KtcM14cAGolrkZurvA+5ai+m1KFQo7rYy0+3RrXQZeVJ87NW00qCN9C4CBNhiXs5hzh7RWFyS+9Cib2SLHpjH9UZ2kLh10XYpICsDw3+n1ToP9KA443QAbkVwQbgM7TIffjEOBic/mvnZTB8MtgepAdPeztbS6dX2H32gG2ylD1jO+w122dDxtk39p39ZL+i9Wg7ehHtnkmjzjJnnV2I6M1vOk/G4A==</latexit><latexit sha1_base64="ZEl+y2mCj4CIcSZ0UNpvtt49GQw=">AAACkHicfVFbaxNBFJ6st7pemtpHXw6GQpUYdkWwfRBb9EFEaavGFrppODs52QydyzIzq4Ylf8pfo4/6S5zdhmoteGDgm2++c/smL6VwPkl+dKIrV69dv7FyM751+87d1e7avU/OVJbTkBtp7FGOjqTQNPTCSzoqLaHKJR3mpy+b98PPZJ0w+qOflzRSWGgxFRx9oMbdt5lCP+Mo4QCew2a2p6hAyKwoZh6tNV+gFeQ5vD9RD0/qc/kHyLxQ5OCc2V2Mu71kkLQBl0G6BD22jP3xWudxNjG8UqQ9l+jccZqUflSj9YJLWsRZ5ahEfooFHQeoMXQc1e3aC9gIzASmxoajPbTs3xk1KtcM14cAGolrkZurvA+5ai+m1KFQo7rYy0+3RrXQZeVJ87NW00qCN9C4CBNhiXs5hzh7RWFyS+9Cib2SLHpjH9UZ2kLh10XYpICsDw3+n1ToP9KA443QAbkVwQbgM7TIffjEOBic/mvnZTB8MtgepAdPeztbS6dX2H32gG2ylD1jO+w122dDxtk39p39ZL+i9Wg7ehHtnkmjzjJnnV2I6M1vOk/G4A==</latexit>

d(Q,Q0) := sups2S,a2A

!2⌦

|!|(Q(s, a,!)�Q0(s, a,!))|<latexit sha1_base64="gUGzftr0VYUvpdgwENC6j0q3CcA=">AAAC7XicfVFbb9MwFHbCbRTYuvHIyxHVRIuyKkGTQJOGxuWBF7RN0G3SXCrHdTNrvkS2g6iy/AzeEK/wl/g3OGmkdVTiSLY/n/Odiz+nueDWxfGfILx1+87de2v3Ow8ePlrf6G5unVhdGMpGVAttzlJimeCKjRx3gp3lhhGZCnaaXr6r46dfmbFcq89unrOxJJniM06J865J9/e0j1MJxxE0x7MB7O0DtkU+Kf2eWkfoZWkxV1gSd0GJgE8RkOX7G4ybXKwly5rIYQ2qCq6u3V9KH3DM+IQKFh371heKllIHsNMOsRoaXE26vXgYNwarIGlBD7V2NNkMdvBU00Iy5agg1p4nce7GJTGOU8GqDi4sy/3zSMbOPVREMjsuG0kr2PaeKcy08Us5aLzLGSWRtlbADyptTbENsnOZRpDK5qJz5QvVrJu93OzVuOQqLxxTdNFqVghwGuofgik3jDoxhw5+z/zkhn30JQ5zZojT5nmJickk+Vb5l2SAI6jx/6hcXVM97mz7DoQa7mUAekEMof5jbMcLnPwr5yo4eTFM4mFyvNs7eNtKvYaeoKeojxL0Eh2gD+gIjRAN1oPdYD94Herwe/gj/LmghkGb8xjdsPDXX2EV5Dc=</latexit><latexit sha1_base64="gUGzftr0VYUvpdgwENC6j0q3CcA=">AAAC7XicfVFbb9MwFHbCbRTYuvHIyxHVRIuyKkGTQJOGxuWBF7RN0G3SXCrHdTNrvkS2g6iy/AzeEK/wl/g3OGmkdVTiSLY/n/Odiz+nueDWxfGfILx1+87de2v3Ow8ePlrf6G5unVhdGMpGVAttzlJimeCKjRx3gp3lhhGZCnaaXr6r46dfmbFcq89unrOxJJniM06J865J9/e0j1MJxxE0x7MB7O0DtkU+Kf2eWkfoZWkxV1gSd0GJgE8RkOX7G4ybXKwly5rIYQ2qCq6u3V9KH3DM+IQKFh371heKllIHsNMOsRoaXE26vXgYNwarIGlBD7V2NNkMdvBU00Iy5agg1p4nce7GJTGOU8GqDi4sy/3zSMbOPVREMjsuG0kr2PaeKcy08Us5aLzLGSWRtlbADyptTbENsnOZRpDK5qJz5QvVrJu93OzVuOQqLxxTdNFqVghwGuofgik3jDoxhw5+z/zkhn30JQ5zZojT5nmJickk+Vb5l2SAI6jx/6hcXVM97mz7DoQa7mUAekEMof5jbMcLnPwr5yo4eTFM4mFyvNs7eNtKvYaeoKeojxL0Eh2gD+gIjRAN1oPdYD94Herwe/gj/LmghkGb8xjdsPDXX2EV5Dc=</latexit><latexit sha1_base64="gUGzftr0VYUvpdgwENC6j0q3CcA=">AAAC7XicfVFbb9MwFHbCbRTYuvHIyxHVRIuyKkGTQJOGxuWBF7RN0G3SXCrHdTNrvkS2g6iy/AzeEK/wl/g3OGmkdVTiSLY/n/Odiz+nueDWxfGfILx1+87de2v3Ow8ePlrf6G5unVhdGMpGVAttzlJimeCKjRx3gp3lhhGZCnaaXr6r46dfmbFcq89unrOxJJniM06J865J9/e0j1MJxxE0x7MB7O0DtkU+Kf2eWkfoZWkxV1gSd0GJgE8RkOX7G4ybXKwly5rIYQ2qCq6u3V9KH3DM+IQKFh371heKllIHsNMOsRoaXE26vXgYNwarIGlBD7V2NNkMdvBU00Iy5agg1p4nce7GJTGOU8GqDi4sy/3zSMbOPVREMjsuG0kr2PaeKcy08Us5aLzLGSWRtlbADyptTbENsnOZRpDK5qJz5QvVrJu93OzVuOQqLxxTdNFqVghwGuofgik3jDoxhw5+z/zkhn30JQ5zZojT5nmJickk+Vb5l2SAI6jx/6hcXVM97mz7DoQa7mUAekEMof5jbMcLnPwr5yo4eTFM4mFyvNs7eNtKvYaeoKeojxL0Eh2gD+gIjRAN1oPdYD94Herwe/gj/LmghkGb8xjdsPDXX2EV5Dc=</latexit><latexit sha1_base64="gUGzftr0VYUvpdgwENC6j0q3CcA=">AAAC7XicfVFbb9MwFHbCbRTYuvHIyxHVRIuyKkGTQJOGxuWBF7RN0G3SXCrHdTNrvkS2g6iy/AzeEK/wl/g3OGmkdVTiSLY/n/Odiz+nueDWxfGfILx1+87de2v3Ow8ePlrf6G5unVhdGMpGVAttzlJimeCKjRx3gp3lhhGZCnaaXr6r46dfmbFcq89unrOxJJniM06J865J9/e0j1MJxxE0x7MB7O0DtkU+Kf2eWkfoZWkxV1gSd0GJgE8RkOX7G4ybXKwly5rIYQ2qCq6u3V9KH3DM+IQKFh371heKllIHsNMOsRoaXE26vXgYNwarIGlBD7V2NNkMdvBU00Iy5agg1p4nce7GJTGOU8GqDi4sy/3zSMbOPVREMjsuG0kr2PaeKcy08Us5aLzLGSWRtlbADyptTbENsnOZRpDK5qJz5QvVrJu93OzVuOQqLxxTdNFqVghwGuofgik3jDoxhw5+z/zkhn30JQ5zZojT5nmJickk+Vb5l2SAI6jx/6hcXVM97mz7DoQa7mUAekEMof5jbMcLnPwr5yo4eTFM4mFyvNs7eNtKvYaeoKeojxL0Eh2gD+gIjRAN1oPdYD94Herwe/gj/LmghkGb8xjdsPDXX2EV5Dc=</latexit>

a generalized contraction with a fixed-point class [Q⇤]

<latexit sha1_base64="ER51OWVAS08VIev0jtr9xR1K0uE=">AAACR3icfZDLShxBFIarx0RNG42XZTZFBkFkHLqDoO4Es3AjKqSjMN2R0zVnxmLq0lRVi0Mzz+A2vpJv4Fu4EpepbgfUCDlQ8NWp/1zqzwvBrYui+6A18+Hj7Nz8p3Dh8+LSl+WV1V9Wl4ZhwrTQ5jwHi4IrTBx3As8LgyBzgWf56KB+P7tCY7lWP924wEzCUPEBZ+B8Kumd/t7MLpbbUTdqgr6HeAptMo2Ti5VgK+1rVkpUjgmwthdHhcsqMI4zgZMwLS0WwEYwxJ5HBRJtVjXbTui6z/TpQBt/lKNN9nVFBdJKcJcd6qGW2IbsWOYdmsvmogvlG9Wqt7PcYDeruCpKh4o9jxqUgjpN68/TPjfInBjTMP2BfnODR77FcYEGnDabVQpmKOF64n8ypGmH1vw/KVcvUs/hup8AzHBvA2WXYIA5733oDY7/tfM9JN+7e934dLu9vzt1ep58Jd/IBonJDtknh+SEJIQRTm7IH3Ib3AUPwWPw9CxtBdOaNfImWsFfYmSufA==</latexit><latexit sha1_base64="ER51OWVAS08VIev0jtr9xR1K0uE=">AAACR3icfZDLShxBFIarx0RNG42XZTZFBkFkHLqDoO4Es3AjKqSjMN2R0zVnxmLq0lRVi0Mzz+A2vpJv4Fu4EpepbgfUCDlQ8NWp/1zqzwvBrYui+6A18+Hj7Nz8p3Dh8+LSl+WV1V9Wl4ZhwrTQ5jwHi4IrTBx3As8LgyBzgWf56KB+P7tCY7lWP924wEzCUPEBZ+B8Kumd/t7MLpbbUTdqgr6HeAptMo2Ti5VgK+1rVkpUjgmwthdHhcsqMI4zgZMwLS0WwEYwxJ5HBRJtVjXbTui6z/TpQBt/lKNN9nVFBdJKcJcd6qGW2IbsWOYdmsvmogvlG9Wqt7PcYDeruCpKh4o9jxqUgjpN68/TPjfInBjTMP2BfnODR77FcYEGnDabVQpmKOF64n8ypGmH1vw/KVcvUs/hup8AzHBvA2WXYIA5733oDY7/tfM9JN+7e934dLu9vzt1ep58Jd/IBonJDtknh+SEJIQRTm7IH3Ib3AUPwWPw9CxtBdOaNfImWsFfYmSufA==</latexit><latexit sha1_base64="ER51OWVAS08VIev0jtr9xR1K0uE=">AAACR3icfZDLShxBFIarx0RNG42XZTZFBkFkHLqDoO4Es3AjKqSjMN2R0zVnxmLq0lRVi0Mzz+A2vpJv4Fu4EpepbgfUCDlQ8NWp/1zqzwvBrYui+6A18+Hj7Nz8p3Dh8+LSl+WV1V9Wl4ZhwrTQ5jwHi4IrTBx3As8LgyBzgWf56KB+P7tCY7lWP924wEzCUPEBZ+B8Kumd/t7MLpbbUTdqgr6HeAptMo2Ti5VgK+1rVkpUjgmwthdHhcsqMI4zgZMwLS0WwEYwxJ5HBRJtVjXbTui6z/TpQBt/lKNN9nVFBdJKcJcd6qGW2IbsWOYdmsvmogvlG9Wqt7PcYDeruCpKh4o9jxqUgjpN68/TPjfInBjTMP2BfnODR77FcYEGnDabVQpmKOF64n8ypGmH1vw/KVcvUs/hup8AzHBvA2WXYIA5733oDY7/tfM9JN+7e934dLu9vzt1ep58Jd/IBonJDtknh+SEJIQRTm7IH3Ib3AUPwWPw9CxtBdOaNfImWsFfYmSufA==</latexit><latexit sha1_base64="ER51OWVAS08VIev0jtr9xR1K0uE=">AAACR3icfZDLShxBFIarx0RNG42XZTZFBkFkHLqDoO4Es3AjKqSjMN2R0zVnxmLq0lRVi0Mzz+A2vpJv4Fu4EpepbgfUCDlQ8NWp/1zqzwvBrYui+6A18+Hj7Nz8p3Dh8+LSl+WV1V9Wl4ZhwrTQ5jwHi4IrTBx3As8LgyBzgWf56KB+P7tCY7lWP924wEzCUPEBZ+B8Kumd/t7MLpbbUTdqgr6HeAptMo2Ti5VgK+1rVkpUjgmwthdHhcsqMI4zgZMwLS0WwEYwxJ5HBRJtVjXbTui6z/TpQBt/lKNN9nVFBdJKcJcd6qGW2IbsWOYdmsvmogvlG9Wqt7PcYDeruCpKh4o9jxqUgjpN68/TPjfInBjTMP2BfnODR77FcYEGnDabVQpmKOF64n8ypGmH1vw/KVcvUs/hup8AzHBvA2WXYIA5733oDY7/tfM9JN+7e934dLu9vzt1ep58Jd/IBonJDtknh+SEJIQRTm7IH3Ib3AUPwWPw9CxtBdOaNfImWsFfYmSufA==</latexit> (HQ)(s,!) := argQ supa0,!0

!|Q(s, a0,!0)<latexit sha1_base64="d5uaXUImBj5ki129FQC0YqFMxy8=">AAACsnicfZDfaxNBEMf3rlXr+SvVR18WQ2kiMdyJYhWEgj70RWzB2EI2hrnN5Lp0fxy7e2I47r/0xX/FJ/cuQY0FBxa+O/uZmZ1vXkrhfJr+iOKd3Rs3b+3dTu7cvXf/QW//4WdnKstxwo009iIHh1JonHjhJV6UFkHlEs/zq3ft+/lXtE4Y/cmvSpwpKLRYCg4+pOY9PWAK/CUHSU8oyxU9Gw7cKAhmFBYwfPOWgS3mdffUMFeV8xoOW4CuicPmN/ylZkJ7tKFZ0/Gh0zY7nPf66Tjtgl4X2Ub0ySZO5/vRM7YwvFKoPZfg3DRLSz+rwXrBJTYJqxyWwK+gwGmQGhS6Wd0Z09CDkFnQpbHhaE+77N8VNSjXrj+iQbSI65RbqXxEc9VdTKlDo5banuWXR7Na6LLyqPl61LKS1Bva+kwXwiL3ckUT9h7Dzy1+CC0+lmjBG/u0bm1V8K0JmxSUjWir/4cK/QcNOjkIE4BbEWyg/BIs8OC8S4LB2b92XheT5+PX4+zsRf/4aOP0HnlMnpABycgrckxOyCmZEE6+k5/RTrQbv4ynMcR8jcbRpuYR2YpY/gKbytGo</latexit><latexit sha1_base64="d5uaXUImBj5ki129FQC0YqFMxy8=">AAACsnicfZDfaxNBEMf3rlXr+SvVR18WQ2kiMdyJYhWEgj70RWzB2EI2hrnN5Lp0fxy7e2I47r/0xX/FJ/cuQY0FBxa+O/uZmZ1vXkrhfJr+iOKd3Rs3b+3dTu7cvXf/QW//4WdnKstxwo009iIHh1JonHjhJV6UFkHlEs/zq3ft+/lXtE4Y/cmvSpwpKLRYCg4+pOY9PWAK/CUHSU8oyxU9Gw7cKAhmFBYwfPOWgS3mdffUMFeV8xoOW4CuicPmN/ylZkJ7tKFZ0/Gh0zY7nPf66Tjtgl4X2Ub0ySZO5/vRM7YwvFKoPZfg3DRLSz+rwXrBJTYJqxyWwK+gwGmQGhS6Wd0Z09CDkFnQpbHhaE+77N8VNSjXrj+iQbSI65RbqXxEc9VdTKlDo5banuWXR7Na6LLyqPl61LKS1Bva+kwXwiL3ckUT9h7Dzy1+CC0+lmjBG/u0bm1V8K0JmxSUjWir/4cK/QcNOjkIE4BbEWyg/BIs8OC8S4LB2b92XheT5+PX4+zsRf/4aOP0HnlMnpABycgrckxOyCmZEE6+k5/RTrQbv4ynMcR8jcbRpuYR2YpY/gKbytGo</latexit><latexit sha1_base64="d5uaXUImBj5ki129FQC0YqFMxy8=">AAACsnicfZDfaxNBEMf3rlXr+SvVR18WQ2kiMdyJYhWEgj70RWzB2EI2hrnN5Lp0fxy7e2I47r/0xX/FJ/cuQY0FBxa+O/uZmZ1vXkrhfJr+iOKd3Rs3b+3dTu7cvXf/QW//4WdnKstxwo009iIHh1JonHjhJV6UFkHlEs/zq3ft+/lXtE4Y/cmvSpwpKLRYCg4+pOY9PWAK/CUHSU8oyxU9Gw7cKAhmFBYwfPOWgS3mdffUMFeV8xoOW4CuicPmN/ylZkJ7tKFZ0/Gh0zY7nPf66Tjtgl4X2Ub0ySZO5/vRM7YwvFKoPZfg3DRLSz+rwXrBJTYJqxyWwK+gwGmQGhS6Wd0Z09CDkFnQpbHhaE+77N8VNSjXrj+iQbSI65RbqXxEc9VdTKlDo5banuWXR7Na6LLyqPl61LKS1Bva+kwXwiL3ckUT9h7Dzy1+CC0+lmjBG/u0bm1V8K0JmxSUjWir/4cK/QcNOjkIE4BbEWyg/BIs8OC8S4LB2b92XheT5+PX4+zsRf/4aOP0HnlMnpABycgrckxOyCmZEE6+k5/RTrQbv4ynMcR8jcbRpuYR2YpY/gKbytGo</latexit><latexit sha1_base64="d5uaXUImBj5ki129FQC0YqFMxy8=">AAACsnicfZDfaxNBEMf3rlXr+SvVR18WQ2kiMdyJYhWEgj70RWzB2EI2hrnN5Lp0fxy7e2I47r/0xX/FJ/cuQY0FBxa+O/uZmZ1vXkrhfJr+iOKd3Rs3b+3dTu7cvXf/QW//4WdnKstxwo009iIHh1JonHjhJV6UFkHlEs/zq3ft+/lXtE4Y/cmvSpwpKLRYCg4+pOY9PWAK/CUHSU8oyxU9Gw7cKAhmFBYwfPOWgS3mdffUMFeV8xoOW4CuicPmN/ylZkJ7tKFZ0/Gh0zY7nPf66Tjtgl4X2Ub0ySZO5/vRM7YwvFKoPZfg3DRLSz+rwXrBJTYJqxyWwK+gwGmQGhS6Wd0Z09CDkFnQpbHhaE+77N8VNSjXrj+iQbSI65RbqXxEc9VdTKlDo5banuWXR7Na6LLyqPl61LKS1Bva+kwXwiL3ckUT9h7Dzy1+CC0+lmjBG/u0bm1V8K0JmxSUjWir/4cK/QcNOjkIE4BbEWyg/BIs8OC8S4LB2b92XheT5+PX4+zsRf/4aOP0HnlMnpABycgrckxOyCmZEE6+k5/RTrQbv4ynMcR8jcbRpuYR2YpY/gKbytGo</latexit>

Optimality Filter

(T Q)(s, a,!) := r(s, a) + �Es0⇠P(·|s,a)(HQ)(s0,!)<latexit sha1_base64="WFfqjoKVVsHwnhVl3tRmURlGPyM=">AAACzXicfVJraxQxFM2Mrzq+tuo3US4upbt1XGZEUASh+IB+kW6h2xaaZbmTzU5DJ8mQZMRlun7Vv+if8DeYmS6ta8ELgXNvzr0nOUlWFsK6JPkVhNeu37h5a+12dOfuvfsPOusPD6yuDOMjpgttjjK0vBCKj5xwBT8qDUeZFfwwO/3Y7B9+5cYKrfbdvORjibkSM8HQ+dKk87NHJboThgXsA80k7PV7NgaM24RqyXPsw7v3bWrarT68AJqjlAhtb5bB50ltN6kV8mLYsEfZVDs4Axtjf3GpsnOhsrmiMel0k0HSBlwF6RJ0yTKGk/XgJZ1qVkmuHCvQ2uM0Kd24RuMEK/giopXlJbJTzPmxhwolt+O6tWwBG74yhZk2fikHbfXvjhqlbY7s7yttQ7EtsnOZxZDJNtGl8oMa1qqWm70d10KVleOKnUvNqgKchuYFYCoMZ66YQ0Q/cX9yw7/4EbslN+i02aopmlzit4W/SQ40hgb/jyrUJdXjaMMrIDPC2wDsBA0y5z9A5A1O/7XzKjh4NUiTQbr3urv9YWn1GnlCnpMeSckbsk12yJCMCCO/g8fB0+BZuBtW4Vn4/ZwaBsueR2Qlwh9/ANui1no=</latexit><latexit sha1_base64="WFfqjoKVVsHwnhVl3tRmURlGPyM=">AAACzXicfVJraxQxFM2Mrzq+tuo3US4upbt1XGZEUASh+IB+kW6h2xaaZbmTzU5DJ8mQZMRlun7Vv+if8DeYmS6ta8ELgXNvzr0nOUlWFsK6JPkVhNeu37h5a+12dOfuvfsPOusPD6yuDOMjpgttjjK0vBCKj5xwBT8qDUeZFfwwO/3Y7B9+5cYKrfbdvORjibkSM8HQ+dKk87NHJboThgXsA80k7PV7NgaM24RqyXPsw7v3bWrarT68AJqjlAhtb5bB50ltN6kV8mLYsEfZVDs4Axtjf3GpsnOhsrmiMel0k0HSBlwF6RJ0yTKGk/XgJZ1qVkmuHCvQ2uM0Kd24RuMEK/giopXlJbJTzPmxhwolt+O6tWwBG74yhZk2fikHbfXvjhqlbY7s7yttQ7EtsnOZxZDJNtGl8oMa1qqWm70d10KVleOKnUvNqgKchuYFYCoMZ66YQ0Q/cX9yw7/4EbslN+i02aopmlzit4W/SQ40hgb/jyrUJdXjaMMrIDPC2wDsBA0y5z9A5A1O/7XzKjh4NUiTQbr3urv9YWn1GnlCnpMeSckbsk12yJCMCCO/g8fB0+BZuBtW4Vn4/ZwaBsueR2Qlwh9/ANui1no=</latexit><latexit sha1_base64="WFfqjoKVVsHwnhVl3tRmURlGPyM=">AAACzXicfVJraxQxFM2Mrzq+tuo3US4upbt1XGZEUASh+IB+kW6h2xaaZbmTzU5DJ8mQZMRlun7Vv+if8DeYmS6ta8ELgXNvzr0nOUlWFsK6JPkVhNeu37h5a+12dOfuvfsPOusPD6yuDOMjpgttjjK0vBCKj5xwBT8qDUeZFfwwO/3Y7B9+5cYKrfbdvORjibkSM8HQ+dKk87NHJboThgXsA80k7PV7NgaM24RqyXPsw7v3bWrarT68AJqjlAhtb5bB50ltN6kV8mLYsEfZVDs4Axtjf3GpsnOhsrmiMel0k0HSBlwF6RJ0yTKGk/XgJZ1qVkmuHCvQ2uM0Kd24RuMEK/giopXlJbJTzPmxhwolt+O6tWwBG74yhZk2fikHbfXvjhqlbY7s7yttQ7EtsnOZxZDJNtGl8oMa1qqWm70d10KVleOKnUvNqgKchuYFYCoMZ66YQ0Q/cX9yw7/4EbslN+i02aopmlzit4W/SQ40hgb/jyrUJdXjaMMrIDPC2wDsBA0y5z9A5A1O/7XzKjh4NUiTQbr3urv9YWn1GnlCnpMeSckbsk12yJCMCCO/g8fB0+BZuBtW4Vn4/ZwaBsueR2Qlwh9/ANui1no=</latexit><latexit sha1_base64="WFfqjoKVVsHwnhVl3tRmURlGPyM=">AAACzXicfVJraxQxFM2Mrzq+tuo3US4upbt1XGZEUASh+IB+kW6h2xaaZbmTzU5DJ8mQZMRlun7Vv+if8DeYmS6ta8ELgXNvzr0nOUlWFsK6JPkVhNeu37h5a+12dOfuvfsPOusPD6yuDOMjpgttjjK0vBCKj5xwBT8qDUeZFfwwO/3Y7B9+5cYKrfbdvORjibkSM8HQ+dKk87NHJboThgXsA80k7PV7NgaM24RqyXPsw7v3bWrarT68AJqjlAhtb5bB50ltN6kV8mLYsEfZVDs4Axtjf3GpsnOhsrmiMel0k0HSBlwF6RJ0yTKGk/XgJZ1qVkmuHCvQ2uM0Kd24RuMEK/giopXlJbJTzPmxhwolt+O6tWwBG74yhZk2fikHbfXvjhqlbY7s7yttQ7EtsnOZxZDJNtGl8oMa1qqWm70d10KVleOKnUvNqgKchuYFYCoMZ66YQ0Q/cX9yw7/4EbslN+i02aopmlzit4W/SQ40hgb/jyrUJdXjaMMrIDPC2wDsBA0y5z9A5A1O/7XzKjh4NUiTQbr3urv9YWn1GnlCnpMeSckbsk12yJCMCCO/g8fB0+BZuBtW4Vn4/ZwaBsueR2Qlwh9/ANui1no=</latexit>

(Pseudo-metric)

AB

C

D

E

F

G

H

K

L


QU

ANTI

TY O

F O

BJEC

TIVE

2 CCS

Non-optimal

Pareto Frontier

F


QU

ANTI

TY O

F O

BJEC

TIVE

2 F Non-preferred

Preference




a. b.

DUtility Projection L


QU

ANTI

TY O

F O

BJEC

TIVE

2

D

F




D F

Optimal Solutions

Sampled Preferences

c.

Advantage of “Envelope” Filter: faster alignment

(HQ)(s,!) := argQ supa0,!0


Maintaining the envelope

allows our method to quickly align one preference with optimal rewards and trajectories that have been explored under other preferences.

1. 2.

Enve

lope

Q-u

pdat

eEx

plor

atio

n


(HQ)(s,!) := argQ supa0,!0


Optimality Filter

(T Q)(s, a,!) := r(s, a) + �Es0⇠P(·|s,a)(HQ)(s0,!)<latexit sha1_base64="WFfqjoKVVsHwnhVl3tRmURlGPyM=">AAACzXicfVJraxQxFM2Mrzq+tuo3US4upbt1XGZEUASh+IB+kW6h2xaaZbmTzU5DJ8mQZMRlun7Vv+if8DeYmS6ta8ELgXNvzr0nOUlWFsK6JPkVhNeu37h5a+12dOfuvfsPOusPD6yuDOMjpgttjjK0vBCKj5xwBT8qDUeZFfwwO/3Y7B9+5cYKrfbdvORjibkSM8HQ+dKk87NHJboThgXsA80k7PV7NgaM24RqyXPsw7v3bWrarT68AJqjlAhtb5bB50ltN6kV8mLYsEfZVDs4Axtjf3GpsnOhsrmiMel0k0HSBlwF6RJ0yTKGk/XgJZ1qVkmuHCvQ2uM0Kd24RuMEK/giopXlJbJTzPmxhwolt+O6tWwBG74yhZk2fikHbfXvjhqlbY7s7yttQ7EtsnOZxZDJNtGl8oMa1qqWm70d10KVleOKnUvNqgKchuYFYCoMZ66YQ0Q/cX9yw7/4EbslN+i02aopmlzit4W/SQ40hgb/jyrUJdXjaMMrIDPC2wDsBA0y5z9A5A1O/7XzKjh4NUiTQbr3urv9YWn1GnlCnpMeSckbsk12yJCMCCO/g8fB0+BZuBtW4Vn4/ZwaBsueR2Qlwh9/ANui1no=</latexit><latexit sha1_base64="WFfqjoKVVsHwnhVl3tRmURlGPyM=">AAACzXicfVJraxQxFM2Mrzq+tuo3US4upbt1XGZEUASh+IB+kW6h2xaaZbmTzU5DJ8mQZMRlun7Vv+if8DeYmS6ta8ELgXNvzr0nOUlWFsK6JPkVhNeu37h5a+12dOfuvfsPOusPD6yuDOMjpgttjjK0vBCKj5xwBT8qDUeZFfwwO/3Y7B9+5cYKrfbdvORjibkSM8HQ+dKk87NHJboThgXsA80k7PV7NgaM24RqyXPsw7v3bWrarT68AJqjlAhtb5bB50ltN6kV8mLYsEfZVDs4Axtjf3GpsnOhsrmiMel0k0HSBlwF6RJ0yTKGk/XgJZ1qVkmuHCvQ2uM0Kd24RuMEK/giopXlJbJTzPmxhwolt+O6tWwBG74yhZk2fikHbfXvjhqlbY7s7yttQ7EtsnOZxZDJNtGl8oMa1qqWm70d10KVleOKnUvNqgKchuYFYCoMZ66YQ0Q/cX9yw7/4EbslN+i02aopmlzit4W/SQ40hgb/jyrUJdXjaMMrIDPC2wDsBA0y5z9A5A1O/7XzKjh4NUiTQbr3urv9YWn1GnlCnpMeSckbsk12yJCMCCO/g8fB0+BZuBtW4Vn4/ZwaBsueR2Qlwh9/ANui1no=</latexit><latexit sha1_base64="WFfqjoKVVsHwnhVl3tRmURlGPyM=">AAACzXicfVJraxQxFM2Mrzq+tuo3US4upbt1XGZEUASh+IB+kW6h2xaaZbmTzU5DJ8mQZMRlun7Vv+if8DeYmS6ta8ELgXNvzr0nOUlWFsK6JPkVhNeu37h5a+12dOfuvfsPOusPD6yuDOMjpgttjjK0vBCKj5xwBT8qDUeZFfwwO/3Y7B9+5cYKrfbdvORjibkSM8HQ+dKk87NHJboThgXsA80k7PV7NgaM24RqyXPsw7v3bWrarT68AJqjlAhtb5bB50ltN6kV8mLYsEfZVDs4Axtjf3GpsnOhsrmiMel0k0HSBlwF6RJ0yTKGk/XgJZ1qVkmuHCvQ2uM0Kd24RuMEK/giopXlJbJTzPmxhwolt+O6tWwBG74yhZk2fikHbfXvjhqlbY7s7yttQ7EtsnOZxZDJNtGl8oMa1qqWm70d10KVleOKnUvNqgKchuYFYCoMZ66YQ0Q/cX9yw7/4EbslN+i02aopmlzit4W/SQ40hgb/jyrUJdXjaMMrIDPC2wDsBA0y5z9A5A1O/7XzKjh4NUiTQbr3urv9YWn1GnlCnpMeSckbsk12yJCMCCO/g8fB0+BZuBtW4Vn4/ZwaBsueR2Qlwh9/ANui1no=</latexit><latexit sha1_base64="WFfqjoKVVsHwnhVl3tRmURlGPyM=">AAACzXicfVJraxQxFM2Mrzq+tuo3US4upbt1XGZEUASh+IB+kW6h2xaaZbmTzU5DJ8mQZMRlun7Vv+if8DeYmS6ta8ELgXNvzr0nOUlWFsK6JPkVhNeu37h5a+12dOfuvfsPOusPD6yuDOMjpgttjjK0vBCKj5xwBT8qDUeZFfwwO/3Y7B9+5cYKrfbdvORjibkSM8HQ+dKk87NHJboThgXsA80k7PV7NgaM24RqyXPsw7v3bWrarT68AJqjlAhtb5bB50ltN6kV8mLYsEfZVDs4Axtjf3GpsnOhsrmiMel0k0HSBlwF6RJ0yTKGk/XgJZ1qVkmuHCvQ2uM0Kd24RuMEK/giopXlJbJTzPmxhwolt+O6tWwBG74yhZk2fikHbfXvjhqlbY7s7yttQ7EtsnOZxZDJNtGl8oMa1qqWm70d10KVleOKnUvNqgKchuYFYCoMZ66YQ0Q/cX9yw7/4EbslN+i02aopmlzit4W/SQ40hgb/jyrUJdXjaMMrIDPC2wDsBA0y5z9A5A1O/7XzKjh4NUiTQbr3urv9YWn1GnlCnpMeSckbsk12yJCMCCO/g8fB0+BZuBtW4Vn4/ZwaBsueR2Qlwh9/ANui1no=</latexit>

...

...

+ Loss Functions:

LAk(✓) = Es,a,!

hkyk �Q(s, a,!; ✓)k22

i

<latexit sha1_base64="qDnv4NRDnuL32pHOCPGDIjmXP/k=">AAACzHicfVFti9NAEN7Et7O+9byPKgyWgyq9khRBQYTzDfzgyx3Yu4NuL2y2k3RpNgm7U7HEfNXf6I/wP7hJK3p34MCyz8w8M7PzbFxmylIQ/PT8S5evXL22db1z4+at23e623ePbLE0EseyyApzEguLmcpxTIoyPCkNCh1neBwvXjf54y9orCryz7QqcapFmqtESUEuFHV/vD+tuBY0J4KXdVQtauhzmiOJR/AC2kwcw9uosgMQA+CxBl5oTEUNPMOEJuuLf2tTq2gBey067J8veA5/+nKj0rkriUano40zjbq9YBi0BhdBuAE9trGDaNvb47NCLjXmJDNh7SQMSppWwpCSGdYdvrRYCrkQKU4czIVGO61axWrYdZEZJIVxJydoo/9WVELbZne3g7YNxbbIrnQ8gFi3TlHmrlHDOjuLkmfTSuXlkjCX61HJMgMqoPkAmCmDkrIVdPgbdC83+MG1+FSiEVSYxxUXJtXia+02SYEPoMH/o6r8L9Xhzq6bIKRRTgaQc2GEJPf/HSdweF7Oi+BoNAyDYXj4pLf/aiP1FrvHHrI+C9lTts/esQM2ZpL98na8+94D/6NPfuXXa6rvbWp22Bnzv/8Gk+/ZHg==</latexit><latexit sha1_base64="qDnv4NRDnuL32pHOCPGDIjmXP/k=">AAACzHicfVFti9NAEN7Et7O+9byPKgyWgyq9khRBQYTzDfzgyx3Yu4NuL2y2k3RpNgm7U7HEfNXf6I/wP7hJK3p34MCyz8w8M7PzbFxmylIQ/PT8S5evXL22db1z4+at23e623ePbLE0EseyyApzEguLmcpxTIoyPCkNCh1neBwvXjf54y9orCryz7QqcapFmqtESUEuFHV/vD+tuBY0J4KXdVQtauhzmiOJR/AC2kwcw9uosgMQA+CxBl5oTEUNPMOEJuuLf2tTq2gBey067J8veA5/+nKj0rkriUano40zjbq9YBi0BhdBuAE9trGDaNvb47NCLjXmJDNh7SQMSppWwpCSGdYdvrRYCrkQKU4czIVGO61axWrYdZEZJIVxJydoo/9WVELbZne3g7YNxbbIrnQ8gFi3TlHmrlHDOjuLkmfTSuXlkjCX61HJMgMqoPkAmCmDkrIVdPgbdC83+MG1+FSiEVSYxxUXJtXia+02SYEPoMH/o6r8L9Xhzq6bIKRRTgaQc2GEJPf/HSdweF7Oi+BoNAyDYXj4pLf/aiP1FrvHHrI+C9lTts/esQM2ZpL98na8+94D/6NPfuXXa6rvbWp22Bnzv/8Gk+/ZHg==</latexit><latexit sha1_base64="qDnv4NRDnuL32pHOCPGDIjmXP/k=">AAACzHicfVFti9NAEN7Et7O+9byPKgyWgyq9khRBQYTzDfzgyx3Yu4NuL2y2k3RpNgm7U7HEfNXf6I/wP7hJK3p34MCyz8w8M7PzbFxmylIQ/PT8S5evXL22db1z4+at23e623ePbLE0EseyyApzEguLmcpxTIoyPCkNCh1neBwvXjf54y9orCryz7QqcapFmqtESUEuFHV/vD+tuBY0J4KXdVQtauhzmiOJR/AC2kwcw9uosgMQA+CxBl5oTEUNPMOEJuuLf2tTq2gBey067J8veA5/+nKj0rkriUano40zjbq9YBi0BhdBuAE9trGDaNvb47NCLjXmJDNh7SQMSppWwpCSGdYdvrRYCrkQKU4czIVGO61axWrYdZEZJIVxJydoo/9WVELbZne3g7YNxbbIrnQ8gFi3TlHmrlHDOjuLkmfTSuXlkjCX61HJMgMqoPkAmCmDkrIVdPgbdC83+MG1+FSiEVSYxxUXJtXia+02SYEPoMH/o6r8L9Xhzq6bIKRRTgaQc2GEJPf/HSdweF7Oi+BoNAyDYXj4pLf/aiP1FrvHHrI+C9lTts/esQM2ZpL98na8+94D/6NPfuXXa6rvbWp22Bnzv/8Gk+/ZHg==</latexit><latexit sha1_base64="qDnv4NRDnuL32pHOCPGDIjmXP/k=">AAACzHicfVFti9NAEN7Et7O+9byPKgyWgyq9khRBQYTzDfzgyx3Yu4NuL2y2k3RpNgm7U7HEfNXf6I/wP7hJK3p34MCyz8w8M7PzbFxmylIQ/PT8S5evXL22db1z4+at23e623ePbLE0EseyyApzEguLmcpxTIoyPCkNCh1neBwvXjf54y9orCryz7QqcapFmqtESUEuFHV/vD+tuBY0J4KXdVQtauhzmiOJR/AC2kwcw9uosgMQA+CxBl5oTEUNPMOEJuuLf2tTq2gBey067J8veA5/+nKj0rkriUano40zjbq9YBi0BhdBuAE9trGDaNvb47NCLjXmJDNh7SQMSppWwpCSGdYdvrRYCrkQKU4czIVGO61axWrYdZEZJIVxJydoo/9WVELbZne3g7YNxbbIrnQ8gFi3TlHmrlHDOjuLkmfTSuXlkjCX61HJMgMqoPkAmCmDkrIVdPgbdC83+MG1+FSiEVSYxxUXJtXia+02SYEPoMH/o6r8L9Xhzq6bIKRRTgaQc2GEJPf/HSdweF7Oi+BoNAyDYXj4pLf/aiP1FrvHHrI+C9lTts/esQM2ZpL98na8+94D/6NPfuXXa6rvbWp22Bnzv/8Gk+/ZHg==</latexit>

LBk(✓) = Es,a,![|!|yk � !|Q(s, a,!; ✓)|]

<latexit sha1_base64="A43OlXOfye2yp1QbdXrxPDOYgNM=">AAAC3XicfVFdixMxFM2MX2v92K4++hKsC1XaMiOCggjLquCD4i7Y3YWmDpn0zjQ0yQzJrVhm59E38VX/mn9GzMwW1rrihcDJuefem5yblko6jKKfQXjp8pWr17aud27cvHV7u7tz58gVSytgLApV2JOUO1DSwBglKjgpLXCdKjhOFy+b/PEnsE4W5gOuSphqnhuZScHRU0n3x9uPFdMc54h0v06qRd1nOAfkD+kL2ibSlL5OKjfgA5ZqygoNOa8nTEGGp+eM7yINghVc1Q27ShZ0SP+db+nD/mbL5+uxzMp8jqfTpNuLRlEb9CKI16BH1nGQ7ARDNivEUoNBobhzkzgqcVpxi1IoqDts6aDkYsFzmHhouAY3rVoLa7rrmRnNCuuPQdqyf1ZUXLvGjQH1oJG4FrmVTgc01e2lKI1v1Kg2Z2H2bFpJUy4RjDgblS0VxYI2G6EzaUGgWtEOewX+5Rbe+RbvS7AcC/uoYtzmmn+u/U9yyga0wf+TSnMu9biz6ydwYaW3gYo5t1z4PbiONzj+286L4OjxKI5G8eGT3t7+2uotco/cJ30Sk6dkj7whB2RMBPkVPAiGwShMwi/h1/DbmTQM1jV3yUaE338D1v/jlw==</latexit><latexit sha1_base64="A43OlXOfye2yp1QbdXrxPDOYgNM=">AAAC3XicfVFdixMxFM2MX2v92K4++hKsC1XaMiOCggjLquCD4i7Y3YWmDpn0zjQ0yQzJrVhm59E38VX/mn9GzMwW1rrihcDJuefem5yblko6jKKfQXjp8pWr17aud27cvHV7u7tz58gVSytgLApV2JOUO1DSwBglKjgpLXCdKjhOFy+b/PEnsE4W5gOuSphqnhuZScHRU0n3x9uPFdMc54h0v06qRd1nOAfkD+kL2ibSlL5OKjfgA5ZqygoNOa8nTEGGp+eM7yINghVc1Q27ShZ0SP+db+nD/mbL5+uxzMp8jqfTpNuLRlEb9CKI16BH1nGQ7ARDNivEUoNBobhzkzgqcVpxi1IoqDts6aDkYsFzmHhouAY3rVoLa7rrmRnNCuuPQdqyf1ZUXLvGjQH1oJG4FrmVTgc01e2lKI1v1Kg2Z2H2bFpJUy4RjDgblS0VxYI2G6EzaUGgWtEOewX+5Rbe+RbvS7AcC/uoYtzmmn+u/U9yyga0wf+TSnMu9biz6ydwYaW3gYo5t1z4PbiONzj+286L4OjxKI5G8eGT3t7+2uotco/cJ30Sk6dkj7whB2RMBPkVPAiGwShMwi/h1/DbmTQM1jV3yUaE338D1v/jlw==</latexit><latexit sha1_base64="A43OlXOfye2yp1QbdXrxPDOYgNM=">AAAC3XicfVFdixMxFM2MX2v92K4++hKsC1XaMiOCggjLquCD4i7Y3YWmDpn0zjQ0yQzJrVhm59E38VX/mn9GzMwW1rrihcDJuefem5yblko6jKKfQXjp8pWr17aud27cvHV7u7tz58gVSytgLApV2JOUO1DSwBglKjgpLXCdKjhOFy+b/PEnsE4W5gOuSphqnhuZScHRU0n3x9uPFdMc54h0v06qRd1nOAfkD+kL2ibSlL5OKjfgA5ZqygoNOa8nTEGGp+eM7yINghVc1Q27ShZ0SP+db+nD/mbL5+uxzMp8jqfTpNuLRlEb9CKI16BH1nGQ7ARDNivEUoNBobhzkzgqcVpxi1IoqDts6aDkYsFzmHhouAY3rVoLa7rrmRnNCuuPQdqyf1ZUXLvGjQH1oJG4FrmVTgc01e2lKI1v1Kg2Z2H2bFpJUy4RjDgblS0VxYI2G6EzaUGgWtEOewX+5Rbe+RbvS7AcC/uoYtzmmn+u/U9yyga0wf+TSnMu9biz6ydwYaW3gYo5t1z4PbiONzj+286L4OjxKI5G8eGT3t7+2uotco/cJ30Sk6dkj7whB2RMBPkVPAiGwShMwi/h1/DbmTQM1jV3yUaE338D1v/jlw==</latexit><latexit sha1_base64="A43OlXOfye2yp1QbdXrxPDOYgNM=">AAAC3XicfVFdixMxFM2MX2v92K4++hKsC1XaMiOCggjLquCD4i7Y3YWmDpn0zjQ0yQzJrVhm59E38VX/mn9GzMwW1rrihcDJuefem5yblko6jKKfQXjp8pWr17aud27cvHV7u7tz58gVSytgLApV2JOUO1DSwBglKjgpLXCdKjhOFy+b/PEnsE4W5gOuSphqnhuZScHRU0n3x9uPFdMc54h0v06qRd1nOAfkD+kL2ibSlL5OKjfgA5ZqygoNOa8nTEGGp+eM7yINghVc1Q27ShZ0SP+db+nD/mbL5+uxzMp8jqfTpNuLRlEb9CKI16BH1nGQ7ARDNivEUoNBobhzkzgqcVpxi1IoqDts6aDkYsFzmHhouAY3rVoLa7rrmRnNCuuPQdqyf1ZUXLvGjQH1oJG4FrmVTgc01e2lKI1v1Kg2Z2H2bFpJUy4RjDgblS0VxYI2G6EzaUGgWtEOewX+5Rbe+RbvS7AcC/uoYtzmmn+u/U9yyga0wf+TSnMu9biz6ydwYaW3gYo5t1z4PbiONzj+286L4OjxKI5G8eGT3t7+2uotco/cJ30Sk6dkj7whB2RMBPkVPAiGwShMwi/h1/DbmTQM1jV3yUaE338D1v/jlw==</latexit>

1.

2.

+ Hindsight Experience Reply (HER)

Envelope MOQ-Algorithm

“Adaptation Phase Algorithm: How to efficiently infer the underlying preference of a new task

through only few-shot interactions?”

Preference Elicitation: few-shot adaptation

Given a new tasks where only scalar feedbacks are available:

Use policy gradient (e.g., REINFORCE) and stochastic search to find:

with only few episodes.

argmaxµ1,...,µm

E!⇠Dm!

"E⌧⇠(P,⇧L(!))

" 1X

t=0

�trt(st, at)

##

<latexit sha1_base64="qXQcvCGvudbNnXwgUMgRPwol82k=">AAAEJnicfVPbjtMwEE1aLku4deGRF4tqpRZVVbtcX0Ar2JV4AFGk7e5KdRs5rpNaGyeRPalaWf4fvoY3BLzxKThpVCirYCnjkzlnZjLxOMhirmAw+Ok2mteu37i5d8u7fefuvfut/QdnKs0lZWOaxqm8CIhiMU/YGDjE7CKTjIggZufB5buCP18yqXianMI6Y1NBooSHnBKwLr/1AxMZCbLyNRa5P+zheQqqV2BhrIvAIgjQibF0IHAqWEQQVlygkqIkRseWKv1mJgyOWQgTtBsIJC9COtuQUQ/hEff11vHBdGx6tMnT7Rq0yeNhlQtfw+uBmWnMkxDWBkdECDLTYKRlTEf50EPEh66HJY8WMEXV7rfag/6gXOgqGFag7VRr5O83XtjmaS5YAjQmSk2GgwymmkjgNGbGw7liGaGXJGITCxMimJrq8ggMOrCeOQpTaZ8EUOn9O0IToYp27ccKVUhUidRaBD0UiPIlzRKbqFDtRK42JXbrQ/hqqnmS5cASuikf5jGCFBWnjOZcMgrxGnn4mNluJPto037KmCSQyie6OnVju4sQ7qEC/0/Kkz9Si+ukSyKNLkydgKZLowtTJ1CRrVSYOgGTtkRh6gRg+dN6OpPVXAPokTT1dVYZ294AfWKF3oH9rYRKbucB0QWRhIK9WZ6dtOG/c3UVnB32h0/7zz8/ax+9rWZuz3nkPHY6ztB56Rw5752RM3ao+8adu8JNml+aX5vfmt830oZbxTx0dlbz12+z0Waz</latexit>







Q(s, a5,!)<latexit sha1_base64="mUw9hAGbN9KmJ5/c4aBiz3nxck8=">AAACWnicfZFdSxwxFIazY6vuWO360StvQhfBynaZEUUvF+uFN0WFrgrOspzJnh2D+RiSjLgM+2O8bX+R0B9jZlxQK/RA4MnJe3JO3qS54NZF0WMjmPvwcX5hsRkufVpe+dxaXbuwujAM+0wLba5SsCi4wr7jTuBVbhBkKvAyvf1RnV/eobFcq19ukuNAQqb4mDNwPjVsfTnfth0Kw/0OTVJJEy0xg2/DVjvqRnXQ9xDPoE1mcTZcbXxPRpoVEpVjAqy9jqPcDUowjjOB0zApLObAbiHDa48KJNpBWc8/pVs+M6JjbfxSjtbZ1xUlSCvB3fhBpa0ktiY7kWmHprLe6Fz5iyrV215ufDgoucoLh4o9txoXgjpNKzvoiBtkTkxomByjn9zgT3/FaY4GnDY7ZQImk3A/9S/JaNKhFf9PytWL1HO45TsAM9zbQNkNGGDO/0boDY7/tfM9XOx246gbn++1e0czqxfJJvlKtklMDkiPnJAz0ieMlOSB/CZ/Gn+DIGgGS8/SoDGrWSdvIth4AqqKsUc=</latexit><latexit sha1_base64="mUw9hAGbN9KmJ5/c4aBiz3nxck8=">AAACWnicfZFdSxwxFIazY6vuWO360StvQhfBynaZEUUvF+uFN0WFrgrOspzJnh2D+RiSjLgM+2O8bX+R0B9jZlxQK/RA4MnJe3JO3qS54NZF0WMjmPvwcX5hsRkufVpe+dxaXbuwujAM+0wLba5SsCi4wr7jTuBVbhBkKvAyvf1RnV/eobFcq19ukuNAQqb4mDNwPjVsfTnfth0Kw/0OTVJJEy0xg2/DVjvqRnXQ9xDPoE1mcTZcbXxPRpoVEpVjAqy9jqPcDUowjjOB0zApLObAbiHDa48KJNpBWc8/pVs+M6JjbfxSjtbZ1xUlSCvB3fhBpa0ktiY7kWmHprLe6Fz5iyrV215ufDgoucoLh4o9txoXgjpNKzvoiBtkTkxomByjn9zgT3/FaY4GnDY7ZQImk3A/9S/JaNKhFf9PytWL1HO45TsAM9zbQNkNGGDO/0boDY7/tfM9XOx246gbn++1e0czqxfJJvlKtklMDkiPnJAz0ieMlOSB/CZ/Gn+DIGgGS8/SoDGrWSdvIth4AqqKsUc=</latexit><latexit sha1_base64="mUw9hAGbN9KmJ5/c4aBiz3nxck8=">AAACWnicfZFdSxwxFIazY6vuWO360StvQhfBynaZEUUvF+uFN0WFrgrOspzJnh2D+RiSjLgM+2O8bX+R0B9jZlxQK/RA4MnJe3JO3qS54NZF0WMjmPvwcX5hsRkufVpe+dxaXbuwujAM+0wLba5SsCi4wr7jTuBVbhBkKvAyvf1RnV/eobFcq19ukuNAQqb4mDNwPjVsfTnfth0Kw/0OTVJJEy0xg2/DVjvqRnXQ9xDPoE1mcTZcbXxPRpoVEpVjAqy9jqPcDUowjjOB0zApLObAbiHDa48KJNpBWc8/pVs+M6JjbfxSjtbZ1xUlSCvB3fhBpa0ktiY7kWmHprLe6Fz5iyrV215ufDgoucoLh4o9txoXgjpNKzvoiBtkTkxomByjn9zgT3/FaY4GnDY7ZQImk3A/9S/JaNKhFf9PytWL1HO45TsAM9zbQNkNGGDO/0boDY7/tfM9XOx246gbn++1e0czqxfJJvlKtklMDkiPnJAz0ieMlOSB/CZ/Gn+DIGgGS8/SoDGrWSdvIth4AqqKsUc=</latexit><latexit sha1_base64="mUw9hAGbN9KmJ5/c4aBiz3nxck8=">AAACWnicfZFdSxwxFIazY6vuWO360StvQhfBynaZEUUvF+uFN0WFrgrOspzJnh2D+RiSjLgM+2O8bX+R0B9jZlxQK/RA4MnJe3JO3qS54NZF0WMjmPvwcX5hsRkufVpe+dxaXbuwujAM+0wLba5SsCi4wr7jTuBVbhBkKvAyvf1RnV/eobFcq19ukuNAQqb4mDNwPjVsfTnfth0Kw/0OTVJJEy0xg2/DVjvqRnXQ9xDPoE1mcTZcbXxPRpoVEpVjAqy9jqPcDUowjjOB0zApLObAbiHDa48KJNpBWc8/pVs+M6JjbfxSjtbZ1xUlSCvB3fhBpa0ktiY7kWmHprLe6Fz5iyrV215ufDgoucoLh4o9txoXgjpNKzvoiBtkTkxomByjn9zgT3/FaY4GnDY7ZQImk3A/9S/JaNKhFf9PytWL1HO45TsAM9zbQNkNGGDO/0boDY7/tfM9XOx246gbn++1e0czqxfJJvlKtklMDkiPnJAz0ieMlOSB/CZ/Gn+DIGgGS8/SoDGrWSdvIth4AqqKsUc=</latexit>






Envelope MOQ-Network

“How to evaluate the performance of multi-objective reinforcement learning algorithms in the learning and

adaptation phases of linear preference scenario?”

Coverage Ratio (CR)

(Learning Phase) An agent's ability to find all the potential optimal solutions in the convex coverage set of the Pareto frontier.

AB

C

D

E

F

G

H

K

M

L


QU

ANTI

TY O

F O

BJEC

TIVE

2

Non-optimal Solutions

a.

L


QU

ANTI

TY O

F O

BJEC

TIVE

2

F



Retrieved Control Frontier

b.CCS

+

+Precision =

Recall =

DN

Optimal Control Frontier

Control Errors

Retrieved Solutions

Adaptation Error (AE)

AB

C

D

E

F

G

H

K

M

L


QU

ANTI

TY O

F O

BJEC

TIVE

2

Non-optimal Solutions

a.

L


QU

ANTI

TY O

F O

BJEC

TIVE

2F



Retrieved Control Frontier

b.CCS

+

+Precision =

Recall =

DN

Optimal Control Frontier

Control Errors

Retrieved Solutions

(Adaptation Phase) An agent's ability to adapt its policy to real-time specified preferences in the adaptation phase.

Experimental Settings

Experimental Domains

Deep Sea Treasure (DST)

Fruit Tree Navigation (FTN)

Task-Oriented Dialog Policy Learning (Dialog)

Multi-Objective SuperMario Game (SuperMario)

INPUT (ASR / SLU)

OUTPUT (NLG / TTS)

Dialogue State Tracking (DST)

Policy Model (parameters θ)

Reward Function

Dialogue Manager

Human User

Actions = {L, R}

2.93.7 3.8

2.93.9

3.21.8

3.6 3.5 4. 3.82.4 2.8 2.7

3.8 3.3 3.5

2.2.7 2.6

3.62.8

3.8 3.2 2.83.5

2.9 3.3 2.9 3.5 3.6 3.62.6

3.3 2.83.5

2.73.3 3.4 3.6 3.1 3.1 2.8

3.7 3.72.6 2.6

3.3 3.2 3.2 2.7 2.1

3.83.1 3.3 3.9

3.1 3.1.9 2.4

3.3 3. 3.1 3.4

Optimal Policy for Preference ω

2.93.7 3.8

2.93.9

3.21.8

3.6 3.5 4. 3.82.4 2.8 2.7

3.8 3.3 3.5

2.2.7 2.6

3.62.8

3.8 3.2 2.83.5

2.9 3.3 2.9 3.5 3.6 3.62.6

3.3 2.83.5

2.73.3 3.4 3.6 3.1 3.1 2.8

3.7 3.72.6 2.6

3.3 3.2 3.2 2.7 2.1

3.83.1 3.3 3.9

3.1 3.1.9 2.4

3.3 3. 3.1 3.4

={ }!|CCS Nutritions ∈ R6, e.g. r=

States = {(row, col)}0

BBBBBB@

4.923924.744252.560424.769361.435234.67811

1

CCCCCCA

Best Utility:Protein CarbsFatsVitaminsMinerals Water

(rewards)

maxr̂02CCS !|r̂0<latexit sha1_base64="UKajn45hrCTJA18bGFZk5iiEP8I=">AAACjHicfZDdahNBFMdP1q+6ak3rpQiDoSgSw64ILRQhGFFvxIqmLXRimJ2cbIbOxzIzWxqWfRdvvPRpvBL0JXwCZzcFWwseGPjNOf8zZ84/K6RwPkl+dKIrV69dv7F2M751+8763e7G5r4zpeU45kYae5gxh1JoHHvhJR4WFpnKJB5kx6OmfnCC1gmjP/llgRPFci3mgjMfUtPuG6rY6bSiC+Yrmili60eECk1C2i+8r0ajj3VNmgo1CnP2uQpVj5YzWZ9vmnZ7ySBpg1yG9Ax6wwe/v34BgL3pRucpnRleKtSeS+bcUZoUflIx6wWXWMe0dFgwfsxyPAqomUI3qdqNa7IVMjMyNzYc7UmbPd9RMeWaDfokQCNxLbmlyvokU+3FFDo81KguzvLznUkldFF61Hw1al5K4g1pDCQzYZF7uSQxfYXh5xbfhSfeF2iZN/ZJRZnNg6d12CQntE8a/p9U6L/SwPFWmMC4FcEGwhfMMh7sdnEwOP3Xzsuw/2yQJoP0Q9obvoRVrMF9eAiPIYVtGMJb2IMxcPgG3+En/IrWo+fRbvRiJY06Zz334EJEr/8AVovJZA==</latexit><latexit sha1_base64="VqXNBjXY4/hWYSj5fMRqTxe6Tyg=">AAACjHicfZDdahNBFMcnWz/qqjWtlyIMhqJIDLtSUJBCMKLeiBVNW+jEMDs52Qydj2XmbDEs+zZ66dN4IYK+hE/g7KZga8EDA78553/mzPlnhZIek+RHJ1q7dPnK1fVr8fUbNzdudTe39r0tnYCxsMq6w4x7UNLAGCUqOCwccJ0pOMiOR0394AScl9Z8wGUBE81zI+dScAypafcV0/zTtGILjhXLNHX1fcqkoSGNC8RqNHpf17SpMKsh5x+rUEVwgqv6bNO020sGSRv0IqSn0Bve/f3ls/y+szfd7DxiMytKDQaF4t4fpUmBk4o7lEJBHbPSQ8HFMc/hKKDhGvykajeu6XbIzOjcunAM0jZ7tqPi2jcb9GmARuJb8kud9Wmm24stTHioUZ2fhfOnk0qaokQwYjVqXiqKljYG0pl0IFAtacxeQPi5gzfhibcFOI7WPawYd3nwtA6b5JT1acP/k0rzVxo43g4TuHAy2EDFgjsugt0+Dgan/9p5EfYfD9JkkL5Le8PnZBXr5A65Rx6QlDwhQ/Ka7JExEeQr+UZ+kl/RRrQTPYt2V9Koc9pzm5yL6OUf1FbK3A==</latexit><latexit sha1_base64="VqXNBjXY4/hWYSj5fMRqTxe6Tyg=">AAACjHicfZDdahNBFMcnWz/qqjWtlyIMhqJIDLtSUJBCMKLeiBVNW+jEMDs52Qydj2XmbDEs+zZ66dN4IYK+hE/g7KZga8EDA78553/mzPlnhZIek+RHJ1q7dPnK1fVr8fUbNzdudTe39r0tnYCxsMq6w4x7UNLAGCUqOCwccJ0pOMiOR0394AScl9Z8wGUBE81zI+dScAypafcV0/zTtGILjhXLNHX1fcqkoSGNC8RqNHpf17SpMKsh5x+rUEVwgqv6bNO020sGSRv0IqSn0Bve/f3ls/y+szfd7DxiMytKDQaF4t4fpUmBk4o7lEJBHbPSQ8HFMc/hKKDhGvykajeu6XbIzOjcunAM0jZ7tqPi2jcb9GmARuJb8kud9Wmm24stTHioUZ2fhfOnk0qaokQwYjVqXiqKljYG0pl0IFAtacxeQPi5gzfhibcFOI7WPawYd3nwtA6b5JT1acP/k0rzVxo43g4TuHAy2EDFgjsugt0+Dgan/9p5EfYfD9JkkL5Le8PnZBXr5A65Rx6QlDwhQ/Ka7JExEeQr+UZ+kl/RRrQTPYt2V9Koc9pzm5yL6OUf1FbK3A==</latexit><latexit sha1_base64="lzpbyKofPpslW1WwdqFdHEuuojY=">AAACjHicfZBtaxNBEMc351M9tab60jeLoSgSw50ICiIUI+obsaJpC90Y5jaTZOk+HLtzYjjuI/lpfCXod3HvGrC14MDCb2f+s7PzL0qtAmXZz15y6fKVq9e2rqc3bt7avt3fuXMQXOUlTqTTzh8VEFArixNSpPGo9Aim0HhYnIzb+uFX9EE5+5nWJU4NLK1aKAkUU7P+W2Hg26wWK6BaFIb75gEXyvKYphVRPR5/ahreVoQzuIQvdawSegm6Ods06w+yUdYFvwj5BgZsE/uznd5jMXeyMmhJagjhOM9KmtbgSUmNTSqqgCXIE1jicUQLBsO07jZu+G7MzPnC+Xgs8S57tqMGE9oNhjxCKwkdhbUphrww3cWVNj7Uqs7PosXzaa1sWRFaeTpqUWlOjrcG8rnyKEmveSpeY/y5x/fxiQ8leiDnH9UC/DJ62sRNllwMecv/kyr7Vxo53Y0TQHoVbeByBR5ktDuk0eD8XzsvwsGTUZ6N8o/5YO/Vxuotdo/dZw9Zzp6xPfaO7bMJk+w7+8F+sd/JdvI0eZG8PJUmvU3PXXYukjd/AO5QxtA=</latexit>

Actions = {L, R}

2.93.7 3.8

2.93.9

3.21.8

3.6 3.5 4. 3.82.4 2.8 2.7

3.8 3.3 3.5

2.2.7 2.6

3.62.8

3.8 3.2 2.83.5

2.9 3.3 2.9 3.5 3.6 3.62.6

3.3 2.83.5

2.73.3 3.4 3.6 3.1 3.1 2.8

3.7 3.72.6 2.6

3.3 3.2 3.2 2.7 2.1

3.83.1 3.3 3.9

3.1 3.1.9 2.4

3.3 3. 3.1 3.4


2.93.7 3.8

2.93.9

3.21.8

3.6 3.5 4. 3.82.4 2.8 2.7

3.8 3.3 3.5

2.2.7 2.6

3.62.8

3.8 3.2 2.83.5

2.9 3.3 2.9 3.5 3.6 3.62.6

3.3 2.83.5

2.73.3 3.4 3.6 3.1 3.1 2.8

3.7 3.72.6 2.6

3.3 3.2 3.2 2.7 2.1

3.83.1 3.3 3.9

3.1 3.1.9 2.4

3.3 3. 3.1 3.4



BBBBBB@

4.923924.744252.560424.769361.435234.67811

1

CCCCCCA


(rewards)


…Actions = {L, R}

2.93.7 3.8

2.93.9

3.21.8

3.6 3.5 4. 3.82.4 2.8 2.7

3.8 3.3 3.5

2.2.7 2.6

3.62.8

3.8 3.2 2.83.5

2.9 3.3 2.9 3.5 3.6 3.62.6

3.3 2.83.5

2.73.3 3.4 3.6 3.1 3.1 2.8

3.7 3.72.6 2.6

3.3 3.2 3.2 2.7 2.1

3.83.1 3.3 3.9

3.1 3.1.9 2.4

3.3 3. 3.1 3.4


2.93.7 3.8

2.93.9

3.21.8

3.6 3.5 4. 3.82.4 2.8 2.7

3.8 3.3 3.5

2.2.7 2.6

3.62.8

3.8 3.2 2.83.5

2.9 3.3 2.9 3.5 3.6 3.62.6

3.3 2.83.5

2.73.3 3.4 3.6 3.1 3.1 2.8

3.7 3.72.6 2.6

3.3 3.2 3.2 2.7 2.1

3.83.1 3.3 3.9

3.1 3.1.9 2.4

3.3 3. 3.1 3.4



BBBBBB@

4.923924.744252.560424.769361.435234.67811

1

CCCCCCA


(rewards)


Actions = {L, R}

2.93.7 3.8

2.93.9

3.21.8

3.6 3.5 4. 3.82.4 2.8 2.7

3.8 3.3 3.5

2.2.7 2.6

3.62.8

3.8 3.2 2.83.5

2.9 3.3 2.9 3.5 3.6 3.62.6

3.3 2.83.5

2.73.3 3.4 3.6 3.1 3.1 2.8

3.7 3.72.6 2.6

3.3 3.2 3.2 2.7 2.1

3.83.1 3.3 3.9

3.1 3.1.9 2.4

3.3 3. 3.1 3.4


2.93.7 3.8

2.93.9

3.21.8

3.6 3.5 4. 3.82.4 2.8 2.7

3.8 3.3 3.5

2.2.7 2.6

3.62.8

3.8 3.2 2.83.5

2.9 3.3 2.9 3.5 3.6 3.62.6

3.3 2.83.5

2.73.3 3.4 3.6 3.1 3.1 2.8

3.7 3.72.6 2.6

3.3 3.2 3.2 2.7 2.1

3.83.1 3.3 3.9

3.1 3.1.9 2.4

3.3 3. 3.1 3.4



BBBBBB@

4.923924.744252.560424.769361.435234.67811

1

CCCCCCA


(rewards)


Synthetic Domains (known ground-truth)

(able to evaluate CR/AE)

Complex Domains (evaluate avg. UT)

Baselines and Performance Comparison

Experimental Results

Fruit Tree Navigation (FTN):

Having the Access to Ground Truth for Evaluation.

Actions = {L, R}

2.93.7 3.8

2.93.9

3.21.8

3.6 3.5 4. 3.82.4 2.8 2.7

3.8 3.3 3.5

2.2.7 2.6

3.62.8

3.8 3.2 2.83.5

2.9 3.3 2.9 3.5 3.6 3.62.6

3.3 2.83.5

2.73.3 3.4 3.6 3.1 3.1 2.8

3.7 3.72.6 2.6

3.3 3.2 3.2 2.7 2.1

3.83.1 3.3 3.9

3.1 3.1.9 2.4

3.3 3. 3.1 3.4


2.93.7 3.8

2.93.9

3.21.8

3.6 3.5 4. 3.82.4 2.8 2.7

3.8 3.3 3.5

2.2.7 2.6

3.62.8

3.8 3.2 2.83.5

2.9 3.3 2.9 3.5 3.6 3.62.6

3.3 2.83.5

2.73.3 3.4 3.6 3.1 3.1 2.8

3.7 3.72.6 2.6

3.3 3.2 3.2 2.7 2.1

3.83.1 3.3 3.9

3.1 3.1.9 2.4

3.3 3. 3.1 3.4



BBBBBB@

4.923924.744252.560424.769361.435234.67811

1

CCCCCCA


(rewards)


Actions = {L, R}

2.93.7 3.8

2.93.9

3.21.8

3.6 3.5 4. 3.82.4 2.8 2.7

3.8 3.3 3.5

2.2.7 2.6

3.62.8

3.8 3.2 2.83.5

2.9 3.3 2.9 3.5 3.6 3.62.6

3.3 2.83.5

2.73.3 3.4 3.6 3.1 3.1 2.8

3.7 3.72.6 2.6

3.3 3.2 3.2 2.7 2.1

3.83.1 3.3 3.9

3.1 3.1.9 2.4

3.3 3. 3.1 3.4


2.93.7 3.8

2.93.9

3.21.8

3.6 3.5 4. 3.82.4 2.8 2.7

3.8 3.3 3.5

2.2.7 2.6

3.62.8

3.8 3.2 2.83.5

2.9 3.3 2.9 3.5 3.6 3.62.6

3.3 2.83.5

2.73.3 3.4 3.6 3.1 3.1 2.8

3.7 3.72.6 2.6

3.3 3.2 3.2 2.7 2.1

3.83.1 3.3 3.9

3.1 3.1.9 2.4

3.3 3. 3.1 3.4



BBBBBB@

4.923924.744252.560424.769361.435234.67811

1

CCCCCCA


(rewards)


…Actions = {L, R}

2.93.7 3.8

2.93.9

3.21.8

3.6 3.5 4. 3.82.4 2.8 2.7

3.8 3.3 3.5

2.2.7 2.6

3.62.8

3.8 3.2 2.83.5

2.9 3.3 2.9 3.5 3.6 3.62.6

3.3 2.83.5

2.73.3 3.4 3.6 3.1 3.1 2.8

3.7 3.72.6 2.6

3.3 3.2 3.2 2.7 2.1

3.83.1 3.3 3.9

3.1 3.1.9 2.4

3.3 3. 3.1 3.4


2.93.7 3.8

2.93.9

3.21.8

3.6 3.5 4. 3.82.4 2.8 2.7

3.8 3.3 3.5

2.2.7 2.6

3.62.8

3.8 3.2 2.83.5

2.9 3.3 2.9 3.5 3.6 3.62.6

3.3 2.83.5

2.73.3 3.4 3.6 3.1 3.1 2.8

3.7 3.72.6 2.6

3.3 3.2 3.2 2.7 2.1

3.83.1 3.3 3.9

3.1 3.1.9 2.4

3.3 3. 3.1 3.4



BBBBBB@

4.923924.744252.560424.769361.435234.67811

1

CCCCCCA


(rewards)


Actions = {L, R}

2.93.7 3.8

2.93.9

3.21.8

3.6 3.5 4. 3.82.4 2.8 2.7

3.8 3.3 3.5

2.2.7 2.6

3.62.8

3.8 3.2 2.83.5

2.9 3.3 2.9 3.5 3.6 3.62.6

3.3 2.83.5

2.73.3 3.4 3.6 3.1 3.1 2.8

3.7 3.72.6 2.6

3.3 3.2 3.2 2.7 2.1

3.83.1 3.3 3.9

3.1 3.1.9 2.4

3.3 3. 3.1 3.4


2.93.7 3.8

2.93.9

3.21.8

3.6 3.5 4. 3.82.4 2.8 2.7

3.8 3.3 3.5

2.2.7 2.6

3.62.8

3.8 3.2 2.83.5

2.9 3.3 2.9 3.5 3.6 3.62.6

3.3 2.83.5

2.73.3 3.4 3.6 3.1 3.1 2.8

3.7 3.72.6 2.6

3.3 3.2 3.2 2.7 2.1

3.83.1 3.3 3.9

3.1 3.1.9 2.4

3.3 3. 3.1 3.4



BBBBBB@

4.923924.744252.560424.769361.435234.67811

1

CCCCCCA


(rewards)


FTN: sample efficiency

Performance Change (Envelope / Scalarized )

FTN: scalability

FTN: scalability

Dialog: adaptation to given user preference

INPUT (ASR / SLU)

OUTPUT (NLG / TTS)

Dialogue State Tracking (DST)

Policy Model (parameters θ)

Reward Function

Dialogue Manager

Human User

Example Dialogue #1 Preference = [0.68 brevity, 0.32 success]


Example Dialogue #2 Preference = [0.28 brevity, 0.72 success]


Envelope MOQ-Net Better Responses to User's Preferences

Scalarized VersionEnvelope Version

Single-Obj(0.5turn+0.5succ)Single-Obj(0.2turn+0.8succ)Single-Obj(0.8turn+0.2succ)

ZHLJKW RI VXFFHVV

Scalarized VersionEnvelope Version

Single-Obj(0.5turn+0.5succ)Single-Obj(0.2turn+0.8succ)Single-Obj(0.8turn+0.2succ)

ZHLJKW RI VXFFHVVWeight of Success

Succ

ess

Rat

e (%

)

95

90

85

80

75

700.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Dialog objectives: brevity / success

** *

SuperMario: infer underlying preferences

SuperMario: infer underlying preferences

Ground Truth Preference Δ Preference = Envelope - Scalarized

Comparison of the inferred preferences of the envelope and scalarized multi-objective A3C algorithm in different game variants with 100 episodes. The ground truth preference is on the diagonal.

More About This Work

http://papers.nips.cc/paper/9605-a-generalized-algorithm-for-multi-objective-reinforcement-learning-and-policy-adaptation





a generalized algorithm for multi-objective reinforcement …runzhey/demo/general_exam.pdf · 2020....

Documents