On gradient regularizers for MMD GANsMichael Arbel1*, Dougal J. Sutherland1*, Mikolaj Binkowski2 and Arthur Gretton1

1Gatsby Computational Neuroscience Unit, University College London 2Department of Mathematics, Imperial College London*Equal contribution

Overview

XMMD-based losses for implicit generative models are effective and principled.

×Previous approaches have bad topological properties.

XWe introduce gradient-regularized MMD loss with better topology.

XNew insight on the desired properties for the discriminator network.

X State-of-the-art results on 64× 64 unconditional ImageNet and 160× 160 CelebA.

Integral Probability Metrics

Integral Probability Metrics (IPMs) are distances between distributions defined by aclass of critic functions F :

D(P,Q) = supf ∈F

EX∼P[f (X )]− EY∼Q[f (Y )]

1-Wasserstein distance: F is the set of 1−Lipschitz functions

F = {f : |f (x)− f (y)| ≤ ‖x − y‖,∀x , y}WGANs approximate f with a critic network φψ. Weight clipping [1] or gradientpenalty [4] used to make φψ approximately Lipschitz.

Maximum Mean Discrepancy (MMD) has F a unit ball in a ReproducingKernel Hilbert Space (RKHS) H with kernel k :

k(realj , fakei)<latexit sha1_base64="mJFEa/WoT8zGGVmonbxjzUiDw1A=">AAACHnicbVDLSsNAFJ3Ud3xVXboZLIKClEQUXRbduFSwrdCGMJnetGMnD2ZuxBLyJW78FTcuFBFc6d84fSzq48DA4Zz7mhOkUmh0nC+rNDM7N7+wuGQvr6yurZc3Nhs6yRSHOk9kom4CpkGKGOooUMJNqoBFgYRm0D8f+s07UFok8TUOUvAi1o1FKDhDI/nlY7u/R9sI9zialQcygyI3E6R/WxxMGQo6RR6yPvii2PfLFafqjED/EndCKmSCS7/80e4kPIsgRi6Z1i3XSdHLmULBJRR2O9OQMt5nXWgZGrMItJePNhd01ygdGibKvBjpSJ3uyFmk9SAKTGXEsKd/e0PxP6+VYXjq5SJOM4SYjxeFmaSY0GFWtCMUcJQDQxhXwtxKeY8pxtEkapsQ3N9f/ksah1XX8KujSu1sEsci2SY7ZI+45ITUyAW5JHXCyQN5Ii/k1Xq0nq03631cWrImPVvkB6zPb1NKo94=</latexit><latexit sha1_base64="mJFEa/WoT8zGGVmonbxjzUiDw1A=">AAACHnicbVDLSsNAFJ3Ud3xVXboZLIKClEQUXRbduFSwrdCGMJnetGMnD2ZuxBLyJW78FTcuFBFc6d84fSzq48DA4Zz7mhOkUmh0nC+rNDM7N7+wuGQvr6yurZc3Nhs6yRSHOk9kom4CpkGKGOooUMJNqoBFgYRm0D8f+s07UFok8TUOUvAi1o1FKDhDI/nlY7u/R9sI9zialQcygyI3E6R/WxxMGQo6RR6yPvii2PfLFafqjED/EndCKmSCS7/80e4kPIsgRi6Z1i3XSdHLmULBJRR2O9OQMt5nXWgZGrMItJePNhd01ygdGibKvBjpSJ3uyFmk9SAKTGXEsKd/e0PxP6+VYXjq5SJOM4SYjxeFmaSY0GFWtCMUcJQDQxhXwtxKeY8pxtEkapsQ3N9f/ksah1XX8KujSu1sEsci2SY7ZI+45ITUyAW5JHXCyQN5Ii/k1Xq0nq03631cWrImPVvkB6zPb1NKo94=</latexit><latexit sha1_base64="mJFEa/WoT8zGGVmonbxjzUiDw1A=">AAACHnicbVDLSsNAFJ3Ud3xVXboZLIKClEQUXRbduFSwrdCGMJnetGMnD2ZuxBLyJW78FTcuFBFc6d84fSzq48DA4Zz7mhOkUmh0nC+rNDM7N7+wuGQvr6yurZc3Nhs6yRSHOk9kom4CpkGKGOooUMJNqoBFgYRm0D8f+s07UFok8TUOUvAi1o1FKDhDI/nlY7u/R9sI9zialQcygyI3E6R/WxxMGQo6RR6yPvii2PfLFafqjED/EndCKmSCS7/80e4kPIsgRi6Z1i3XSdHLmULBJRR2O9OQMt5nXWgZGrMItJePNhd01ygdGibKvBjpSJ3uyFmk9SAKTGXEsKd/e0PxP6+VYXjq5SJOM4SYjxeFmaSY0GFWtCMUcJQDQxhXwtxKeY8pxtEkapsQ3N9f/ksah1XX8KujSu1sEsci2SY7ZI+45ITUyAW5JHXCyQN5Ii/k1Xq0nq03631cWrImPVvkB6zPb1NKo94=</latexit><latexit sha1_base64="mJFEa/WoT8zGGVmonbxjzUiDw1A=">AAACHnicbVDLSsNAFJ3Ud3xVXboZLIKClEQUXRbduFSwrdCGMJnetGMnD2ZuxBLyJW78FTcuFBFc6d84fSzq48DA4Zz7mhOkUmh0nC+rNDM7N7+wuGQvr6yurZc3Nhs6yRSHOk9kom4CpkGKGOooUMJNqoBFgYRm0D8f+s07UFok8TUOUvAi1o1FKDhDI/nlY7u/R9sI9zialQcygyI3E6R/WxxMGQo6RR6yPvii2PfLFafqjED/EndCKmSCS7/80e4kPIsgRi6Z1i3XSdHLmULBJRR2O9OQMt5nXWgZGrMItJePNhd01ygdGibKvBjpSJ3uyFmk9SAKTGXEsKd/e0PxP6+VYXjq5SJOM4SYjxeFmaSY0GFWtCMUcJQDQxhXwtxKeY8pxtEkapsQ3N9f/ksah1XX8KujSu1sEsci2SY7ZI+45ITUyAW5JHXCyQN5Ii/k1Xq0nq03631cWrImPVvkB6zPb1NKo94=</latexit>

k(fakei, fakej)<latexit sha1_base64="EaRXWyt4MxQfjkd3aSBpkV/o7DA=">AAACHXicbZDLSgMxFIYz9VbrrerSTbAIFaTMSEGXRTcuK9gLtKVk0jNtbOZCckYsw7yIG1/FjQtFXLgR38b0stC2BwIf/39OTvK7kRQabfvHyqysrq1vZDdzW9s7u3v5/YO6DmPFocZDGaqmyzRIEUANBUpoRgqY70pouMPrsd94AKVFGNzhKIKOz/qB8ARnaKRuvpwbFmkb4REndyUKemnisSF0RXq2VL9PT7v5gl2yJ0UXwZlBgcyq2s1/tXshj30IkEumdcuxI+wkTKHgEtJcO9YQMT5kfWgZDJgPupNMNqf0xCg96oXKnADpRP07kTBf65Hvmk6f4UDPe2NxmdeK0bvsJCKIYoSATxd5saQY0nFUtCcUcJQjA4wrYd5K+YApxtEEmjMhOPNfXoT6eckxfFsuVK5mcWTJETkmReKQC1IhN6RKaoSTJ/JC3si79Wy9Wh/W57Q1Y81mDsm/sr5/AV4oo1o=</latexit><latexit sha1_base64="EaRXWyt4MxQfjkd3aSBpkV/o7DA=">AAACHXicbZDLSgMxFIYz9VbrrerSTbAIFaTMSEGXRTcuK9gLtKVk0jNtbOZCckYsw7yIG1/FjQtFXLgR38b0stC2BwIf/39OTvK7kRQabfvHyqysrq1vZDdzW9s7u3v5/YO6DmPFocZDGaqmyzRIEUANBUpoRgqY70pouMPrsd94AKVFGNzhKIKOz/qB8ARnaKRuvpwbFmkb4REndyUKemnisSF0RXq2VL9PT7v5gl2yJ0UXwZlBgcyq2s1/tXshj30IkEumdcuxI+wkTKHgEtJcO9YQMT5kfWgZDJgPupNMNqf0xCg96oXKnADpRP07kTBf65Hvmk6f4UDPe2NxmdeK0bvsJCKIYoSATxd5saQY0nFUtCcUcJQjA4wrYd5K+YApxtEEmjMhOPNfXoT6eckxfFsuVK5mcWTJETkmReKQC1IhN6RKaoSTJ/JC3si79Wy9Wh/W57Q1Y81mDsm/sr5/AV4oo1o=</latexit><latexit sha1_base64="EaRXWyt4MxQfjkd3aSBpkV/o7DA=">AAACHXicbZDLSgMxFIYz9VbrrerSTbAIFaTMSEGXRTcuK9gLtKVk0jNtbOZCckYsw7yIG1/FjQtFXLgR38b0stC2BwIf/39OTvK7kRQabfvHyqysrq1vZDdzW9s7u3v5/YO6DmPFocZDGaqmyzRIEUANBUpoRgqY70pouMPrsd94AKVFGNzhKIKOz/qB8ARnaKRuvpwbFmkb4REndyUKemnisSF0RXq2VL9PT7v5gl2yJ0UXwZlBgcyq2s1/tXshj30IkEumdcuxI+wkTKHgEtJcO9YQMT5kfWgZDJgPupNMNqf0xCg96oXKnADpRP07kTBf65Hvmk6f4UDPe2NxmdeK0bvsJCKIYoSATxd5saQY0nFUtCcUcJQjA4wrYd5K+YApxtEEmjMhOPNfXoT6eckxfFsuVK5mcWTJETkmReKQC1IhN6RKaoSTJ/JC3si79Wy9Wh/W57Q1Y81mDsm/sr5/AV4oo1o=</latexit><latexit sha1_base64="EaRXWyt4MxQfjkd3aSBpkV/o7DA=">AAACHXicbZDLSgMxFIYz9VbrrerSTbAIFaTMSEGXRTcuK9gLtKVk0jNtbOZCckYsw7yIG1/FjQtFXLgR38b0stC2BwIf/39OTvK7kRQabfvHyqysrq1vZDdzW9s7u3v5/YO6DmPFocZDGaqmyzRIEUANBUpoRgqY70pouMPrsd94AKVFGNzhKIKOz/qB8ARnaKRuvpwbFmkb4REndyUKemnisSF0RXq2VL9PT7v5gl2yJ0UXwZlBgcyq2s1/tXshj30IkEumdcuxI+wkTKHgEtJcO9YQMT5kfWgZDJgPupNMNqf0xCg96oXKnADpRP07kTBf65Hvmk6f4UDPe2NxmdeK0bvsJCKIYoSATxd5saQY0nFUtCcUcJQjA4wrYd5K+YApxtEEmjMhOPNfXoT6eckxfFsuVK5mcWTJETkmReKQC1IhN6RKaoSTJ/JC3si79Wy9Wh/W57Q1Y81mDsm/sr5/AV4oo1o=</latexit>

k(reali, realj)<latexit sha1_base64="7BZkgUJbjSOsYar4HNNAXTbcHLQ=">AAACH3icbVBLS0JBGJ1rL7OX1bLNkAQGIfdGVEupTUuDfICKzB0/dXLug5nvRnK5/6RNf6VNiyKinf+m8eqi1AMDh3O+1xw3lEKjbY+tzMrq2vpGdjO3tb2zu5ffP6jpIFIcqjyQgWq4TIMUPlRRoIRGqIB5roS6O7yd+PUnUFoE/gOOQmh7rO+LnuAMjdTJX+aGRdpCeMZ0VuzKCJLYTJAdkZwtNx6T006+YJfsFHSRODNSIDNUOvmfVjfgkQc+csm0bjp2iO2YKRRcQpJrRRpCxoesD01DfeaBbsfp6oSeGKVLe4Eyz0eaqn87YuZpPfJcU+kxHOh5byIu85oR9q7bsfDDCMHn00W9SFIM6CQs2hUKOMqRIYwrYW6lfMAU42gizZkQnPkvL5Laeckx/P6iUL6ZxZElR+SYFIlDrkiZ3JEKqRJOXsgb+SCf1qv1bn1Z39PSjDXrOST/YI1/AUc+pGI=</latexit><latexit sha1_base64="7BZkgUJbjSOsYar4HNNAXTbcHLQ=">AAACH3icbVBLS0JBGJ1rL7OX1bLNkAQGIfdGVEupTUuDfICKzB0/dXLug5nvRnK5/6RNf6VNiyKinf+m8eqi1AMDh3O+1xw3lEKjbY+tzMrq2vpGdjO3tb2zu5ffP6jpIFIcqjyQgWq4TIMUPlRRoIRGqIB5roS6O7yd+PUnUFoE/gOOQmh7rO+LnuAMjdTJX+aGRdpCeMZ0VuzKCJLYTJAdkZwtNx6T006+YJfsFHSRODNSIDNUOvmfVjfgkQc+csm0bjp2iO2YKRRcQpJrRRpCxoesD01DfeaBbsfp6oSeGKVLe4Eyz0eaqn87YuZpPfJcU+kxHOh5byIu85oR9q7bsfDDCMHn00W9SFIM6CQs2hUKOMqRIYwrYW6lfMAU42gizZkQnPkvL5Laeckx/P6iUL6ZxZElR+SYFIlDrkiZ3JEKqRJOXsgb+SCf1qv1bn1Z39PSjDXrOST/YI1/AUc+pGI=</latexit><latexit sha1_base64="7BZkgUJbjSOsYar4HNNAXTbcHLQ=">AAACH3icbVBLS0JBGJ1rL7OX1bLNkAQGIfdGVEupTUuDfICKzB0/dXLug5nvRnK5/6RNf6VNiyKinf+m8eqi1AMDh3O+1xw3lEKjbY+tzMrq2vpGdjO3tb2zu5ffP6jpIFIcqjyQgWq4TIMUPlRRoIRGqIB5roS6O7yd+PUnUFoE/gOOQmh7rO+LnuAMjdTJX+aGRdpCeMZ0VuzKCJLYTJAdkZwtNx6T006+YJfsFHSRODNSIDNUOvmfVjfgkQc+csm0bjp2iO2YKRRcQpJrRRpCxoesD01DfeaBbsfp6oSeGKVLe4Eyz0eaqn87YuZpPfJcU+kxHOh5byIu85oR9q7bsfDDCMHn00W9SFIM6CQs2hUKOMqRIYwrYW6lfMAU42gizZkQnPkvL5Laeckx/P6iUL6ZxZElR+SYFIlDrkiZ3JEKqRJOXsgb+SCf1qv1bn1Z39PSjDXrOST/YI1/AUc+pGI=</latexit><latexit sha1_base64="7BZkgUJbjSOsYar4HNNAXTbcHLQ=">AAACH3icbVBLS0JBGJ1rL7OX1bLNkAQGIfdGVEupTUuDfICKzB0/dXLug5nvRnK5/6RNf6VNiyKinf+m8eqi1AMDh3O+1xw3lEKjbY+tzMrq2vpGdjO3tb2zu5ffP6jpIFIcqjyQgWq4TIMUPlRRoIRGqIB5roS6O7yd+PUnUFoE/gOOQmh7rO+LnuAMjdTJX+aGRdpCeMZ0VuzKCJLYTJAdkZwtNx6T006+YJfsFHSRODNSIDNUOvmfVjfgkQc+csm0bjp2iO2YKRRcQpJrRRpCxoesD01DfeaBbsfp6oSeGKVLe4Eyz0eaqn87YuZpPfJcU+kxHOh5byIu85oR9q7bsfDDCMHn00W9SFIM6CQs2hUKOMqRIYwrYW6lfMAU42gizZkQnPkvL5Laeckx/P6iUL6ZxZElR+SYFIlDrkiZ3JEKqRJOXsgb+SCf1qv1bn1Z39PSjDXrOST/YI1/AUc+pGI=</latexit>

Closed form solution:

f ?(t) ∝ EP[k(X , t)]− EQ[k(Y , t)]

Unbiased estimator:

MMD2

= 1n(n−1)

∑i 6=j

k(reali, realj) + k(fakei, fakej)

− 2n2

∑i ,j

k(reali, fakej)

Smooth optimal critic:

sample-optimal critics population-optimal critics

Wasserstein MMD

Maximum Mean Discrepancy for GANs

MMD GANs optimize critic in kernel:

kψ(x , y) = kbase(φψ(x), φψ(y))

infθ

supψ

MMD2kψ

(P,Qθ)︸ ︷︷ ︸DMMD(P,Qθ)

MMD� <latexit sha1_base64="VOUevIlyKh9xmvvFY/TPO0OQzkc=">AAACM3icbVC7TsMwFHV4lvJqy8hiUSExVQlCArYKFsYi0YeURJXjOK1VJ7FspyKK8hms8BV8DGJDrPwDbpKBtlzJ1vE5914fHY8zKpVpfhgbm1vbO7u1vfr+weHRcaPZGsg4EZj0ccxiMfKQJIxGpK+oYmTEBUGhx8jQm90v9OGcCEnj6EmlnLghmkQ0oBgpTdkOn9Jx5nBJ83GjbXbMouA6sCrQBlX1xk2j5fgxTkISKcyQlLZlcuVmSCiKGcnrTiIJR3iGJsTWMEIhkW5WeM7huWZ8GMRCn0jBgv07kaFQyjT0dGeI1FSuagvyP81OVHDjZjTiiSIRLj8KEgZVDBcBQJ8KghVLNUBYUO0V4ikSCCsd09Imf065rFw/l7brjk8CHXXxysJ0IlCaZ+Vtdm7zuk7RWs1sHQwuO5bGj1ft7l2VZw2cgjNwASxwDbrgAfRAH2AQgxfwCt6Md+PT+DK+y9YNo5o5AUtl/PwC7Uyrxg==</latexit><latexit sha1_base64="VOUevIlyKh9xmvvFY/TPO0OQzkc=">AAACM3icbVC7TsMwFHV4lvJqy8hiUSExVQlCArYKFsYi0YeURJXjOK1VJ7FspyKK8hms8BV8DGJDrPwDbpKBtlzJ1vE5914fHY8zKpVpfhgbm1vbO7u1vfr+weHRcaPZGsg4EZj0ccxiMfKQJIxGpK+oYmTEBUGhx8jQm90v9OGcCEnj6EmlnLghmkQ0oBgpTdkOn9Jx5nBJ83GjbXbMouA6sCrQBlX1xk2j5fgxTkISKcyQlLZlcuVmSCiKGcnrTiIJR3iGJsTWMEIhkW5WeM7huWZ8GMRCn0jBgv07kaFQyjT0dGeI1FSuagvyP81OVHDjZjTiiSIRLj8KEgZVDBcBQJ8KghVLNUBYUO0V4ikSCCsd09Imf065rFw/l7brjk8CHXXxysJ0IlCaZ+Vtdm7zuk7RWs1sHQwuO5bGj1ft7l2VZw2cgjNwASxwDbrgAfRAH2AQgxfwCt6Md+PT+DK+y9YNo5o5AUtl/PwC7Uyrxg==</latexit><latexit sha1_base64="VOUevIlyKh9xmvvFY/TPO0OQzkc=">AAACM3icbVC7TsMwFHV4lvJqy8hiUSExVQlCArYKFsYi0YeURJXjOK1VJ7FspyKK8hms8BV8DGJDrPwDbpKBtlzJ1vE5914fHY8zKpVpfhgbm1vbO7u1vfr+weHRcaPZGsg4EZj0ccxiMfKQJIxGpK+oYmTEBUGhx8jQm90v9OGcCEnj6EmlnLghmkQ0oBgpTdkOn9Jx5nBJ83GjbXbMouA6sCrQBlX1xk2j5fgxTkISKcyQlLZlcuVmSCiKGcnrTiIJR3iGJsTWMEIhkW5WeM7huWZ8GMRCn0jBgv07kaFQyjT0dGeI1FSuagvyP81OVHDjZjTiiSIRLj8KEgZVDBcBQJ8KghVLNUBYUO0V4ikSCCsd09Imf065rFw/l7brjk8CHXXxysJ0IlCaZ+Vtdm7zuk7RWs1sHQwuO5bGj1ft7l2VZw2cgjNwASxwDbrgAfRAH2AQgxfwCt6Md+PT+DK+y9YNo5o5AUtl/PwC7Uyrxg==</latexit><latexit sha1_base64="VOUevIlyKh9xmvvFY/TPO0OQzkc=">AAACM3icbVC7TsMwFHV4lvJqy8hiUSExVQlCArYKFsYi0YeURJXjOK1VJ7FspyKK8hms8BV8DGJDrPwDbpKBtlzJ1vE5914fHY8zKpVpfhgbm1vbO7u1vfr+weHRcaPZGsg4EZj0ccxiMfKQJIxGpK+oYmTEBUGhx8jQm90v9OGcCEnj6EmlnLghmkQ0oBgpTdkOn9Jx5nBJ83GjbXbMouA6sCrQBlX1xk2j5fgxTkISKcyQlLZlcuVmSCiKGcnrTiIJR3iGJsTWMEIhkW5WeM7huWZ8GMRCn0jBgv07kaFQyjT0dGeI1FSuagvyP81OVHDjZjTiiSIRLj8KEgZVDBcBQJ8KghVLNUBYUO0V4ikSCCsd09Imf065rFw/l7brjk8CHXXxysJ0IlCaZ+Vtdm7zuk7RWs1sHQwuO5bGj1ft7l2VZw2cgjNwASxwDbrgAfRAH2AQgxfwCt6Md+PT+DK+y9YNo5o5AUtl/PwC7Uyrxg==</latexit>

� <latexit sha1_base64="VOUevIlyKh9xmvvFY/TPO0OQzkc=">AAACM3icbVC7TsMwFHV4lvJqy8hiUSExVQlCArYKFsYi0YeURJXjOK1VJ7FspyKK8hms8BV8DGJDrPwDbpKBtlzJ1vE5914fHY8zKpVpfhgbm1vbO7u1vfr+weHRcaPZGsg4EZj0ccxiMfKQJIxGpK+oYmTEBUGhx8jQm90v9OGcCEnj6EmlnLghmkQ0oBgpTdkOn9Jx5nBJ83GjbXbMouA6sCrQBlX1xk2j5fgxTkISKcyQlLZlcuVmSCiKGcnrTiIJR3iGJsTWMEIhkW5WeM7huWZ8GMRCn0jBgv07kaFQyjT0dGeI1FSuagvyP81OVHDjZjTiiSIRLj8KEgZVDBcBQJ8KghVLNUBYUO0V4ikSCCsd09Imf065rFw/l7brjk8CHXXxysJ0IlCaZ+Vtdm7zuk7RWs1sHQwuO5bGj1ft7l2VZw2cgjNwASxwDbrgAfRAH2AQgxfwCt6Md+PT+DK+y9YNo5o5AUtl/PwC7Uyrxg==</latexit><latexit sha1_base64="VOUevIlyKh9xmvvFY/TPO0OQzkc=">AAACM3icbVC7TsMwFHV4lvJqy8hiUSExVQlCArYKFsYi0YeURJXjOK1VJ7FspyKK8hms8BV8DGJDrPwDbpKBtlzJ1vE5914fHY8zKpVpfhgbm1vbO7u1vfr+weHRcaPZGsg4EZj0ccxiMfKQJIxGpK+oYmTEBUGhx8jQm90v9OGcCEnj6EmlnLghmkQ0oBgpTdkOn9Jx5nBJ83GjbXbMouA6sCrQBlX1xk2j5fgxTkISKcyQlLZlcuVmSCiKGcnrTiIJR3iGJsTWMEIhkW5WeM7huWZ8GMRCn0jBgv07kaFQyjT0dGeI1FSuagvyP81OVHDjZjTiiSIRLj8KEgZVDBcBQJ8KghVLNUBYUO0V4ikSCCsd09Imf065rFw/l7brjk8CHXXxysJ0IlCaZ+Vtdm7zuk7RWs1sHQwuO5bGj1ft7l2VZw2cgjNwASxwDbrgAfRAH2AQgxfwCt6Md+PT+DK+y9YNo5o5AUtl/PwC7Uyrxg==</latexit><latexit sha1_base64="VOUevIlyKh9xmvvFY/TPO0OQzkc=">AAACM3icbVC7TsMwFHV4lvJqy8hiUSExVQlCArYKFsYi0YeURJXjOK1VJ7FspyKK8hms8BV8DGJDrPwDbpKBtlzJ1vE5914fHY8zKpVpfhgbm1vbO7u1vfr+weHRcaPZGsg4EZj0ccxiMfKQJIxGpK+oYmTEBUGhx8jQm90v9OGcCEnj6EmlnLghmkQ0oBgpTdkOn9Jx5nBJ83GjbXbMouA6sCrQBlX1xk2j5fgxTkISKcyQlLZlcuVmSCiKGcnrTiIJR3iGJsTWMEIhkW5WeM7huWZ8GMRCn0jBgv07kaFQyjT0dGeI1FSuagvyP81OVHDjZjTiiSIRLj8KEgZVDBcBQJ8KghVLNUBYUO0V4ikSCCsd09Imf065rFw/l7brjk8CHXXxysJ0IlCaZ+Vtdm7zuk7RWs1sHQwuO5bGj1ft7l2VZw2cgjNwASxwDbrgAfRAH2AQgxfwCt6Md+PT+DK+y9YNo5o5AUtl/PwC7Uyrxg==</latexit><latexit sha1_base64="VOUevIlyKh9xmvvFY/TPO0OQzkc=">AAACM3icbVC7TsMwFHV4lvJqy8hiUSExVQlCArYKFsYi0YeURJXjOK1VJ7FspyKK8hms8BV8DGJDrPwDbpKBtlzJ1vE5914fHY8zKpVpfhgbm1vbO7u1vfr+weHRcaPZGsg4EZj0ccxiMfKQJIxGpK+oYmTEBUGhx8jQm90v9OGcCEnj6EmlnLghmkQ0oBgpTdkOn9Jx5nBJ83GjbXbMouA6sCrQBlX1xk2j5fgxTkISKcyQlLZlcuVmSCiKGcnrTiIJR3iGJsTWMEIhkW5WeM7huWZ8GMRCn0jBgv07kaFQyjT0dGeI1FSuagvyP81OVHDjZjTiiSIRLj8KEgZVDBcBQJ8KghVLNUBYUO0V4ikSCCsd09Imf065rFw/l7brjk8CHXXxysJ0IlCaZ+Vtdm7zuk7RWs1sHQwuO5bGj1ft7l2VZw2cgjNwASxwDbrgAfRAH2AQgxfwCt6Md+PT+DK+y9YNo5o5AUtl/PwC7Uyrxg==</latexit>

Can also use gradient penalty [2].

Continuity under weak topology

DMMD not continuous / differentiable in general:

x<latexit sha1_base64="IArQpDG4Gw7Ax+5Wri9CZWKD4Bo=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxBfsBbSib7aRdu9mE3Y1YQn+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e2s3n5EpXks780kQT+iQ8lDzqixVuOpX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu1mzyOIpzAKZyDB1dQgzuoQxMYIDzDK7w5D86L8+58LFoLTj5zDH/kfP4A5juM/A==</latexit><latexit sha1_base64="IArQpDG4Gw7Ax+5Wri9CZWKD4Bo=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxBfsBbSib7aRdu9mE3Y1YQn+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e2s3n5EpXks780kQT+iQ8lDzqixVuOpX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu1mzyOIpzAKZyDB1dQgzuoQxMYIDzDK7w5D86L8+58LFoLTj5zDH/kfP4A5juM/A==</latexit><latexit sha1_base64="IArQpDG4Gw7Ax+5Wri9CZWKD4Bo=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxBfsBbSib7aRdu9mE3Y1YQn+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e2s3n5EpXks780kQT+iQ8lDzqixVuOpX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu1mzyOIpzAKZyDB1dQgzuoQxMYIDzDK7w5D86L8+58LFoLTj5zDH/kfP4A5juM/A==</latexit><latexit sha1_base64="IArQpDG4Gw7Ax+5Wri9CZWKD4Bo=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxBfsBbSib7aRdu9mE3Y1YQn+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e2s3n5EpXks780kQT+iQ8lDzqixVuOpX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu1mzyOIpzAKZyDB1dQgzuoQxMYIDzDK7w5D86L8+58LFoLTj5zDH/kfP4A5juM/A==</latexit>

P = �0<latexit sha1_base64="5NTfpX2cu/5hSk4hrfl2Ks7B01o=">AAACEXicbVC7SgNBFJ2Nrxhfq5Y2g0FIFXZF0EYI2lhGMA/ILmF2cpMMmX0wc1cMy/6Cjb9iY6GIrZ2df+PkUWjigYHDOffOnDlBIoVGx/m2Ciura+sbxc3S1vbO7p69f9DUcao4NHgsY9UOmAYpImigQAntRAELAwmtYHQ98Vv3oLSIozscJ+CHbBCJvuAMjdS1Kx7CA07vyQKZQp55IcNhEGT1nF5SrwcSWdfJadcuO1VnCrpM3DkpkznqXfvL68U8DSFCLpnWHddJ0M+YQsEl5CUv1ZAwPmID6BgasRC0n02T5PTEKD3aj5U5EdKp+nsjY6HW4zAwk5O4etGbiP95nRT7F34moiRFiPjsoX4qKcZ0Ug/tCQUc5dgQxpUwWSkfMsU4mhJLpgR38cvLpHladQ2/PSvXruZ1FMkROSYV4pJzUiM3pE4ahJNH8kxeyZv1ZL1Y79bHbLRgzXcOyR9Ynz8J8Z23</latexit><latexit sha1_base64="5NTfpX2cu/5hSk4hrfl2Ks7B01o=">AAACEXicbVC7SgNBFJ2Nrxhfq5Y2g0FIFXZF0EYI2lhGMA/ILmF2cpMMmX0wc1cMy/6Cjb9iY6GIrZ2df+PkUWjigYHDOffOnDlBIoVGx/m2Ciura+sbxc3S1vbO7p69f9DUcao4NHgsY9UOmAYpImigQAntRAELAwmtYHQ98Vv3oLSIozscJ+CHbBCJvuAMjdS1Kx7CA07vyQKZQp55IcNhEGT1nF5SrwcSWdfJadcuO1VnCrpM3DkpkznqXfvL68U8DSFCLpnWHddJ0M+YQsEl5CUv1ZAwPmID6BgasRC0n02T5PTEKD3aj5U5EdKp+nsjY6HW4zAwk5O4etGbiP95nRT7F34moiRFiPjsoX4qKcZ0Ug/tCQUc5dgQxpUwWSkfMsU4mhJLpgR38cvLpHladQ2/PSvXruZ1FMkROSYV4pJzUiM3pE4ahJNH8kxeyZv1ZL1Y79bHbLRgzXcOyR9Ynz8J8Z23</latexit><latexit sha1_base64="5NTfpX2cu/5hSk4hrfl2Ks7B01o=">AAACEXicbVC7SgNBFJ2Nrxhfq5Y2g0FIFXZF0EYI2lhGMA/ILmF2cpMMmX0wc1cMy/6Cjb9iY6GIrZ2df+PkUWjigYHDOffOnDlBIoVGx/m2Ciura+sbxc3S1vbO7p69f9DUcao4NHgsY9UOmAYpImigQAntRAELAwmtYHQ98Vv3oLSIozscJ+CHbBCJvuAMjdS1Kx7CA07vyQKZQp55IcNhEGT1nF5SrwcSWdfJadcuO1VnCrpM3DkpkznqXfvL68U8DSFCLpnWHddJ0M+YQsEl5CUv1ZAwPmID6BgasRC0n02T5PTEKD3aj5U5EdKp+nsjY6HW4zAwk5O4etGbiP95nRT7F34moiRFiPjsoX4qKcZ0Ug/tCQUc5dgQxpUwWSkfMsU4mhJLpgR38cvLpHladQ2/PSvXruZ1FMkROSYV4pJzUiM3pE4ahJNH8kxeyZv1ZL1Y79bHbLRgzXcOyR9Ynz8J8Z23</latexit><latexit sha1_base64="5NTfpX2cu/5hSk4hrfl2Ks7B01o=">AAACEXicbVC7SgNBFJ2Nrxhfq5Y2g0FIFXZF0EYI2lhGMA/ILmF2cpMMmX0wc1cMy/6Cjb9iY6GIrZ2df+PkUWjigYHDOffOnDlBIoVGx/m2Ciura+sbxc3S1vbO7p69f9DUcao4NHgsY9UOmAYpImigQAntRAELAwmtYHQ98Vv3oLSIozscJ+CHbBCJvuAMjdS1Kx7CA07vyQKZQp55IcNhEGT1nF5SrwcSWdfJadcuO1VnCrpM3DkpkznqXfvL68U8DSFCLpnWHddJ0M+YQsEl5CUv1ZAwPmID6BgasRC0n02T5PTEKD3aj5U5EdKp+nsjY6HW4zAwk5O4etGbiP95nRT7F34moiRFiPjsoX4qKcZ0Ug/tCQUc5dgQxpUwWSkfMsU4mhJLpgR38cvLpHladQ2/PSvXruZ1FMkROSYV4pJzUiM3pE4ahJNH8kxeyZv1ZL1Y79bHbLRgzXcOyR9Ynz8J8Z23</latexit>

Q✓ = �✓<latexit sha1_base64="p/S50GX6DibhmsVwx9LpstUKV8Y=">AAACIHicbVDLSgMxFM3UV62vqks3wSK4KjMi1I1QdOOyBfuATimZzG0bmnmQ3BHLMJ/ixl9x40IR3enXmD4QbT0QOJxzb3JyvFgKjbb9aeVWVtfWN/Kbha3tnd294v5BU0eJ4tDgkYxU22MapAihgQIltGMFLPAktLzR9cRv3YHSIgpvcRxDN2CDUPQFZ2ikXrHiItzj9J5UgZ+lbsBw6HlpPeulLg4BWUYvqeuDRPajZLRXLNllewq6TJw5KZE5ar3ih+tHPAkgRC6Z1h3HjrGbMoWCS8gKbqIhZnzEBtAxNGQB6G46DZbRE6P4tB8pc0KkU/X3RsoCrceBZyYn8fWiNxH/8zoJ9i+6qQjjBCHks4f6iaQY0Ulb1BcKOMqxIYwrYbJSPmSKcTSdFkwJzuKXl0nzrOwYXj8vVa/mdeTJETkmp8QhFVIlN6RGGoSTB/JEXsir9Wg9W2/W+2w0Z813DskfWF/fW2Wk5A==</latexit><latexit sha1_base64="p/S50GX6DibhmsVwx9LpstUKV8Y=">AAACIHicbVDLSgMxFM3UV62vqks3wSK4KjMi1I1QdOOyBfuATimZzG0bmnmQ3BHLMJ/ixl9x40IR3enXmD4QbT0QOJxzb3JyvFgKjbb9aeVWVtfWN/Kbha3tnd294v5BU0eJ4tDgkYxU22MapAihgQIltGMFLPAktLzR9cRv3YHSIgpvcRxDN2CDUPQFZ2ikXrHiItzj9J5UgZ+lbsBw6HlpPeulLg4BWUYvqeuDRPajZLRXLNllewq6TJw5KZE5ar3ih+tHPAkgRC6Z1h3HjrGbMoWCS8gKbqIhZnzEBtAxNGQB6G46DZbRE6P4tB8pc0KkU/X3RsoCrceBZyYn8fWiNxH/8zoJ9i+6qQjjBCHks4f6iaQY0Ulb1BcKOMqxIYwrYbJSPmSKcTSdFkwJzuKXl0nzrOwYXj8vVa/mdeTJETkmp8QhFVIlN6RGGoSTB/JEXsir9Wg9W2/W+2w0Z813DskfWF/fW2Wk5A==</latexit><latexit sha1_base64="p/S50GX6DibhmsVwx9LpstUKV8Y=">AAACIHicbVDLSgMxFM3UV62vqks3wSK4KjMi1I1QdOOyBfuATimZzG0bmnmQ3BHLMJ/ixl9x40IR3enXmD4QbT0QOJxzb3JyvFgKjbb9aeVWVtfWN/Kbha3tnd294v5BU0eJ4tDgkYxU22MapAihgQIltGMFLPAktLzR9cRv3YHSIgpvcRxDN2CDUPQFZ2ikXrHiItzj9J5UgZ+lbsBw6HlpPeulLg4BWUYvqeuDRPajZLRXLNllewq6TJw5KZE5ar3ih+tHPAkgRC6Z1h3HjrGbMoWCS8gKbqIhZnzEBtAxNGQB6G46DZbRE6P4tB8pc0KkU/X3RsoCrceBZyYn8fWiNxH/8zoJ9i+6qQjjBCHks4f6iaQY0Ulb1BcKOMqxIYwrYbJSPmSKcTSdFkwJzuKXl0nzrOwYXj8vVa/mdeTJETkmp8QhFVIlN6RGGoSTB/JEXsir9Wg9W2/W+2w0Z813DskfWF/fW2Wk5A==</latexit><latexit sha1_base64="p/S50GX6DibhmsVwx9LpstUKV8Y=">AAACIHicbVDLSgMxFM3UV62vqks3wSK4KjMi1I1QdOOyBfuATimZzG0bmnmQ3BHLMJ/ixl9x40IR3enXmD4QbT0QOJxzb3JyvFgKjbb9aeVWVtfWN/Kbha3tnd294v5BU0eJ4tDgkYxU22MapAihgQIltGMFLPAktLzR9cRv3YHSIgpvcRxDN2CDUPQFZ2ikXrHiItzj9J5UgZ+lbsBw6HlpPeulLg4BWUYvqeuDRPajZLRXLNllewq6TJw5KZE5ar3ih+tHPAkgRC6Z1h3HjrGbMoWCS8gKbqIhZnzEBtAxNGQB6G46DZbRE6P4tB8pc0KkU/X3RsoCrceBZyYn8fWiNxH/8zoJ9i+6qQjjBCHks4f6iaQY0Ulb1BcKOMqxIYwrYbJSPmSKcTSdFkwJzuKXl0nzrOwYXj8vVa/mdeTJETkmp8QhFVIlN6RGGoSTB/JEXsir9Wg9W2/W+2w0Z813DskfWF/fW2Wk5A==</latexit>

✓<latexit sha1_base64="9nw3eiH0vxpghXegSZAQaK2DTls=">AAAB7nicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3biMYB6QDKGnU0ma9DzorhHCkI9w40IRt36PO//GnmQWmnih4XCriq66QaKkIdf9dtbWNza3tks75d29/YPDytFxy8SpFtgUsYp1J+AGlYywSZIUdhKNPAwUtoPJXV5vP6E2Mo4eaZqgH/JRJIdScLJWu0djJF7uV6puzZ2LrYJXQBUKNfqVr94gFmmIEQnFjel6bkJ+xjVJoXBW7qUGEy4mfIRdixEP0fjZfN0ZO7fOgA1jbV9EbO7+nsh4aMw0DGxnyGlslmu5+V+tm9Lwxs9klKSEkVh8NEwVo5jlt7OB1ChITS1woaXdlYkx11yQTSgPwVs+eRValzXP8sNVtX5bxFGCUziDC/DgGupwDw1ogoAJPMMrvDmJ8+K8Ox+L1jWnmDmBP3I+fwDagY88</latexit><latexit sha1_base64="9nw3eiH0vxpghXegSZAQaK2DTls=">AAAB7nicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3biMYB6QDKGnU0ma9DzorhHCkI9w40IRt36PO//GnmQWmnih4XCriq66QaKkIdf9dtbWNza3tks75d29/YPDytFxy8SpFtgUsYp1J+AGlYywSZIUdhKNPAwUtoPJXV5vP6E2Mo4eaZqgH/JRJIdScLJWu0djJF7uV6puzZ2LrYJXQBUKNfqVr94gFmmIEQnFjel6bkJ+xjVJoXBW7qUGEy4mfIRdixEP0fjZfN0ZO7fOgA1jbV9EbO7+nsh4aMw0DGxnyGlslmu5+V+tm9Lwxs9klKSEkVh8NEwVo5jlt7OB1ChITS1woaXdlYkx11yQTSgPwVs+eRValzXP8sNVtX5bxFGCUziDC/DgGupwDw1ogoAJPMMrvDmJ8+K8Ox+L1jWnmDmBP3I+fwDagY88</latexit><latexit sha1_base64="9nw3eiH0vxpghXegSZAQaK2DTls=">AAAB7nicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3biMYB6QDKGnU0ma9DzorhHCkI9w40IRt36PO//GnmQWmnih4XCriq66QaKkIdf9dtbWNza3tks75d29/YPDytFxy8SpFtgUsYp1J+AGlYywSZIUdhKNPAwUtoPJXV5vP6E2Mo4eaZqgH/JRJIdScLJWu0djJF7uV6puzZ2LrYJXQBUKNfqVr94gFmmIEQnFjel6bkJ+xjVJoXBW7qUGEy4mfIRdixEP0fjZfN0ZO7fOgA1jbV9EbO7+nsh4aMw0DGxnyGlslmu5+V+tm9Lwxs9klKSEkVh8NEwVo5jlt7OB1ChITS1woaXdlYkx11yQTSgPwVs+eRValzXP8sNVtX5bxFGCUziDC/DgGupwDw1ogoAJPMMrvDmJ8+K8Ox+L1jWnmDmBP3I+fwDagY88</latexit><latexit sha1_base64="9nw3eiH0vxpghXegSZAQaK2DTls=">AAAB7nicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3biMYB6QDKGnU0ma9DzorhHCkI9w40IRt36PO//GnmQWmnih4XCriq66QaKkIdf9dtbWNza3tks75d29/YPDytFxy8SpFtgUsYp1J+AGlYywSZIUdhKNPAwUtoPJXV5vP6E2Mo4eaZqgH/JRJIdScLJWu0djJF7uV6puzZ2LrYJXQBUKNfqVr94gFmmIEQnFjel6bkJ+xjVJoXBW7qUGEy4mfIRdixEP0fjZfN0ZO7fOgA1jbV9EbO7+nsh4aMw0DGxnyGlslmu5+V+tm9Lwxs9klKSEkVh8NEwVo5jlt7OB1ChITS1woaXdlYkx11yQTSgPwVs+eRValzXP8sNVtX5bxFGCUziDC/DgGupwDw1ogoAJPMMrvDmJ8+K8Ox+L1jWnmDmBP3I+fwDagY88</latexit>

0<latexit sha1_base64="mSC7jR5dlnc4mUSSs2Y7zLybdHk=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjraY7KFfcqrsQWQcvhwrkagzKX/1hzNIIpWGCat3z3MT4GVWGM4GzUj/VmFA2oSPsWZQ0Qu1ni0Vn5MI6QxLGyj5pyML9PZHRSOtpFNjOiJqxXq3Nzf9qvdSEN37GZZIalGz5UZgKYmIyv5oMuUJmxNQCZYrbXQkbU0WZsdmUbAje6snr0L6qepab15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh85nz95G4y0</latexit><latexit sha1_base64="mSC7jR5dlnc4mUSSs2Y7zLybdHk=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjraY7KFfcqrsQWQcvhwrkagzKX/1hzNIIpWGCat3z3MT4GVWGM4GzUj/VmFA2oSPsWZQ0Qu1ni0Vn5MI6QxLGyj5pyML9PZHRSOtpFNjOiJqxXq3Nzf9qvdSEN37GZZIalGz5UZgKYmIyv5oMuUJmxNQCZYrbXQkbU0WZsdmUbAje6snr0L6qepab15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh85nz95G4y0</latexit><latexit sha1_base64="mSC7jR5dlnc4mUSSs2Y7zLybdHk=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjraY7KFfcqrsQWQcvhwrkagzKX/1hzNIIpWGCat3z3MT4GVWGM4GzUj/VmFA2oSPsWZQ0Qu1ni0Vn5MI6QxLGyj5pyML9PZHRSOtpFNjOiJqxXq3Nzf9qvdSEN37GZZIalGz5UZgKYmIyv5oMuUJmxNQCZYrbXQkbU0WZsdmUbAje6snr0L6qepab15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh85nz95G4y0</latexit><latexit sha1_base64="mSC7jR5dlnc4mUSSs2Y7zLybdHk=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjraY7KFfcqrsQWQcvhwrkagzKX/1hzNIIpWGCat3z3MT4GVWGM4GzUj/VmFA2oSPsWZQ0Qu1ni0Vn5MI6QxLGyj5pyML9PZHRSOtpFNjOiJqxXq3Nzf9qvdSEN37GZZIalGz5UZgKYmIyv5oMuUJmxNQCZYrbXQkbU0WZsdmUbAje6snr0L6qepab15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh85nz95G4y0</latexit>

� (x) = x<latexit sha1_base64="yC47CR6xz6bMMbsCM/qVyQIvACI=">AAACJXicbZDLSgMxFIYzXmu9VV26CRah3ZQZEXQjFN24rGAv0JaSSU/bYCYzJGekZZhn8DF8Arf6BO5EcOXK9zC9LGzrgYSP/1z5/UgKg6775aysrq1vbGa2sts7u3v7uYPDmgljzaHKQxnqhs8MSKGgigIlNCINLPAl1P2Hm3G+/gjaiFDd4yiCdsD6SvQEZ2ilTq7YQhjiZE4Saqb6kCataCA69jciLQyLV2Ogw7STy7sldxJ0GbwZ5MksKp3cT6sb8jgAhVwyY5qeG2E7YRoFl5BmW7GBiPEH1oemRcUCMO1kckpKT63Spb1Q26eQTtS/HQkLjBkFvq0MGA7MYm4s/pdrxti7bCdCRTGC4tNFvVhSDOnYH9oVGjjKkQXGtbC3Uj5gmnG0Ls5tGU5Ptb54iy4sQ+2s5Fm+O8+Xr2cOZcgxOSEF4pELUia3pEKqhJMn8kJeyZvz7Lw7H87ntHTFmfUckblwvn8BtuOnSQ==</latexit><latexit sha1_base64="yC47CR6xz6bMMbsCM/qVyQIvACI=">AAACJXicbZDLSgMxFIYzXmu9VV26CRah3ZQZEXQjFN24rGAv0JaSSU/bYCYzJGekZZhn8DF8Arf6BO5EcOXK9zC9LGzrgYSP/1z5/UgKg6775aysrq1vbGa2sts7u3v7uYPDmgljzaHKQxnqhs8MSKGgigIlNCINLPAl1P2Hm3G+/gjaiFDd4yiCdsD6SvQEZ2ilTq7YQhjiZE4Saqb6kCataCA69jciLQyLV2Ogw7STy7sldxJ0GbwZ5MksKp3cT6sb8jgAhVwyY5qeG2E7YRoFl5BmW7GBiPEH1oemRcUCMO1kckpKT63Spb1Q26eQTtS/HQkLjBkFvq0MGA7MYm4s/pdrxti7bCdCRTGC4tNFvVhSDOnYH9oVGjjKkQXGtbC3Uj5gmnG0Ls5tGU5Ptb54iy4sQ+2s5Fm+O8+Xr2cOZcgxOSEF4pELUia3pEKqhJMn8kJeyZvz7Lw7H87ntHTFmfUckblwvn8BtuOnSQ==</latexit><latexit sha1_base64="yC47CR6xz6bMMbsCM/qVyQIvACI=">AAACJXicbZDLSgMxFIYzXmu9VV26CRah3ZQZEXQjFN24rGAv0JaSSU/bYCYzJGekZZhn8DF8Arf6BO5EcOXK9zC9LGzrgYSP/1z5/UgKg6775aysrq1vbGa2sts7u3v7uYPDmgljzaHKQxnqhs8MSKGgigIlNCINLPAl1P2Hm3G+/gjaiFDd4yiCdsD6SvQEZ2ilTq7YQhjiZE4Saqb6kCataCA69jciLQyLV2Ogw7STy7sldxJ0GbwZ5MksKp3cT6sb8jgAhVwyY5qeG2E7YRoFl5BmW7GBiPEH1oemRcUCMO1kckpKT63Spb1Q26eQTtS/HQkLjBkFvq0MGA7MYm4s/pdrxti7bCdCRTGC4tNFvVhSDOnYH9oVGjjKkQXGtbC3Uj5gmnG0Ls5tGU5Ptb54iy4sQ+2s5Fm+O8+Xr2cOZcgxOSEF4pELUia3pEKqhJMn8kJeyZvz7Lw7H87ntHTFmfUckblwvn8BtuOnSQ==</latexit><latexit sha1_base64="yC47CR6xz6bMMbsCM/qVyQIvACI=">AAACJXicbZDLSgMxFIYzXmu9VV26CRah3ZQZEXQjFN24rGAv0JaSSU/bYCYzJGekZZhn8DF8Arf6BO5EcOXK9zC9LGzrgYSP/1z5/UgKg6775aysrq1vbGa2sts7u3v7uYPDmgljzaHKQxnqhs8MSKGgigIlNCINLPAl1P2Hm3G+/gjaiFDd4yiCdsD6SvQEZ2ilTq7YQhjiZE4Saqb6kCataCA69jciLQyLV2Ogw7STy7sldxJ0GbwZ5MksKp3cT6sb8jgAhVwyY5qeG2E7YRoFl5BmW7GBiPEH1oemRcUCMO1kckpKT63Spb1Q26eQTtS/HQkLjBkFvq0MGA7MYm4s/pdrxti7bCdCRTGC4tNFvVhSDOnYH9oVGjjKkQXGtbC3Uj5gmnG0Ls5tGU5Ptb54iy4sQ+2s5Fm+O8+Xr2cOZcgxOSEF4pELUia3pEKqhJMn8kJeyZvz7Lw7H87ntHTFmfUckblwvn8BtuOnSQ==</latexit>

DiracGAN [7]

Gradient Constrained MMD

Adjust the radius of the RKHS ball according to the smoothness of k :

FS = {f ∈ Hk : ‖f ‖Hk≤ σk}

SMMDk(P,Q) := supf ∈FS

EP[f (X )]− EQ[f (X )] = σk MMDk(P,Q)

σk :=

(λ + EX∼S

[k(X ,X ) +

∑di=1

∂2k(y ,z)∂yi∂zi

∣∣∣(y ,z)=(X ,X )

])−12

Optimal f ? satisfies EX∼S[‖∇f ?(X )‖2] ≤ 1

Other possible choices for F :

FLip := {f ∈ Hk : ‖f ‖2Lip + λ‖f ‖2 ≤ 1} FGC := {f ∈ Hk : ‖f ‖2

L2(S) + ‖∇f ‖2L2(S) + λ‖f ‖2 ≤ 1}

F FLip FGC FS

Effectiveness

Tractability

4 2 0 2 40.000.250.500.751.001.251.50

MMDSMMD

Theory: Continuity under weak topology

DSMMD(P,Q) is continuous in weak topology if:

S has a density (can depend on P, Q)

φψ is fully connected, Leaky-ReLU activations,non-increasing width

Each layer of φψ has weights with boundedcondition number

kbase is “reasonable” (Gaussian, linear, . . . )0 20K 40K 60K

0

200

400

600

Condition Number: Layer 1SMMDGANSN-SMMDGANMMDGANSN-MMDGAN

Orthogonal Normalization [3] or Spectral Normalization [8] control the condition number in practice.

Experimental Comparison

Scaled MMD GANs outperform other GANs (WGAN-GP, MMD-GAN, SN-GAN).

(a) Scaled MMD GAN, SN (b) SN-GAN

ImageNet, 64× 64.No labels.Generator: 10-layer ResNet.Critic: 10-layer ResNet.

(a) Scaled MMD GAN, SN (b) MMD GAN, GP+L2

CelebA, 160× 160.Generator: 10-layer ResNet.Critic: 5-layer DCGAN.

0 20 40FID [2]

WGAN-GP [1, NeurIPS-17]BGAN [6, ICLR-18]

SN-GAN [8, ICLR-18]MMDGAN-GP-L2 [2, ICLR-18]

Sobolev-GAN [9, ICLR-18]SN-SMMDGAN (ours)

ImageNet CelebA

0 20 40KID ×103 [5]

0 20FID [2]

0 10 20KID ×103 [5]

Implementation at github.com/MichaelArbel/Scaled-MMD-GAN

Faster training and better complexity control

0 2 4 6 8 10generator iterations ×104

10

15

20

25

30KID×103

SMMDGANSN-SMMDGANSN-SWGANSobolev-GANSN-GANWGAN-GPMMDGAN-GP-L2

0 1 2 3 4 5 6generator iterations ×104

101

102

103

104Critic Complexity: 2

k SMMDGANSN-SMMDGANMMDGANSN-MMDGAN

BibliographyM. Arjovsky, S. Chintala, and L. Bottou. “Wasserstein Generative Adversarial Networks”. ICML. 2017.

M. Binkowski, D. J. Sutherland, M. Arbel, and A. Gretton. “Demystifying MMD GANs”. ICLR. 2018.

A. Brock, T. Lim, J. M. Ritchie, and N. Weston. “Neural Photo Editing with Introspective Adversarial Networks”. ICLR. 2017.

I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville. “Improved Training of Wasserstein GANs”. NeurIPS. 2017.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, G. Klambauer, and S. Hochreiter. “GANs Trained by a Two Time-Scale Update RuleConverge to a Nash Equilibrium”. NeurIPS. 2017.

R. Hjelm, A. Jacob, T. Che, A. Trischler, K. Cho, and Y. Bengio. “Boundary-Seeking Generative Adversarial Networks”. ICLR. 2018.

L. Mescheder, A. Geiger, and S. Nowozin. “Which Training Methods for GANs do actually Converge?” ICML. 2018.

T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. “Spectral Normalization for Generative Adversarial Networks”. ICLR. 2018.

Y. Mroueh, C.-L. Li, T. Sercu, A. Raj, and Y. Cheng. “Sobolev GAN”. ICLR. 2018.

This work was funded by the Gatsby Charitable Foundation. Contact: michael.n.arbel@gmail.com