On gradient regularizers for MMD GANsMichael Arbel1*, Dougal J. Sutherland1*, Mikolaj Binkowski2 and Arthur Gretton1
1Gatsby Computational Neuroscience Unit, University College London 2Department of Mathematics, Imperial College London*Equal contribution
Overview
XMMD-based losses for implicit generative models are effective and principled.
×Previous approaches have bad topological properties.
XWe introduce gradient-regularized MMD loss with better topology.
XNew insight on the desired properties for the discriminator network.
X State-of-the-art results on 64× 64 unconditional ImageNet and 160× 160 CelebA.
Integral Probability Metrics
Integral Probability Metrics (IPMs) are distances between distributions defined by aclass of critic functions F :
D(P,Q) = supf ∈F
EX∼P[f (X )]− EY∼Q[f (Y )]
1-Wasserstein distance: F is the set of 1−Lipschitz functions
F = {f : |f (x)− f (y)| ≤ ‖x − y‖,∀x , y}WGANs approximate f with a critic network φψ. Weight clipping [1] or gradientpenalty [4] used to make φψ approximately Lipschitz.
Maximum Mean Discrepancy (MMD) has F a unit ball in a ReproducingKernel Hilbert Space (RKHS) H with kernel k :
k(realj , fakei)<latexit sha1_base64="mJFEa/WoT8zGGVmonbxjzUiDw1A=">AAACHnicbVDLSsNAFJ3Ud3xVXboZLIKClEQUXRbduFSwrdCGMJnetGMnD2ZuxBLyJW78FTcuFBFc6d84fSzq48DA4Zz7mhOkUmh0nC+rNDM7N7+wuGQvr6yurZc3Nhs6yRSHOk9kom4CpkGKGOooUMJNqoBFgYRm0D8f+s07UFok8TUOUvAi1o1FKDhDI/nlY7u/R9sI9zialQcygyI3E6R/WxxMGQo6RR6yPvii2PfLFafqjED/EndCKmSCS7/80e4kPIsgRi6Z1i3XSdHLmULBJRR2O9OQMt5nXWgZGrMItJePNhd01ygdGibKvBjpSJ3uyFmk9SAKTGXEsKd/e0PxP6+VYXjq5SJOM4SYjxeFmaSY0GFWtCMUcJQDQxhXwtxKeY8pxtEkapsQ3N9f/ksah1XX8KujSu1sEsci2SY7ZI+45ITUyAW5JHXCyQN5Ii/k1Xq0nq03631cWrImPVvkB6zPb1NKo94=</latexit><latexit sha1_base64="mJFEa/WoT8zGGVmonbxjzUiDw1A=">AAACHnicbVDLSsNAFJ3Ud3xVXboZLIKClEQUXRbduFSwrdCGMJnetGMnD2ZuxBLyJW78FTcuFBFc6d84fSzq48DA4Zz7mhOkUmh0nC+rNDM7N7+wuGQvr6yurZc3Nhs6yRSHOk9kom4CpkGKGOooUMJNqoBFgYRm0D8f+s07UFok8TUOUvAi1o1FKDhDI/nlY7u/R9sI9zialQcygyI3E6R/WxxMGQo6RR6yPvii2PfLFafqjED/EndCKmSCS7/80e4kPIsgRi6Z1i3XSdHLmULBJRR2O9OQMt5nXWgZGrMItJePNhd01ygdGibKvBjpSJ3uyFmk9SAKTGXEsKd/e0PxP6+VYXjq5SJOM4SYjxeFmaSY0GFWtCMUcJQDQxhXwtxKeY8pxtEkapsQ3N9f/ksah1XX8KujSu1sEsci2SY7ZI+45ITUyAW5JHXCyQN5Ii/k1Xq0nq03631cWrImPVvkB6zPb1NKo94=</latexit><latexit sha1_base64="mJFEa/WoT8zGGVmonbxjzUiDw1A=">AAACHnicbVDLSsNAFJ3Ud3xVXboZLIKClEQUXRbduFSwrdCGMJnetGMnD2ZuxBLyJW78FTcuFBFc6d84fSzq48DA4Zz7mhOkUmh0nC+rNDM7N7+wuGQvr6yurZc3Nhs6yRSHOk9kom4CpkGKGOooUMJNqoBFgYRm0D8f+s07UFok8TUOUvAi1o1FKDhDI/nlY7u/R9sI9zialQcygyI3E6R/WxxMGQo6RR6yPvii2PfLFafqjED/EndCKmSCS7/80e4kPIsgRi6Z1i3XSdHLmULBJRR2O9OQMt5nXWgZGrMItJePNhd01ygdGibKvBjpSJ3uyFmk9SAKTGXEsKd/e0PxP6+VYXjq5SJOM4SYjxeFmaSY0GFWtCMUcJQDQxhXwtxKeY8pxtEkapsQ3N9f/ksah1XX8KujSu1sEsci2SY7ZI+45ITUyAW5JHXCyQN5Ii/k1Xq0nq03631cWrImPVvkB6zPb1NKo94=</latexit><latexit sha1_base64="mJFEa/WoT8zGGVmonbxjzUiDw1A=">AAACHnicbVDLSsNAFJ3Ud3xVXboZLIKClEQUXRbduFSwrdCGMJnetGMnD2ZuxBLyJW78FTcuFBFc6d84fSzq48DA4Zz7mhOkUmh0nC+rNDM7N7+wuGQvr6yurZc3Nhs6yRSHOk9kom4CpkGKGOooUMJNqoBFgYRm0D8f+s07UFok8TUOUvAi1o1FKDhDI/nlY7u/R9sI9zialQcygyI3E6R/WxxMGQo6RR6yPvii2PfLFafqjED/EndCKmSCS7/80e4kPIsgRi6Z1i3XSdHLmULBJRR2O9OQMt5nXWgZGrMItJePNhd01ygdGibKvBjpSJ3uyFmk9SAKTGXEsKd/e0PxP6+VYXjq5SJOM4SYjxeFmaSY0GFWtCMUcJQDQxhXwtxKeY8pxtEkapsQ3N9f/ksah1XX8KujSu1sEsci2SY7ZI+45ITUyAW5JHXCyQN5Ii/k1Xq0nq03631cWrImPVvkB6zPb1NKo94=</latexit>
k(fakei, fakej)<latexit sha1_base64="EaRXWyt4MxQfjkd3aSBpkV/o7DA=">AAACHXicbZDLSgMxFIYz9VbrrerSTbAIFaTMSEGXRTcuK9gLtKVk0jNtbOZCckYsw7yIG1/FjQtFXLgR38b0stC2BwIf/39OTvK7kRQabfvHyqysrq1vZDdzW9s7u3v5/YO6DmPFocZDGaqmyzRIEUANBUpoRgqY70pouMPrsd94AKVFGNzhKIKOz/qB8ARnaKRuvpwbFmkb4REndyUKemnisSF0RXq2VL9PT7v5gl2yJ0UXwZlBgcyq2s1/tXshj30IkEumdcuxI+wkTKHgEtJcO9YQMT5kfWgZDJgPupNMNqf0xCg96oXKnADpRP07kTBf65Hvmk6f4UDPe2NxmdeK0bvsJCKIYoSATxd5saQY0nFUtCcUcJQjA4wrYd5K+YApxtEEmjMhOPNfXoT6eckxfFsuVK5mcWTJETkmReKQC1IhN6RKaoSTJ/JC3si79Wy9Wh/W57Q1Y81mDsm/sr5/AV4oo1o=</latexit><latexit sha1_base64="EaRXWyt4MxQfjkd3aSBpkV/o7DA=">AAACHXicbZDLSgMxFIYz9VbrrerSTbAIFaTMSEGXRTcuK9gLtKVk0jNtbOZCckYsw7yIG1/FjQtFXLgR38b0stC2BwIf/39OTvK7kRQabfvHyqysrq1vZDdzW9s7u3v5/YO6DmPFocZDGaqmyzRIEUANBUpoRgqY70pouMPrsd94AKVFGNzhKIKOz/qB8ARnaKRuvpwbFmkb4REndyUKemnisSF0RXq2VL9PT7v5gl2yJ0UXwZlBgcyq2s1/tXshj30IkEumdcuxI+wkTKHgEtJcO9YQMT5kfWgZDJgPupNMNqf0xCg96oXKnADpRP07kTBf65Hvmk6f4UDPe2NxmdeK0bvsJCKIYoSATxd5saQY0nFUtCcUcJQjA4wrYd5K+YApxtEEmjMhOPNfXoT6eckxfFsuVK5mcWTJETkmReKQC1IhN6RKaoSTJ/JC3si79Wy9Wh/W57Q1Y81mDsm/sr5/AV4oo1o=</latexit><latexit sha1_base64="EaRXWyt4MxQfjkd3aSBpkV/o7DA=">AAACHXicbZDLSgMxFIYz9VbrrerSTbAIFaTMSEGXRTcuK9gLtKVk0jNtbOZCckYsw7yIG1/FjQtFXLgR38b0stC2BwIf/39OTvK7kRQabfvHyqysrq1vZDdzW9s7u3v5/YO6DmPFocZDGaqmyzRIEUANBUpoRgqY70pouMPrsd94AKVFGNzhKIKOz/qB8ARnaKRuvpwbFmkb4REndyUKemnisSF0RXq2VL9PT7v5gl2yJ0UXwZlBgcyq2s1/tXshj30IkEumdcuxI+wkTKHgEtJcO9YQMT5kfWgZDJgPupNMNqf0xCg96oXKnADpRP07kTBf65Hvmk6f4UDPe2NxmdeK0bvsJCKIYoSATxd5saQY0nFUtCcUcJQjA4wrYd5K+YApxtEEmjMhOPNfXoT6eckxfFsuVK5mcWTJETkmReKQC1IhN6RKaoSTJ/JC3si79Wy9Wh/W57Q1Y81mDsm/sr5/AV4oo1o=</latexit><latexit sha1_base64="EaRXWyt4MxQfjkd3aSBpkV/o7DA=">AAACHXicbZDLSgMxFIYz9VbrrerSTbAIFaTMSEGXRTcuK9gLtKVk0jNtbOZCckYsw7yIG1/FjQtFXLgR38b0stC2BwIf/39OTvK7kRQabfvHyqysrq1vZDdzW9s7u3v5/YO6DmPFocZDGaqmyzRIEUANBUpoRgqY70pouMPrsd94AKVFGNzhKIKOz/qB8ARnaKRuvpwbFmkb4REndyUKemnisSF0RXq2VL9PT7v5gl2yJ0UXwZlBgcyq2s1/tXshj30IkEumdcuxI+wkTKHgEtJcO9YQMT5kfWgZDJgPupNMNqf0xCg96oXKnADpRP07kTBf65Hvmk6f4UDPe2NxmdeK0bvsJCKIYoSATxd5saQY0nFUtCcUcJQjA4wrYd5K+YApxtEEmjMhOPNfXoT6eckxfFsuVK5mcWTJETkmReKQC1IhN6RKaoSTJ/JC3si79Wy9Wh/W57Q1Y81mDsm/sr5/AV4oo1o=</latexit>
k(reali, realj)<latexit sha1_base64="7BZkgUJbjSOsYar4HNNAXTbcHLQ=">AAACH3icbVBLS0JBGJ1rL7OX1bLNkAQGIfdGVEupTUuDfICKzB0/dXLug5nvRnK5/6RNf6VNiyKinf+m8eqi1AMDh3O+1xw3lEKjbY+tzMrq2vpGdjO3tb2zu5ffP6jpIFIcqjyQgWq4TIMUPlRRoIRGqIB5roS6O7yd+PUnUFoE/gOOQmh7rO+LnuAMjdTJX+aGRdpCeMZ0VuzKCJLYTJAdkZwtNx6T006+YJfsFHSRODNSIDNUOvmfVjfgkQc+csm0bjp2iO2YKRRcQpJrRRpCxoesD01DfeaBbsfp6oSeGKVLe4Eyz0eaqn87YuZpPfJcU+kxHOh5byIu85oR9q7bsfDDCMHn00W9SFIM6CQs2hUKOMqRIYwrYW6lfMAU42gizZkQnPkvL5Laeckx/P6iUL6ZxZElR+SYFIlDrkiZ3JEKqRJOXsgb+SCf1qv1bn1Z39PSjDXrOST/YI1/AUc+pGI=</latexit><latexit sha1_base64="7BZkgUJbjSOsYar4HNNAXTbcHLQ=">AAACH3icbVBLS0JBGJ1rL7OX1bLNkAQGIfdGVEupTUuDfICKzB0/dXLug5nvRnK5/6RNf6VNiyKinf+m8eqi1AMDh3O+1xw3lEKjbY+tzMrq2vpGdjO3tb2zu5ffP6jpIFIcqjyQgWq4TIMUPlRRoIRGqIB5roS6O7yd+PUnUFoE/gOOQmh7rO+LnuAMjdTJX+aGRdpCeMZ0VuzKCJLYTJAdkZwtNx6T006+YJfsFHSRODNSIDNUOvmfVjfgkQc+csm0bjp2iO2YKRRcQpJrRRpCxoesD01DfeaBbsfp6oSeGKVLe4Eyz0eaqn87YuZpPfJcU+kxHOh5byIu85oR9q7bsfDDCMHn00W9SFIM6CQs2hUKOMqRIYwrYW6lfMAU42gizZkQnPkvL5Laeckx/P6iUL6ZxZElR+SYFIlDrkiZ3JEKqRJOXsgb+SCf1qv1bn1Z39PSjDXrOST/YI1/AUc+pGI=</latexit><latexit sha1_base64="7BZkgUJbjSOsYar4HNNAXTbcHLQ=">AAACH3icbVBLS0JBGJ1rL7OX1bLNkAQGIfdGVEupTUuDfICKzB0/dXLug5nvRnK5/6RNf6VNiyKinf+m8eqi1AMDh3O+1xw3lEKjbY+tzMrq2vpGdjO3tb2zu5ffP6jpIFIcqjyQgWq4TIMUPlRRoIRGqIB5roS6O7yd+PUnUFoE/gOOQmh7rO+LnuAMjdTJX+aGRdpCeMZ0VuzKCJLYTJAdkZwtNx6T006+YJfsFHSRODNSIDNUOvmfVjfgkQc+csm0bjp2iO2YKRRcQpJrRRpCxoesD01DfeaBbsfp6oSeGKVLe4Eyz0eaqn87YuZpPfJcU+kxHOh5byIu85oR9q7bsfDDCMHn00W9SFIM6CQs2hUKOMqRIYwrYW6lfMAU42gizZkQnPkvL5Laeckx/P6iUL6ZxZElR+SYFIlDrkiZ3JEKqRJOXsgb+SCf1qv1bn1Z39PSjDXrOST/YI1/AUc+pGI=</latexit><latexit sha1_base64="7BZkgUJbjSOsYar4HNNAXTbcHLQ=">AAACH3icbVBLS0JBGJ1rL7OX1bLNkAQGIfdGVEupTUuDfICKzB0/dXLug5nvRnK5/6RNf6VNiyKinf+m8eqi1AMDh3O+1xw3lEKjbY+tzMrq2vpGdjO3tb2zu5ffP6jpIFIcqjyQgWq4TIMUPlRRoIRGqIB5roS6O7yd+PUnUFoE/gOOQmh7rO+LnuAMjdTJX+aGRdpCeMZ0VuzKCJLYTJAdkZwtNx6T006+YJfsFHSRODNSIDNUOvmfVjfgkQc+csm0bjp2iO2YKRRcQpJrRRpCxoesD01DfeaBbsfp6oSeGKVLe4Eyz0eaqn87YuZpPfJcU+kxHOh5byIu85oR9q7bsfDDCMHn00W9SFIM6CQs2hUKOMqRIYwrYW6lfMAU42gizZkQnPkvL5Laeckx/P6iUL6ZxZElR+SYFIlDrkiZ3JEKqRJOXsgb+SCf1qv1bn1Z39PSjDXrOST/YI1/AUc+pGI=</latexit>
Closed form solution:
f ?(t) ∝ EP[k(X , t)]− EQ[k(Y , t)]
Unbiased estimator:
MMD2
= 1n(n−1)
∑i 6=j
k(reali, realj) + k(fakei, fakej)
− 2n2
∑i ,j
k(reali, fakej)
Smooth optimal critic:
sample-optimal critics population-optimal critics
Wasserstein MMD
Maximum Mean Discrepancy for GANs
MMD GANs optimize critic in kernel:
kψ(x , y) = kbase(φψ(x), φψ(y))
infθ
supψ
MMD2kψ
(P,Qθ)︸ ︷︷ ︸DMMD(P,Qθ)
MMD� <latexit sha1_base64="VOUevIlyKh9xmvvFY/TPO0OQzkc=">AAACM3icbVC7TsMwFHV4lvJqy8hiUSExVQlCArYKFsYi0YeURJXjOK1VJ7FspyKK8hms8BV8DGJDrPwDbpKBtlzJ1vE5914fHY8zKpVpfhgbm1vbO7u1vfr+weHRcaPZGsg4EZj0ccxiMfKQJIxGpK+oYmTEBUGhx8jQm90v9OGcCEnj6EmlnLghmkQ0oBgpTdkOn9Jx5nBJ83GjbXbMouA6sCrQBlX1xk2j5fgxTkISKcyQlLZlcuVmSCiKGcnrTiIJR3iGJsTWMEIhkW5WeM7huWZ8GMRCn0jBgv07kaFQyjT0dGeI1FSuagvyP81OVHDjZjTiiSIRLj8KEgZVDBcBQJ8KghVLNUBYUO0V4ikSCCsd09Imf065rFw/l7brjk8CHXXxysJ0IlCaZ+Vtdm7zuk7RWs1sHQwuO5bGj1ft7l2VZw2cgjNwASxwDbrgAfRAH2AQgxfwCt6Md+PT+DK+y9YNo5o5AUtl/PwC7Uyrxg==</latexit><latexit sha1_base64="VOUevIlyKh9xmvvFY/TPO0OQzkc=">AAACM3icbVC7TsMwFHV4lvJqy8hiUSExVQlCArYKFsYi0YeURJXjOK1VJ7FspyKK8hms8BV8DGJDrPwDbpKBtlzJ1vE5914fHY8zKpVpfhgbm1vbO7u1vfr+weHRcaPZGsg4EZj0ccxiMfKQJIxGpK+oYmTEBUGhx8jQm90v9OGcCEnj6EmlnLghmkQ0oBgpTdkOn9Jx5nBJ83GjbXbMouA6sCrQBlX1xk2j5fgxTkISKcyQlLZlcuVmSCiKGcnrTiIJR3iGJsTWMEIhkW5WeM7huWZ8GMRCn0jBgv07kaFQyjT0dGeI1FSuagvyP81OVHDjZjTiiSIRLj8KEgZVDBcBQJ8KghVLNUBYUO0V4ikSCCsd09Imf065rFw/l7brjk8CHXXxysJ0IlCaZ+Vtdm7zuk7RWs1sHQwuO5bGj1ft7l2VZw2cgjNwASxwDbrgAfRAH2AQgxfwCt6Md+PT+DK+y9YNo5o5AUtl/PwC7Uyrxg==</latexit><latexit sha1_base64="VOUevIlyKh9xmvvFY/TPO0OQzkc=">AAACM3icbVC7TsMwFHV4lvJqy8hiUSExVQlCArYKFsYi0YeURJXjOK1VJ7FspyKK8hms8BV8DGJDrPwDbpKBtlzJ1vE5914fHY8zKpVpfhgbm1vbO7u1vfr+weHRcaPZGsg4EZj0ccxiMfKQJIxGpK+oYmTEBUGhx8jQm90v9OGcCEnj6EmlnLghmkQ0oBgpTdkOn9Jx5nBJ83GjbXbMouA6sCrQBlX1xk2j5fgxTkISKcyQlLZlcuVmSCiKGcnrTiIJR3iGJsTWMEIhkW5WeM7huWZ8GMRCn0jBgv07kaFQyjT0dGeI1FSuagvyP81OVHDjZjTiiSIRLj8KEgZVDBcBQJ8KghVLNUBYUO0V4ikSCCsd09Imf065rFw/l7brjk8CHXXxysJ0IlCaZ+Vtdm7zuk7RWs1sHQwuO5bGj1ft7l2VZw2cgjNwASxwDbrgAfRAH2AQgxfwCt6Md+PT+DK+y9YNo5o5AUtl/PwC7Uyrxg==</latexit><latexit sha1_base64="VOUevIlyKh9xmvvFY/TPO0OQzkc=">AAACM3icbVC7TsMwFHV4lvJqy8hiUSExVQlCArYKFsYi0YeURJXjOK1VJ7FspyKK8hms8BV8DGJDrPwDbpKBtlzJ1vE5914fHY8zKpVpfhgbm1vbO7u1vfr+weHRcaPZGsg4EZj0ccxiMfKQJIxGpK+oYmTEBUGhx8jQm90v9OGcCEnj6EmlnLghmkQ0oBgpTdkOn9Jx5nBJ83GjbXbMouA6sCrQBlX1xk2j5fgxTkISKcyQlLZlcuVmSCiKGcnrTiIJR3iGJsTWMEIhkW5WeM7huWZ8GMRCn0jBgv07kaFQyjT0dGeI1FSuagvyP81OVHDjZjTiiSIRLj8KEgZVDBcBQJ8KghVLNUBYUO0V4ikSCCsd09Imf065rFw/l7brjk8CHXXxysJ0IlCaZ+Vtdm7zuk7RWs1sHQwuO5bGj1ft7l2VZw2cgjNwASxwDbrgAfRAH2AQgxfwCt6Md+PT+DK+y9YNo5o5AUtl/PwC7Uyrxg==</latexit>
� <latexit sha1_base64="VOUevIlyKh9xmvvFY/TPO0OQzkc=">AAACM3icbVC7TsMwFHV4lvJqy8hiUSExVQlCArYKFsYi0YeURJXjOK1VJ7FspyKK8hms8BV8DGJDrPwDbpKBtlzJ1vE5914fHY8zKpVpfhgbm1vbO7u1vfr+weHRcaPZGsg4EZj0ccxiMfKQJIxGpK+oYmTEBUGhx8jQm90v9OGcCEnj6EmlnLghmkQ0oBgpTdkOn9Jx5nBJ83GjbXbMouA6sCrQBlX1xk2j5fgxTkISKcyQlLZlcuVmSCiKGcnrTiIJR3iGJsTWMEIhkW5WeM7huWZ8GMRCn0jBgv07kaFQyjT0dGeI1FSuagvyP81OVHDjZjTiiSIRLj8KEgZVDBcBQJ8KghVLNUBYUO0V4ikSCCsd09Imf065rFw/l7brjk8CHXXxysJ0IlCaZ+Vtdm7zuk7RWs1sHQwuO5bGj1ft7l2VZw2cgjNwASxwDbrgAfRAH2AQgxfwCt6Md+PT+DK+y9YNo5o5AUtl/PwC7Uyrxg==</latexit><latexit sha1_base64="VOUevIlyKh9xmvvFY/TPO0OQzkc=">AAACM3icbVC7TsMwFHV4lvJqy8hiUSExVQlCArYKFsYi0YeURJXjOK1VJ7FspyKK8hms8BV8DGJDrPwDbpKBtlzJ1vE5914fHY8zKpVpfhgbm1vbO7u1vfr+weHRcaPZGsg4EZj0ccxiMfKQJIxGpK+oYmTEBUGhx8jQm90v9OGcCEnj6EmlnLghmkQ0oBgpTdkOn9Jx5nBJ83GjbXbMouA6sCrQBlX1xk2j5fgxTkISKcyQlLZlcuVmSCiKGcnrTiIJR3iGJsTWMEIhkW5WeM7huWZ8GMRCn0jBgv07kaFQyjT0dGeI1FSuagvyP81OVHDjZjTiiSIRLj8KEgZVDBcBQJ8KghVLNUBYUO0V4ikSCCsd09Imf065rFw/l7brjk8CHXXxysJ0IlCaZ+Vtdm7zuk7RWs1sHQwuO5bGj1ft7l2VZw2cgjNwASxwDbrgAfRAH2AQgxfwCt6Md+PT+DK+y9YNo5o5AUtl/PwC7Uyrxg==</latexit><latexit sha1_base64="VOUevIlyKh9xmvvFY/TPO0OQzkc=">AAACM3icbVC7TsMwFHV4lvJqy8hiUSExVQlCArYKFsYi0YeURJXjOK1VJ7FspyKK8hms8BV8DGJDrPwDbpKBtlzJ1vE5914fHY8zKpVpfhgbm1vbO7u1vfr+weHRcaPZGsg4EZj0ccxiMfKQJIxGpK+oYmTEBUGhx8jQm90v9OGcCEnj6EmlnLghmkQ0oBgpTdkOn9Jx5nBJ83GjbXbMouA6sCrQBlX1xk2j5fgxTkISKcyQlLZlcuVmSCiKGcnrTiIJR3iGJsTWMEIhkW5WeM7huWZ8GMRCn0jBgv07kaFQyjT0dGeI1FSuagvyP81OVHDjZjTiiSIRLj8KEgZVDBcBQJ8KghVLNUBYUO0V4ikSCCsd09Imf065rFw/l7brjk8CHXXxysJ0IlCaZ+Vtdm7zuk7RWs1sHQwuO5bGj1ft7l2VZw2cgjNwASxwDbrgAfRAH2AQgxfwCt6Md+PT+DK+y9YNo5o5AUtl/PwC7Uyrxg==</latexit><latexit sha1_base64="VOUevIlyKh9xmvvFY/TPO0OQzkc=">AAACM3icbVC7TsMwFHV4lvJqy8hiUSExVQlCArYKFsYi0YeURJXjOK1VJ7FspyKK8hms8BV8DGJDrPwDbpKBtlzJ1vE5914fHY8zKpVpfhgbm1vbO7u1vfr+weHRcaPZGsg4EZj0ccxiMfKQJIxGpK+oYmTEBUGhx8jQm90v9OGcCEnj6EmlnLghmkQ0oBgpTdkOn9Jx5nBJ83GjbXbMouA6sCrQBlX1xk2j5fgxTkISKcyQlLZlcuVmSCiKGcnrTiIJR3iGJsTWMEIhkW5WeM7huWZ8GMRCn0jBgv07kaFQyjT0dGeI1FSuagvyP81OVHDjZjTiiSIRLj8KEgZVDBcBQJ8KghVLNUBYUO0V4ikSCCsd09Imf065rFw/l7brjk8CHXXxysJ0IlCaZ+Vtdm7zuk7RWs1sHQwuO5bGj1ft7l2VZw2cgjNwASxwDbrgAfRAH2AQgxfwCt6Md+PT+DK+y9YNo5o5AUtl/PwC7Uyrxg==</latexit>
Can also use gradient penalty [2].
Continuity under weak topology
DMMD not continuous / differentiable in general:
x<latexit sha1_base64="IArQpDG4Gw7Ax+5Wri9CZWKD4Bo=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxBfsBbSib7aRdu9mE3Y1YQn+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e2s3n5EpXks780kQT+iQ8lDzqixVuOpX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu1mzyOIpzAKZyDB1dQgzuoQxMYIDzDK7w5D86L8+58LFoLTj5zDH/kfP4A5juM/A==</latexit><latexit sha1_base64="IArQpDG4Gw7Ax+5Wri9CZWKD4Bo=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxBfsBbSib7aRdu9mE3Y1YQn+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e2s3n5EpXks780kQT+iQ8lDzqixVuOpX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu1mzyOIpzAKZyDB1dQgzuoQxMYIDzDK7w5D86L8+58LFoLTj5zDH/kfP4A5juM/A==</latexit><latexit sha1_base64="IArQpDG4Gw7Ax+5Wri9CZWKD4Bo=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxBfsBbSib7aRdu9mE3Y1YQn+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e2s3n5EpXks780kQT+iQ8lDzqixVuOpX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu1mzyOIpzAKZyDB1dQgzuoQxMYIDzDK7w5D86L8+58LFoLTj5zDH/kfP4A5juM/A==</latexit><latexit sha1_base64="IArQpDG4Gw7Ax+5Wri9CZWKD4Bo=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEfRY9OKxBfsBbSib7aRdu9mE3Y1YQn+BFw+KePUnefPfuG1z0NYXFh7emWFn3iARXBvX/XYKa+sbm1vF7dLO7t7+QfnwqKXjVDFssljEqhNQjYJLbBpuBHYShTQKBLaD8e2s3n5EpXks780kQT+iQ8lDzqixVuOpX664VXcusgpeDhXIVe+Xv3qDmKURSsME1brruYnxM6oMZwKnpV6qMaFsTIfYtShphNrP5otOyZl1BiSMlX3SkLn7eyKjkdaTKLCdETUjvVybmf/VuqkJr/2MyyQ1KNniozAVxMRkdjUZcIXMiIkFyhS3uxI2oooyY7Mp2RC85ZNXoXVR9Sw3Liu1mzyOIpzAKZyDB1dQgzuoQxMYIDzDK7w5D86L8+58LFoLTj5zDH/kfP4A5juM/A==</latexit>
P = �0<latexit sha1_base64="5NTfpX2cu/5hSk4hrfl2Ks7B01o=">AAACEXicbVC7SgNBFJ2Nrxhfq5Y2g0FIFXZF0EYI2lhGMA/ILmF2cpMMmX0wc1cMy/6Cjb9iY6GIrZ2df+PkUWjigYHDOffOnDlBIoVGx/m2Ciura+sbxc3S1vbO7p69f9DUcao4NHgsY9UOmAYpImigQAntRAELAwmtYHQ98Vv3oLSIozscJ+CHbBCJvuAMjdS1Kx7CA07vyQKZQp55IcNhEGT1nF5SrwcSWdfJadcuO1VnCrpM3DkpkznqXfvL68U8DSFCLpnWHddJ0M+YQsEl5CUv1ZAwPmID6BgasRC0n02T5PTEKD3aj5U5EdKp+nsjY6HW4zAwk5O4etGbiP95nRT7F34moiRFiPjsoX4qKcZ0Ug/tCQUc5dgQxpUwWSkfMsU4mhJLpgR38cvLpHladQ2/PSvXruZ1FMkROSYV4pJzUiM3pE4ahJNH8kxeyZv1ZL1Y79bHbLRgzXcOyR9Ynz8J8Z23</latexit><latexit sha1_base64="5NTfpX2cu/5hSk4hrfl2Ks7B01o=">AAACEXicbVC7SgNBFJ2Nrxhfq5Y2g0FIFXZF0EYI2lhGMA/ILmF2cpMMmX0wc1cMy/6Cjb9iY6GIrZ2df+PkUWjigYHDOffOnDlBIoVGx/m2Ciura+sbxc3S1vbO7p69f9DUcao4NHgsY9UOmAYpImigQAntRAELAwmtYHQ98Vv3oLSIozscJ+CHbBCJvuAMjdS1Kx7CA07vyQKZQp55IcNhEGT1nF5SrwcSWdfJadcuO1VnCrpM3DkpkznqXfvL68U8DSFCLpnWHddJ0M+YQsEl5CUv1ZAwPmID6BgasRC0n02T5PTEKD3aj5U5EdKp+nsjY6HW4zAwk5O4etGbiP95nRT7F34moiRFiPjsoX4qKcZ0Ug/tCQUc5dgQxpUwWSkfMsU4mhJLpgR38cvLpHladQ2/PSvXruZ1FMkROSYV4pJzUiM3pE4ahJNH8kxeyZv1ZL1Y79bHbLRgzXcOyR9Ynz8J8Z23</latexit><latexit sha1_base64="5NTfpX2cu/5hSk4hrfl2Ks7B01o=">AAACEXicbVC7SgNBFJ2Nrxhfq5Y2g0FIFXZF0EYI2lhGMA/ILmF2cpMMmX0wc1cMy/6Cjb9iY6GIrZ2df+PkUWjigYHDOffOnDlBIoVGx/m2Ciura+sbxc3S1vbO7p69f9DUcao4NHgsY9UOmAYpImigQAntRAELAwmtYHQ98Vv3oLSIozscJ+CHbBCJvuAMjdS1Kx7CA07vyQKZQp55IcNhEGT1nF5SrwcSWdfJadcuO1VnCrpM3DkpkznqXfvL68U8DSFCLpnWHddJ0M+YQsEl5CUv1ZAwPmID6BgasRC0n02T5PTEKD3aj5U5EdKp+nsjY6HW4zAwk5O4etGbiP95nRT7F34moiRFiPjsoX4qKcZ0Ug/tCQUc5dgQxpUwWSkfMsU4mhJLpgR38cvLpHladQ2/PSvXruZ1FMkROSYV4pJzUiM3pE4ahJNH8kxeyZv1ZL1Y79bHbLRgzXcOyR9Ynz8J8Z23</latexit><latexit sha1_base64="5NTfpX2cu/5hSk4hrfl2Ks7B01o=">AAACEXicbVC7SgNBFJ2Nrxhfq5Y2g0FIFXZF0EYI2lhGMA/ILmF2cpMMmX0wc1cMy/6Cjb9iY6GIrZ2df+PkUWjigYHDOffOnDlBIoVGx/m2Ciura+sbxc3S1vbO7p69f9DUcao4NHgsY9UOmAYpImigQAntRAELAwmtYHQ98Vv3oLSIozscJ+CHbBCJvuAMjdS1Kx7CA07vyQKZQp55IcNhEGT1nF5SrwcSWdfJadcuO1VnCrpM3DkpkznqXfvL68U8DSFCLpnWHddJ0M+YQsEl5CUv1ZAwPmID6BgasRC0n02T5PTEKD3aj5U5EdKp+nsjY6HW4zAwk5O4etGbiP95nRT7F34moiRFiPjsoX4qKcZ0Ug/tCQUc5dgQxpUwWSkfMsU4mhJLpgR38cvLpHladQ2/PSvXruZ1FMkROSYV4pJzUiM3pE4ahJNH8kxeyZv1ZL1Y79bHbLRgzXcOyR9Ynz8J8Z23</latexit>
Q✓ = �✓<latexit sha1_base64="p/S50GX6DibhmsVwx9LpstUKV8Y=">AAACIHicbVDLSgMxFM3UV62vqks3wSK4KjMi1I1QdOOyBfuATimZzG0bmnmQ3BHLMJ/ixl9x40IR3enXmD4QbT0QOJxzb3JyvFgKjbb9aeVWVtfWN/Kbha3tnd294v5BU0eJ4tDgkYxU22MapAihgQIltGMFLPAktLzR9cRv3YHSIgpvcRxDN2CDUPQFZ2ikXrHiItzj9J5UgZ+lbsBw6HlpPeulLg4BWUYvqeuDRPajZLRXLNllewq6TJw5KZE5ar3ih+tHPAkgRC6Z1h3HjrGbMoWCS8gKbqIhZnzEBtAxNGQB6G46DZbRE6P4tB8pc0KkU/X3RsoCrceBZyYn8fWiNxH/8zoJ9i+6qQjjBCHks4f6iaQY0Ulb1BcKOMqxIYwrYbJSPmSKcTSdFkwJzuKXl0nzrOwYXj8vVa/mdeTJETkmp8QhFVIlN6RGGoSTB/JEXsir9Wg9W2/W+2w0Z813DskfWF/fW2Wk5A==</latexit><latexit sha1_base64="p/S50GX6DibhmsVwx9LpstUKV8Y=">AAACIHicbVDLSgMxFM3UV62vqks3wSK4KjMi1I1QdOOyBfuATimZzG0bmnmQ3BHLMJ/ixl9x40IR3enXmD4QbT0QOJxzb3JyvFgKjbb9aeVWVtfWN/Kbha3tnd294v5BU0eJ4tDgkYxU22MapAihgQIltGMFLPAktLzR9cRv3YHSIgpvcRxDN2CDUPQFZ2ikXrHiItzj9J5UgZ+lbsBw6HlpPeulLg4BWUYvqeuDRPajZLRXLNllewq6TJw5KZE5ar3ih+tHPAkgRC6Z1h3HjrGbMoWCS8gKbqIhZnzEBtAxNGQB6G46DZbRE6P4tB8pc0KkU/X3RsoCrceBZyYn8fWiNxH/8zoJ9i+6qQjjBCHks4f6iaQY0Ulb1BcKOMqxIYwrYbJSPmSKcTSdFkwJzuKXl0nzrOwYXj8vVa/mdeTJETkmp8QhFVIlN6RGGoSTB/JEXsir9Wg9W2/W+2w0Z813DskfWF/fW2Wk5A==</latexit><latexit sha1_base64="p/S50GX6DibhmsVwx9LpstUKV8Y=">AAACIHicbVDLSgMxFM3UV62vqks3wSK4KjMi1I1QdOOyBfuATimZzG0bmnmQ3BHLMJ/ixl9x40IR3enXmD4QbT0QOJxzb3JyvFgKjbb9aeVWVtfWN/Kbha3tnd294v5BU0eJ4tDgkYxU22MapAihgQIltGMFLPAktLzR9cRv3YHSIgpvcRxDN2CDUPQFZ2ikXrHiItzj9J5UgZ+lbsBw6HlpPeulLg4BWUYvqeuDRPajZLRXLNllewq6TJw5KZE5ar3ih+tHPAkgRC6Z1h3HjrGbMoWCS8gKbqIhZnzEBtAxNGQB6G46DZbRE6P4tB8pc0KkU/X3RsoCrceBZyYn8fWiNxH/8zoJ9i+6qQjjBCHks4f6iaQY0Ulb1BcKOMqxIYwrYbJSPmSKcTSdFkwJzuKXl0nzrOwYXj8vVa/mdeTJETkmp8QhFVIlN6RGGoSTB/JEXsir9Wg9W2/W+2w0Z813DskfWF/fW2Wk5A==</latexit><latexit sha1_base64="p/S50GX6DibhmsVwx9LpstUKV8Y=">AAACIHicbVDLSgMxFM3UV62vqks3wSK4KjMi1I1QdOOyBfuATimZzG0bmnmQ3BHLMJ/ixl9x40IR3enXmD4QbT0QOJxzb3JyvFgKjbb9aeVWVtfWN/Kbha3tnd294v5BU0eJ4tDgkYxU22MapAihgQIltGMFLPAktLzR9cRv3YHSIgpvcRxDN2CDUPQFZ2ikXrHiItzj9J5UgZ+lbsBw6HlpPeulLg4BWUYvqeuDRPajZLRXLNllewq6TJw5KZE5ar3ih+tHPAkgRC6Z1h3HjrGbMoWCS8gKbqIhZnzEBtAxNGQB6G46DZbRE6P4tB8pc0KkU/X3RsoCrceBZyYn8fWiNxH/8zoJ9i+6qQjjBCHks4f6iaQY0Ulb1BcKOMqxIYwrYbJSPmSKcTSdFkwJzuKXl0nzrOwYXj8vVa/mdeTJETkmp8QhFVIlN6RGGoSTB/JEXsir9Wg9W2/W+2w0Z813DskfWF/fW2Wk5A==</latexit>
✓<latexit sha1_base64="9nw3eiH0vxpghXegSZAQaK2DTls=">AAAB7nicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3biMYB6QDKGnU0ma9DzorhHCkI9w40IRt36PO//GnmQWmnih4XCriq66QaKkIdf9dtbWNza3tks75d29/YPDytFxy8SpFtgUsYp1J+AGlYywSZIUdhKNPAwUtoPJXV5vP6E2Mo4eaZqgH/JRJIdScLJWu0djJF7uV6puzZ2LrYJXQBUKNfqVr94gFmmIEQnFjel6bkJ+xjVJoXBW7qUGEy4mfIRdixEP0fjZfN0ZO7fOgA1jbV9EbO7+nsh4aMw0DGxnyGlslmu5+V+tm9Lwxs9klKSEkVh8NEwVo5jlt7OB1ChITS1woaXdlYkx11yQTSgPwVs+eRValzXP8sNVtX5bxFGCUziDC/DgGupwDw1ogoAJPMMrvDmJ8+K8Ox+L1jWnmDmBP3I+fwDagY88</latexit><latexit sha1_base64="9nw3eiH0vxpghXegSZAQaK2DTls=">AAAB7nicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3biMYB6QDKGnU0ma9DzorhHCkI9w40IRt36PO//GnmQWmnih4XCriq66QaKkIdf9dtbWNza3tks75d29/YPDytFxy8SpFtgUsYp1J+AGlYywSZIUdhKNPAwUtoPJXV5vP6E2Mo4eaZqgH/JRJIdScLJWu0djJF7uV6puzZ2LrYJXQBUKNfqVr94gFmmIEQnFjel6bkJ+xjVJoXBW7qUGEy4mfIRdixEP0fjZfN0ZO7fOgA1jbV9EbO7+nsh4aMw0DGxnyGlslmu5+V+tm9Lwxs9klKSEkVh8NEwVo5jlt7OB1ChITS1woaXdlYkx11yQTSgPwVs+eRValzXP8sNVtX5bxFGCUziDC/DgGupwDw1ogoAJPMMrvDmJ8+K8Ox+L1jWnmDmBP3I+fwDagY88</latexit><latexit sha1_base64="9nw3eiH0vxpghXegSZAQaK2DTls=">AAAB7nicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3biMYB6QDKGnU0ma9DzorhHCkI9w40IRt36PO//GnmQWmnih4XCriq66QaKkIdf9dtbWNza3tks75d29/YPDytFxy8SpFtgUsYp1J+AGlYywSZIUdhKNPAwUtoPJXV5vP6E2Mo4eaZqgH/JRJIdScLJWu0djJF7uV6puzZ2LrYJXQBUKNfqVr94gFmmIEQnFjel6bkJ+xjVJoXBW7qUGEy4mfIRdixEP0fjZfN0ZO7fOgA1jbV9EbO7+nsh4aMw0DGxnyGlslmu5+V+tm9Lwxs9klKSEkVh8NEwVo5jlt7OB1ChITS1woaXdlYkx11yQTSgPwVs+eRValzXP8sNVtX5bxFGCUziDC/DgGupwDw1ogoAJPMMrvDmJ8+K8Ox+L1jWnmDmBP3I+fwDagY88</latexit><latexit sha1_base64="9nw3eiH0vxpghXegSZAQaK2DTls=">AAAB7nicbZDLSgNBEEVrfMb4irp00xgEV2FGBF0G3biMYB6QDKGnU0ma9DzorhHCkI9w40IRt36PO//GnmQWmnih4XCriq66QaKkIdf9dtbWNza3tks75d29/YPDytFxy8SpFtgUsYp1J+AGlYywSZIUdhKNPAwUtoPJXV5vP6E2Mo4eaZqgH/JRJIdScLJWu0djJF7uV6puzZ2LrYJXQBUKNfqVr94gFmmIEQnFjel6bkJ+xjVJoXBW7qUGEy4mfIRdixEP0fjZfN0ZO7fOgA1jbV9EbO7+nsh4aMw0DGxnyGlslmu5+V+tm9Lwxs9klKSEkVh8NEwVo5jlt7OB1ChITS1woaXdlYkx11yQTSgPwVs+eRValzXP8sNVtX5bxFGCUziDC/DgGupwDw1ogoAJPMMrvDmJ8+K8Ox+L1jWnmDmBP3I+fwDagY88</latexit>
0<latexit sha1_base64="mSC7jR5dlnc4mUSSs2Y7zLybdHk=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjraY7KFfcqrsQWQcvhwrkagzKX/1hzNIIpWGCat3z3MT4GVWGM4GzUj/VmFA2oSPsWZQ0Qu1ni0Vn5MI6QxLGyj5pyML9PZHRSOtpFNjOiJqxXq3Nzf9qvdSEN37GZZIalGz5UZgKYmIyv5oMuUJmxNQCZYrbXQkbU0WZsdmUbAje6snr0L6qepab15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh85nz95G4y0</latexit><latexit sha1_base64="mSC7jR5dlnc4mUSSs2Y7zLybdHk=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjraY7KFfcqrsQWQcvhwrkagzKX/1hzNIIpWGCat3z3MT4GVWGM4GzUj/VmFA2oSPsWZQ0Qu1ni0Vn5MI6QxLGyj5pyML9PZHRSOtpFNjOiJqxXq3Nzf9qvdSEN37GZZIalGz5UZgKYmIyv5oMuUJmxNQCZYrbXQkbU0WZsdmUbAje6snr0L6qepab15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh85nz95G4y0</latexit><latexit sha1_base64="mSC7jR5dlnc4mUSSs2Y7zLybdHk=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjraY7KFfcqrsQWQcvhwrkagzKX/1hzNIIpWGCat3z3MT4GVWGM4GzUj/VmFA2oSPsWZQ0Qu1ni0Vn5MI6QxLGyj5pyML9PZHRSOtpFNjOiJqxXq3Nzf9qvdSEN37GZZIalGz5UZgKYmIyv5oMuUJmxNQCZYrbXQkbU0WZsdmUbAje6snr0L6qepab15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh85nz95G4y0</latexit><latexit sha1_base64="mSC7jR5dlnc4mUSSs2Y7zLybdHk=">AAAB6HicbZBNS8NAEIYn9avWr6pHL4tF8FQSEeqx6MVjC/YD2lA220m7drMJuxuhhP4CLx4U8epP8ua/cdvmoK0vLDy8M8POvEEiuDau++0UNja3tneKu6W9/YPDo/LxSVvHqWLYYrGIVTegGgWX2DLcCOwmCmkUCOwEk7t5vfOESvNYPphpgn5ER5KHnFFjraY7KFfcqrsQWQcvhwrkagzKX/1hzNIIpWGCat3z3MT4GVWGM4GzUj/VmFA2oSPsWZQ0Qu1ni0Vn5MI6QxLGyj5pyML9PZHRSOtpFNjOiJqxXq3Nzf9qvdSEN37GZZIalGz5UZgKYmIyv5oMuUJmxNQCZYrbXQkbU0WZsdmUbAje6snr0L6qepab15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh85nz95G4y0</latexit>
� (x) = x<latexit sha1_base64="yC47CR6xz6bMMbsCM/qVyQIvACI=">AAACJXicbZDLSgMxFIYzXmu9VV26CRah3ZQZEXQjFN24rGAv0JaSSU/bYCYzJGekZZhn8DF8Arf6BO5EcOXK9zC9LGzrgYSP/1z5/UgKg6775aysrq1vbGa2sts7u3v7uYPDmgljzaHKQxnqhs8MSKGgigIlNCINLPAl1P2Hm3G+/gjaiFDd4yiCdsD6SvQEZ2ilTq7YQhjiZE4Saqb6kCataCA69jciLQyLV2Ogw7STy7sldxJ0GbwZ5MksKp3cT6sb8jgAhVwyY5qeG2E7YRoFl5BmW7GBiPEH1oemRcUCMO1kckpKT63Spb1Q26eQTtS/HQkLjBkFvq0MGA7MYm4s/pdrxti7bCdCRTGC4tNFvVhSDOnYH9oVGjjKkQXGtbC3Uj5gmnG0Ls5tGU5Ptb54iy4sQ+2s5Fm+O8+Xr2cOZcgxOSEF4pELUia3pEKqhJMn8kJeyZvz7Lw7H87ntHTFmfUckblwvn8BtuOnSQ==</latexit><latexit sha1_base64="yC47CR6xz6bMMbsCM/qVyQIvACI=">AAACJXicbZDLSgMxFIYzXmu9VV26CRah3ZQZEXQjFN24rGAv0JaSSU/bYCYzJGekZZhn8DF8Arf6BO5EcOXK9zC9LGzrgYSP/1z5/UgKg6775aysrq1vbGa2sts7u3v7uYPDmgljzaHKQxnqhs8MSKGgigIlNCINLPAl1P2Hm3G+/gjaiFDd4yiCdsD6SvQEZ2ilTq7YQhjiZE4Saqb6kCataCA69jciLQyLV2Ogw7STy7sldxJ0GbwZ5MksKp3cT6sb8jgAhVwyY5qeG2E7YRoFl5BmW7GBiPEH1oemRcUCMO1kckpKT63Spb1Q26eQTtS/HQkLjBkFvq0MGA7MYm4s/pdrxti7bCdCRTGC4tNFvVhSDOnYH9oVGjjKkQXGtbC3Uj5gmnG0Ls5tGU5Ptb54iy4sQ+2s5Fm+O8+Xr2cOZcgxOSEF4pELUia3pEKqhJMn8kJeyZvz7Lw7H87ntHTFmfUckblwvn8BtuOnSQ==</latexit><latexit sha1_base64="yC47CR6xz6bMMbsCM/qVyQIvACI=">AAACJXicbZDLSgMxFIYzXmu9VV26CRah3ZQZEXQjFN24rGAv0JaSSU/bYCYzJGekZZhn8DF8Arf6BO5EcOXK9zC9LGzrgYSP/1z5/UgKg6775aysrq1vbGa2sts7u3v7uYPDmgljzaHKQxnqhs8MSKGgigIlNCINLPAl1P2Hm3G+/gjaiFDd4yiCdsD6SvQEZ2ilTq7YQhjiZE4Saqb6kCataCA69jciLQyLV2Ogw7STy7sldxJ0GbwZ5MksKp3cT6sb8jgAhVwyY5qeG2E7YRoFl5BmW7GBiPEH1oemRcUCMO1kckpKT63Spb1Q26eQTtS/HQkLjBkFvq0MGA7MYm4s/pdrxti7bCdCRTGC4tNFvVhSDOnYH9oVGjjKkQXGtbC3Uj5gmnG0Ls5tGU5Ptb54iy4sQ+2s5Fm+O8+Xr2cOZcgxOSEF4pELUia3pEKqhJMn8kJeyZvz7Lw7H87ntHTFmfUckblwvn8BtuOnSQ==</latexit><latexit sha1_base64="yC47CR6xz6bMMbsCM/qVyQIvACI=">AAACJXicbZDLSgMxFIYzXmu9VV26CRah3ZQZEXQjFN24rGAv0JaSSU/bYCYzJGekZZhn8DF8Arf6BO5EcOXK9zC9LGzrgYSP/1z5/UgKg6775aysrq1vbGa2sts7u3v7uYPDmgljzaHKQxnqhs8MSKGgigIlNCINLPAl1P2Hm3G+/gjaiFDd4yiCdsD6SvQEZ2ilTq7YQhjiZE4Saqb6kCataCA69jciLQyLV2Ogw7STy7sldxJ0GbwZ5MksKp3cT6sb8jgAhVwyY5qeG2E7YRoFl5BmW7GBiPEH1oemRcUCMO1kckpKT63Spb1Q26eQTtS/HQkLjBkFvq0MGA7MYm4s/pdrxti7bCdCRTGC4tNFvVhSDOnYH9oVGjjKkQXGtbC3Uj5gmnG0Ls5tGU5Ptb54iy4sQ+2s5Fm+O8+Xr2cOZcgxOSEF4pELUia3pEKqhJMn8kJeyZvz7Lw7H87ntHTFmfUckblwvn8BtuOnSQ==</latexit>
DiracGAN [7]
Gradient Constrained MMD
Adjust the radius of the RKHS ball according to the smoothness of k :
FS = {f ∈ Hk : ‖f ‖Hk≤ σk}
SMMDk(P,Q) := supf ∈FS
EP[f (X )]− EQ[f (X )] = σk MMDk(P,Q)
σk :=
(λ + EX∼S
[k(X ,X ) +
∑di=1
∂2k(y ,z)∂yi∂zi
∣∣∣(y ,z)=(X ,X )
])−12
Optimal f ? satisfies EX∼S[‖∇f ?(X )‖2] ≤ 1
Other possible choices for F :
FLip := {f ∈ Hk : ‖f ‖2Lip + λ‖f ‖2 ≤ 1} FGC := {f ∈ Hk : ‖f ‖2
L2(S) + ‖∇f ‖2L2(S) + λ‖f ‖2 ≤ 1}
F FLip FGC FS
Effectiveness
Tractability
4 2 0 2 40.000.250.500.751.001.251.50
MMDSMMD
Theory: Continuity under weak topology
DSMMD(P,Q) is continuous in weak topology if:
S has a density (can depend on P, Q)
φψ is fully connected, Leaky-ReLU activations,non-increasing width
Each layer of φψ has weights with boundedcondition number
kbase is “reasonable” (Gaussian, linear, . . . )0 20K 40K 60K
0
200
400
600
Condition Number: Layer 1SMMDGANSN-SMMDGANMMDGANSN-MMDGAN
Orthogonal Normalization [3] or Spectral Normalization [8] control the condition number in practice.
Experimental Comparison
Scaled MMD GANs outperform other GANs (WGAN-GP, MMD-GAN, SN-GAN).
(a) Scaled MMD GAN, SN (b) SN-GAN
ImageNet, 64× 64.No labels.Generator: 10-layer ResNet.Critic: 10-layer ResNet.
(a) Scaled MMD GAN, SN (b) MMD GAN, GP+L2
CelebA, 160× 160.Generator: 10-layer ResNet.Critic: 5-layer DCGAN.
0 20 40FID [2]
WGAN-GP [1, NeurIPS-17]BGAN [6, ICLR-18]
SN-GAN [8, ICLR-18]MMDGAN-GP-L2 [2, ICLR-18]
Sobolev-GAN [9, ICLR-18]SN-SMMDGAN (ours)
ImageNet CelebA
0 20 40KID ×103 [5]
0 20FID [2]
0 10 20KID ×103 [5]
Implementation at github.com/MichaelArbel/Scaled-MMD-GAN
Faster training and better complexity control
0 2 4 6 8 10generator iterations ×104
10
15
20
25
30KID×103
SMMDGANSN-SMMDGANSN-SWGANSobolev-GANSN-GANWGAN-GPMMDGAN-GP-L2
0 1 2 3 4 5 6generator iterations ×104
101
102
103
104Critic Complexity: 2
k SMMDGANSN-SMMDGANMMDGANSN-MMDGAN
BibliographyM. Arjovsky, S. Chintala, and L. Bottou. “Wasserstein Generative Adversarial Networks”. ICML. 2017.
M. Binkowski, D. J. Sutherland, M. Arbel, and A. Gretton. “Demystifying MMD GANs”. ICLR. 2018.
A. Brock, T. Lim, J. M. Ritchie, and N. Weston. “Neural Photo Editing with Introspective Adversarial Networks”. ICLR. 2017.
I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville. “Improved Training of Wasserstein GANs”. NeurIPS. 2017.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, G. Klambauer, and S. Hochreiter. “GANs Trained by a Two Time-Scale Update RuleConverge to a Nash Equilibrium”. NeurIPS. 2017.
R. Hjelm, A. Jacob, T. Che, A. Trischler, K. Cho, and Y. Bengio. “Boundary-Seeking Generative Adversarial Networks”. ICLR. 2018.
L. Mescheder, A. Geiger, and S. Nowozin. “Which Training Methods for GANs do actually Converge?” ICML. 2018.
T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. “Spectral Normalization for Generative Adversarial Networks”. ICLR. 2018.
Y. Mroueh, C.-L. Li, T. Sercu, A. Raj, and Y. Cheng. “Sobolev GAN”. ICLR. 2018.
This work was funded by the Gatsby Charitable Foundation. Contact: [email protected]
