Upload File

uncertainty in bayesian neural nets · bnn •variational inference •maximize lower bound on the...

  • Home
  • Documents
  • Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior
41
Uncertainty in Bayesian Neural Nets August 4 2017

Upload: others

Post on 18-Aug-2020

2 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

UncertaintyinBayesianNeuralNets

August42017

Page 2: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

Overview

• BNNreview• Visualizationexperiments• BNNresults

Page 3: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

BNN

Prior:p(W)Likelihood:p(Y|X,W)ApproximatePosterior:q(W)PosteriorPredictive:𝐸"($)[𝑝(𝑦|𝑥,𝑊)]

Page 4: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

BNN

• VariationalInference• Maximizelowerboundonthemarginallog-likelihood

log 𝑝 𝑌 𝑋 ≥ 𝐸" $ [log 𝑝 𝑌 𝑋,𝑊 + log 𝑝 𝑊 − log 𝑞 𝑊 ]

Prior PosteriorApprox

Likelihood

Y

X W

Dependentonthenumberofdatapoints

1𝑀9 log 𝑝 𝑌: 𝑋:,𝑊

;

:<=

+1𝑁 log

𝑝(𝑊)𝑞(𝑊)

Page 5: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

Differentpriorsandposteriorapproximations

• Priorsp(W):• 𝑁(0, 𝜎A)• Scale-mixturesofNormals• SparsityInducing

• PosteriorApproximationsq(W):• Deltapeak q W = 𝛿𝑊• FullyFactorizedGaussiansq W =∏𝑁(𝑤I|𝜇I, 𝜎IA)�

�• BernoulliDropout• GaussianDropout• MNF

Page 6: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

MultiplicativeNormalizingFlows(MNF)

• Augmentmodelwithauxiliaryvariable

ChristosLouizos,MaxWellingICML2017

Y

X W

Z

GenerativeModel

InferenceModel

W

Z

𝑧~𝑞 𝑧 𝑊~𝑞 𝑊 𝑧

𝑞 𝑊 = N𝑞 𝑊 𝑧 𝑞 𝑧 𝑑𝑧�

�

𝑞 𝑊 𝑧 =PP𝑁(𝑧I𝜇IQ, 𝜎IQA)RSTU

Q<=

RVW

I<=

log 𝑝 𝑌 𝑋 ≥ 𝐸" $ [log 𝑝 𝑌 𝑋,𝑊 + log 𝑝 𝑊 − log 𝑞 𝑊|𝑧 + log 𝑟 𝑧 𝑤 − log 𝑞(𝑧)]

NewlowerboundNormalizingFlows

Page 7: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

PredictiveDistributions

Page 8: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

Uncertainties

• Modeluncertainty(Epistemicuncertainty)• Capturesignoranceaboutthemodelthatismostsuitabletoexplainthedata• Reducesastheamountofobserveddataincreases• Summarizedbygeneratingfunctionrealizationsfromourdistribution

• MeasurementNoise(Aleatoric uncertainty)• Noiseinherentintheenvironment,capturedinlikelihoodfunction

• Predictiveuncertainty• Entropyofprediction=H[p(y|x)]

Page 9: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

VisualizationExperiments

• 1Dregression• ClassificationofMNIST(visualizein2D)

• Questions:• Activations• Numberofsamples• Heldoutclasses• Typeofuncertainties

Page 10: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

Sigmoid:(1+e-x)-1 Tanh

Softplus:ln(1+ex) ReLU:max(0,x)

BNNswithDifferentActivationFunctions

Page 11: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

UncertaintyofDecisionBoundaries

• Setup:• ClassificationofMNIST• Train:50000Test:10000

784-100-2-100-10

BNN:FFG,N(0,1)Activations:Softplus

NN BNN

Page 12: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

DecisionBoundaries– 3Samples

PlotofArgmax p(y|x)ateachpoint

Page 13: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

UncertaintyofDecisionBoundaries:HeldOutClasses• Setup:• Classificationofdigits0to4(5to9heldout)

784-100-100-2-100-100-10

BNN:FFG,N(0,1)Activations:Softplus

NN BNN

Page 14: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

Wheredoyouthinktheheldoutclasseswillgo?

InsideorOutsidetheCircle?

Page 15: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

Wheredoyouthinktheheldoutclasseswillgo?

Page 16: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

HeldOutClasses

Unseenclassesdon’tgetencodedassomethingfaraway,insteadencodednearmean

Page 17: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

ConfidenceofPredictions?MaybelargeareashavehighentropyArgmax vsMax

Page 18: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

ClassBoundaries- Confidences

SharptransitionsThereisn’tmuchuncertainspace:mostlyuniform,highconfidence

Page 19: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

EntropyArgmax

Max

Entropy

Page 20: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

AffectofChoiceofActivationFunction

• Softplus• ReLU• Tanh

Page 21: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

Softplus

Sample1 Sample2 Sample3 Meanofq(W) 𝐸"($)[𝑝(𝑦|𝑥, 𝑤)]

Page 22: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

ReLU

Sample1 Sample2 Sample3 Meanofq(W) 𝐸"($)[𝑝(𝑦|𝑥, 𝑤)]

Page 23: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

TanhSample1 Sample2 Sample3 Meanofq(W) 𝐸"($)[𝑝(𝑦|𝑥, 𝑤)]

Page 24: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

Mix(Softplus,ReLu,Tanh)Sample1 Sample2 Sample3 Meanofq(W) 𝐸"($)[𝑝(𝑦|𝑥)]

Page 25: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

NumberofDatapoints25000 10000 1000 100

Argmax

Max

Entropy

𝐸"($)[𝑝(𝑦|𝑥)]

Page 26: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

ModelvsOutputUncertainty

• PredictiveUncertainty=𝐻[𝑝(𝑦|𝑥)]

OutputUncertainty

ModelUncertainty

𝐻[𝑝(𝑦|𝑥, 𝑤Z)]where𝑤Z=meanofq(w)

𝐻[𝐸"($)[𝑝(𝑦|𝑥, 𝑤)]]

Outputhighentropy(ondecisionboundary)

Highvariancepredictions

Page 27: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

ModelvsOutputUncertainty

Train Test HeldOut

ModelUncertainty .07 .26 .43

OutputUncertainty .03 .15 .25

Train Test HeldOut

ModelUncertainty .06 .06 .43

OutputUncertainty .05 .05 .36

100trainingdatapoints

25000trainingdatapoints

Smalldata:modeluncertainty

Largedata:outputuncertainty

Page 28: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

NN BNN GP+NN

AdversarialExamples,Uncertainty,andTransferTestingRobustnessinGaussianProcessHybridDeepNetworks(July2017)

Page 29: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

Visualizelandscapeoflikelihood

w1

w2

p(ytrain|xtrain,W)

DimensionofWislarge,sousean2Dauxiliaryvariable

Page 30: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

Visualizelandscapeoflikelihood

• AuxiliaryVariableModel

Y

X W

Z

GenerativeModel

InferenceModel

W

Z

784-100-100-2-10-10-10NN BNN

(2D)𝑧~𝑞 𝑧

𝑞 𝑊 = N𝛿 𝑊 𝑧 𝑞 𝑧 𝑑𝑧�

�

𝑊~𝑞 𝑊 𝑧r 𝑧 𝑊

log 𝑝 𝑌 𝑋 ≥ 𝐸" $ [log 𝑝 𝑌 𝑋,𝑊 + log 𝑝 𝑊 − log 𝑞 𝑊|𝑧 + log 𝑟 𝑧 𝑤 − log 𝑞(𝑧)]

𝑞 𝑊 𝑧 = 𝛿(𝑊|𝑧)

hyper-network hypo-network

Page 31: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

DecisionBoundariesz1 z2 z3 𝐸"([)[𝑝(𝑦|𝑥, 𝑧)]

Page 32: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

LikelihoodLandscape

Logp(ytrain|xtrain,W,z) Logp(ytest|xtest,W,z)

z1

z2 z2

Page 33: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

LikelihoodLandscape

logp(ytrain|xtrain,W,z) logp(ytest|xtest,W,z)

logp(ytrain|xtrain,W,z)+logr(z|W)- logq(z)

z1

z2

Page 34: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

LikelihoodLandscape

Logp(ytrain|xtrain,W,z) Logp(ytest|xtest,W,z)

logp(ytrain|xtrain,W,z)+logr(z|W)- logq(z)

z1

z2

Page 35: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

LikelihoodLandscape

Logp(ytrain|xtrain,W,z) Logp(ytest|xtest,W,z)

logp(ytrain|xtrain,W,z)+logr(z|W)- logq(z)

z1

z2

Page 36: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

RecentBNNPapers

• MultiplicativeNormalizingFlowsforVariationalBayesianNeuralNetworks(2017)• VariationalDropoutSparsifies DeepNeuralNetworks(2017)• BayesianCompressionforDeepLearning(2017)

• AdversarialPerturbations• Compression

Page 37: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

Adversarialperturbations

MNIST CIFAR10

Page 38: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

CompressionvsUncertainty

H[P]

Page 39: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

Conclusion

• UsedvisualizationstohelpunderstanduncertaintyinBNNs• Goal:improveuncertaintyestimatesandgeneralization

Applications• Activelearning• BayesOpt• RL

• Safety• Efficiency

Page 40: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

References

• WeightUncertaintyinNeuralNetworks(2015)

• VariationalDropoutandtheLocalReparameterization Trick(2015)

• DropoutasaBayesianApproximation:RepresentingModelUncertaintyinDeepLearning(2016)

• VariationalDropoutSparsifies DeepNeuralNetworks(2017)

• OnCalibrationofModernNeuralNetworks(2017)

• MultiplicativeNormalizingFlowsforVariationalBayesianNeuralNetworks(2017)

Page 41: Uncertainty in Bayesian Neural Nets · BNN •Variational Inference •Maximize lower bound on the marginal log-likelihood log’12 ≥!"$[log’12,,+log’,−log6,] Prior Posterior

ThankYou


BNN Bigband Workshop 2012 - jkg-bruchsal.de

lsamathwilson.weebly.com · log4 10t2uv-3 In 30. 32. 34. 36. 38. In 4df5 In ... log-2 8 50. 52. 54. 56. 58. log3 10 ... (y + 8) 5 log6 2 2 (z 9) — 3 Iny — In

s6: atff)/§ IQD-BNN - PC e:tt- am · II d

LP-BNN: Ultra-low-Latency BNN Inference with Layer …LP-BNN: Ultra-low-Latency BNN Inference with Layer Parallelism ... CONV1 CONV2 CONV3 CONV4 CONV5 CONV6 CONV7 CONV8 CONV9 CONV10

BNN Bussiness Proposal - Romy Verstraeten

BNN Toolbox

BNN Water Solutions - Tallgrass Energy · BNN Water Solutions Other Tallgrass Owned Assets BNN Water Solutions 22 May 2019. Created Date: 20191008102740Z

BNN BigBand Workshop 2011 - jkg-bruchsal.de

azso.podpierwiastkiem.plazso.podpierwiastkiem.pl/materialy/zz33(log)pto.pdf · 84 15. 16. 18. Wyraž liczbe a za pomocQ p. a) a log6 4, p log6 9 b) a log6 8, p log6 3 c) a = log 25,

Mariners Weather Log6 Mariners Weather Log Hurricane Avoidance Hurricane Avoidance Continued from Page 5 pressure system is detected. An example of this technique is diagramed in Figure

Presentatie BNN Pecha Kucha van Johnny

Z10035 BNN AMIGO O1 Lime Z10035v25prokcssmedia.blob.core.windows.net/sys-master... · BNN RECADO O2 BNN FILIPO O2 Z20162 BNN SALVADOR High Z90601 Z90701 Z20263 Low comfortable shoes

2014 - 2015 BNN Rights Catalog

Final presentation about BNN and Social Media

Pechakucha Bnn Vs. Social Media

BNN @ Immovator Cross Media Cafe

2011 BNN Book Catalog

O log6/26/12 webcache .googleusercontent.com/search?q =cach e:vlTT gJ51 C60J : . Download (168k) Link to this page Edit a …

Standard Life-BNN Retail

BELIZE NETWORK OF NGOsrepresentatives of several BNN member agencies, as well as experts who were invited to serve as advisors. This strategy from the BNN aims to leverage this moment

Summary of formulae Powers - Home - Springer978-1-349-13551-6/1.pdf · ... 9 729 3 (vi) 16 0.25 -! (ix) log 1 q = x (x) ... = log2 + log3 . log6 119 g1Vmg x = log4.5 = · (vi)

ROAD MAP REFORMASI BIROKRASI BNN 2015 - 2019 · 2015-12-07 · Tentang Keanggotaan Tim Analisis dan Evaluasi Jabatan BNN Periode 2015 –2019 b. ... untuk seluruh pegawai dalam rangka

BNN Energy - Basic Overview - Q4 2015kls

BNN Technology plc Final Results for the 12 months ended .../media/Files/B/BNN... · Xinhuatong and Xinhua News Agency for the exclusive rights for payment processing and key

Mitigating Model Non-Identifiability in BNN with Latent ... · Mitigating Model Non-Identifiability in BNN with Latent Variables Yaniv Yacoby * 1Weiwei Pan Finale Doshi-Velez1 Abstract

Training Deep Gaussian Processes using Stochastic ...approximateinference.org/accepted/BuiEtAl2015_poster.pdf · FITC-DGP-2 FITC-DGP-1 FITC-GP BNN-Dropout BNN-PBP BNN-VI Generative

Ppp Proposal Bnn

Paparan kabag humas bnn-jenis dan dampak narkoba-22052014

Final Presentation on BNN and Social Media

CANADA’S ONLY ALL BUSINESS AND FINANCIAL NEWS · PDF file4 INTRODUCTION RESEARCH PROGRAMMING BNN GUESTS CONTACT PARTNERSHIPS SUMMARY • 59% of C-Suite Executives watch BNN at least

Logscms8.revize.com/revize/lakelandschools/schools/lakelandhigh/Math/... · ... log2 = X 8) log3 — 1 = X 2 3 1 9) logr 3— = X 3 ... —log 6 36 = log6 n 2) log4 64 —log 4 16

Trends & Strategies In The Creative Industries; BNN & Social Media

CAT 2016 Question Paper- Solutions - Collegedunia · 13. Alternative Solution: log6. b log7 1095 . — log6 NOW, the product abcd 10g6 log7 log8 log5 log6 log7 log 25 10924 log24

BNN, Social Media & Interactive Television Solutions

2013 BNN Rights Catalog