![Page 1: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/1.jpg)
SummaryOfSeveralAutoencodermodels
Presentor:JiGao
DepartmentofComputerScience,UniversityofVirginiahttps://qdata.github.io/deep2Read/
![Page 2: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/2.jpg)
List
• AdversarialAutoencoders• PixelGAN Autoencoders• GeneratinganddesigningDNAwithdeepgenerativemodels• FeedbackGAN(FBGAN)forDNA:aNovelFeedback-LoopArchitectureforOptimizingProteinFunctions• AutoregressiveGenerativeAdversarialNetworks
![Page 3: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/3.jpg)
AdversarialautoencodersAlireza Makhzani, JonathonShlens,Navdeep Jaitly, IanGoodfellow,BrendanFrey
• Useadversariallearningintrainingautoencoders
![Page 4: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/4.jpg)
Autoencoders
• Autoencoder
• Decoder=Generator:Startfromaprior(oftennormaldistribution),producesample
![Page 5: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/5.jpg)
Autoencoders
WhyDecoderwork?• Anydistributionin𝑑 dimensioncanbegeneratedbyasufficientlycomplicatedfunctionon𝑑 normallydistributionalvariables.
![Page 6: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/6.jpg)
Whynotdirectlyoptimizedecoder?
• Ifdirectlyoptimizedecoderviasampling,itwilltakeexponentiallynumberofsamples(Andalsoexponentiallyparameters)• Alotofthesamplingareuseless,foraX,weonlyneedthepartofzthatarelikelytoproduceX• FindmostlikelyztoproduceXcansavehugeamountoftimeandmaketheprocesstractable
![Page 7: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/7.jpg)
VariationalAutoencoder
𝐷#$(𝑄(𝑧)||𝑃(𝑧|𝑥)) = 𝐸.~0[log𝑄 𝑧 − log𝑃(𝑧|𝑥)]
= 𝐸.~0[log𝑄 𝑧 − log𝑃 𝑥 𝑧 − log𝑃 𝑧 + log𝑃(𝑥)]
Latentvariable𝑧~𝑃(𝑧) Ifwesample𝑄(𝑧) toapproximate𝑃(𝑥),wehave
log𝑃(𝑥) − 𝐷#$(𝑄(𝑧)||𝑃(𝑧|𝑥)) = 𝐸.~0[log𝑃 𝑧|𝑥 ] − 𝐷#$(𝑄(𝑧)||𝑃(𝑧))
Bayesian
Reasonabletolet𝑄(𝑧) conditionedonx.
Wehave:log𝑃(𝑥) ≥ 𝐸.~0[log𝑃 𝑧|𝑥 ] − 𝐷#$(𝑄(𝑧|𝑥)||𝑃(𝑧))
Variationalbound
![Page 8: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/8.jpg)
VariationalAutoencoderlog𝑃(𝑥) ≥ 𝐸.~0[log𝑃 𝑧|𝑥 ] − 𝐷#$(𝑄(𝑧|𝑥)||𝑃(𝑧))InVAE,welet𝑄 𝑧 𝑥 = 𝑁(𝑧|𝜇 𝑥;𝜃 , Σ(𝑥; 𝜃))
Inthiscase:
Samplexandz,wehave
![Page 9: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/9.jpg)
Posterier
• GaussianPosterier
![Page 10: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/10.jpg)
VariationalAutoencoder
FromTutorialonVariationalAutoencoders https://arxiv.org/abs/1606.05908
![Page 11: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/11.jpg)
Adversarialautoencoder
• VAEworkson
• 𝐷#$(𝑄 𝑧 𝑥 ||𝑃 𝑧 ) termcanbeoptimizedinadversarialtraining• Trainrepeatedlyintwosteps:1.Maximize𝐸.~0[log𝑃 𝑥|𝑧 ]2.Minimizethedistancebetween𝑄(𝑧|𝑥) and𝑃(𝑧)
log𝑃(𝑥) ≥ 𝐸.~0[log𝑃 𝑥|𝑧 ] − 𝐷#$(𝑄(𝑧|𝑥)||𝑃(𝑧))
![Page 12: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/12.jpg)
Freedomofchoosingq()
• ComparetoVAE,inthisformitcanbeoptimizedusingseveraldifferentways:• 1.Deterministic:𝑞 𝑧 𝑥 isadeterministicfunctiononx• 2.Gaussianposterior:𝑄 𝑧 𝑥 = 𝑁 𝑧 𝜇 𝑥; 𝜃 , Σ 𝑥; 𝜃 similartoVAE.Canusethesamereparameterization• 3.Universalapproximator posterior,𝑞 𝑧 𝑥, 𝜂 = 𝛿(𝑧 − 𝑓(𝑥, 𝜂))
![Page 13: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/13.jpg)
Adversarialautoencoderperformance
![Page 14: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/14.jpg)
Loglikelihood
![Page 15: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/15.jpg)
Supervisedlearning
• Fullysupervisedlearningtogeneratesamplesinaparticularway
![Page 16: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/16.jpg)
Semi-supervisedlearning
• 2adversarialnets:Onewithcategoricaldata• Traininthreephases:• 1.Reconstructionphase• 2.Regularizationphase• 3.Semi-supervisedphase
![Page 17: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/17.jpg)
PixelGAN AutoencodersAlirezaMakhzani,BrendanFrey
![Page 18: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/18.jpg)
PixelGAN Autoencoders
• UsePixelCNN asthegenerativepath• PixelCNN conditionedonq(z|x)
![Page 19: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/19.jpg)
Categoricalprior
![Page 20: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/20.jpg)
Experiment
![Page 21: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/21.jpg)
GeneratinganddesigningDNAwithdeepgenerativemodelsNathanKilloran,LeoJ.Lee,AndrewDelong,DavidDuvenaud,BrendanJ.Frey
• 2017• ThreeapproachestogenerateDNAsequence:• 1.GAN• 2.Activationmaximization(DeepDream)• 3.Ajointof1and2
![Page 22: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/22.jpg)
GANondiscreteoutput
• DNAsequenceisdiscrete,similartoNLPtask• WGAN-GPcangeneratethesequenceinthedirectway:
LetGANdirectlyoutputone-hotcharacterembeddings fromalatentvectorwithoutanydiscretesamplingstep.Softmax directlypassedtocritic.
![Page 23: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/23.jpg)
GANonDNA
• UsesuchmethodonDNA:
![Page 24: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/24.jpg)
ActivationMaximization
• ThemethodisactuallyDeepDream:• Startfromsamplex,makeittargetatacertainpropertyt(output)• 𝑥 → 𝑥 + 𝜖∇E𝑡• Worksoncontinuouscase,soneedtorelaxdiscretesymbolsintocontinuouscase
![Page 25: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/25.jpg)
Jointmethod
• UseGANtogeneratesample• Useactivationmaximizationtooptimizeasampletocertainproperties
![Page 26: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/26.jpg)
Experiment:Motif
• Samplesequencestunedtohaveahighpredictorscore
![Page 27: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/27.jpg)
Experiment
![Page 28: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/28.jpg)
FeedbackGAN(FBGAN)forDNA:aNovelFeedback-LoopArchitectureforOptimizingProteinFunctionsAnvita Gupta,JamesZou
• 2018• Target:DesignDNAautomaticallyfollowingsomeproperties
![Page 29: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/29.jpg)
FeedbackGAN
• (a)WGAN-GPasgenerator• (b)Analyzer:supposetobeanyfunction• Ratethegeneratedsamples• Markthetopsortedsamplesasrealsamples
• (c)Feedbackscheme• Sendthetopsortedsamplebacktothediscriminator
![Page 30: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/30.jpg)
Evaluation
• Beforetraining,3.125%ofsequencesinitially followedthecorrectgenestructure• Aftertraining,77.08%ofsampledsequencescontainedthecorrectgenestructure
![Page 31: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/31.jpg)
AutoregressiveGenerativeAdversarialNetworksYasin Yazici,Kim-HuiYap,StefanWinkler
• ICLR18Workshop
![Page 32: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/32.jpg)
ARGAN
• ReplacediscriminatorintoaCNN+Autoregressivemodel• Motivation:anautoregressivemodelwouldmodelthefeaturedistributionbetterthanfullyconnectedlayers
![Page 33: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/33.jpg)
S-ARGANandC-ARGAN
![Page 34: Summary Of Several Autoencoder models - GitHub Pages€¦ · Adversarial autoencoders Alireza Makhzani, Jonathon Shlens, NavdeepJaitly, Ian Goodfellow, Brendan Frey • Use adversarial](https://reader033.vdocuments.us/reader033/viewer/2022042220/5ec5f5aa90ca1d693c70618a/html5/thumbnails/34.jpg)
Result