generative adversarial networks (gans) for discrete...
TRANSCRIPT
![Page 1: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/1.jpg)
GenerativeAdversarialNetworks(GANs)forDiscreteData
Lantao YuShanghaiJiaoTongUniversity
http://lantaoyu.comJuly 26, 2017
![Page 2: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/2.jpg)
SelfIntroduction– Lantao Yu• Position
• Research Assistant atCSDept.ofSJTU2015-now• ApexDataandKnowledgeManagementLab• Researchondeep learning, reinforcement learning andmulti-agent systems
• Education• Third-year Undergraduate Student, Computer ScienceDept., ShanghaiJiaoTongUniversity,China,2014-now
• Contact• [email protected]• [email protected]
![Page 3: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/3.jpg)
ApexData&KnowledgeManagementLab
• Students• 8PhDs• 8Masters• 24Undergrads
• www.apexlab.orgProfessor YongYu A.P.WeinanZhang
• Machinelearninganddatascience• withapplicationsofrecommendersystems,computationalads,socialnetworks,crowdsourcing,urbancomputingetc.
![Page 4: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/4.jpg)
Content
• Fundamentals- GenerativeAdversarialNetworks• Connection and difference between generating discretedata and continuous data with GANs
• Advances- GANsforDiscreteData• SeqGAN:SequenceGenerationviaGANswithPolicyGradient
• IRGAN:AMinimaxGameforUnifyingGenerativeandDiscriminativeInformationRetrievalModels
![Page 5: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/5.jpg)
GenerativeAdversarialNetworks(GANs)[Goodfellow,I.,etal.2014.Generativeadversarialnets.InNIPS2014.]
![Page 6: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/6.jpg)
ProblemDefinition• Givenadataset,buildamodelofthedatadistributionthatfitsthetrueone
• Traditionalobjective:maximumlikelihoodestimation(MLE)
• Checkwhetheratruedataiswithahighmassdensityofthelearnedmodel
D = {x} q(x)p(x)
max
q
1
|D|X
x2D
log q(x) ⇡ max
q
Ex⇠p(x)[log q(x)]
![Page 7: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/7.jpg)
Problems of MLE
• Checkwhetheratruedataiswithahighmassdensityofthelearnedmodel
• Approximatedby
Training/evaluation Use (Turing Test)
• Checkwhetheramodel-generateddataisconsideredastrueaspossible
• Morestraightforwardbutitishardorimpossibletodirectlycalculate
• InconsistencyofEvaluationandUse
max
q
Ex⇠p(x)[log q(x)] max
q
Ex⇠q(x)[log p(x)]
max
q
1
|D|X
x2D
log q(x)
p(x)
![Page 8: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/8.jpg)
Problems of MLE• Equivalent to minimizing asymmetric :
• When but , theintegrandinsidetheKLgrowsquicklytoinfinity, which means MLE assigns anextremely high cost to such situation
• When but , the value inside the KLdivergence goes to 0, which means MLE pays extremely lowcost for generating fake looking samples
[Arjovsky M,Bottou L.Towardsprincipledmethodsfortraininggenerativeadversarialnetworks.arXiv,2017.]
KL(p(x)||q(x))
p(x) > 0 q(x) ! 0
p(x) ! 0 q(x) > 0
KL(p(x)||q(x)) =Z
Xp(x) log
p(x)
q(x)
dx
![Page 9: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/9.jpg)
Problems of MLE• Exposure bias in sequential data generation:
Training InferenceWhen generating the nexttoken , sample from:yt
max
✓EY⇠ptrue
X
t
logG✓(yt|Y1:t�1) G✓(yt|Y1:t�1)
Update the model as follows:
The real prefix The guessed prefix
![Page 10: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/10.jpg)
GenerativeAdversarialNets(GANs)
• Whatwereallywant
• Butwecannotdirectlycalculate p(x)p(x)
• Idea:whatifwebuildadiscriminatortojudgewhetheradatainstanceistrueorfake(artificiallygenerated)?
• Leveragethestrongpowerofdeeplearningbaseddiscriminativemodels
max
q
Ex⇠q(x)[log p(x)]
![Page 11: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/11.jpg)
GenerativeAdversarialNets(GANs)
• Discriminatortriestocorrectlydistinguishthetruedataandthefakemodel-generateddata
• Generatortriestogeneratehigh-qualitydatatofooldiscriminator
• G&Dcanbeimplementedvianeuralnetworks• Ideally,whenDcannotdistinguishthetrueandgenerateddata,Gnicelyfitsthetrueunderlyingdatadistribution
GD
RealWorld
Generator
Discriminator
Data
![Page 12: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/12.jpg)
GANs:AMinimaxGame• The most general form:
ProbabilityDensityFunction
min
G
max
D
V (D,G)
= Ex⇠pdata(x)[logD(x)] + E
x⇠G(x)[log(1�D(x))]
=
Z
x
pdata(x) · log(D(x)) +G(x) · log(1�D(x))dx
![Page 13: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/13.jpg)
Generatingcontinuousdata
• Thegenerativemodelisadifferentiable mappingfromthepriornoisespacetodataspace
• Firstsamplefromasimpleprior,thenapplyadeterministicfunction
• NoexplicitProbabilityDensityFunctionfordata
z ⇠ p(z)G : Z ! X
x
![Page 14: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/14.jpg)
• WithoutexplicitP.D.F.,wecanrewritetheminimaxgameas:
• The most general form:min
G
max
D
V (D,G) = Ex⇠pdata(x)[logD(x)] + E
x⇠G(x)[log(1�D(x))]
=
Z
x
pdata(x) · log(D(x)) +G(x) · log(1�D(x))dx
ProbabilityDensityFunction
min
G
max
D
V (D,G)
= Ex⇠pdata(x)[logD(x)] + E
z⇠pz(z)[log(1�D(G(z)))]
=
Z
x
pdata(x) · log(D(x))dx+
Z
z
p
z
(z) · log(1�D(G(z)))dx
GANsforcontinuousdata
Directlyoptimizethedifferentiablemapping!
![Page 15: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/15.jpg)
• Inordertotakegradientonthegeneratorparameter,x hastobecontinuous
xx
zz
pp
1.Generation
2.Discrimination3.Gradientongenerateddata
4.Furthergradientongenerator
minGmaxDJ (D )min
GmaxDJ (D ) max
DJ (D )max
DJ (D )Generator Discriminator
P (True|x) = D(x;�(D))
x = G(z; ✓(G))
rL(p)
rx
rL(p)
rx
rx
r✓
(G)
J
(D)= E
x⇠pdata(x)[logD(x)] + Ez⇠pz(z)[log(1�D(G(z)))]
GANsforcontinuousdata
![Page 16: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/16.jpg)
GANsforcontinuousdata
Discriminator
Data
Generator
min
G
max
D
V (D,G)
= Ex⇠pdata(x)[logD(x)] + E
z⇠pz(z)[log(1�D(G(z)))]
=
Z
x
pdata(x) · log(D(x))dx+
Z
z
p
z
(z) · log(1�D(G(z)))dx
Directlyoptimizethedifferentiablemapping!
![Page 17: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/17.jpg)
IdealFinalEquilibrium
• Generatorgeneratesperfectdatadistribution
• Discriminatorcannotdistinguishthetrueandgenerateddata
![Page 18: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/18.jpg)
TrainingGANsTrainingdiscriminator
![Page 19: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/19.jpg)
TrainingGANs
Traininggenerator
![Page 20: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/20.jpg)
OptimalStrategyforDiscriminator
• Optimalforany andisalways Discriminator
Data
Generator
D(x)pdata(x) pG(x)
D(x) =pdata(x)
pdata(x) + pG(x)
![Page 21: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/21.jpg)
ReformulatetheMinimaxGame
issomethingbetween and
[Huszár, Ferenc. "How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?." arXiv (2015).]
J
(D)= E
x⇠pdata(x)[logD(x)] + Ez⇠pz(z)[log(1�D(G(z)))]
= Ex⇠pdata(x)[logD(x)] + E
x⇠pG(x)[log(1�D(x))]
= Ex⇠pdata(x) log
pdata(x)
pdata(x) + p
G
(x)
+ Ex⇠pG(x) log
p
G
(x)
pdata(x) + p
G
(x)
= log(4) +KL(pdata||pdata + p
G
2
) +KL(p
G
||pdata + p
G
2
)
![Page 22: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/22.jpg)
CaseStudy
![Page 23: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/23.jpg)
Generatingdiscretedata
• Thegenerativemodelcomputesaprobabilitydistributionoverthecandidatechoices
• FirstcomputestheProbabilityDensityFunctionofdata(e.g.softmax overthevocabulary),thensamplefromit.
x
![Page 24: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/24.jpg)
• WithexplicitP.D.F.wecansimplystartwiththe mostgeneral formoftheminimaxgame:
• Now,insteadofoptimizingatransformation,wecandirectlyoptimizetheP.D.F.withtheguidanceofthediscriminator.
• Notethatevenfordiscretedata,isdifferentiable!
GANsfordiscretedata
ProbabilityDensityFunction
min
G
max
D
V (D,G)
= Ex⇠pdata(x)[logD(x)] + E
x⇠G(x)[log(1�D(x))]
=
Z
x
pdata(x) · log(D(x)) +G(x) · log(1�D(x))dx
G(x)
![Page 25: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/25.jpg)
GANsforDiscreteData• Wecoulddirectbuildaparametricdistributionfordiscretedata
• Forexampleofthediscretedata
• ThedataP.D.F.couldbedefinedas
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
A1 A2 A3 A4 A5
Ai Probability
wherethescoringfunctioncouldbedefinedbasedondomainknowledge,e.g.,aneuralnetworkwithAiembeddingastheinput
{A1;A2;A3;A4;A5}
p(Ai) =ef(Ai)
Pj e
f(Aj)
![Page 26: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/26.jpg)
BorrowtheIdeafromRL• Foragenerator• Intuition
• lowertheprobabilityofthechoicethatleadstolowvalue/reward• highertheprobabilityofthechoicethatleadstohighvalue/reward
• Theone-field5-categoryexample
0
0.1
0.2
0.3
A1 A2 A3 A4 A5
Ai Probability
0
0.1
0.2
0.3
0.4
A1 A2 A3 A4 A5
Ai Probability
0
0.1
0.2
0.3
0.4
A1 A2 A3 A4 A5
Ai Probability
2.GenerateA2Observepositivereward(fromD)
4.GenerateA3Observenegativereward(fromD)
1.Initializeθ 3.Updateθ bypolicygradient 5.Updateθ bypolicygradient
G(Ai) = P (Ai; ✓(G))
![Page 27: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/27.jpg)
Advances:GANsforDiscreteData• Lantao Yu,WeinanZhang,JunWang,YongYu.SeqGAN:Sequence
GenerativeAdversarialNetswithPolicyGradient. AAAI2017.• JunWang,Lantao Yu,WeinanZhang,YuGong,Yinghui Xu,Benyou
Wang,PengZhangandDellZhang.IRGAN:AMinimaxGameforUnifyingGenerativeandDiscriminativeInformationRetrievalModels.SIGIR2017.
![Page 28: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/28.jpg)
SeqGAN:SequenceGenerationviaGANswithPolicyGradient[Lantao Yu,WeinanZhang,JunWang,YongYu.SeqGAN:SequenceGenerativeAdversarialNetswithPolicyGradient.AAAI2017.]
![Page 29: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/29.jpg)
ProblemforDiscreteData• Oncontinuousdata,thereisdirectgradient
• Guidethegeneratorto(slightly)modifytheoutput
• Nodirectgradientondiscretedata• Textgenerationexample
• “床前明月光”• “Icaughtapenguininthepark”• FromIanGoodfellow:“Ifyououtputtheword‘penguin’,youcan'tchangethatto"penguin+.001"onthenextstep,becausethereisnosuchwordas"penguin+.001".Youhavetogoallthewayfrom"penguin"to"ostrich".”
P (t rue)P (t rue)
G
D
[https://www.reddit.com/r/MachineLearning/comments/40ldq6/generative_adversarial_networks_for_text/]
r✓(G)
1
|m|
mX
i=1
log(1�D(G(z(i))))
![Page 30: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/30.jpg)
SeqGAN
• Generatorisareinforcementlearningpolicyofgeneratingasequence
• decidethenextwordtogenerategiventhepreviousones• Discriminatorprovidesthereward(i.e.theprobabilityofbeingtruedata)forthewholesequence
G(yt|Y1:t�1)
D(Y n1:T )
![Page 31: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/31.jpg)
SequenceGenerator• Objective:tomaximizetheexpectedreward
• State-actionvaluefunction istheexpectedaccumulativerewardthat
• Startfromstates• Takingaction a• Andfollowingpolicy Guntiltheend
• Rewardisonlyoncompletedsequence(noimmediatereward)
J(✓) = E[RT |s0, ✓] =X
y12YG✓(y1|s0) ·QG✓
D�(s0, y1)
QG✓D�
(s, a)
QG✓D�
(s = Y1:T�1, a = yT ) = D�(Y1:T )
![Page 32: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/32.jpg)
State-ActionValueSetting• Rewardisonlyoncompletedsequence
• Noimmediatereward• Thenthelast-stepstate-actionvalue
• Forintermediatestate-actionvalue• UseMonteCarlosearchtoestimate
• Followingaroll-outpolicy GG
QG✓D�
(s = Y1:T�1, a = yT ) = D�(Y1:T )
QG✓D�
(s = Y1:t�1, a = yt) =⇢
1N
PNn=1 D�(Y n
1:T ), Y n1:T 2 MC
G�(Y1:t;N) for t < T
D�(Y1:t) for t = T ,
�Y 11:T , . . . , Y
N1:T
= MCG� (Y1:t;N)
![Page 33: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/33.jpg)
TrainingSequenceGenerator• Policygradient(REINFORCE)
[RichardSuttonetal.PolicyGradientMethodsforReinforcementLearningwithFunctionApproximation.NIPS1999.]
✓ ✓ + ↵hr✓J(✓)
r✓J(✓) =TX
t=1
EY1:t�1⇠G✓ [
X
yt2Yr✓G✓(yt|Y1:t�1) ·QG✓
D�(Y1:t�1, yt)]
'TX
t=1
X
yt2Yr✓G✓(yt|Y1:t�1) ·QG✓
D�(Y1:t�1, yt)
=
TX
t=1
X
yt2YG✓(yt|Y1:t�1)r✓ logG✓(yt|Y1:t�1) ·QG✓
D�(Y1:t�1, yt)
=
TX
t=1
Eyt⇠G✓(yt|Y1:t�1)[r✓ logG✓(yt|Y1:t�1) ·QG✓D�
(Y1:t�1, yt)],
![Page 34: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/34.jpg)
TrainingSequenceDiscriminator• Objective:standardbi-classification
min
��EY⇠pdata [logD�(Y )]� EY⇠G✓ [log(1�D�(Y ))]
![Page 35: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/35.jpg)
OverallAlgorithm
![Page 36: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/36.jpg)
SequenceGeneratorModel
• RNNwithLSTMcells
[Hochreiter,S.,andSchmidhuber,J.1997.Longshort-termmemory.Neuralcomputation9(8):1735–1780.]
Shanghai is incredibly
is incrediblySoftmax samplingovervocabulary
?
![Page 37: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/37.jpg)
SequenceDiscriminatorModel
[Kim,Y.2014.Convolutionalneuralnetworksforsentenceclassification.EMNLP2014.]
![Page 38: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/38.jpg)
ExperimentsonSyntheticData• EvaluationmeasurewithOracle
• An oracle model (e.g. the randomly initialized LSTM)• Firstly, the oracle model produces some sequences astraining data for the generative model
• Secondly the oracle model can be considered as thehuman observer to accurately evaluate the perceptualquality of the generative model
NLL
oracle
= �EY1:T⇠G✓
h TX
t=1
logGoracle
(yt|Y1:t�1
)
i
![Page 39: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/39.jpg)
ExperimentsonSyntheticData• EvaluationmeasurewithOracle
NLL
oracle
= �EY1:T⇠G✓
h TX
t=1
logGoracle
(yt|Y1:t�1
)
i
![Page 40: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/40.jpg)
ExperimentsonReal-WorldData• Chinesepoemgeneration
• Obamapoliticalspeechtextgeneration
• Midimusicgeneration
![Page 41: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/41.jpg)
ExperimentsonReal-WorldData• Chinesepoemgeneration
南陌春风早,东邻去日斜。
紫陌追随日,青门相见时。
胡风不开花,四气多作雪。
山夜有雪寒,桂里逢客时。
此时人且饮,酒愁一节梦。
四面客归路,桂花开青竹。
Canyoudistinguishwhichpartisfromhumanormachine?
![Page 42: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/42.jpg)
ExperimentsonReal-WorldData• Chinesepoemgeneration
南陌春风早,东邻去日斜。
紫陌追随日,青门相见时。
胡风不开花,四气多作雪。
山夜有雪寒,桂里逢客时。
此时人且饮,酒愁一节梦。
四面客归路,桂花开青竹。
Human Machine
![Page 43: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/43.jpg)
ObamaSpeechTextGeneration• i stoodheretodayi haveoneandmostimportantthingthatnotonviolencethroughoutthehorizonisOTHERSamericanfireandOTHERSbutweneedyouareastrongsource
• forthisbusinessleadershipwillremembernowi cantaffordtostartwithjustthewayoureuropean supportfortherightthingtoprotectthoseamericanstoryfromtheworldand
• i wanttoacknowledgeyouweregoingtobeanoutstandingjobtimesforstudentmedicaleducationandwarmtherepublicanswholikemytimesifhesaidisthatbroughtthe
• WhenhewastoldofthisextraordinaryhonorthathewasthemosttrustedmaninAmerica
• ButwealsorememberandcelebratethejournalismthatWalterpracticed-- astandardofhonestyandintegrityandresponsibilitytowhichsomanyofyouhavecommittedyourcareers.It'sastandardthat'salittlebithardertofindtoday
• Iamhonoredtobeheretopaytributetothelifeandtimesofthemanwhochronicledourtime.
Human Machine
![Page 44: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/44.jpg)
IRGAN:AMinimaxGameforInformationRetrievalJunWang,Lantao Yu,WeinanZhang,YuGong,Yinghui Xu,Benyou Wang,PengZhangandDellZhang.IRGAN:AMinimaxGameforUnifyingGenerativeandDiscriminativeInformationRetrievalModels.SIGIR2017.
![Page 45: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/45.jpg)
Two schools of thinking in IR modelingGenerative Retrieval Discriminative Retrieval
Query1
document1
document2
document3
document4
(Query1, document1)
relevance
• Assumethereisanunderlyingstochasticgenerativeprocessbetweendocumentsandqueries
• Generate/Select relevant documentsgiven a query
• Learnsfromlabeledrelevantjudgments
• Predict therelevancegivenaquery-documentpair
![Page 46: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/46.jpg)
ThreeparadigmsinLearningtoRank(LTR)
• Pointwise: learntoapproximatetherelevanceestimationofeachdocumenttothehumanrating
• Pairwise:distinguishthemore-relevantdocumentfromadocumentpair
• Listwise:learntooptimise the(smoothed)lossfunctiondefinedoverthewholerankinglistforeachquery
![Page 47: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/47.jpg)
IRGAN:Aminimaxgameunifyingbothmodels
• Takeadvantageofbothschoolsofthinking:• Thegenerativemodellearnstofittherelevancedistributionoverdocumentsviathesignalfromthediscriminativemodel.
• Thediscriminativemodelisabletoexploittheunlabeleddataselectedbythegenerativemodeltoachieveabetterestimationfordocumentranking.
![Page 48: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/48.jpg)
IRGANFormulation
• Theunderlyingtruerelevancedistributiondepictstheuser’s relevancepreferencedistributionoverthecandidatedocumentswithrespecttohissubmittedquery
• Trainingset:Asetofsamplesfrom• Generativeretrievalmodel
• Goal:approximatethetruerelevancedistribution
• Discriminativeretrievalmodel• Goal:distinguishbetweenrelevantdocumentsandnon-relevantdocuments
ptrue(d|q, r)
ptrue(d|q, r)p✓(d|q, r)
f�(q, d)
![Page 49: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/49.jpg)
IRGANFormulation
• OverallObjective
JG⇤,D⇤= min
✓max
�
NX
n=1
⇣Ed⇠ptrue(d|qn,r) [logD(d|qn)] +
Ed⇠p✓(d|qn,r) [log(1�D(d|qn))]⌘
D(d|q) = �(f�(d, q)) =exp(f�(d, q))
1 + exp(f�(d, q))where
![Page 50: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/50.jpg)
IRGANFormulation
• OptimizingDiscriminativeRetrieval
�⇤= argmax
�
NX
n=1
⇣Ed⇠ptrue(d|qn,r) [log(�(f�(d, qn))] +
Ed⇠p✓⇤ (d|qn,r) [log(1� �(f�(d, qn)))]⌘
![Page 51: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/51.jpg)
IRGANFormulation
• OptimizingGenerativeRetrieval• Samplesdocumentsfromthewholedocumentsettofoolitsopponent
• REINFORCE(AdvantageFunction)
RewardTerm
![Page 52: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/52.jpg)
IRGANFormulation
• Algorithm
![Page 53: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/53.jpg)
IRGANFormulation
• ExtensiontoPairwiseCase• It is common that the dataset is a set of ordereddocument pairs for each query rather than a set ofrelevant documents.
• Capture relativepreferencejudgements rather thanabsoluterelevancejudgements
• Now, for each query ,we have asetoflabelleddocumentpairs
qnRn = {hdi, dji|di � dj}
![Page 54: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/54.jpg)
IRGANFormulation
• ExtensiontoPairwiseCase• Discriminator would try to predict if a document pair iscorrectly ranked, which can be implemented as manypairwise ranking loss function:
• RankNet:• Ranking SVM (Hinge Loss):• RankBoost:
z = f�(du, q)� f�(dv, q)
(1� z)+
log(1 + exp(�z))
exp(�z)
where
![Page 55: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/55.jpg)
IRGANFormulation
• ExtensiontoPairwiseCase• Generator wouldtrytogeneratedocumentpairsthatare similartothosein ,i.e.,withthecorrectranking.
• A softmax function over the Cartesian Product of thedocument sets, where the logits is the advantage ofover in a document pair
Rn
didj (di, dj)
![Page 56: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/56.jpg)
An Intuitive Explanation of IRGAN
![Page 57: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/57.jpg)
An Intuitive Explanation of IRGAN
• Thegenerativeretrieval modelisguidedbythesignalprovidedfromthediscriminative retrievalmodel,whichmakesitmorefavorablethanthenon-learning methodsortheMaximumLikelihoodEstimation (MLE)scheme.
• Thediscriminativeretrievalmodelcouldbeenhancedtobetter ranktopdocumentsviaastrategicnegativesamplingfromthe generator.
![Page 58: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/58.jpg)
Experiments: Web Search
![Page 59: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/59.jpg)
Experiments: Web Search
![Page 60: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/60.jpg)
Experiments: Item Recommendation
![Page 61: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/61.jpg)
Experiments: Item Recommendation
![Page 62: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/62.jpg)
Experiments: Question Answering
![Page 63: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/63.jpg)
SummaryofIRGAN• We proposed IRGANframeworkthatunifiestwoschoolsofinformationretrievalmethodologies, viaadversarialtraininginaminimax game, which takesadvantage of both schools of thinking.
• Significant performance gains were observed inthree typical information retrieval tasks.
• Experiments suggest that different equilibriacouldbereachedintheenddependingonthetasks andsettings.
![Page 64: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/64.jpg)
SummaryofThisTalk• GenerativeAdversarialNetworks(GANs)aresopopularindeepunsupervisedlearningresearch
Training/evaluation Use (Turing Test)
• Fordiscretedatageneration,onecoulddirectlydefinetheparametricdistributionandoptimizetheP.D.F.bypolicygradientmethods,e.g.REINFORCE
• GANsprovideanewtrainingframeworkclosertothetargetuseofthegenerativemodel
maxEx⇠p(x)[log q(x)] maxE
x⇠q(x)[log p(x)]
![Page 65: Generative Adversarial Networks (GANs) for Discrete Datalantaoyu.com/files/2017-07-26-gan-for-discrete-data.pdf · 7/26/2017 · Real World Generator Discriminator Data. GANs: A](https://reader033.vdocuments.us/reader033/viewer/2022051916/60076ec4aab37172aa2ac741/html5/thumbnails/65.jpg)
ThankYou
Lantao YuShanghaiJiaoTongUniversity
http://lantaoyu.com
• Lantao Yu,WeinanZhang,JunWang,YongYu.SeqGAN:SequenceGenerativeAdversarialNetswithPolicyGradient.AAAI2017.
• JunWang,Lantao Yu,WeinanZhang,YuGong,Yinghui Xu,Benyou Wang,PengZhangandDellZhang.IRGAN:AMinimaxGameforUnifyingGenerativeandDiscriminativeInformationRetrievalModels.SIGIR2017.