![Page 1: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/1.jpg)
Learning to Generate Product Reviews from Attributes
Authors: Li Dong, Shaohan Huang, Furu Wei, Mirella Lapata, Ming Zhou and Ke Xu
Presenter: Yimeng Zhou
![Page 2: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/2.jpg)
IntroductionX Presents an attention-enhanced attribute-to-
sequence model to generate product reviews for given attribute information such as user, product and rating.
![Page 3: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/3.jpg)
IntroductionXChallenges:XVariety of candidate reviews that satisfy the input
attributes.XUnknown or latent factors that influence the
generated reviews, which renders the generation process non-deterministic.
XRating explicitly determine the usage of sentiment words.
XUser and product implicitly influence word usage.
![Page 4: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/4.jpg)
Compared to Prior workXMost previous work focuses on using rule-based
methods or machine learning techniques for sentiment classification, which classifies reviews into different sentiment categories
X In contrast, this model is mainly evaluated on the review generation task rather than classification. Moreover, it uses an attention mechanism in encoder-decoder model
![Page 5: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/5.jpg)
Model - OverviewX Input attributesXGenerate product review to
maximize the conditional probability p(r|a)X|a| is fixed to 3 with userID, productid and
rating.
![Page 6: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/6.jpg)
Model - Overview
X The model learns to compute the likelihood of generated reviews given input attributes.
X This conditional probability p(r|a) is decomposed to
![Page 7: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/7.jpg)
Model – Three parts
X Attribute Encoder
X Sequence Decoder
X Attention MechanismX Att2seq model without attention mechanism
![Page 8: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/8.jpg)
Model – Attribute EncoderX Use multilayer perceptrons to encode input attributes
into vector representations that are used as latent factors for generating reviews.
X Input attributes a are represented by low-dimensional vectors. The attribute ai‘s vector g(ai) is computed via
X Where is a parameter matrix and e(ai) is a one-hot vector representing the presence or absence of ai.
![Page 9: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/9.jpg)
Model – Attribute EncoderX Then these attribute vectors are concatenated and
fed into a hidden layer which outputs the encoding vector. The output of the hidden layer is computed as:
![Page 10: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/10.jpg)
Model – Sequence DecoderX The decoder is built by stacking multiple layers of
recurrent neural networks with long short-term memory units to better handle long sequences.
X RNNs use vectors to represent information for the current time step and recurrently compute the next hidden states.
![Page 11: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/11.jpg)
Model – Sequence DecoderX The LSTM introduces several gates and explicit
memory cells to memorize or forget information, which enables networks learn more complicated patterns
X The n-dimensional hidden vector in layer l and time step t is computed via
![Page 12: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/12.jpg)
Model – Sequence DecoderX The LSTM unit is given by
![Page 13: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/13.jpg)
Model – Sequence DecoderX Finally, for the vanilla model without using an
attention mechanism, the predicted distribution of the t-th output word is:
![Page 14: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/14.jpg)
Model – Attention MechanismX Better utilize encoder-side informationX The attention mechanism learns soft alignments
between generated words and attributes, and adaptively computes encoder-side context vectors used to predict the next tokens.
![Page 15: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/15.jpg)
Model – Attention Mechanism
![Page 16: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/16.jpg)
Model – Attention MechanismX For the t-th time step of the decoder, we compute
the attention score of attribute ai via
X Z is a normalization term that ensures
![Page 17: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/17.jpg)
Model – Attention MechanismX Then the attention context vector ct is obtained by
which is a weighted sum of attribute vectors.
![Page 18: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/18.jpg)
Model – Attention MechanismX Further employ the vector to predict the t-th output
token as
![Page 19: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/19.jpg)
Model – Attention MechanismX Aim at maximizing the likelihood of generated
reviews given input attributes for the training data. X The optimization problem is to maximize
X Avoid overfitting: insert dropout layers between different LSTM layers as suggested in Zaremba et al. (2015).
![Page 20: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/20.jpg)
ExperimentsX Dataset: built upon Amazon product data including
reviews and metadata spanning.X The whole dataset is randomly split into three parts
TRAIN, DEV and TEST (70%. 10%, 20%)X Parameter settings:
X Dimension of Attributes vectors:64X Dimension of word embeddings and hidden vectors:512X Uniform distribution [-0.08,0.08]X Batch size, smoothing constant, learning rate: 50, 0.95, 0.0002X Dropout rate: 0.2X Gradient values: [-5, 5]
![Page 21: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/21.jpg)
Results
![Page 22: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/22.jpg)
Results - Polarities
![Page 23: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/23.jpg)
Results – Ablation
![Page 24: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/24.jpg)
Results – Attention Scores
![Page 25: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/25.jpg)
Results – Control Variable
![Page 26: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/26.jpg)
ImprovementsX Use more fine-grained attributes as the input of our
model.XConditioned on device specification, brand,
user’s gender, product description, etc.XLeverage review texts without attributes to
improve the sequence decoder.
![Page 27: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/27.jpg)
ConclusionX Proposed a novel product review generation task, in
which generated reviews are conditioned on input attributes,
X Formulated a neural network based attribute-to-sequence model that uses multilayer perceptrons to encode input attributes and employs recurrent neural networks to generate reviews.
X Introduced an attention mechanism to better utilize input attribute information.
![Page 28: Learning to Generate Product Reviews from Attributes · Experiments XDataset: built upon Amazon product data including reviews and metadata spanning. XThe whole dataset is randomly](https://reader033.vdocuments.us/reader033/viewer/2022060303/5f08aff77e708231d4233c25/html5/thumbnails/28.jpg)
Thank you!