assignment 2: statistical machine translationfrank/csc401/tutorials/csc401_a2_1.pdf · assignment...
TRANSCRIPT
![Page 1: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/1.jpg)
Assignment 2:
Statistical Machine
Translation
CSC401/2511 Tutorial, Winter 2019
Mohamed AbdallaBased on slides by Patricia Thaine, Jackie Cheung, Alex Fraser and Frank Rudzicz
![Page 2: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/2.jpg)
Today’s Schedule
• History of Machine Translation
• A2: the big picture
• Task 1: BLEU Score
• Task 2: Encoder-Decoder
• Task 3: Training and Testing
• Marking: How each tasks are evaluated
![Page 3: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/3.jpg)
Today’s Schedule
• History of Machine Translation
• A2: the big picture
• Task 1: BLEU Score
• Task 2: Encoder-Decoder
• Task 3: Training and Testing
• Marking: How each tasks are evaluated
![Page 4: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/4.jpg)
The early
years of MT
Rough History of Machine Translation Research
When I look at an article in
Russian, I say: "This is really
written in English, but has been
coded in some strange symbols. I
will now proceed to decode."
[Warren Weaver, 1947]
Cromieres, Fabien, Toshiaki Nakazawa, and Raj Dabre. "Neural Machine Translation:
Basics, Practical Aspects and Recent Trends." Proceedings of the IJCNLP 2017
![Page 5: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/5.jpg)
It turns out MT is
difficult
Rough History of Machine Translation Research
Beginning of
MT
Cromieres, Fabien, Toshiaki Nakazawa, and Raj Dabre. "Neural Machine Translation:
Basics, Practical Aspects and Recent Trends." Proceedings of the IJCNLP 2017
![Page 6: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/6.jpg)
Rough History of Machine Translation Research
It turns out MT is
difficult
Beginning of
MT
Disinterest in MT
research
(esp. in USA after
ALPAC report)
Cromieres, Fabien, Toshiaki Nakazawa, and Raj Dabre. "Neural Machine Translation:
Basics, Practical Aspects and Recent Trends." Proceedings of the IJCNLP 2017
![Page 7: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/7.jpg)
Rough History of Machine Translation Research
It turns out MT is
difficult
Beginning of
MT
Disinterest in MT
research
(esp. in USA after
ALPAC report)
Diversity and the
number of installed
systems for MT had
increased
Cromieres, Fabien, Toshiaki Nakazawa, and Raj Dabre. "Neural Machine Translation:
Basics, Practical Aspects and Recent Trends." Proceedings of the IJCNLP 2017
![Page 8: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/8.jpg)
Example-based
Machine Translation
(EBMT) [Nagao, 1981]
Rough History of Machine Translation Research
Cromieres, Fabien, Toshiaki Nakazawa, and Raj Dabre. "Neural Machine Translation:
Basics, Practical Aspects and Recent Trends." Proceedings of the IJCNLP 2017
![Page 9: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/9.jpg)
Example-based
Machine Translation
(EBMT) [Nagao, 1981]
Statistical Machine
Translation (SMT)
[Brown et al., 1993]
Rough History of Machine Translation Research
Cromieres, Fabien, Toshiaki Nakazawa, and Raj Dabre. "Neural Machine Translation:
Basics, Practical Aspects and Recent Trends." Proceedings of the IJCNLP 2017
![Page 10: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/10.jpg)
Example-based
Machine Translation
(EBMT) [Nagao, 1981]
Statistical Machine
Translation (SMT)
[Brown et al., 1993]
Rough History of Machine Translation Research
Phrase-based SMT
[Koehn et al., 2003]
Cromieres, Fabien, Toshiaki Nakazawa, and Raj Dabre. "Neural Machine Translation:
Basics, Practical Aspects and Recent Trends." Proceedings of the IJCNLP 2017
![Page 11: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/11.jpg)
Example-based
Machine Translation
(EBMT) [Nagao, 1981]
Statistical Machine
Translation (SMT)
[Brown et al., 1993]
Rough History of Machine Translation Research
Phrase-based SMT
[Koehn et al., 2003]
Neural Machine
Translation (NMT)
[Sutskever et al., 2014]
Cromieres, Fabien, Toshiaki Nakazawa, and Raj Dabre. "Neural Machine Translation:
Basics, Practical Aspects and Recent Trends." Proceedings of the IJCNLP 2017
![Page 12: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/12.jpg)
Example-based
Machine Translation
(EBMT) [Nagao, 1981]
Statistical Machine
Translation (SMT)
[Brown et al., 1993]
Rough History of Machine Translation Research
Phrase-based SMT
[Koehn et al., 2003]
Neural Machine
Translation (NMT)
[Sutskever et al., 2014]
Cromieres, Fabien, Toshiaki Nakazawa, and Raj Dabre. "Neural Machine Translation:
Basics, Practical Aspects and Recent Trends." Proceedings of the IJCNLP 2017
![Page 13: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/13.jpg)
Today’s Schedule
• History of Machine Translation
• A2: the big picture
• Task 1: Preprocess Inputs
• Task 2: Compute n-gram counts
• Marking: How each tasks are evaluated
![Page 14: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/14.jpg)
A2 – Historically:
Noisy Channel Model
We need a language model, a translation
model, and a decoder.
your tasks!
![Page 15: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/15.jpg)
A2 – Currently:
Noisy Channel Model
We need an encoder, a decoder, and a search
algorithm.
![Page 16: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/16.jpg)
Task 1. BLEU Score
File: “a2_bleu_score.py”
4 functions:
1. grouper(seq, n):
2. n_gram_precision(reference, candidate, n):
3. brevity_penalty(reference, candidate):
4. BLEU_score(reference, hypothesis, n):
![Page 17: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/17.jpg)
Task 1. BLEU Score
BLEU Score is calculated like in the lecture
slides.
Notes:
- No capping.
- Only 1 reference and 1 candidate at a time.
- Don’t include SOS and EOS tokens in the
calculations.
![Page 18: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/18.jpg)
Task 2. Encoder and Decoder
File: “a2_encoder_decoder.py”
4 Classes:
1. Encoder(EncoderBase):
2. DecoderWithoutAttention(DecoderBase):
3. DecoderWithAttention(DecoderWithoutAttention):
4. EncoderDecoder(EncoderDecoderBase):
![Page 19: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/19.jpg)
Task 2. Encoder and Decoder
Notes:
• Does not follow 100% from the slides. Look at the
handouts!
• To correctly implement the code you’ll need to look
and understand other files (a2_abcs.py).
DO NOT CHANGE ANYTHING IN THE FILE.
![Page 20: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/20.jpg)
Task 3. Train and Test
File: “a2_training_and_testing.py”
3 Functions:
1. def train_for_epoch
2. def compute_batch_total_bleu
3. def compute_average_bleu_over_dataset
![Page 21: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/21.jpg)
General Development
- You will not need any imports that are not already
specified.
- Do not change any files that come implemented.
- Check Piazza for updates to the code.
- When dealing with matrices (i.e., tensors) in the
neural network, use vectorized/matrix operations.
![Page 22: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/22.jpg)
Marking
(A large) Portion of it will be auto-marked.
- Your code will be tested individually (by file).
- Your code must adhere to the specifications
for function calls to work.
- Do NOT hardcode any paths.
- It must work on CDF (test your code).
- There is a GPU component which will be
tough to finish on time if you leave it to the
last second.
![Page 23: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/23.jpg)
Marking (Cont)
Initially:
B C D
A B C D
Correct Code:
Your Code:
A
![Page 24: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/24.jpg)
Marking (Cont)
If we are testing “A”:
B C D
A B C D
Correct Code:
Your Code:
A
![Page 25: Assignment 2: Statistical Machine Translationfrank/csc401/tutorials/csc401_a2_1.pdf · Assignment 2: Statistical Machine Translation CSC401/2511 Tutorial, Winter 2019 Mohamed Abdalla](https://reader030.vdocuments.us/reader030/viewer/2022040215/5f0a97047e708231d42c60df/html5/thumbnails/25.jpg)
Marking (Cont)
If we are testing “B”:
A C D
A B C D
Correct Code:
Your Code:
B