multi-task deep learning in the software development domain · multi-task deep learning (multi-task...
TRANSCRIPT
![Page 1: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/1.jpg)
Chair of Software Engineering for Business Information Systems (sebis) Faculty of InformaticsTechnische Universität Münchenwwwmatthes.in.tum.de
Multi-task Deep Learning in the Software Development domainSilvia Severini, Garching, 04.11.19Advisor: Ahmed Elnaggar and Davide Bacciu
![Page 2: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/2.jpg)
▪ Introduction▪ Background▪ Research questions▪ Approach▪ Datasets ▪ Results and discussion▪ Conclusion▪ Future research
Outline
© sebis 1Final Presentation Master Thesis – Silvia Severini
![Page 3: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/3.jpg)
The Software Development Domain
© sebis 2
High-quality software:
● nowadays almost every aspect of life relies on it (health, transportation, entertainment)
● Costly
● Hard-work for developers
=> Application of Machine Learning techniques
“Deep learning has achieved competitive performance against previous
algorithms on about 40 SE tasks” [cit]
Final Presentation Master Thesis – Silvia Severini
![Page 4: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/4.jpg)
Problem
When applying Deep learning to software development domain tasks, we face three main problems:
● Data scarcity
● Energy consumption
● Manipulation of source code
Source code summarization for Python
© sebis 3Final Presentation Master Thesis – Silvia Severini
![Page 5: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/5.jpg)
▪ Introduction▪ Background▪ Research questions▪ Approach▪ Datasets ▪ Results and discussion▪ Conclusion▪ Future research
Outline
© sebis 4Final Presentation Master Thesis – Silvia Severini
![Page 6: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/6.jpg)
Natural Language Processing on Source Code
● Context awareness
● Unlimited vocabulary
● Data preprocessing and tokenization
public class HelloWorld { public static void main ( String [ ] args ) { System . out . println ( " Hello, World " ) ; } }
© sebis 5Final Presentation Master Thesis – Silvia Severini
![Page 7: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/7.jpg)
Multi-task Deep Learning
(Multi-task Deep Learning) given m learning tasks {Ti }i=1m where all the tasks or a subset of them
are related, multi-task learning aims to help improve the learning of a model for Ti by using the knowledge contained in all or some of the m tasks.
Why?
● Implicit data augmentation● Attention focusing● Representation bias● Regularization
=> Augment of the generalization capabilities [1]
© sebis 6Final Presentation Master Thesis – Silvia Severini
![Page 8: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/8.jpg)
▪ Introduction▪ Background▪ Research questions▪ Approach▪ Datasets ▪ Results and discussion▪ Conclusion▪ Future research
Outline
© sebis 7Final Presentation Master Thesis – Silvia Severini
![Page 9: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/9.jpg)
Research questions
Can multi-task deep learning be beneficial for tasks in the software development domain?1
How far is multi-task deep learning from state-of-the-art solutions in the software development domain?
2
Could the model be trained with the English language and programming languages together?
3
How does training on multiple tasks of the software development domain simultaneously compare to training on each task separately act?
4
© sebis 8Final Presentation Master Thesis – Silvia Severini
![Page 10: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/10.jpg)
▪ Introduction▪ Background▪ Research questions▪ Approach▪ Datasets ▪ Results and discussion▪ Conclusion▪ Future research
Outline
© sebis 9Final Presentation Master Thesis – Silvia Severini
![Page 11: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/11.jpg)
Transformer model
State-of-the-art for sequence-to-sequence problem manipulation:
● Parallelizable computation
● Faster training
● Manipulation of long-range dependencies
● Encoder - Decoder
Transformer model by Vaswani et al. (2017)
© sebis 10Final Presentation Master Thesis – Silvia Severini
![Page 12: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/12.jpg)
Model architecture
● Multi-task
○ Decoder-only with attention
○ Vocabulary from the language model
○ Multiple loss functions optimized concurrently
● Single-task
○ Each task is trained independently
○ No language model objective
○ Encoder-decoder with attention
© sebis 11Final Presentation Master Thesis – Silvia Severini
![Page 13: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/13.jpg)
Experimental setup
Leibniz-Rechenzentrum (LRZ) available machines
● TensorFlow based library maintained by the Google Brain team
● Three phases:○ Data generation○ Training○ Decoding and evaluation
Hardware:
© sebis 12Final Presentation Master Thesis – Silvia Severini
![Page 14: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/14.jpg)
▪ Introduction▪ Background▪ Research questions▪ Approach▪ Datasets ▪ Results and discussion▪ Conclusion▪ Future research
Outline
© sebis 13Final Presentation Master Thesis – Silvia Severini
![Page 15: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/15.jpg)
Dataset
© sebis 14Final Presentation Master Thesis – Silvia Severini
![Page 16: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/16.jpg)
Dataset and preprocessing
1. 7 supervised tasks from “Deep learning in
software engineering” [Li et al.]
2. Unsupervised Language Model with 5
languages
● Substitution of char, string and numbers with
specific tokens
● Linearization of the code snippets
● Specific Tokenizer for each language
© sebis 15Final Presentation Master Thesis – Silvia Severini
![Page 17: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/17.jpg)
▪ Introduction▪ Background▪ Research questions▪ Approach▪ Datasets ▪ Results and discussion▪ Conclusion▪ Future research
Outline
© sebis 16Final Presentation Master Thesis – Silvia Severini
![Page 18: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/18.jpg)
Single-task learning
© sebis 17Final Presentation Master Thesis – Silvia Severini
![Page 19: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/19.jpg)
Multi-task learning
0.78
0.74
0.70
0.66
0.62
0.58
50k 100k 150k 200k 250k 300k 350k 400k 450k 500k 550k 600k
© sebis 18Final Presentation Master Thesis – Silvia Severini
![Page 20: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/20.jpg)
Single-task vs Multi-task
● MTL performed better on
summarization tasks
● Overfitting avoided for MTL
© sebis 19Final Presentation Master Thesis – Silvia Severini
![Page 21: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/21.jpg)
Comparison with state-of-the-art
● First and last do not have BLEU
counterpart
● Improvement over 4 tasks
© sebis 20Final Presentation Master Thesis – Silvia Severini
Program Synthesis
![Page 22: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/22.jpg)
Partial results
Small MTL Big MTL
50k 100k 150k 200k 250k 300k 350k 400k 450k 500k 550k 600k 650k
65%
74%
0.78
0.74
0.70
0.66
0.62
0.58
© sebis 21Final Presentation Master Thesis – Silvia Severini
![Page 23: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/23.jpg)
▪ Introduction▪ Background▪ Research questions▪ Approach▪ Datasets ▪ Results and discussion▪ Conclusion▪ Future research
Outline
© sebis 22Final Presentation Master Thesis – Silvia Severini
![Page 24: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/24.jpg)
Conclusions
● We investigate the application of Multi-task learning in the software development domain
● We trained single-task and multi-tasks models based on the Transformer architecture
● Consider the structural information of the source code to deal with English and programming
languages
● We achieved better performances than the state-of-the-art papers on 4 tasks
● Multi-task learning is promising for this domain, given enough resources available
© sebis 23Final Presentation Master Thesis – Silvia Severini
![Page 25: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/25.jpg)
▪ Introduction▪ Background▪ Research questions▪ Approach▪ Datasets ▪ Results and discussion▪ Conclusion▪ Future research
Outline
© sebis 24Final Presentation Master Thesis – Silvia Severini
![Page 26: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/26.jpg)
Future research
● Deeper hyperparameters exploration
● Combination of different tasks to understand which give benefits
● Better evaluate the effectiveness of the language model
● Starting point for future research in the software development domain
© sebis 25Final Presentation Master Thesis – Silvia Severini
![Page 27: Multi-task Deep Learning in the Software Development domain · Multi-task Deep Learning (Multi-task Deep Learning) given m learning tasks {T i} i=1 m where all the tasks or a subset](https://reader034.vdocuments.us/reader034/viewer/2022042620/5f45479f25535f04fe6940d8/html5/thumbnails/27.jpg)
Technische Universität MünchenFaculty of InformaticsChair of Software Engineering for Business Information Systems
Boltzmannstraße 385748 Garching bei München
Tel +49.89.289.Fax +49.89.289.17136
wwwmatthes.in.tum.de
Silvia Severini
17132