accomplishments and challenges - york...
TRANSCRIPT
![Page 1: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/1.jpg)
Performance Regression AnalysisPerformance Regression Analysis
Cor-Paul Bezemer
Accomplishments and ChallengesAccomplishments and Challenges
![Page 2: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/2.jpg)
““A burnt child dreads fire”A burnt child dreads fire”
![Page 3: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/3.jpg)
Regression testing makes sure we do Regression testing makes sure we do not make the same mistake twicenot make the same mistake twice
Write codeExecute regressiontest
Deploy or fix changes
![Page 4: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/4.jpg)
Regression testing also helps us to make Regression testing also helps us to make sure we are not re-introducing more bugs sure we are not re-introducing more bugs
![Page 5: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/5.jpg)
Performance regression testing helps ensure Performance regression testing helps ensure we are not breaking performancewe are not breaking performance
Write codeExecute performance test
Deploy if performance is OK or fix changes
![Page 6: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/6.jpg)
Performance regression testing Performance regression testing can/should be done on several metricscan/should be done on several metrics
![Page 7: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/7.jpg)
Performance regression testing Performance regression testing can/should be done on several metricscan/should be done on several metrics
![Page 8: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/8.jpg)
Performance regression testing Performance regression testing can/should be done on several metricscan/should be done on several metrics
![Page 9: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/9.jpg)
Performance regression testing Performance regression testing can/should be done on several metricscan/should be done on several metrics
![Page 10: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/10.jpg)
Directions in research on Directions in research on performance regression analysisperformance regression analysis
![Page 11: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/11.jpg)
Directions in performance Directions in performance regression analysis researchregression analysis research
1. Execution profile comparison
2. Source code analysis
3. Visualization
![Page 12: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/12.jpg)
Execution profile comparisonExecution profile comparison
![Page 13: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/13.jpg)
Execution profile comparisonExecution profile comparison
![Page 14: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/14.jpg)
Disadvantages of execution Disadvantages of execution profile comparisonprofile comparison
- Need to run performance tests several times because of variation
- Small changes may be difficult to interpret
![Page 15: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/15.jpg)
Source code analysisSource code analysis
Identify risky commits and recommend those for testing (Huang et al. @ ICSE 2014)
![Page 16: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/16.jpg)
Disadvantages of source Disadvantages of source code analysiscode analysis
- Can not replace 'real' performance measurements (yet)
![Page 17: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/17.jpg)
Visualization of performance Visualization of performance regression analysis resultsregression analysis results
![Page 18: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/18.jpg)
Control charts Control charts (Nguyen et al. @ MSR 2014)(Nguyen et al. @ MSR 2014)
![Page 19: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/19.jpg)
Differential flame graphsDifferential flame graphs(Bezemer et al. @ SANER 2015)(Bezemer et al. @ SANER 2015)
![Page 20: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/20.jpg)
![Page 21: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/21.jpg)
Deciding on the baseline Deciding on the baseline performanceperformance
![Page 22: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/22.jpg)
Deciding on the acceptable Deciding on the acceptable performance deviationperformance deviation
![Page 23: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/23.jpg)
Reporting results to the Reporting results to the developerdeveloper
![Page 24: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/24.jpg)
Availability of performance Availability of performance teststests
![Page 25: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/25.jpg)
Execution time of testsExecution time of tests
![Page 26: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/26.jpg)
Performance regression testing helps ensure Performance regression testing helps ensure we are not breaking performancewe are not breaking performance
Write codeExecute performance test
Deploy if performance is OK or fix changes
![Page 27: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/27.jpg)
![Page 28: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/28.jpg)
Directions in performance Directions in performance regression analysis researchregression analysis research
1. Execution profile comparison
2. Source code analysis
3. Visualization
![Page 29: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/29.jpg)
![Page 30: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/30.jpg)
Availability of performance Availability of performance teststests
![Page 31: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/31.jpg)
![Page 32: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/32.jpg)
Execution time of testsExecution time of tests
![Page 33: Accomplishments and Challenges - York Universitylt2016.eecs.yorku.ca/slides/performance_regression... · 2016. 3. 14. · Accomplishments and Challenges ... Directions in performance](https://reader036.vdocuments.us/reader036/viewer/2022062607/60485a3a3fe7493f96188dc0/html5/thumbnails/33.jpg)