Mining Co-Change Information to Understand when Build Changes are Necessary
@shane_mcintosh
Shane McIntosh
Bram Adams
Meiyappan Nagappan
Ahmed E. Hassan
What is a build system?
Source code
2
What is a build system?
Source code
Deliverable
2
.tex
.c
.cc
.o
.o
.dvi
.a
.exe
.deb
Build systems describe how sources are!translated into deliverables
3
Continuous Integration:!Enabled by the
build system
4
.c .mk
Continuous Integration:!Enabled by the
build system
Commit
4
Commit 9719cf0
.c .mk
Continuous Integration:!Enabled by the
build system
Commit
4
BuildCommit 9719cf0
.c .mk
Continuous Integration:!Enabled by the
build system
Commit
4
Build
Test
Commit 9719cf0
.c .mk
Continuous Integration:!Enabled by the
build system
Commit
4
Build
Test
ReportCommit 9719cf0 was successfully integrated
Commit 9719cf0
.c .mk
Continuous Integration:!Enabled by the
build system
Commit
4
Build
Test
ReportCommit 9719cf0 was successfully integrated
Commit 9719cf0
.c .mk
“...nothing can be said to be certain, except death and taxes” - Benjamin Franklin
The Build “Tax”
An Empirical Study of Build Maintenance Effort!
S. McIntosh, B. Adams, T. H. D. Nguyen, Y. Kamei, A. E. Hassan
[ICSE 2011]
Up to 27% of source changes require build
changes, too!
5
Neglected build maintenance!is a frequent cause build breaks
6
.c .mk
Neglected build maintenance!is a frequent cause build breaks
Commit
6
Commit aedd38
.c
.mk
Neglected build maintenance!is a frequent cause build breaks
Commit
6
Commit aedd38
.c
.mk
Neglected build maintenance!is a frequent cause build breaks
Commit
6
BuildCommit aedd38
.c
.mk
Neglected build maintenance!is a frequent cause build breaks
Commit
6
Build
Test
Commit aedd38
.c
.mk
Neglected build maintenance!is a frequent cause build breaks
Commit
6
Build
Test
Commit aedd38
.c
.mk
Neglected build maintenance!is a frequent cause build breaks
Commit
6
Build
Test
Report
Commit aedd38
.c
.mk
Commit aedd38 broke the build!
Neglected build maintenance!can even impact end users
7
Neglected build maintenance!can even impact end users
7
Not working due to linking of
incorrect SQLite library version
Neglected build maintenance!can even impact end users
7
Not working due to linking of
incorrect SQLite library version
When are build!changes necessary?
8
Overview of the studied systems
C++ Java
8
Overview of the studied systems
C++ Java
8
Overview of the studied systems
13 years of
historical data
C++ Java
8
Overview of the studied systems
13 years of
historical data
Single build system for Firefox, Thunderbird, etc.
C++ Java
8
Overview of the studied systems
13 years of
historical data
Single build system for Firefox, Thunderbird, etc.
C++ Java
Total of 16 years of historical data
8
Overview of the studied systems
13 years of
historical data
Single build system for Firefox, Thunderbird, etc.
C++ Java
Proprietary and open
source systems
Total of 16 years of historical data
9
Semi-automatic identification of file types using naming conventions
.mk.c.c.ac.dat.h
9
Semi-automatic identification of file types using naming conventions
.mk.cTest
.cBuildSource
.ac.dat.hBuildTestSource
Grouping related changes according to the work items that they address
10
Grouping related changes according to the work items that they address
.c .c .cChanges .mk
10
Missed code!in #2121
Add feature!#2121
Fix for!bug #1234
Grouping related changes according to the work items that they address
.c .c .c
Transactions
Changes .mk
10
2121
Missed code!in #2121
Add feature!#2121
1234
Fix for!bug #1234
Grouping related changes according to the work items that they address
.c .c .c
Transactions
Work items
Changes .mk
10
11
!
Case study structure
11
!
Case study structure
11
(RQ1)!Co-change frequency
.c .mk?
!
Case study structure
11
(RQ2)!Co-change prediction accuracy
(RQ1)!Co-change frequency
.c .mk?
!
Case study structure
11
(RQ2)!Co-change prediction accuracy
(RQ1)!Co-change frequency
.c .mk?
(RQ3)!Most powerful
co-change indicators
!
Case study structure
12
(RQ2)!Co-change prediction accuracy
(RQ1)!Co-change frequency
.c .mk?
(RQ3)!Most powerful
co-change indicators
13
The proportion of build changes that are accompanied by source code changes
.c .mk?
13
The proportion of build changes that are accompanied by source code changes
.c .mk?
.c .c.mk .mk .c.mk
13
The proportion of build changes that are accompanied by source code changes
.c .mk?
.c .cWork!items
.mk
1
.mk .c.mk
32
13
The proportion of build changes that are accompanied by source code changes
.c .mk?
.c .cWork!items
.mk
1
.mk .c.mk
32
2 build co-changes
3 total build changes
14
Build changes are often accompanied by changes to the source code
Mozilla Eclipse Lucene Jazz
Build ⇒ Source 86% 82% 44% 72%
Build ⇒ Test 29% 36% 41% 36%
Build ⇒ Src/Test 88% 82% 53% 77%
.c .mk?
15
Build changes are occasionally accompanied by changes to test code
Mozilla Eclipse Lucene Jazz
Build ⇒ Source 86% 82% 44% 72%
Build ⇒ Test 29% 36% 41% 36%
Build ⇒ Src/Test 88% 82% 53% 77%
.c .mk?
16
53%-88% of build changes are accompanied by changes to source or test code
Mozilla Eclipse Lucene Jazz
Build ⇒ Source 86% 82% 44% 72%
Build ⇒ Test 29% 36% 41% 36%
Build ⇒ Src/Test 88% 82% 53% 77%
.c .mk?
!
Case study structure
17
(RQ2)!Co-change prediction accuracy
(RQ1)!Co-change frequency
.c .mk?
(RQ3)!Most powerful
co-change indicators
Most build changes are
accompanied by source/test code changes
!
Case study structure
18
(RQ2)!Co-change prediction accuracy
(RQ1)!Co-change frequency
.c .mk?
(RQ3)!Most powerful
co-change indicators
Most build changes are
accompanied by source/test code changes
19
Metrics that may indicate whether a code change will require an accompanying build co-change
19
Metrics that may indicate whether a code change will require an accompanying build co-change
Language-agnostic
19
Metrics that may indicate whether a code change will require an accompanying build co-change
Language-agnostic
+++ File added
19
Metrics that may indicate whether a code change will require an accompanying build co-change
Language-agnostic
+++ File added
.mkHistorical tendencies .c
19
Metrics that may indicate whether a code change will require an accompanying build co-change
Language-agnostic
Language-aware
+++ File added
.mkHistorical tendencies .c
19
Metrics that may indicate whether a code change will require an accompanying build co-change
Language-agnostic
Language-aware
+++ File added
.mkHistorical tendencies .c
Changed dependencies
import
19
Metrics that may indicate whether a code change will require an accompanying build co-change
Language-agnostic
Language-aware
+++ File added
.mkHistorical tendencies .c
Changed dependencies
import
Changed!conditional compilation
#ifdef
1 2
.mk
20
We train classifiers to identify code changes that require build changes
Work!items
.c.c .c
Random forest classification model
Build change necessary
No build change necessary
1 2
.mk
20
We train classifiers to identify code changes that require build changes
Work!items
.c
.c .cRandom forest classification model
Build change necessary
No build change necessary
1 2
.mk
21
We train classifiers to identify code changes that require build changes
Work!items
.c
Random forest classification model
Build change!necessary
No build change necessary
1 2
.mk
22
We train classifiers to identify code changes that require build changes
Work!items
.c.c .c
Random forest classification model
Build change necessary
No build change necessary
1 2
.mk
22
We train classifiers to identify code changes that require build changes
Work!items
.c
.c .c
Random forest classification model
Build change necessary
No build change necessary
1 2
.mk
23
We train classifiers to identify code changes that require build changes
Work!items
.c .c
Random forest classification model
Build change necessary
No build change!necessary
24
Language-aware metrics add significant explanatory power
Mozilla Eclipse Lucene Jazz
AUC (language-agnostic metrics) 0.84 0.66 0.71 0.57
AUC (language-agnostic and
aware metrics)0.88 0.69 0.78 0.60
25
Mozilla classifiers are highly accurate
Mozilla Eclipse Lucene Jazz
AUC (language-agnostic metrics) 0.84 0.66 0.71 0.57
AUC (language-agnostic and
aware metrics)0.88 0.69 0.78 0.60
26
Java classifiers are less accurate than Mozilla one
Mozilla Eclipse Lucene Jazz
AUC (language-agnostic metrics) 0.84 0.66 0.71 0.57
AUC (language-agnostic and
aware metrics)0.88 0.69 0.78 0.60
27
Java build changes are rarely!driven by code changes
Category Task Amount Correctly classifiedSystem structure Refactorings 19 (25%) 8
General build maintenance
Build tool configuration 15 (20%) 0
Build defects 6 (8%) 0
Release engineering
Add platform support 12 (16%) 2
Packaging fixes 12 (16%) 3
Library versioning 8 (11%) 0
Test maintenance Test infrastructure 3 (4%) 0
!
Case study structure
28
(RQ2)!Co-change prediction accuracy
(RQ1)!Co-change frequency
.c .mk?
(RQ3)!Most powerful
co-change indicators
Most build changes are
accompanied by source/test code changes
AUCC++ 0.88, AUCJava 0.60-0.88
Java build changes are rarely motivated by code changes
!
Case study structure
29
(RQ2)!Co-change prediction accuracy
(RQ1)!Co-change frequency
.c .mk?
(RQ3)!Most powerful
co-change indicators
Most build changes are
accompanied by source/test code changes
AUCC++ 0.88, AUCJava 0.60-0.88
Java build changes are rarely motivated by code changes
30
Measuring variable importance in random forest classifiers
Random Forests!L. Breiman
[Machine Learning 2001]
500
Var1 Var2
150
Build!change?
NoNoYes
Var3
2456777
11165
413
Random forest classification model
ClassificationWork!Item
123
30
Measuring variable importance in random forest classifiers
Random Forests!L. Breiman
[Machine Learning 2001]
500
Var1 Var2
150
Build!change?
NoNoYes
Var3
2456777
11165
413
Random forest classification model
ClassificationWork!Item
123
NoYesYes
30
Measuring variable importance in random forest classifiers
Random Forests!L. Breiman
[Machine Learning 2001]
500
Var1 Var2
150
Build!change?
NoNoYes
Var3
2456777
11165
413
Random forest classification model
ClassificationWork!Item
123
NoYesYes
Misclassification rate of 1 ÷ 3 = 0.33
30
Measuring variable importance in random forest classifiers
Random Forests!L. Breiman
[Machine Learning 2001]
500
Var1 Var2
150
Build!change?
NoNoYes
Var3
2456777
11165
413
Random forest classification model
ClassificationWork!Item
123
NoYesYes
Randomly
permute values
31
Measuring variable importance in random forest classifiers
Random Forests!L. Breiman
[Machine Learning 2001]
500
Var1 Var2
150
Build!change?
NoNoYes
Var3
2
456
77711165
413
Random forest classification model
ClassificationWork!Item
123
31
Measuring variable importance in random forest classifiers
Random Forests!L. Breiman
[Machine Learning 2001]
500
Var1 Var2
150
Build!change?
NoNoYes
Var3
2
456
77711165
413
Random forest classification model
ClassificationWork!Item
123
YesYesYes
31
Measuring variable importance in random forest classifiers
Random Forests!L. Breiman
[Machine Learning 2001]
500
Var1 Var2
150
Build!change?
NoNoYes
Var3
2
456
77711165
413
Random forest classification model
ClassificationWork!Item
123
YesYesYes
Misclassification rate of 2 ÷ 3 = 0.67
31
Measuring variable importance in random forest classifiers
Random Forests!L. Breiman
[Machine Learning 2001]
500
Var1 Var2
150
Build!change?
NoNoYes
Var3
2
456
77711165
413
Random forest classification model
ClassificationWork!Item
123
YesYesYes
Misclassification rate of 2 ÷ 3 = 0.67
Var1 randomness increases misclassification rate by 0.33
Mozilla Eclipse−core Lucene Jazz
●
●●
●● ●●
●●●●
●
●
●
0.00
0.05
0.10
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Conditional co
mpilation
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Varia
ble
Impo
rtanc
e Sc
ore
32
Variable importance in the!Mozilla classifier
Mozilla Eclipse−core Lucene Jazz
●
●●
●● ●●
●●●●
●
●
●
0.00
0.05
0.10
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Conditional co
mpilation
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Varia
ble
Impo
rtanc
e Sc
ore
32
Variable importance in the!Mozilla classifier
Structure-altering changes are
most important in C++ classifiers
Mozilla Eclipse−core Lucene Jazz
●
●●
●● ●●
●●●●
●
●
●
0.00
0.05
0.10
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Conditional co
mpilation
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Varia
ble
Impo
rtanc
e Sc
ore
32
Variable importance in the!Mozilla classifier
Structure-altering changes are
most important in C++ classifiers
Historical co-change tendencies are also important
33
Variable importance in Java classifiers
Mozilla Eclipse−core Lucene Jazz
●
●●
●● ●●
●●●●
●
●
●
0.00
0.05
0.10
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Conditional co
mpilation
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Varia
ble
Impo
rtanc
e Sc
ore
33
Variable importance in Java classifiers
Mozilla Eclipse−core Lucene Jazz
●
●●
●● ●●
●●●●
●
●
●
0.00
0.05
0.10
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Conditional co
mpilation
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Varia
ble
Impo
rtanc
e Sc
ore Structure-altering is less
important than in Mozilla
33
Variable importance in Java classifiers
Mozilla Eclipse−core Lucene Jazz
●
●●
●● ●●
●●●●
●
●
●
0.00
0.05
0.10
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Conditional co
mpilation
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Prior build co−ch
anges
Number of files
Source added
Test added
Source deleted
Test deleted
Source modifie
d
Test modifie
d
Source renamed
Test renamed
Non−core dependencie
s
Varia
ble
Impo
rtanc
e Sc
ore Structure-altering is less
important than in Mozilla
Java classifiers rely more on historical co-change
tendencies
!
Case study structure
34
(RQ2)!Co-change prediction accuracy
(RQ1)!Co-change frequency
.c .mk?
(RQ3)!Most powerful
co-change indicators
Most build changes are
accompanied by source/test code changes
AUCC++ 0.88, AUCJava 0.60-0.88
Java build changes are rarely motivated by code changes
C++: Structure-altering changes
Java: Historical co-change tendencies
