iwsm2014 extracting dependencies from software changes (thomas wetzlmaier - claus klammer - rudolf...

SCCH is an initiative of SCCH is located in

Extracting Dependencies from Software Changes

An Industry Experience Report

Thomas Wetzlmaier

+43 7236 3343 [email protected]

Claus Klammer


Rudolf Ramler


2© Software Competence Center Hagenberg GmbH

Key Facts of our Research Organization

Application-oriented research organization Initiated by institutes of the Johannes

Kepler University (JKU) Linz ~ 65 employees

SAE: Software Analytics and Evolution Software Architecture Domain-specific languages Document extraction from code Automated Software Testing

5 Research Foci


Industry Case: ERP System

Project background ERP System for the building and construction industry Supersedes an IBM System i (AS/400) implemented software Client-server application with a modular rich client Technology: Java, JBoss, Hibernate Tool support: Subversion, Bugzilla, Mylyn Repository contains not only source code, but also other

document types (reports, data models, images, libs) We analyzed ~6 years of development and maintenance

We identified 4 categories with total of 31 features Client: 14 features Server: 4 features

Platform: 6 features Development: 7 features


Results of the Repository Analysis

Repository measures:

Distribution of transaction sizes:

x-axis: size in number of files; y-axis: number of transactions


Dependency Categories

Static vs. dynamic Dependencies Static: analyze source code (inheritance, method usage,

imports) Dynamic: analyze program execution (coverage, log files)

Physical vs. logical Dependencies Physical: obvious, static, strong Logical: not obvious, valuable,

no structural relation available, weak, any type of file (i.e. detected from co-changes)


Detected dependency types

TransactionChanges that have been committed together, i.e., in the same transaction

Change-IDChanges that are related to the same development task, e.g., enhancement or bug fix

CommentChanges that have the same commit message

Distribution of detected dependency types


Change History

11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8

A 10 2 46 24 10 3 3 4 1 27 26 16 17 3 12 89 1 2 10 20 20 9 4 27 11 2 16 14 4 16 2 25 12 13 9 4 6 6 17 18 7 3 21 2 1 6 5 11 5 1 2 6 5 14 7 12 5 6 3

B 50 2 201 5 313 119 23 3 171 103 3 18 1 1

C 33 1 3 26 1 4 1 36 14 42 1 6 51 26 31 2 3 5 16 1 1 17 29 8 16 12 13 5 6 14 23 29 7 5 2 2 29 2 1

D 40 15 87 56 19 15 46 6 18 6 4 45 4 13 2 5 3 1 1 2 2 1 1 4 7 10

E 9 9 105 1 12 5 4 43 5 2 2 1 1 2 1

F 138 161 68 43 7 41 15 81 7 12

G 100 7 96 1 1 1 1 116 207 86

H 205 94 27 7 9 188 137 67 162 83 24 284 149 55 80 57 17 51 22 100 183 106 153 67 102 83 442 125 31 84 7 31 187 394 124 63 44 112 3 50 122 45 129 1 14 67 85 23 102 61 21 268 80 26 32 41 36 87 501 4 168 164 83

I 29 327 23 52 49 8 17 12 23 21 25 16 217 33 6 17 15 5 26 57 156 10 64 130 65 8 6 63 28 77 82 27 71 23 161 78 33 153 15 20 56 9 129 3 14 6 68 6 26 84 2 20 25 12 11 46 29 39 29 41 5 8 1

J 24 8 9 14 9 3 13 11 19 48 13 1 1 4 1 3 6 8 1 11 11 3 2 45 11 12 8 16 7 2 12 14 27 11 5 4 6 1 3 6 57 102 12 1 4 13

K 884 324 884 884

L 136 160

M 858 808 4 934 ## ## 2 1 2 1515 1519 57 9

N 29 238 112 18 122 77 62 16 23 11 74 55 3 8 14 3 50 9 54 55 53 1 44 1 30 8 1 1 7 21 19 2 83 26 3 46 22 13 3 11 24 1 10

O 39 177 116 63 17 9 26 265 588 94 100 412 293 506 520 84 52 139 518 79 89 23 159 424 14 30 227 46 10 56 20 23 21 10 56 44 68 24 41 30 11 22 446 47 111 16 49 40 190 49 3 10 3 13 37 26 49 489 40 151 4 29 60 129 64

P 123 1 2 133 10 6 2 13 36 16 3 13 4 1 2 192 149 8 3 14 1 1 2 1 4 6 163 7 4 18 297 7

Q 12 19 76 73 4 20 78 35

R1 272 183 237 246 15 3 45 95 23 86 84 24 152 88 156 125 81 58 112 23 170 85 102 71 88 1139 24 369 73 4 148 4 282 17 94 48 399 429 412 32 100 155 70 740 551 479 83 42 297 173 90 44 32 2 81 310 14 22 22 16 155 92

R2 168 144 166 117 2 18 3 16 19 8 33 1 1 1 11 2 54 7 11 23 77 48 183 89 159 21 27 85 73 2 46 78 81 52 41 9 89 52 44 69 4 28 58 56 39 29 58 1 47 96 19 7 8 46 17 9 11 10 8 16

R3 100 625 158 356 693 361 157 951 165 187 88 435 217 293 46 193 125 216 131 285 121 276 292 156 248 13 183 38 133 216 311 193 109 157 147 134 91 91 20 29 125 153 94 75 155 2 111 53 26 9 15 30 21 33 62 2 7 38

R4 15 28 31 14 12 2 4 2 3 24 1 22 2 9 4 2 5 4 107 4 21 12 47 9 7 41 3 20 3 1 5 6 7 11 14 14 7 1

S 26 43 49 5 166 58 126 447 121 93 426 30 177 317 233 78 261 101 111 70 9 61 30 11 2 99 198 58 12 14 74 114 148 459 45 110 56 197 61 446 168 98 4 27 22 813 400 58 72 37 31 2 44 15 41 14 60 17 9 61 94 128

T1 176 141 59 38 19 8 119 59 58 5 76 5 8 6 19 25 7 76 29 22 1 19 18 6 16 3 1 12 61 61 1 52 34 14 25 84 45 25 6 9 12 1 84 5 2 51 23 8 5 1 2

T2 14 437 31 34 3 12 2 65 17 16 38 3 7 1

T3 ## 1 42 57

U 15 7 9 2 2 24 1 1 2 1 367 213 116 27 6 17 10 19 38 2 10 57 1 52 134 38 8 1 34 5 26 5 1 32 9 4 1 2 23 7

V 104 71 257 102 142 198 69 74 4 281 379 397 283 106 51 42 40 23 84 28

W 95 340 635 245 326 101 134 8 1 8 2 29 15 7 30 33 9 2 10

X 465 637 4 205 57 47 14 276 310 208 18 3

Y 228 96 7

Z 93 1

60 33 46 ### ### ### ### ### 43 ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### 96 ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ###

1 1 198 394 591 788 985 1181 ## 1575 1771 ## 2000

0

Activity

4000

Year 6

Feature

Dev.

Platform

Server

Client

Y 0 Year 1 Year 2 Year 3 Year 4 Year 5

number of changes to different files in distinct commits per month


11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8

A 6 568 340 4 12 66 246 21 20 50 75 2 4 34 30 117 31 ## 11 27 56 24 26 142 23 20 67 22 2 45 ## 2 71 1 3 25 56 252 18 25 29 19 4 70 29

B 228 2 ## 237 ## 804 4 24 12 1 ## ## 1165 42 25 119 1 16 32

C 120 346 4 252 8 266 10 2 140 47 1 8 7 166 269 180 2 12 4 6 469 ## 160 1 12 6 133 2

D 118 43 510 609 260 335 152 328 100 13 185 834 114 642 6 86 3 95 23 120 82 102 80 52 791 7 18 3 1

E 81 ## 2 172 16 1550 961 506 23 6 1 119 1 2 171

F ## 46 1572 332 38 140 816 8

G ## 169 ## 119 3 1 ## ## ##

H ## ## 406 264 253 3 ## ## ## ## 755 348 114 ## 1127 454 ## 145 651 182 ## ## 855 ## ## 2161 ## ## ## ## ## 10 123 ## ## ## ## 59 ## 79 685 ## 778 ## 546 ## ## 236 ## ## 385 ## 668 356 ## ## 767 ## ## 16 ## ## ##

I 732 ## 376 ## ## 30 98 62 62 449 488 219 95 64 545 190 82 494 6 141 284 ## 486 ## ## ## 437 20 ## 180 ## ## 278 891 504 ## 1719 57 ## 17 232 ## 327 ## 2 48 133 516 68 698 ## 4 198 194 158 67 ## 422 190 355 312 128 23 7

J ## 329 99 623 165 160 119 338 96 113 11 20 102 3 165 195 2 60 247 30 ## 124 10 1132 295 199 603 100 13 ## 196 ## 139 70 70 1518 22 709 ## ## 115 1 23 160

K ##

L ## ##

M 85 2 747 9 34 108 269 720 5 ## 157

N ## 656 53 2 221 30 23 ## 1361 24 398 94 98 45 ## ## ## 1019 234 10 28 240 468 5 ## 1316 720 ## ## ## 9 783 344 100

O 513 ## 1553 224 46 755 73 ## ## 207 596 ## ## ## ## 393 476 873 ## ## ## 652 732 ## 369 27 3715 ## 78 ## ## 292 1193 655 689 ## 210 733 347 363 252 ## 884 ## 121 920 216 ## 1315 10 256 9 37 385 404 ## ## 445 ## 17 ## 1219 ## ##

P ## 2 445 552 57 355 112 167 39 151 51 32 ## ## 554 36 120 4 6 6 1 75 155 ## 103 88 376 ## 396

Q ## 252 1471 ## 100 417 ## 837

R1 ## ## ## ## 156 15 170 120 722 ## 449 136 446 453 ## 1477 ## 325 1616 355 ## ## ## 1587 ## ## 365 ## ## 141 ## 6 251 351 1186 ## ## ## ## 355 ## ## ## ## 853 ## 880 ## ## ## ## 1149 320 71 ## ## 473 346 61 190 450 560 9 9

R2 ## 1291 5571 ## 18 20 167 ## 194 98 13 51 26 411 23 728 31 372 ## ## ## ## ## ## 996 ## ## 161 2 ## ## 928 327 544 125 ## ## ## ## 25 519 ## ## ## 1621 ## 34 764 1195 76 4 265 ## 59 177 590 145 224 494

R3 ## ## 513 ## ## ## ## ## ## ## 672 ## ## ## 254 ## ## ## ## ## ## ## ## ## ## 24 236 641 ## ## ## 562 831 4117 ## ## ## ## 28 136 ## ## ## ## ## 5 ## 788 864 677 712 973 80 555 ## 174 ##

R4 47 789 180 102 89 7 143 53 17 22 ## 300 14 238 48 38 ## 21 231 178 ## 97 292 822 9 ## 51 25 7 835 165 57 202 320 6

S 218 545 ## 94 ## 199 ## ## ## ## 35 296 ## ## ## ## ## ## ## ## 112 ## ## 867 ## ## ## 160 68 473 ## ## ## 517 ## ## ## 1153 ## ## ## 6 165 195 ## ## 318 4711 647 189 23 1777 ## ## 90 421 77 68 ## 1163 ##

T1 ## 283 945 579 127 11 371 ## 1578 94 ## 42 241 127 466 ## 133 ## 223 342 ## 619 227 32 165 160 513 469 541 160 ## ## ## 243 503 57 ## 4191 73 ## 228 29 2 7 1 4

T2 196 ## 712 809 536 ## 51 ## 163 425 894 7 52 1 4

T3 65 14 ## ##

U 75 176 274 6 24 2 11 20 22 3 ## 6101 ## ## 126 29 635 ## ## 8 2 36 42 33 1164 ## 319 44 5 1151 415 ## 52 2 ## 434 7 238 3 188 237

V ## 5710 ## ## ## ## ## 3157 345 ## ## ## ## 500 245 609 511 63 1677 420

W 8011 ## ## ## ## ## ## 161 21 181 64 689 749 141 239 450 291 98 178

X ## ## 114 ## 964 1116 58 ## ## ## 191 100

Y ## 815 8

Z ##

### 2 ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ###

1 1 ## ## ## ## ## ## ## ## ## ## 237K

Activity

0

Feature

Dev.

Platform

Server

Client

280000

Y 0 Year 1 Year 2 Year 3 Year 4 Year 5 Year 6

Derived Dependencies from Change History

number of distinct dependencies per file


Dependencies at different levels

java 4E +06 34528 2E +05 76079 258 8067 14129 3350 826 2110 1248

jrxml 34528 3E +05 12001 3188 76 54 5 54 282

gif 2E +05 12001 2E +05 8142 2082 196 2984 98 237 1607 62 223

xml 76079 3188 8142 9448 240 1404 703 4 159 142 199 102

jpg 258 76 2082 240 41344 19404 590 66 3168 887 6

sql 8067 1404 40884

htm 196 19404 9506 2 1568 392

html 14129 54 2984 703 590 2 3418 44 160 220 6 53

cs 66 5860 250 353

js 98 4 3168 1568 44 240 14 74 2

png 3350 5 237 159 160 14 492 32

css 826 54 1607 142 887 392 220 74 32 118 5

xsd 2110 62 199 6 250 328 18

txt 1248 282 223 102 6 53 353 2 5 18 88

Dependency graph at feature levelshows high but mostly weak coupling

Java files are also frequently related with non-code files


Dependency distribution shows low strength of the dependencies

x-axis: detected multiple times; y-axis: number of dependencies

~160.000 out of 1.85 million dependencies were detected twice

~1.000 dependencies were detected 10-times


Summary and Conclusions

We detected dependencies that could not easily be derived from the static structure of the software system

We found a high number of dependencies with a low strength (confidence)

With the ongoing development the number of dependencies increased steadily

We consider the detected dependencies as “raw data” for the computation and visualization of logical coupling measures

Future Work: Further post-processing and filtering to adjust the dependency data to the intended application scenarios

iwsm2014 extracting dependencies from software changes (thomas wetzlmaier - claus klammer - rudolf...

Software

software analytics

features development

initiative of scch

features client

features server

features platform

dynamic dependencies

logical dependencies