engineering software analytics studiessofteng.polito.it/eseiw2014/esem/softaneng-2upspinellis.pdf1...
TRANSCRIPT
![Page 1: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/1.jpg)
1
18/9/2014 1
Engineering
Software Analytics Studies
Diomidis Spinellis
Department of Management Science and Technology
Athens University of Economics and Business
http://www.spinellis.gr/
@CoolSWEng
![Page 2: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/2.jpg)
2
![Page 3: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/3.jpg)
3
![Page 4: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/4.jpg)
4
curiosity
![Page 5: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/5.jpg)
5
serendipity
![Page 6: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/6.jpg)
6
Outline
• Data gathering
• Processing
– Methods
– Advice
• Performance
Data gathering
![Page 7: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/7.jpg)
7
Alliances
![Page 8: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/8.jpg)
8
![Page 9: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/9.jpg)
9
Google hacks
![Page 10: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/10.jpg)
10
![Page 11: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/11.jpg)
11
Instrumentation
![Page 12: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/12.jpg)
12
![Page 13: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/13.jpg)
13
4266 utimensat(AT_FDCWD,
"boost_1_44_0/libs/bind/test/bind_stdcall_mf_test.cpp", {{1336157319,
67677176}, {1090801932, 0}}, 0) = 0
4266 chown("boost_1_44_0/libs/bind/test/bind_stdcall_mf_test.cpp", 0, 0)
= 0
4266 chmod("boost_1_44_0/libs/bind/test/bind_stdcall_mf_test.cpp", 0644)
= 0
4266 open("boost_1_44_0/libs/bind/test/bind_stdcall_test.cpp",
O_WRONLY|O_CREAT|O_EXCL, 0600) = 4
4266 write(4, ""..., 2789) = 2789
4266 close(4) = 0
4266 utimensat(AT_FDCWD,
"boost_1_44_0/libs/bind/test/bind_stdcall_test.cpp", {{1336157319,
67677176}, {1090801932, 0}}, 0) = 0
4266 chown("boost_1_44_0/libs/bind/test/bind_stdcall_test.cpp", 0, 0) =
0
4266 chmod("boost_1_44_0/libs/bind/test/bind_stdcall_test.cpp", 0644) =
0
4266 open("boost_1_44_0/libs/bind/test/bind_test.cpp",
O_WRONLY|O_CREAT|O_EXCL, 0600) = 4
4266 read(3, <unfinished ...>
![Page 14: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/14.jpg)
14
Recursion
![Page 15: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/15.jpg)
15
![Page 16: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/16.jpg)
16
![Page 17: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/17.jpg)
17
Synthesis
cvs2svn
![Page 18: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/18.jpg)
18
git fast-import
![Page 19: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/19.jpg)
19
$ git log --reverse --date-order ``` commit 94a21182365ebb258eeee2aa41c5fbcb1f7fd566 Author: Ken Thompson and Dennis Ritchie <research!{ken,dmr}> Date: Tue Jun 20 04:00:00 1972 -0500 Research V1 development Work on file u5.s commit b7b2640b9e27415d453a8fbe975a87902a01849d […] Author: Ken Thompson <research!ken> Date: Tue Nov 26 18:13:21 1974 -0500 Research V5 development Work on file usr/sys/ken/slp.c commit 3d19667a65d35a411d911282ed8b87e32a56a349 […] Author: Dennis Ritchie <research!dmr> Date: Mon Dec 2 18:18:02 1974 -0500 Research V5 development Work on file usr/sys/dmr/kl.c […] commit 171931a3f6f28ce4d196c20fdec6a4413a843f89 Author: Brian W. Kernighan <research!bwk> Date: Tue May 13 19:43:47 1975 -0500 Research V6 development Work on file rat/r.g […] commit ac4b13bca433a44a97689af10247970118834696 Author: S. R. Bourne <research!srb> Date: Fri Jan 12 02:17:45 1979 -0500 Research V7 development Work on file usr/src/cmd/sh/blok.c […] Author: Eric Schmidt <[email protected]> Date: Sat Jan 5 22:49:18 1980 -0800 BSD 3 development Work on file usr/src/cmd/net/sub.c
$ git blame -C -C usr/sys/sys/pipe.c 3cc1108b usr/sys/ken/pipe.c (Ken Thompson 1974-11-26 18:13:21 -0500 53) rf->f_flag = FREAD|FPIPE; 3cc1108b usr/sys/ken/pipe.c (Ken Thompson 1974-11-26 18:13:21 -0500 54) rf->f_inode = ip; 3cc1108b usr/sys/ken/pipe.c (Ken Thompson 1974-11-26 18:13:21 -0500 55) ip->i_count = 2; [...] 1f183be2 usr/sys/sys/pipe.c (Ken Thompson 1979-01-10 15:19:35 -0500 122) register struct inode *ip; 1f183be2 usr/sys/sys/pipe.c (Ken Thompson 1979-01-10 15:19:35 -0500 123) 1f183be2 usr/sys/sys/pipe.c (Ken Thompson 1979-01-10 15:19:35 -0500 124) ip = fp->f_inode; 1f183be2 usr/sys/sys/pipe.c (Ken Thompson 1979-01-10 15:19:35 -0500 125) c = u.u_count; 1f183be2 usr/sys/sys/pipe.c (Ken Thompson 1979-01-10 15:19:35 -0500 126) 1f183be2 usr/sys/sys/pipe.c (Ken Thompson 1979-01-10 15:19:35 -0500 127) loop: 1f183be2 usr/sys/sys/pipe.c (Ken Thompson 1979-01-10 15:19:35 -0500 128) 1f183be2 usr/sys/sys/pipe.c (Ken Thompson 1979-01-10 15:19:35 -0500 129) /* 9a9f6b22 usr/src/sys/sys/pipe.c (Bill Joy 1980-01-05 05:51:18 -0800 130) * If error or all done, return. 9a9f6b22 usr/src/sys/sys/pipe.c (Bill Joy 1980-01-05 05:51:18 -0800 131) */ 9a9f6b22 usr/src/sys/sys/pipe.c (Bill Joy 1980-01-05 05:51:18 -0800 132) 9a9f6b22 usr/src/sys/sys/pipe.c (Bill Joy 1980-01-05 05:51:18 -0800 133) if (u.u_error) 9a9f6b22 usr/src/sys/sys/pipe.c (Bill Joy 1980-01-05 05:51:18 -0800 134) return; 6d632e85 usr/sys/ken/pipe.c (Ken Thompson 1975-07-17 10:33:37 -0500 135) plock(ip); 6d632e85 usr/sys/ken/pipe.c (Ken Thompson 1975-07-17 10:33:37 -0500 136) if(c == 0) { 6d632e85 usr/sys/ken/pipe.c (Ken Thompson 1975-07-17 10:33:37 -0500 137) prele(ip); 6d632e85 usr/sys/ken/pipe.c (Ken Thompson 1975-07-17 10:33:37 -0500 138) u.u_count = 0; 6d632e85 usr/sys/ken/pipe.c (Ken Thompson 1975-07-17 10:33:37 -0500 139) return; 6d632e85 usr/sys/ken/pipe.c (Ken Thompson 1975-07-17 10:33:37 -0500 140) } ```
![Page 20: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/20.jpg)
20
Probing
nmap
![Page 21: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/21.jpg)
21
Processing
methods
![Page 22: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/22.jpg)
22
KISS
Count word occurrences
• 20GB in 36,260 files
• Hadoop: 12:25:20
![Page 23: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/23.jpg)
23
sgsh
Count word occurrences
• 20GB in 36,260 files
• Hadoop: 12:25:20
• sgsh: 00:34:38
![Page 24: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/24.jpg)
24
find Gutenberg -type f |
xargs cat |
tr -s ' \t\n\r\f' \\n |
sort -S 6G |
uniq -c
Count word occurrences
• 20GB in 36,260 files
• Hadoop: 12:25:20
• sgsh: 00:34:38
• Unix pipeline: 00:21:37
![Page 25: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/25.jpg)
25
Processing
pipelines
Fetching
Get
![Page 26: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/26.jpg)
26
Selecting
Select
Processing
Process
![Page 27: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/27.jpg)
27
Summarizing
Summarize
Fetching
Get
![Page 28: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/28.jpg)
28
curl
wget
git log
git blame
![Page 29: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/29.jpg)
29
$ # Print author names
git log --format='%an' |
# Order them by author
sort |
# Count number of commits for each author
uniq --count |
# Order them by number of commits
sort --numeric-sort --reverse |
# Print top ten
head -10
20131 Linus Torvalds
8445 David S. Miller
7692 Andrew Morton
5156 Greg Kroah-Hartman
5116 Mark Brown
4723 Russell King
4584 Takashi Iwai
4385 Al Viro
4220 Ingo Molnar
3276 Tejun Heo
![Page 30: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/30.jpg)
30
0
20
40
60
80
100
0 10 20 30 40 50 60 70 80 90
% o
f no
n-ad
heri
ng li
nes
Number of developers in the annotated file
![Page 31: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/31.jpg)
31
Selecting
Select
![Page 32: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/32.jpg)
32
grep
cut
awk
sed
Processing
Process
![Page 33: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/33.jpg)
33
sort
-n
-r
+2nr
-u
-t:
![Page 34: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/34.jpg)
34
diff
$ diff file1 file2 |
grep ’^[<>]’ |
wc -l
![Page 35: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/35.jpg)
35
comm
$ comm Linux FreeBSD
[
arch
ash
bash
cat
chflags
chgrp
chio
chmod
chown
cp
cpio
csh
date
dd
df
dir
dmesg
dnsdomainname
domainname
echo
ed
egrep
expr
false
getfacl
![Page 36: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/36.jpg)
36
join
gvpr
![Page 37: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/37.jpg)
37
digraph D {
"archivers/arj" -> "devel/gmake"
"archivers/arj" -> "devel/autoconf259"
"archivers/dact" -> "devel/gmake"
"archivers/dact" -> "security/libmcrypt"
"archivers/dact" -> "archivers/lzo"
"archivers/deepforest" -> "x11-toolkits/tkstep80"
"archivers/deepforest" -> "graphics/libimg"
"archivers/deepforest" -> "x11/xorg-libraries"
"archivers/fastjar" -> "devel/gmake"
"archivers/fastjar" -> "lang/perl5.8"
"archivers/gtar" -> "converters/libiconv"
"archivers/gtar" -> "devel/gettext"
BEG_G {
$tvtype = TV_fwd;
$tvroot = node($, ARGV[0]);
}
N [$tvroot = NULL; 1]
END_G {
induce($T);
write($T);
exit(0);
}
![Page 38: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/38.jpg)
38
![Page 39: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/39.jpg)
39
Summarizing
Summarize
![Page 40: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/40.jpg)
40
wc
uniq
head
tail
![Page 41: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/41.jpg)
41
fmt
awk
![Page 42: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/42.jpg)
42
awk '{print $2}' licenses |
sort |
uniq -c |
sort -rn
![Page 43: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/43.jpg)
43
![Page 44: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/44.jpg)
44
0
2
4
6
8
10
12
14
16
18 01
/06/
1974
01/0
6/19
75
01/0
6/19
76
01/0
6/19
77
01/0
6/19
78
01/0
6/19
79
01/0
6/19
80
01/0
6/19
81
01/0
6/19
82
01/0
6/19
83
01/0
6/19
84
01/0
6/19
85
01/0
6/19
86
01/0
6/19
87
01/0
6/19
88
01/0
6/19
89
01/0
6/19
90
01/0
6/19
91
01/0
6/19
92
01/0
6/19
93
01/0
6/19
94
01/0
6/19
95
01/0
6/19
96
01/0
6/19
97
01/0
6/19
98
01/0
6/19
99
01/0
6/20
00
01/0
6/20
01
01/0
6/20
02
01/0
6/20
03
01/0
6/20
04
01/0
6/20
05
01/0
6/20
06
01/0
6/20
07
01/0
6/20
08
01/0
6/20
09
01/0
6/20
10
01/0
6/20
11
01/0
6/20
12
FNAMELEN
IDLEN
0
0.005
0.01
0.015
0.02
0.025
0.03
0.035
0.04
0.045
0.05
01/0
6/19
74
01/0
6/19
75
01/0
6/19
76
01/0
6/19
77
01/0
6/19
78
01/0
6/19
79
01/0
6/19
80
01/0
6/19
81
01/0
6/19
82
01/0
6/19
83
01/0
6/19
84
01/0
6/19
85
01/0
6/19
86
01/0
6/19
87
01/0
6/19
88
01/0
6/19
89
01/0
6/19
90
01/0
6/19
91
01/0
6/19
92
01/0
6/19
93
01/0
6/19
94
01/0
6/19
95
01/0
6/19
96
01/0
6/19
97
01/0
6/19
98
01/0
6/19
99
01/0
6/20
00
01/0
6/20
01
01/0
6/20
02
01/0
6/20
03
01/0
6/20
04
01/0
6/20
05
01/0
6/20
06
01/0
6/20
07
01/0
6/20
08
01/0
6/20
09
01/0
6/20
10
01/0
6/20
11
01/0
6/20
12
goto / line
register / line
![Page 45: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/45.jpg)
45
Scripting languages
github.com/dspinellis/unix-history-repo
• Research Edition Unix: V1, V3–V7 • Unix 32V • BSD 1, 2, 3, 4, 4.1, 4.2, 4.3 *, 4.4 * • 386BSD 0.0, 0.1 • FreeBSD 1.0–10.0 • Tags • Contributors • Branches and merges
![Page 46: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/46.jpg)
46
Tool front ends
find /usr/ports/ -name Makefile
-maxdepth 3 |
sed 's,/Makefile,,' |
while read dir
do
cd $dir
make -V PORTNAME\
-V RUN_DEPENDS -V BUILD_DEPENDS\
-V LIB_DEPENDS -V FETCH_DEPENDS\
-V DEPENDS
done
![Page 47: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/47.jpg)
47
C Preprocessor
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
01/0
6/19
74
01/0
5/19
75
01/0
4/19
76
01/0
3/19
77
01/0
2/19
78
01/0
1/19
79
01/1
2/19
79
01/1
1/19
80
01/1
0/19
81
01/0
9/19
82
01/0
8/19
83
01/0
7/19
84
01/0
6/19
85
01/0
5/19
86
01/0
4/19
87
01/0
3/19
88
01/0
2/19
89
01/0
1/19
90
01/1
2/19
90
01/1
1/19
91
01/1
0/19
92
01/0
9/19
93
01/0
8/19
94
01/0
7/19
95
01/0
6/19
96
01/0
5/19
97
01/0
4/19
98
01/0
3/19
99
01/0
2/20
00
01/0
1/20
01
01/1
2/20
01
01/1
1/20
02
01/1
0/20
03
01/0
9/20
04
01/0
8/20
05
01/0
7/20
06
01/0
6/20
07
01/0
5/20
08
01/0
4/20
09
01/0
3/20
10
01/0
2/20
11
01/0
1/20
12
01/1
2/20
12
comment char %
![Page 48: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/48.jpg)
48
Analyze Binaries
nm,readelf
ar,ldd
dumpbin
javap
![Page 49: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/49.jpg)
49
$ cd /usr/bin
$ ldd * 2>/dev/null |
awk '/=>/{print $3}' |
sort |
uniq -c |
sort -rn |
head
541 /lib/libc.so.6
541 /lib/ld-linux.so.2
104 /lib/libdl.so.2
92 /lib/libm.so.6
62 /lib/libncurses.so.5
59 /lib/libnsl.so.1
50 /lib/libcrypt.so.1
34 /usr/lib/libstdc++-
libc6.2-2.so.3
34 /lib/libpam.so.0
28 /usr/lib/libz.so.1
![Page 50: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/50.jpg)
50
![Page 51: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/51.jpg)
51
Tool Interoperability
![Page 52: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/52.jpg)
52
Combine
data sources
![Page 53: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/53.jpg)
53
http://geonames.usgs.gov/
http://earth-info.nga.mil/
![Page 54: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/54.jpg)
54
![Page 55: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/55.jpg)
55
Processing
advice
![Page 56: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/56.jpg)
56
AITMOAF
Automate
![Page 57: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/57.jpg)
57
make
set –e
a && b && c
Monitor
![Page 58: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/58.jpg)
58
top/htop
logger
pv
monitor (www.spinellis.gr/blog/20081027/)
Take notes
![Page 59: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/59.jpg)
59
Checkpoint
Handle
broken
connections
![Page 60: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/60.jpg)
60
nohup
mosh
Performance
![Page 61: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/61.jpg)
61
Optimize
Parallelize
![Page 62: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/62.jpg)
62
parallel
![Page 63: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/63.jpg)
63
find . -name \*.c -type f -print0 |
xargs -0 -n 1 \
analyze-file.sh
find . -name \*.c -type f -print0 |
parallel --keep-order -d '\0' -n 1 –gnu \
analyze-file.sh
![Page 64: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/64.jpg)
64
Low-level processing
![Page 65: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/65.jpg)
65
class RangeMap {
private:
vector<bool> active;
public:
static const int NMONTH = (2009 - 2001) * 12;
RangeMap() : active(NMONTH, false) {}
};
class EntryDetails {
private:
RangeMap defined, stub;
vector <int> nrefs;
string definer, referrer, firstRef;
time_t firstDef;
int numReferences, numContributors;
int numRevisions, numReverts;
public:
EntryDetails() : defined(), stub(),
nrefs(RangeMap::NMONTH, 0), firstDef(-1),
numReferences(0), numContributors(0), numRevisions(0),
numReverts(0) {}
};
![Page 66: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/66.jpg)
66
class RangeMap {
private BitSet active;
public static final int NMONTH = (2009 - 2001) * 12;
public RangeMap() {
active = new BitSet(NMONTH);
}
}
class EntryDetails {
RangeMap defined, stub;
ArrayList <Integer> nrefs;
String definer, referrer, firstRef;
Date firstDef;
int numReferences, numContributors;
int numRevisions, numReverts;
public EntryDetails() {
defined = new RangeMap();
stub = new RangeMap();
nrefs = new ArrayList<Integer>(RangeMap.NMONTH);
}
}
MB
7,212
12,287
C++ Java
![Page 67: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/67.jpg)
67
![Page 68: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/68.jpg)
68
Experiment
on a subset
Sample
![Page 69: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/69.jpg)
69
![Page 70: Engineering Software Analytics Studiessofteng.polito.it/ESEIW2014/ESEM/SoftAnEng-2upSpinellis.pdf1 18/9/2014 1 Engineering Software Analytics Studies Diomidis Spinellis Department](https://reader036.vdocuments.us/reader036/viewer/2022081513/5e42c46c3f225e61df65ddce/html5/thumbnails/70.jpg)
70
Thank you!
http://www.spinellis.gr
@CoolSWEng
Image Credits
• Aero Icarus • Chrysler Group • Jose Maria Minarro Vivancos • Kirti Poddar • Lab Science Career • Lachlan Donald • Luigi Chiesa • Marco40134 • Mark Sze • Meats Meyer • NASA • NTSB • PSParrot
• Q-Wing • Rachel Kramer • Sam Rayner • Tim Bouwer • Tudor Barker • Wikimedia Commons • William Avery, W. Heath Robinson • kaktuslampan • kenmainr • max guitare • patrasTimes.gr • vlc team, ubuntu, Hidro • zzpza