presentation of the quantitative software engineering (quase) lab, university of alberta
DESCRIPTION
Honolulu, October 8, 2000. Presentation of the Quantitative Software Engineering (QuaSE) Lab, University of Alberta. Giancarlo Succi Department of Electrical and Computer Engineering University of Alberta. ISERN 2000 Meeting. The group. James Miller Petr Musilek Marek Reformat - PowerPoint PPT PresentationTRANSCRIPT
Presentation of thePresentation of theQuantitative Software Quantitative Software
Engineering (QuaSE) Lab, Engineering (QuaSE) Lab, University of AlbertaUniversity of Alberta
Giancarlo SucciDepartment of Electrical and Computer Engineering
University of Alberta
Honolulu, October 8, 2000Honolulu, October 8, 2000
ISERN 2000 MeetingISERN 2000 Meeting
October 8, 2000 G. Succi – QuaSE Lab – UoA 2
The group James Miller Petr Musilek Marek Reformat Witold Pedrycz Giancarlo Succi 2 Visiting profs 1 PDF 12 Graduate students
October 8, 2000 G. Succi – QuaSE Lab – UoA 3
Research interests
Software metrics (definitions and tools) Advanced (Statistical and CI) models Sensitivity of cost models Certification of components and analysis of product lines Inspections Analysis of the nature of flexible methodologies …
““Application of quantitative Application of quantitative methods to software methods to software
engineering”engineering”
October 8, 2000 G. Succi – QuaSE Lab – UoA 4
Sponsors The University of Alberta The Alberta government ASRA NSERC Nortel CFI WaveRider Valmet
October 8, 2000 G. Succi – QuaSE Lab – UoA 5
Brief overview of 2 projects Analysis of the ability of predicting
defects using OO metrics Study of the occurrences of software
service requests
October 8, 2000 G. Succi – QuaSE Lab – UoA 6
Analysis of the ability of predicting defects using OO metrics Investigate and quantifyquantify the impact of the
object-oriented design on the defect-proneness of classes
Empirically validatevalidate the ability of the object-oriented design metrics to identify classes with high number defects in commercial software applications
Build and evaluateevaluate explanatory statistical models applicable for the count data
October 8, 2000 G. Succi – QuaSE Lab – UoA 7
The adopted measures Object-oriented design metrics
(Chidamber and Kemerer, 1991)• Coupling Between Objects
(CBO)• Weighted Methods per
Class (WMC/NOM)• Lack of Cohesion in
Methods (LCOM) Our tool: WebMetricsWebMetrics - metrics collection system Dependent variable – number of defects for a class
• Depth of Inheritance Tree (DIT)
• Number Of Children (NOC)
• Response For a Class (RFC)
October 8, 2000 G. Succi – QuaSE Lab – UoA 8
Issues in the Statistical Analysis
Distribution of the dependent variable?
• Count data
Defects
120100806040200
80
70
60
50
40
30
20
10
0
Poisson Regression• EquidispersionEquidispersion
Negative Binomial Regression• Gamma-distributed
mean• UnderpredictionUnderprediction of zero values
Zero-inflated Negative Binomial regression• Two processes – different distributions
October 8, 2000 G. Succi – QuaSE Lab – UoA 9
Building the models Ordinal Least Squares (OLS)
vs. Maximum Likelihood (ML) ML - general solution for fitting model parameters
• Consistency: the probability that the ML estimator differs from the true parameter by an arbitrary small amount tends toward zero as the sample size grows
• Asymptotic efficiency: The variance of the ML estimator is the smallest possible
Selection of predictors• Stepwise regression based on the statistical significance
Resulting models• Univariate: RFC• Bivariate: RFC and DIT
October 8, 2000 G. Succi – QuaSE Lab – UoA 10
Comparison of the models
Univariate r α PRM OLS 1 0.40 13.47 PRM ML 1 0.71 2.07
NBRM 1 0.42 0.51 ZINB 1 0.78 0.46
Bivariate r α PRM OLS 2 0.41 13.24 PRM ML 2 0.74 1.85
NBRM 2 0.46 0.42 ZINB 2 0.78 0.39
0
20
40
60
80
100
0 20 40 60 80 100
Model % PRM OLS 1 45.33 PRM ML 1 45.33 NBRM 1 45.33 ZINB 1 45.33
Model % PRM OLS 2 43.33 PRM ML 2 43.33 NBRM 2 44.00 ZINB 2 44.00
Criticality prediction - Alberg diagram
Goodness of fit and overdispersion
October 8, 2000 G. Succi – QuaSE Lab – UoA 11
Study of the occurrences of software service requests
Service Request demand for a modification of the software system
behavior (early) life-cycle process attribute measure
Our goal:Our goal: To define and validate a framework for pre-release SR analysis on three industrial datasets: Predict efforteffort, resourcesresources, and timetime to be allocated
for a project Predict the final number of SRsnumber of SRs for a project Provide a basis for comparisonbasis for comparison and assessmentand assessment of
different projects and development processes
October 8, 2000 G. Succi – QuaSE Lab – UoA 12
SRs and Reliability Reliability
failure-centric quality measure that views the software system as a whole from a customer perspective
Software Reliability Growth Models (SRGM) models describing failure detection over time using calendar time, the number of tests run, or
execution time The evaluated models:
GO S-ShapedGO S-Shaped Goel-OkumotoGoel-Okumoto GompertzGompertz Hossain-Dahiya/GOHossain-Dahiya/GO LogisticLogistic
WeibullWeibull Weibull S-shapedWeibull S-shaped Yamada ExponentialYamada Exponential Yamada RaleighYamada Raleigh
October 8, 2000 G. Succi – QuaSE Lab – UoA 13
CriteriaCompany B, product 1, L model
0
10
20
30
40
50
60
4/23
/92
5/7/
92
5/21
/92
6/4/
92
6/18
/92
7/2/
92
7/16
/92
7/30
/92
8/13
/92
8/27
/92
9/10
/92
9/24
/92
10/8
/92
10/2
2/92
11/5
/92
11/1
9/92
Calendar time
SRs
Company B, product 1, GO model
0
10
20
30
40
50
60
70
4/23
/92
5/7/
92
5/21
/92
6/4/
92
6/18
/92
7/2/
92
7/16
/92
7/30
/92
8/13
/92
8/27
/92
9/10
/92
9/24
/92
10/8
/92
10/2
2/92
11/5
/92
11/1
9/92
Calendar time
SRs
Goodness of fit Accuracy of the final
point Relative precision of fit Coverage of fit Predictive ability
Company B, product 1, HD model
0
10
20
30
40
50
60
4/23
/92
5/7/
92
5/21
/92
6/4/
92
6/18
/92
7/2/
92
7/16
/92
7/30
/92
8/13
/92
8/27
/92
9/10
/92
9/24
/92
10/8
/92
10/2
2/92
11/5
/92
11/1
9/92
Calendar time
SRs
October 8, 2000 G. Succi – QuaSE Lab – UoA 14
Summary of Results
Square Error Accuracy Conf. Interval Percentile Predict. Ability Rank Pr. A Pr. B Pr. A Pr. B Pr. A Pr. B Pr. A Pr. B Pr. A Pr. B
1 W-S W-S W-S W-S HD GO-S YR GO W W 2 W W G GO-S G G W-S YE G L 3 YR G HD G L L G W-S HD HD 4 G HD W W GO-S HD W YR GO-S G 5 HD YR YR HD W W-S YE W W-S YE 6 L GO-S L L GO GO L L GO W-S 7 GO-S L GO-S YR W-S W GO HD YE GO-S 8 GO GO GO GO YE YE GO-S G YR YR 9 YE YE YE YE YR YR HD GO-S L GO
Severity 3, HD Model Predictions
0
50
100
150
200
250
300
4/16
/99
5/16
/99
6/16
/99
7/16
/99
8/16
/99
9/16
/99
10/1
6/99
11/1
6/99
12/1
6/99
Time
Erro
rs C
ompl
eted SRs
HD(8)HD(3)HD(2)HD(1)
95% Confidence Interval - GO-S Model
0
20
40
60
80
100
120
140
160
180
200
3/29
/99
4/29
/99
5/29
/99
6/29
/99
7/29
/99
8/29
/99
9/29
/99
10/2
9/99
11/2
9/99
12/2
9/99
SRs
GO-S (L)
GO-S (U)
October 8, 2000 G. Succi – QuaSE Lab – UoA 15
SRGM Sensitivity to Noise How sensitive are
the models to the human factor in the SRs data recording?
Monte Carlo analysis with normally distributed noise N(0,) added to the original data
Percentile of the model parameters within the 95% interval
0
20
40
60
80
100
0 2 4 6 8 10
log2(k)
Perc
entil
e
GO-SW-SHDGWL
October 8, 2000 G. Succi – QuaSE Lab – UoA 16
Response Time and Gamma Analysis
When compared with other models, linear regression has the best performance
S 4 S 3 S 1 S 2 P r o d u c t A
P r o d u c t B
S 4 S 3 S 2 S 1
T i m e S 4 S 2 S 3 S 1 P r o d u c t C
S 4 P r o d u c t D S 3 S 1 S 2
T i m e
Product A
0
50
100
150
200
250
300
350
400
Calendar time
Tim
e to
ser
ve
October 8, 2000 G. Succi – QuaSE Lab – UoA 17
Future research plans Extend the analysis of software models
with “more advanced” techniques Perform domain-specific studies, such as
in product lines and extreme programming
Analyse the evolution of the software market to determine driving forces of software development processes
…
October 8, 2000 G. Succi – QuaSE Lab – UoA 18
Netcraft survey: web server usage over the Internet
October 8, 2000 G. Succi – QuaSE Lab – UoA 19
Expectation from ISERN In- and out- flow of ideas Sharing of experimental data and
experimental protocols Exchange of visits Partnerships in projects (and funding
proposals when possible) …