towards the characterization of realistic models: evaluation of multidisciplinary graph metrics
TRANSCRIPT
Budapest University of Technology and EconomicsDepartment of Measurement and Information Systems
MTA-BME Lendület Research Group on Cyber-Physical SystemsBudapest University of Technology and Economics
Fault Tolerant Systems Research Group
Towards the Characterization of Realistic Models: Evaluation of Multidisciplinary Graph Metrics
Gábor Szárnyas, Zsolt Kővári, Ágnes Salánki, Dániel Varró
Motivation
Research Community
Problems of experimental evaluation of MDE papers
Difficult to find real industrial model
Tool Providers
Test generation for modeling tools
Scalability evaluationand stress testing of MDE tools
Smart CPS
Synthesis of prototypical test context/environment
Testing of autonomous robots(R3COP project)
Motivation
Research Community
Problems of experimental evaluation of MDE papers
Difficult to find real industrial model
Tool Providers
Test generation for modeling tools
Scalability evaluationand stress testing of MDE tools
Smart CPS
Synthesis of prototypical test context/environment
Testing of autonomous robots(R3COP project)
How to automatically synthesize graph models…?
Research Question and Objectives
• All well-formedness constraints satisfied
• Designated seed fragments included Consistent
• How to characterize realistic models?
• How to distinguish real and generated models?Realistic
• Guaranteed test coverage
• Required for tool qualificationDiverse
• Performance benchmarks
• Stress testing of tools and control algorithmsScalable
How to automatically synthesize graph models which are...
Research Question and Objectives
• All well-formedness constraints satisfied
• Designated seed fragments included Consistent
• How to characterize realistic models?
• How to distinguish real and generated models?Realistic
• Guaranteed test coverage
• Required for tool qualificationDiverse
• Performance benchmarks
• Stress testing of tools and control algorithmsScalable
How to automatically synthesize graph models which are...
Performance Experiments
Performance Experiments
„I would like to benchmark my tool on real models”
Performance Experiments
„I would like to benchmark my tool on real models”
o Industrial models are difficult to obtain.
Performance Experiments
„I would like to benchmark my tool on real models”
o Industrial models are difficult to obtain.
Workaround #1: „Never mind, my tool has very good performance for the TTC 2038 case.”
Performance Experiments
„I would like to benchmark my tool on real models”
o Industrial models are difficult to obtain.
Workaround #1: „Never mind, my tool has very good performance for the TTC 2038 case.”
o Great, but what does that imply for real use cases?
Performance Experiments
„I would like to benchmark my tool on real models”
o Industrial models are difficult to obtain.
Workaround #1: „Never mind, my tool has very good performance for the TTC 2038 case.”
o Great, but what does that imply for real use cases?
Workaround #2: Implement a custom benchmark
Performance Experiments
„I would like to benchmark my tool on real models”
o Industrial models are difficult to obtain.
Workaround #1: „Never mind, my tool has very good performance for the TTC 2038 case.”
o Great, but what does that imply for real use cases?
Workaround #2: Implement a custom benchmark
o Again, what does that imply for real use cases?
Performance Experiments
„I would like to benchmark my tool on real models”
o Industrial models are difficult to obtain.
Workaround #1: „Never mind, my tool has very good performance for the TTC 2038 case.”
o Great, but what does that imply for real use cases?
Workaround #2: Implement a custom benchmark
o Again, what does that imply for real use cases?
Qualitative description of models is required
How to Obtain Models for Benchmarking?
• Difficult to obtain
• Obfuscated modelsIndustrial
• Quality of models?Student work
• Good quality models
• Small in sizeTutorial
• How realistic are these models?Generated
What Makes a Model Realistic?
How to decide if a model is realistic
without domain-specific knowledge?
Statecharts with Attributes
RedRed &
Orange
GreenOrange
RedRed &
Orange
GreenOrange
Statecharts with Attributes
RedRed &
Orange
GreenOrange
RedRed &
Orange
GreenOrange
Statecharts
S1 S2
S3S4
S1 S2
S3S4
Statecharts
S1 S2
S3S4
S1 S2
S3S4
Typed Graphs of the Models
S1 S2 S3 S4
T1 T2 T3 T4 T5
E S4 S1 S2 S3
T1 T2 T3 T4 T5
E
Typed Graphs of the Models
S1 S2 S3 S4
T1 T2 T3 T4 T5
E S4 S1 S2 S3
T1 T2 T3 T4 T5
E
Which is the graphof a real model?
Graph Metrics
Use graph metrics for characterizingthe graph of the model.
Graph Metrics
Graph Metrics
Number of vertices
Graph Metrics
05
10152025
Number of vertices
Graph Metrics
05
10152025
Number of vertices
Number of edges
Graph Metrics
05
10152025
Number of vertices
0
10
20
30
40
Number of edges
Graph Metrics
05
10152025
Number of vertices
0
10
20
30
40
Number of edges
Average shortestpath
Graph Metrics
05
10152025
Number of vertices
0
10
20
30
40
Number of edges
Average shortestpath
0123456
Graph Metrics
05
10152025
Number of vertices
0
10
20
30
40
Number of edges
Average shortestpath
0123456
Graph Metrics
05
10152025
Number of vertices
0
10
20
30
40
Number of edges
Average shortestpath
0123456
Clusteredness
One-Dimensional Graph Metrics
05
10152025
Number of vertices
0
10
20
30
40
Number of edges
Average shortestpath
0123456
Clusteredness
00.20.40.60.8
1
Graph Metrics
05
10152025
Number of vertices
0
10
20
30
40
Number of edges
Average shortestpath
0123456
Clusteredness
00.20.40.60.8
1
Centrality
Graph Metrics
05
10152025
Number of vertices
0
10
20
30
40
Number of edges
Average shortestpath
0123456
Clusteredness
00.20.40.60.8
1
Centrality
00.20.40.60.8
1
Graph Metrics
S1 S2 S3 S4
T1 T2 T3 T4 T5
E S4 S1 S2 S3
T1 T2 T3 T4 T5
E
Which is the graphof a real model?
Graph Metrics
S1 S2 S3 S4
T1 T2 T3 T4 T5
E S4 S1 S2 S3
T1 T2 T3 T4 T5
E
Which is the graphof a real model?
Graph Metrics
S1 S2 S3 S4
T1 T2 T3 T4 T5
E S4 S1 S2 S3
T1 T2 T3 T4 T5
E
They are isomorphic.
Which is the graphof a real model?
Graph Metrics
S1 S2 S3 S4
T1 T2 T3 T4 T5
E S4 S1 S2 S3
T1 T2 T3 T4 T5
E
They are isomorphic.
Which is the graphof a real model?
Related finding: simplegraph metrics are unable topredict query performance
Network Theory
Mid ‘90s, László Albert-Barabási et al.
o Preferential attachment: „the rich gets richer”
Scale-free networks (web, power grid, etc.)
Most approaches only consider untyped graphs.
Network Theory
Mid ‘90s, László Albert-Barabási et al.
o Preferential attachment: „the rich gets richer”
Scale-free networks (web, power grid, etc.)
Most approaches only consider untyped graphs.
S1
S2
S3
S4
T1
T2
T3
T4
T5
E
Network Theory
Mid ‘90s, László Albert-Barabási et al.
o Preferential attachment: „the rich gets richer”
Scale-free networks (web, power grid, etc.)
Most approaches only consider untyped graphs.
S1
S2
S3
S4
T1
T2
T3
T4
T5
E S4 S1 S2 S3
T1 T2 T3 T4 T5
E
„Evaluation of Multidisciplinary Graph Metrics”
Typed graph (computer science)
Multi-layered networks (social network analysis)
Multidimensional networks (network theory)
Multiplex networks (physics)
Source: Wikipedia, Multidimensionalnetwork
Multidimensional Metrics
Dimensional degree distributions
Node dimension connectivity
o ratio of nodes in the that belong to a dimension
Multiplex participation coefficient
o the connections of v are uniformly distributed among D
Node activity & pairwise multiplexity
o the ratio of nodes, which are active in both d1 and d2
Methodology
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
Methodology
1. Collect models
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
Methodology
1. Collect models
2. Data Cleansing: removeRed
Red-Orange
Green Orange
T1 T2 T3 T4 T5
Entry
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information
o attributes
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information
o attributes
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information
o attributes
o object types
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information
o attributes
o object types
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information
o attributes
o object types
o small models
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information
o attributes
o object types
o small models
o derived references
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information
o attributes
o object types
o small models
o derived references
3. Calculate graph metrics
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
Methodology
1. Collect models
2. Data Cleansing: remove
o layout information
o attributes
o object types
o small models
o derived references
3. Calculate graph metrics
4. Analyze results
o Statistical + exploratory
RedRed-
OrangeGreen Orange
T1 T2 T3 T4 T5
Entry
Domains
AutoFOCUS
Building Information Model
Capella
JaMoPP
Train Benchmark
Yakindu
real
real
tutorial
synthetic
tutorial
tutorial
Domain 1
Statistical Analysis
Domain 1
Statistical Analysis
0
1
0 2 4 6
0
1
0 2 4 6
Domain 1
Domain 2
Statistical Analysis
0
1
0 2 4 6
0
1
0 2 4 6
0
1
0 2 4 6
0
1
0 2 4 6
Domain 1
Domain 2
Statistical Analysis
0
1
0 2 4 6
0
1
0 2 4 6
0
1
0 2 4 6
0
1
0 2 4 6
0
1
0 2 4 6
Statistical Analysis
0
1
0 2 4 6
Homogeneity
Statistical Analysis
0
1
0 2 4 6
0
1
0 1 2 3 40
1
0 2 4 6
Homogeneity
Statistical Analysis
0
1
0 2 4 6
0
1
0 1 2 3 40
1
0 2 4 6
Homogeneity
Statistical Analysis
0
1
0 2 4 6
0
1
0 1 2 3 40
1
0 2 4 6
Kolmogorov-Smirnov distance
Homogeneity
Statistical Analysis
0
1
0 2 4 6
0
1
0 1 2 3 40
1
0 2 4 6
Kolmogorov-Smirnov distance
Homogeneity
Distinctiveness
Statistical Analysis
0
1
0 2 4 6
0
1
0 1 2 3 40
1
0 2 4 6
0
1
0 2 4 6
Kolmogorov-Smirnov distance
Homogeneity
Distinctiveness
Statistical Analysis
0
1
0 2 4 6
0
1
0 1 2 3 40
1
0 2 4 6
0
1
0 2 4 6
Kolmogorov-Smirnov distance
Dimensional Clustering Coefficients
Dimensional Clustering Coefficients
KS distance
Findings
1. Metamodel-level information is insufficient
Findings
1. Metamodel-level information is insufficient
1. The ratio of containment edge types in the Capella metamodels: 75%
2. The ratio of containment edges in the Capella models: 42–50 %
Findings
1. Metamodel-level information is insufficient
2. Containment edges dominate distributions
Findings
1. Metamodel-level information is insufficient
2. Containment edges dominate distributions
3. Many edges follow the locality principle
Future Directions
Use metrics for
o Instance model generators
o Query optimization
Improve performance of calculating metrics:incremental calculation
o https://github.com/ftsrg/model-analyzer
oWorks for both EMF and RDF models
All analysis results & code are available online:
o http://docs.inf.mit.bme.hu/model-metrics/
The Train Benchmark
SOSYM paper – The Train Benchmark: Cross-Technology Performance Evaluation of Continuous Model Validation
o 6 queries, 12 transformations
o EMF, property graphs, RDF, SQL
o 12+ tools
o Automated visualization & reporting
http://github.com/ftsrg/trainbenchmark
Ω