automating the formalization of product comparison matrices
DESCRIPTION
Automating the Formalization of Product Comparison Matrices ASE 2014TRANSCRIPT
Automating the Formalization of
Product Comparison Matrices
Guillaume Bécan, Nicolas Sannier, Mathieu Acher,
Olivier Barais, Arnaud Blouin, Benoit Baudry
Product lines everywhere
Automating the Formalization of Product Comparison Matrices - 2
Product Comparison Matrices (PCMs)
Automating the Formalization of Product Comparison Matrices - 3
Services on top of PCMs
Automating the Formalization of Product Comparison Matrices - 4
Edit
Compare
Visualize
Filter
Rank
Merge
Configure
Multi-objective optimization
Problem
Automating the Formalization of Product Comparison Matrices - 5
Edit
Compare
Visualize
…
Information is:
• Uncontrolled
• Heterogeneous
• Ambiguous
[Sannier et al, ASE 2013]
| [[Acer Inc.|Acer]]
| [[Acer beTouch E110|beTouch E110]]
| {{dts|format=dmy|2010|2|15}}
| 1.5
| [[320x240|320x240 QVGA]]
| {{convert|2.8|in|mm|abbr=on}}
| Touch, accelerometer
|
* [[GSM]]/​GPRS/​[[Enhanced Data
Rates for GSM Evolution|EDGE]]
* [[Universal Mobile Telecommunications
System|UMTS]] 850 1900
* CSD
Problem
Automating the Formalization of Product Comparison Matrices - 6
Common
language Transformation
Edit
Compare
Visualize
…
• How to formalize data contained in natural language PCMs?
• How to automate the formalization of PCMs?
• What tools and services can be built on top of this formalization?
Contributions
Automating the Formalization of Product Comparison Matrices - 7
1. Design of a metamodel for product comparison matrices
2. Automated techniques for formalizing raw data into formalized
product comparison matrix model
3. Evaluation on 30,000+ cells from Wikipedia
Metamodeling driven by (lots of) data
Automating the Formalization of Product Comparison Matrices - 8
Designed for data (lots of examples + personal experience)
Designed for applications (edit, compare, visualize…)
Objectives:
• A metamodel that can contain every PCM of Wikipedia
• A metamodel for building services on top of these PCMs
Categorization of patterns (ASE 2013)
Refinement of the patterns
Realization of the metamodel (2 intensive weeks)
Formalizing some examples to adjust the metamodel
Driven by statistics and manual review of lots of PCMs New concept
Statistics
Brainstorming
Working on the metamodel since February 2013
300+ PCMs – 300,000 cells
Numerous domains
Manual review of 50 PCMs (thousands of cells)
Statistics on all PCMs
Analysis of Wikipedia syntax for tables
Automated transformation of all PCMs to PCM models
PCM metamodel
Automating the Formalization of Product Comparison Matrices - 9
PCM metamodel
Automating the Formalization of Product Comparison Matrices - 10
Structure of a PCM
PCM metamodel
Automating the Formalization of Product Comparison Matrices - 11
Feature/Product oriented
Automating the Formalization of Product Comparison Matrices - 12
Formalized interpretation of a cell
Data types: Boolean, Integer, Real
Special values: Unknown, Empty, Inconsistent, Partial
PCM metamodel
row string formalized integer
Contributions
Automating the Formalization of Product Comparison Matrices - 13
1. Design of a metamodel for product comparison matrices
2. Automated techniques for formalizing raw data into
formalized product comparison matrix model
3. Evaluation on 30,000+ cells from Wikipedia
Approach
Automating the Formalization of Product Comparison Matrices - 14
Parsing: transform a PCM artefact in a PCM model
PCM PCM
model
parsing preprocessing extracting
information exploiting
PCM
model
PCM
model
PCM metamodel S
E
R
V
I
C
E
S
| [[Acer Inc.|Acer]]
| [[Acer beTouch E110|beTouch E110]]
| {{dts|format=dmy|2010|2|15}}
| 1.5
| [[320x240|320x240 QVGA]]
| {{convert|2.8|in|mm|abbr=on}}
| Touch, accelerometer
|
* [[GSM]]/​GPRS/​[[Enhanced
Data Rates for GSM Evolution|EDGE]]
* [[Universal Mobile Telecommunications
System|UMTS]] 850 1900
* CSD
Enable the development of a
generic formalization process
Approach
Automating the Formalization of Product Comparison Matrices - 15
Preprocessing:
Contributors cannot be trusted: missing cells, headers everywhere
We have to normalize the matrix and identify headers
Default strategy: first line and first column are headers
PCM PCM
model
parsing preprocessing extracting
information exploiting
PCM
model
PCM
model
PCM metamodel S
E
R
V
I
C
E
S
Approach
Automating the Formalization of Product Comparison Matrices - 16
Extracting information:
• Identify features and products
• Interpret cells based on a set of syntactic rules (regex)
PCM PCM
model
parsing preprocessing extracting
information exploiting
PCM
model
PCM
model
PCM metamodel S
E
R
V
I
C
E
S
List of rules:
…
"\d+" => Integer
…
match Integer(100)
Same process as the metamodel for creating the rules
Contributions
Automating the Formalization of Product Comparison Matrices - 17
1. Design of a metamodel for product comparison matrices
2. Automated techniques for formalizing raw data into formalized
product comparison matrix model
3. Evaluation on 30,000+ cells from Wikipedia
Evaluation
Automating the Formalization of Product Comparison Matrices - 18
Experimental settings:
• 75 PCMs from Wikipedia
• Headers specified manually
• Automated extraction of information
PCM PCM
model
parsing preprocessing extracting
information
PCM
model
PCM
model
PCM metamodel
exploiting
S
E
R
V
I
C
E
S
RQ1
RQ2
RQ3
Evaluation
Automating the Formalization of Product Comparison Matrices - 19
Task: check interpretation of each cell (30,000+)
• Validate
• Correct it with existing concept
• Correct it with a new concept
• I don’t know / there is no interpretation
20 evaluators
Online editor
Evaluation
Automating the Formalization of Product Comparison Matrices - 20
Metrics:
• Number of valid cells
• Number of cells corrected with concepts from the metamodel
• Number of cells corrected with new concepts
• List of new concepts
Evaluation
Automating the Formalization of Product Comparison Matrices - 21
RQ1: To what extent can PCMs be formalized?
93.11% of the cells are valid
2.61% are corrected with concepts from the metamodel
4.28% are invalid and the evaluators proposed a new concept
• Dates
• Dimensions and units
• Versions
Solution:
• Add corresponding data types to the
metamodel
• Create new rules for interpreting cells
Evaluation
Automating the Formalization of Product Comparison Matrices - 22
RQ2: To what extent can the formalization be automated?
93,11% of the cells are correctly formalized
Formalization errors may arise from 4 main areas:
• Overlapping concepts (e.g. what does an empty cell mean?)
• Missing concepts (e.g. dates, versions…)
• Missing interpretation rules
• Bad rules
Evaluation
Automating the Formalization of Product Comparison Matrices - 23
RQ3: What services can be built on top of formalized PCMs?
Editing and formalizing PCMs
Warnings during edition (inconsistent cells)
Filtering capabilities
Translate PCMs to variability models
The metamodel provides
• Feature/product oriented perspective
• Clear semantics
Results of the evaluation
Automating the Formalization of Product Comparison Matrices - 24
We now have a common language for PCMs
• validated by humans
• validated by transformation
• validated by the editor
A large proportion of the formalization can be automated
BUT human is necessary
Good news: the editor can help formalizing the data
Future work
Automating the Formalization of Product Comparison Matrices - 26
Universal editor
Support large datasets Community of PCM contributors
Synchronization with Wikipedia
Questions?
Automating the Formalization of Product Comparison Matrices - 27
PCM PCM
model
parsing preprocessing extracting
information exploiting
PCM
model
PCM
model
PCM metamodel S
E
R
V
I
C
E
S