sqa-mashup @csmr-wcre 2014
TRANSCRIPT
Supporting Continuous Integration by Mashing-Up Software Quality Information
software evolution & architecture lab
Department of Informatics
Martin Brandtner, Emanuel Giger, and Harald Gall University of Zurich, Switzerland http://seal.ifi.uzh.ch IEEE CSMR-‐WCRE, Antwerp, Belgium, February 3-‐6, 2014
Continuous Integration (CI)
“[...] you want to ensure that everyone can easily see the state of the system and the changes that have been made to it.”
Martin Fowler
Martin Brandtner - IEEE CSMR-WCRE 2014 1
CI – Stakeholders, State & Changes
Martin Brandtner - IEEE CSMR-WCRE 2014 2
Dave (Developer)
Trinity (Tester)
So3ware System
State: The source code compiles without warnings.
State: Every integra>on test passed successfully.
Changes: Focus on the source code of the system.
Changes: Focus on the source code of the test cases.
CI – Toolchain
Commits + Issues
Build informa>on
Quality metrics
Martin Brandtner - IEEE CSMR-WCRE 2014 3
Research Question
How do stakeholders perform in answering questions about software quality with an integrated approach, such as SQA-Mashup, compared to the use of standalone CI-tools?
Martin Brandtner - IEEE CSMR-WCRE 2014 4
SQA-Mashup – View – Developer
Martin Brandtner - IEEE CSMR-WCRE 2014 5
SQA-Mashup – View – Tester
Martin Brandtner - IEEE CSMR-WCRE 2014 6
SQA-Mashup – Timeline
Martin Brandtner - IEEE CSMR-WCRE 2014 7
SQA-Mashup – Pipe Configuration
Martin Brandtner - IEEE CSMR-WCRE 2014 8
SQA-Mashup – Pipe
Action 1 – Fetch
– Endpoint Handler – URL to RESTful Service
Action 2 – Format (List) – Mapping
Martin Brandtner - IEEE CSMR-WCRE 2014 9
SQA-Mashup – User interface
Martin Brandtner - IEEE CSMR-WCRE 2014 10
Controlled user study – Setting JUnit project on GitHub
16 participants (8 control, 8 experimental) – 14 advanced graduate students,
work at least part-time in industry – 1 PhD + 1 PostDoc – Group assignment: random
Control group: GitHub, Jenkins-CI, SonarQube Experimental group: SQA-Mashup
Martin Brandtner - IEEE CSMR-WCRE 2014 11
Controlled user study – Tasks Total: 9 tasks (3 tasks in each category)
Task categories: – Program comprehension (e.g. software evolution) – Testing (e.g. testing coverage) – Cross-Domain (e.g. time-period related tasks)
Preserve in structure and semantics
1. What have my coworkers been doing? * 2. What have my coworkers been doing between 2013/07/16 and
2013/08/01?
Martin Brandtner - IEEE CSMR-WCRE 2014 12
* T. Fritz and G. C. Murphy, “Using informaNon fragments to answer the quesNons developers ask,” ser. ICSE, Cape Town, 2010, pp. 175–184.
Controlled user study – Hypotheses
H10: There is no difference in the total score per subjects between the experimental and the control group. H20: There is no difference in the total time per subjects between the experimental and the control group.
Martin Brandtner - IEEE CSMR-WCRE 2014 13
Controlled user study – Results
(sec)
Martin Brandtner - IEEE CSMR-WCRE 2014 14
Task Analysis – Score Where did the advancements come from? Total score in all other tasks: Experimental group at least as good as control group
Martin Brandtner - IEEE CSMR-WCRE 2014 15
Experimental
group: sig
nificantly h
igher
score compared
to the con
trol group
Where did the advancements come from? Total time in all other tasks: Experimental group at least as good as control group
Task Analysis – Time
Martin Brandtner - IEEE CSMR-WCRE 2014 16
Experimental
group: sig
nificantly
faster compared
to the con
trol group
The results of a user study showed that participants solved tasks with 21.6% higher correctness and 57% faster with SQA-Mashup compared to standalone CI-tools.
Martin Brandtner - IEEE CSMR-WCRE 2014 17
hHp://www.ifi.uzh.ch/seal/research/tools/sqa-‐mashup.html