static analyzer debugging and quality assurance...
TRANSCRIPT
![Page 1: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/1.jpg)
Static analyzer debugging and quality assuranceapproaches
Maxim Menshikov
28.05.2020
St.Petersburg State University
![Page 2: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/2.jpg)
About the author & the project
Maxim Menshikov —
• PhD student at St.Petersburg State University.• Software engineer.• (ex. security analyst; participated in commercial debuggerproject).
Equid1 — a static analyzer for C/C++/RuC based on ModelChecking and Abstract Interpretation. It verifies contracts andfinds common defects.
1Engine for performing queries on unified intermediate representations ofprogram and domain models 1
![Page 3: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/3.jpg)
What’s special in analyzers?
Not much.
• Many equivalent transformations:input format ̸= intermediate format ̸= output format.
• Intermediate representations are mostly internal.
• The code is usually consistent and has high integrity, butthere are logical mistakes, unprocessed parts → the biggestdefects are logical.
2
![Page 4: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/4.jpg)
The problem
• There are many debugging & quality assurance methods.
• None of them are specialized enough for static analysis.
• Every project brings its own set of hardly formalizedmethods.
What if we find a right specialization of the methods to the staticanalysis field?
3
![Page 5: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/5.jpg)
The paper’s goal
4
![Page 6: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/6.jpg)
Defect sources (observations)
• Missing support for the specific syntax/intermediaterepresentation (IR) construction in submodules.
• Small differences in implementations for repeating parts(classes).
• Transformation and ordering issues.
5
![Page 7: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/7.jpg)
Defect reasons (observations)
• Low visuality of the transformation passes and thedevelopment process.
• Unattainable cross-dependencies between modules.
• Low quality of tests.
6
![Page 8: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/8.jpg)
Proposed solutions
Proposing the solutions of these three groups:
• Code generation:Generated code usage verification.
• Testing:Goal-driven random test case generation.
• Logging:Log fusion and visual representation.
7
![Page 9: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/9.jpg)
Code generation
• One model, several interpretations, many output source files.• Perform a simple integrity check.
8
![Page 10: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/10.jpg)
Code generation: enumeration example
9
![Page 11: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/11.jpg)
Goal-driven random test case generation
The idea is: generate input programs with an integratedverification goal (assertion).
10
![Page 12: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/12.jpg)
Goal-driven random test case generation
1. The tool generates a random goal and asserts it → anexpression.
2. The expression is repeatedly rolled intoif/switch/for/while/... random blocks → a block.
3. The meaningful blocks are shuffled using equivalenttransformations.
11
![Page 13: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/13.jpg)
Goal-driven random test case generation
In result we get:
1. A completely random program.
2. A set of shuffled random programs.
By that, it is possible to verify:
1. Logical issues in transformations.
2. Ordering issues.
3. Runtime failures.
12
![Page 14: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/14.jpg)
Log fusion
Fuse separate logs, set up cross-references, so the final log is atechnical documentation of the run. Allows for easy navigation.
13
![Page 15: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/15.jpg)
Log fusion: reverse recording assistance
The Log Fusion also helps break right after the specific log lineusing reversible debugger like RR2, UndoDB, etc. That is achievedusing logging engine traps and GDB scripts.
2https://rr‐project.org
14
![Page 16: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/16.jpg)
Visual representation: steps
Visualize steps — present all transformations in one window,allow to debug specific transformations.
15
![Page 17: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/17.jpg)
Visual representation: log health
Log health — visualize the time allocation for different modules.
In result, it is possible to determine whether the specific part isunintentially skipped.
16
![Page 18: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/18.jpg)
Random test case generation: discovered issues & theirseverity
The method detected many performance, ordering, logical issues,and even runtime failures.
Defect type Number of issues SeverityPerformance 3 MediumOrdering 5 HighRuntime failure 1 HighLogical issues 1 Medium
17
![Page 19: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/19.jpg)
Log fusion: (rough) time to resolve the issues
Average improvement rate: 2.8.
Defect type Time to resolve before (h) Time to resolve after (h)
Performance 25 13Ordering 5 1Runtime failure 1 0.3Logical issues 1 1
18
![Page 20: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/20.jpg)
Results: code generation and visual representation
The improvement is hard to examine.
In our experiment, developing the same feature twice took 7times less time than on previous iteration - thanks to codegeneration.
Visual representation allowed to discover at least 2 performanceissues, and overall provided an enormous help during defectresolution.
19
![Page 21: Static analyzer debugging and quality assurance approachessyrcose.ispras.ru/2020/presentations/SYRCoSE_2020_slides... · 2020. 6. 3. · Random test case generation: discovered issues](https://reader034.vdocuments.us/reader034/viewer/2022051914/6005215d10ecf642933e2987/html5/thumbnails/21.jpg)
Conclusion
• The specialization of the proposed methods helps find realissues in the static analyzer.
• The combination of approaches dramatically decreases thedefect resolution time.
20