automated testing: past, present and future · 2020. 1. 4. · automated testing: past, present and...
TRANSCRIPT
![Page 2: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/2.jpg)
2
Goals
● NetBSD is where testing is happening● Show what's coming in the near future
![Page 3: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/3.jpg)
3
Goals
● NetBSD is where testing is happening● Show what's coming in the near future
● Get other BSD systems involved!
![Page 4: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/4.jpg)
4
Contents
Past What led to ATF?
PresentWhat is ATF?
Other testing technologies
FutureA critique of ATF
Meet Kyua
![Page 5: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/5.jpg)
5
Why testing?
● Software has bugs● Automated testing helps reducing bugs
![Page 6: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/6.jpg)
6
Origins: src/regress
● No consistency● Not used; instead, ad-hoc testing● What about release engineering?
![Page 7: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/7.jpg)
7
What is ATF?
● Simply put: a testing framework● Libraries: C, C++, Shell● Tools: run, report
● Since 2007 (SoC) until now
![Page 8: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/8.jpg)
8
ATF in NetBSD
● For developers:● Maintain stable current● Help release engineering
● For users:● Confidence
![Page 9: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/9.jpg)
9
Related technologies
● rump● puffs, pud and others● anita
● Development policies● Dashboards
![Page 10: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/10.jpg)
10
Continuous Builds
![Page 11: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/11.jpg)
11
Continuous Builds
![Page 12: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/12.jpg)
12
Releng dashboard
![Page 13: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/13.jpg)
13
A critique of ATF
● Ugly codebase● Desired features are hard to implement
● Parallel execution● Integrated dashboards
● High entry barrier for contributors
![Page 14: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/14.jpg)
14
Refactor vs. Rewrite
● The eternal dilemma
![Page 15: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/15.jpg)
15
Refactor vs. Rewrite
● The eternal dilemma
● Decision:● Rewrite user tools● Reuse bindings (atf-{c,c++,sh}) for now
![Page 16: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/16.jpg)
16
What is Kyua?
● Think of it as ATF v2
● Component separation● “Better” code● Easier entry for contributors● New features!
![Page 17: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/17.jpg)
17
From ATF to Kyua
● Smooth migration plan● Rewrite the tools (easy!)● Later: rewrite the libraries (hard)
![Page 18: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/18.jpg)
18
Kyua: Components
Test program(atf-c)
Test program(atf-c++)
Test program(atf-sh)
KyuaRuntime engine
KyuaStore
KyuaInspection tools
Reports Dashboards
![Page 20: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/20.jpg)
20
Test coverage
Test cases Run time
ATF
All components542 683
Kyua
Runtime engine
48s real19s user16s sys
23s real4s user5s sys
![Page 21: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/21.jpg)
21
Kyua examples
$ cd /usr/tests
$ kyua test --session_id=bsdcan_session
$ kyua debug fs/tmpfs/t_mount
$ kyua inspect fs/tmpfs/t_mount
$ kyua export -–session_id=bsdcan_session
-o html
![Page 22: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/22.jpg)
22
The short-term future
● Kyua 0.1: run-time engine, no store● Kyua 0.2: with store, exporting of results
● Lua binding for test programs?● Adoption by other BSDs?
![Page 23: Automated testing: Past, Present and Future · 2020. 1. 4. · Automated testing: Past, Present and Future Julio Merino for BSDCan 2011](https://reader036.vdocuments.us/reader036/viewer/2022063018/5fdcf5e24df0685cad2b0df9/html5/thumbnails/23.jpg)
23
Conclusions
● Tests make NetBSD more robust● NetBSD has many exciting technologies● Kyua is the evolution of ATF
● Other BSDs should join the party!