11 auto regression analysis shuang he intel linux graphics validation team open source technology...
TRANSCRIPT
![Page 1: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/1.jpg)
11
Auto Regression Analysis
Shuang HeIntel Linux Graphics Validation Team
Open Source Technology Center
2011-08-16
![Page 2: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/2.jpg)
22
Legal DisclaimerINFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY RELATING TO SALE AND/OR USE OF INTEL PRODUCTS, INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT, OR OTHER INTELLECTUAL PROPERTY RIGHT.
Intel may make changes to specifications, product descriptions, and plans at any time, without notice. All dates provided are subject to change without notice.Intel is a trademark of Intel Corporation in the U.S. and other countries.*Other names and brands may be claimed as the property of others.Copyright © 2007-2011, Intel Corporation. All rights are protected.
![Page 3: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/3.jpg)
33
Agenda• Overview• Architecture• Auto Nightly Test Infrastructure• Auto Nightly Analysis• Plan• Reference
Catch and analyze every regression!
![Page 4: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/4.jpg)
44
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 310
50
100
150
200
250
3002011 Intel linux graphics driver commit
(week)
(commits)
![Page 5: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/5.jpg)
55
Background• Integrated Test suites:
– Intel-gpu-tools (11 cases)– Rendercheck (10 cases)– Piglit (7500+ cases)– Cairo_test (1272 cases)– OpenGL conformance (4565 cases)– OpenGL ES 2.0 conformance (1198 cases)– Webglc (5685 cases)
![Page 6: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/6.jpg)
66
Architecture
DATABASE
Auto Build
Auto Test
Auto Analysi
s
Test Result Page
gfx-ab-at
infrastructure
auto_bisect
![Page 7: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/7.jpg)
77
Architecture
`
Testing Machine-1
Testing Machine-2
Testing Machine-3
Building MachineExternal Source Repository Internal Source Repository
IntranetInternet
Running AutoBuild & AutoTest
Running test case with Infrastructure
Running test case result analysis with
auto_bisectBisect machines
![Page 8: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/8.jpg)
88
Information required for analysis
• Hardware information– GPU, CPU, Memory
• Build information– Software version
• Test information– Test case version– Test case result
![Page 9: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/9.jpg)
99
Auto Nightly Build
• Build & Distribute Xorg components• Build & Distribute Kernel package• Build & Distribute Test suites• Archives
– X11R7.*.tgz– Kernel.* package
• Build info -> DATABASE– Component branch/commit– Infrastructure branch/commit– Test suite branch/commit
![Page 10: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/10.jpg)
1010
Auto Nightly Test
• Run test cases one by one• Test log collection
– Run time– Output– Dmesg per case
• test result ->DATABASE• Test status watchdog
– TIMEOUT -> killed– GPU hang -> reboot -> resume to next case– System hang -> reboot -> resume to next case
• Pre/post-condition check
![Page 11: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/11.jpg)
1111
Auto Nightly Test Process
Target Test machine
GPU hang check
TIMEOUT control
Running test case
Build/control machine
System hang check
TIMEOUT control
Resume after reboot
Auto Test Control
Init & control
Finish & collect logs
![Page 12: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/12.jpg)
1212
Auto Nightly Analysis
• Track result change– Regression– Improvement
• Locate culprit component– Xorg component– Infrastructure– Testsuite
• Locate culprit commit– Xorg component– Infrastructure– Testsuite
![Page 13: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/13.jpg)
1313
Auto Nightly Analysis Process
Test suite regression analysis
Infrastructure regression analysis
Kernel regression analysis
Auto bisect
Component regression
analysis
package regression
analysis
![Page 14: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/14.jpg)
1414
Component regression analysis
• Purpose– Locating culprit component– uncommon component break
• difficulties– Too many components– Library dependency
![Page 15: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/15.jpg)
1515
Locate culprit component
![Page 16: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/16.jpg)
1616
Locate culprit component
• Reduce analysis effort – Only analyze libraries in use– Only analyze changed libraries
• Reduce dependency– Group related library
![Page 17: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/17.jpg)
1717
Locate culprit commit
![Page 18: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/18.jpg)
1818
bad branch first algorithmIf A is bad commit, and C fixed it, and then F is bad again, A -> B -> C -> D -> E -> F -> G -> H (master)
\ \ / O -> P... Q -> R -> S->T (feature 1) Start with H as bad commit, and D as good commit, it's possible git-bisect would jump to Q, and it will lead to wrong direction
![Page 19: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/19.jpg)
1919
Bad branch first algorithmEXAPMLE 2: Given this condition: A -> B -> C -> D -> E -> F -> G -> H (master) \ O -> P -> Q -> R -> S / (feature 1) \ X -> Y -> Z/ (feature 2) start with H as bad commit, and A as good commit, if y is the target bad commit.
![Page 20: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/20.jpg)
2020
![Page 21: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/21.jpg)
2121
Auto Nightly Analysis Archi965 platforms
Build/control machine
Test machine Bisect machine
Test machine
Test machine
enqueue
enqueue
enqueue
Summary report
Test report&Bisect report
![Page 22: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/22.jpg)
2222
Auto Nightly Analysis
Build machine
Test machine
Regression analysis
Engineers
Build&dist
Case 1
Analyze case 1
Analysis report
request
Case 2
All case completeTest
result Test report summary
Case n
Analyze case n
Analysis report
request
regression
regression
![Page 23: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/23.jpg)
2323
Nightly Test Result Page
![Page 24: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/24.jpg)
2424
Analysis enhancement
• Analysis accuracy– Verification
• Make sure all parents commit are good
• Analysis acceleration– Archive built binaries– Analyze regression in batch– Archive testing result
![Page 25: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/25.jpg)
2525
Some results
Regressionauto
bisected
Improvementauto
bisected
2011-Aug 62 633
2011-Jul 202 381
2011-Jun 247 329
2011-May 97 35
2011-Apr 45 258
2011-Mar 90 102
2011-Feb 19 87
2011-Jan 19 33
![Page 26: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/26.jpg)
2626
Risk
• Unstable– System may hang– GPU hang– Test case unstable
![Page 27: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/27.jpg)
2727
Plan• Email triggered patch test system• PRTS (Patch Regression Test Suite)
– Select test case set based on code modification• Nightly Auto Test
– Capture more info when case crashed• Auto bisect for kernel/GPU hang• Nightly test result page
– Single case test result query– Quality trend indicators
• Pass rate• Regression rate• …
![Page 28: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/28.jpg)
2828
Q&A
![Page 29: 11 Auto Regression Analysis Shuang He Intel Linux Graphics Validation Team Open Source Technology Center 2011-08-16](https://reader036.vdocuments.us/reader036/viewer/2022062417/5516b1cb550346f6208b52a3/html5/thumbnails/29.jpg)