11
Auto Regression Analysis
Shuang HeIntel Linux Graphics Validation Team
Open Source Technology Center
2011-08-16
22
Legal DisclaimerINFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY RELATING TO SALE AND/OR USE OF INTEL PRODUCTS, INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT, OR OTHER INTELLECTUAL PROPERTY RIGHT.
Intel may make changes to specifications, product descriptions, and plans at any time, without notice. All dates provided are subject to change without notice.Intel is a trademark of Intel Corporation in the U.S. and other countries.*Other names and brands may be claimed as the property of others.Copyright © 2007-2011, Intel Corporation. All rights are protected.
33
Agenda• Overview• Architecture• Auto Nightly Test Infrastructure• Auto Nightly Analysis• Plan• Reference
Catch and analyze every regression!
44
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 310
50
100
150
200
250
3002011 Intel linux graphics driver commit
(week)
(commits)
55
Background• Integrated Test suites:
– Intel-gpu-tools (11 cases)– Rendercheck (10 cases)– Piglit (7500+ cases)– Cairo_test (1272 cases)– OpenGL conformance (4565 cases)– OpenGL ES 2.0 conformance (1198 cases)– Webglc (5685 cases)
66
Architecture
DATABASE
Auto Build
Auto Test
Auto Analysi
s
Test Result Page
gfx-ab-at
infrastructure
auto_bisect
77
Architecture
`
Testing Machine-1
Testing Machine-2
Testing Machine-3
Building MachineExternal Source Repository Internal Source Repository
IntranetInternet
Running AutoBuild & AutoTest
Running test case with Infrastructure
Running test case result analysis with
auto_bisectBisect machines
88
Information required for analysis
• Hardware information– GPU, CPU, Memory
• Build information– Software version
• Test information– Test case version– Test case result
99
Auto Nightly Build
• Build & Distribute Xorg components• Build & Distribute Kernel package• Build & Distribute Test suites• Archives
– X11R7.*.tgz– Kernel.* package
• Build info -> DATABASE– Component branch/commit– Infrastructure branch/commit– Test suite branch/commit
1010
Auto Nightly Test
• Run test cases one by one• Test log collection
– Run time– Output– Dmesg per case
• test result ->DATABASE• Test status watchdog
– TIMEOUT -> killed– GPU hang -> reboot -> resume to next case– System hang -> reboot -> resume to next case
• Pre/post-condition check
1111
Auto Nightly Test Process
Target Test machine
GPU hang check
TIMEOUT control
Running test case
Build/control machine
System hang check
TIMEOUT control
Resume after reboot
Auto Test Control
Init & control
Finish & collect logs
1212
Auto Nightly Analysis
• Track result change– Regression– Improvement
• Locate culprit component– Xorg component– Infrastructure– Testsuite
• Locate culprit commit– Xorg component– Infrastructure– Testsuite
1313
Auto Nightly Analysis Process
Test suite regression analysis
Infrastructure regression analysis
Kernel regression analysis
Auto bisect
Component regression
analysis
package regression
analysis
1414
Component regression analysis
• Purpose– Locating culprit component– uncommon component break
• difficulties– Too many components– Library dependency
1515
Locate culprit component
1616
Locate culprit component
• Reduce analysis effort – Only analyze libraries in use– Only analyze changed libraries
• Reduce dependency– Group related library
1717
Locate culprit commit
1818
bad branch first algorithmIf A is bad commit, and C fixed it, and then F is bad again, A -> B -> C -> D -> E -> F -> G -> H (master)
\ \ / O -> P... Q -> R -> S->T (feature 1) Start with H as bad commit, and D as good commit, it's possible git-bisect would jump to Q, and it will lead to wrong direction
1919
Bad branch first algorithmEXAPMLE 2: Given this condition: A -> B -> C -> D -> E -> F -> G -> H (master) \ O -> P -> Q -> R -> S / (feature 1) \ X -> Y -> Z/ (feature 2) start with H as bad commit, and A as good commit, if y is the target bad commit.
2020
2121
Auto Nightly Analysis Archi965 platforms
Build/control machine
Test machine Bisect machine
Test machine
Test machine
enqueue
enqueue
enqueue
Summary report
Test report&Bisect report
2222
Auto Nightly Analysis
Build machine
Test machine
Regression analysis
Engineers
Build&dist
Case 1
Analyze case 1
Analysis report
request
Case 2
All case completeTest
result Test report summary
Case n
Analyze case n
Analysis report
request
regression
regression
2323
Nightly Test Result Page
2424
Analysis enhancement
• Analysis accuracy– Verification
• Make sure all parents commit are good
• Analysis acceleration– Archive built binaries– Analyze regression in batch– Archive testing result
2525
Some results
Regressionauto
bisected
Improvementauto
bisected
2011-Aug 62 633
2011-Jul 202 381
2011-Jun 247 329
2011-May 97 35
2011-Apr 45 258
2011-Mar 90 102
2011-Feb 19 87
2011-Jan 19 33
2626
Risk
• Unstable– System may hang– GPU hang– Test case unstable
2727
Plan• Email triggered patch test system• PRTS (Patch Regression Test Suite)
– Select test case set based on code modification• Nightly Auto Test
– Capture more info when case crashed• Auto bisect for kernel/GPU hang• Nightly test result page
– Single case test result query– Quality trend indicators
• Pass rate• Regression rate• …
2828
Q&A