how i stopped worrying about and loved dumprendertree
DESCRIPTION
For the talk at shibuya.js on Test.jsTRANSCRIPT
![Page 1: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/1.jpg)
2011/03/08MORITA Hajime <[email protected]>
http://steps.dodgson.org/
How I stopped worryingandloved DumpRenderTree
Tuesday, March 8, 2011
![Page 2: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/2.jpg)
2011/03/08MORITA Hajime <[email protected]>
http://steps.dodgson.org/
a bit
How I stopped worryingandloved DumpRenderTree
Tuesday, March 8, 2011
![Page 3: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/3.jpg)
あらすじ
• WebKit のテストは歴史的事情からうまれたTest Smellの山を工夫と根性でなんとかしています
• .js よりは Test の話です
• 個人の見解であり(略)
Tuesday, March 8, 2011
![Page 4: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/4.jpg)
DumpRenderTree (DRT)
Demo
Tuesday, March 8, 2011
![Page 5: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/5.jpg)
DumpRenderTree (DRT)
• コマンドラインツール
• HTMLをロードしてレイアウト情報をダンプ (描画結果も利用可)
• 期待値 “Golden File” と比較
• OS 毎に用意
Tuesday, March 8, 2011
![Page 6: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/6.jpg)
歴史的想定 (-2002)
• ブラウザエンジンの仕事=レイアウト計算
• DumpRenderTree, LayoutTest
• WebKit は Mac で動かすもの
Tuesday, March 8, 2011
![Page 7: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/7.jpg)
今日このごろ
• 言語処理系JavaScript / DOM
• 通信プロトコル ex: ws://
• アニメーションex: -webkit-transition:
• OS, メディア統合ex: drag-n-drop, media, storage
• 対話的デバイスex: <input type=”speech”>
• WYSIWYG 編集contentEditable
• OS/GUI いろいろ (Ports)Win, Qt, GTK, EFL, Skia ...
• Mac もいろいろTiger, Leopard, SL, Lion
レイアウト計算? Mac でうごかすもの?
Tuesday, March 8, 2011
![Page 8: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/8.jpg)
3 Test Smells in WebKit
• Obscure Tests 読みにくいテスト
• Fragile Tests 壊れやすいテスト
• High Test Maintenance Cost メンテ大変
Tuesday, March 8, 2011
![Page 9: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/9.jpg)
(1/3): Obscure Tests
• JavaScript
• HTML5-ish (File, XHR, WebSocket, etc.)
• クラッシュバグの再現
レイアウトとは無関係なものも Dump?
Tuesday, March 8, 2011
![Page 10: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/10.jpg)
if (window.layoutTestController) { layoutTestController.dumpAsText(); ...}
function log(msg){ document.getElementById('log').innerHTML += (msg + “\n”);}
... sel = window.getSelection(); actualText = sel.getRangeAt(0).toString();
if (expectedText == actualText) { log("Passed " + testId + " (with contentEditable)"); } else { log("Failed " + testId + " (with contentEditable)"); log(" Expected: " + expectedText); log(" Actual: " + actualText); }...
LayoutTestController.dumpAsText()
Tuesday, March 8, 2011
![Page 11: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/11.jpg)
Passed totest_minimumPassed totest_minimum (with contentEditable)Passed totest_before_afterPassed totest_before_after (with contentEditable)Passed totest_linebreak_outPassed totest_linebreak_out (with contentEditable)Passed totest_tab_outPassed totest_tab_out (with contentEditable)Passed totest_multiple_whitespaces_outPassed totest_multiple_whitespaces_out (with contentEditable)Passed totest_linebreak_inPassed totest_linebreak_in (with contentEditable)
実行結果:
• よみやすい!
• どの Port でも結果が同じ!
Tuesday, March 8, 2011
![Page 12: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/12.jpg)
(1/3): Obscure Tests
• LayoutTestController.dumpAsText()
• js-test-pre.js/js-test-post.jsassertion やログ出力をまとめたライブラリ
• dump-as-markup.jsDOM だけ表示 (built w/ dumpAsText())contentEditable のテストなどで便利
Tuesday, March 8, 2011
![Page 13: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/13.jpg)
(2/3): Fragile Tests
• レイアウト結果が少しだけ変わったら?• 新OSでフォントのデザインがかわったら?
• 対応 OS が増えたら?
• 組版アルゴリズムを改善したら?• GPU がちがったら?
• 同じテストに項目を追加したら?• DRT のフォーマットを変えたら?
Golden File の欠点: Over specification
とにかくよく壊れるTuesday, March 8, 2011
![Page 14: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/14.jpg)
Tree Redness
http://build.webkit.org/waterfall
• いつも赤いと誰も気にしなくなる(割れ窓)
• 本当のバグなのか期待値が古いのかわからないTuesday, March 8, 2011
![Page 15: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/15.jpg)
“Skipped” list
• テストを実行しない(=赤くならない)
• 期待値の変化が予想どおりの時のみ追記可• 期待値ファイルが古い(無い)旨バグ登録する
• Bug = TODO list
• 誰かが直してくれると信じる• 全Portの Golden Fileを更新なんてムリだから...
Port 毎に無視するテストの一覧
Tuesday, March 8, 2011
![Page 16: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/16.jpg)
Rebaselining Tool (Chromium only)
自分でテストを動かさなくていい!
Buildbot Server
Local PC SVN
result.zip...
expected.txt...
...expected.txt
...
scriptDownload
Commit (by dev)Update
Tuesday, March 8, 2011
![Page 17: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/17.jpg)
(3/3): High Test Maintenance Cost
• すぐ変化する Golden Files
• 試行錯誤の難しい遅さ• なぜか失敗するテスト• 複数 Port が別々に実装するテストドライバ
• ...
Tuesday, March 8, 2011
![Page 18: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/18.jpg)
Gardener (for Chromium)
• ビルドやテストがこわれたら対処する仕事Rebaseline, Skip, Revert, ..
• 当番制
Tuesday, March 8, 2011
![Page 19: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/19.jpg)
Reftests (work in progress)
• 二つの HTML を比較するテスト
• Port 非依存、変化に強い!
• Mozilla 由来https://developer.mozilla.org/en/Creating_reftest-based_unit_tests
• W3C でも採用を検討中
Tuesday, March 8, 2011
![Page 20: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/20.jpg)
まとめ
• WebKit のテストは歴史的事情からうまれたTest Smellの山を工夫と根性でなんとかしています
Tuesday, March 8, 2011
![Page 21: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/21.jpg)
Backup Slides
Tuesday, March 8, 2011
![Page 22: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/22.jpg)
7 Test Smells in WebKit• Obscure Tests 読みにくいテスト
• Fragile Tests 壊れやすいテスト
• Erratic Tests よく失敗するテスト
• Manual Intervention 手動でテスト
• Slow Tests 遅いテスト
• High Test Maintenance Cost メンテ大変
• Developers Not Writing (Unit) Tests
Tuesday, March 8, 2011
![Page 23: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/23.jpg)
(x/x): Erratic Tests
• タイムアウトが早過ぎる• OSが不安定(メディア系など)
• 通信エラー• 再現性のないクラッシュ• 毎回微妙に位置が違う... (アニメーションなど)
aka. Flaky Tests
なんだか知らんがたまに失敗するテスト
Tuesday, March 8, 2011
![Page 24: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/24.jpg)
再実行する (Chromium only)
unexpected flaky 1
http://build.chromium.org/p/chromium.webkit/waterfall
Tuesday, March 8, 2011
![Page 25: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/25.jpg)
(x/x): Manual Intervention
• ユーザ操作• DOM からもレイアウトからも見えない状態
• ex: 正しくコールバックが呼ばれたか?
• デバイス
素の JS では自動化コードを書けないテスト
Tuesday, March 8, 2011
![Page 26: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/26.jpg)
DRT に API を追加する// from editing/input/emacs-ctrl-o.html... if (window.eventSender) eventSender.keyDown("o", ["ctrlKey"]);...
// from editing/spelling/.../spelling-attribute-at-child.js... var marked = layoutTestController.hasSpellingMarker(1, 2);...
// from fast/speech/input-onspeechchange-event.html... if (window.layoutTestController && window.eventSender) { layoutTestController.addMockSpeechInputResult(....);...
Tuesday, March 8, 2011
![Page 27: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/27.jpg)
でも...
Tuesday, March 8, 2011
![Page 28: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/28.jpg)
(x/x): Slow Tests
• 2万件
• テスト毎に毎回 HTML/JS をロード
Tuesday, March 8, 2011
![Page 29: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/29.jpg)
並列化したいけど...
• GUI はスケールしにくい専有資源: “Active Window”, “Clipboard”, ...
• 競合するリソース (ex. localStorage)
• 暗黙の依存関係 (テストサーバの状態とか)
• Chromium WebKit はだいたい並列化
• Window-less by design
• ...誰かが頑張ってテストドライバを書いた
Tuesday, March 8, 2011
![Page 30: How I stopped worrying about and loved DumpRenderTree](https://reader034.vdocuments.us/reader034/viewer/2022051015/55389081550346bf308b47ce/html5/thumbnails/30.jpg)
各テストの高速化$ ./Tools/Scripts/run-webkit-tests --helpUsage: old-run-webkit-tests [options] [testdir|testpath ...] .... --slowest Report the 10 slowest tests ....
Tuesday, March 8, 2011