itt 2015 - simon stewart - building android apps at speed and scale
TRANSCRIPT
![Page 1: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/1.jpg)
Building Android Apps At Scale and Speed
Simon Stewart @shs96c
Software Engineer, Facebook
![Page 2: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/2.jpg)
One Guiding Principle
![Page 3: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/3.jpg)
We don't want people waiting on computers
<
![Page 4: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/4.jpg)
We want computers waiting on people
<
![Page 5: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/5.jpg)
Scalable Source Organisation
![Page 6: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/6.jpg)
Server
Sharing Code Today
Android
c++
iOS
![Page 7: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/7.jpg)
Server
Sharing Code Today
AndroidiOS
c++c++
![Page 8: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/8.jpg)
Server
Sharing Code Today
AndroidiOS
c++ c++c++
![Page 9: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/9.jpg)
Sharing Code Tomorrow
fbsource
c++
![Page 10: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/10.jpg)
All our source in a single repository (a "monorepo")
![Page 11: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/11.jpg)
Why a monorepo?
Easier: code sharing, versioning, contributing
Simpler: large scale changes
Consistent: uniform tooling
![Page 12: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/12.jpg)
Scalable Source Control
![Page 13: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/13.jpg)
Mercurial
>1M >100KSource control commands
per daycommits per week
![Page 14: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/14.jpg)
Large Repos Are Slow
▪Lots of commits a day
Pulls are slow
Checkouts are slow
▪Years of history
Clones are slow
Requires lots of disk space
![Page 15: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/15.jpg)
We want computers waiting on people
<
![Page 16: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/16.jpg)
Shallow CheckoutsClone just the latest version
Work locally without complete history
Need more history? Download on demand
HEAD
HEAD-1
HEAD-2
HEAD-3
![Page 17: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/17.jpg)
Sparse CheckoutsWork on just the files you need
Build system integrations knows how to check out more
~/fbsource ios/... common/...
~/fbsource/.hg
![Page 18: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/18.jpg)
WatchmanOS independent filesystem change notification
Integrates with mercurial
File checking operations scale linearly with number of files changed
![Page 19: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/19.jpg)
Remote FilelogClone and pull just download metadata
Download files when required
Clones and pulls: 10x faster
![Page 20: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/20.jpg)
Large Repos Are Fast
50times faster than git
2000hg improvements
Up to More than
![Page 21: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/21.jpg)
Scalable Builds
![Page 22: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/22.jpg)
Buck▪Next generation build system
▪Designed for modern hardware
▪SSD
▪Plentiful RAM
▪Many cores
▪Automatically parallelizes builds
▪And it's Open Source
![Page 23: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/23.jpg)
![Page 24: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/24.jpg)
Build Files
java_binary( name = 'example-binary', deps = [ ':example', ], )
java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )
Build file. Typically named BUCK
![Page 25: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/25.jpg)
Build Files
java_binary( name = 'example-binary', deps = [ ':example', ], )
java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )
Build file. Typically named BUCK
A build rule of type "java_binary"
![Page 26: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/26.jpg)
Build Files
java_binary( name = 'example-binary', deps = [ ':example', ], )
java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )
Build file. Typically named BUCK
A build rule of type "java_binary"
A target defined in the same build file
![Page 27: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/27.jpg)
Build Files
java_binary( name = 'example-binary', deps = [ ':example', ], )
java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )
Build file. Typically named BUCK
A build rule of type "java_binary"
A target defined in the same build file
Build rule of type "java_library"
![Page 28: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/28.jpg)
Build Files
java_binary( name = 'example-binary', deps = [ ':example', ], )
java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )
Build file. Typically named BUCK
A build rule of type "java_binary"
A target defined in the same build file
A target defined in another build file
Build rule of type "java_library"
![Page 29: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/29.jpg)
Build Files
java_binary( name = 'example-binary', deps = [ ':example', ], )
java_library( name = 'example', srcs = glob(['*.java']), deps = [ '//third-party/guava:guava', ], visibility = [ '//javatests/...', ], )
Build file. Typically named BUCK
A build rule of type "java_binary"
A target defined in the same build file
A target defined in another build file
Build rule of type "java_library"
All targets have visibility
![Page 30: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/30.jpg)
Target and Action Graph
Target Graph: a naive representation of targets defined in build files.
Action Graph: an optimized graph of actions to perform as the build
![Page 31: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/31.jpg)
Language-Aware Smartspublic class Foo {
private String someField;
public void doSomething(int times) {
System.out.println("Wow! " + times);
}
}
Buck calculates the ABI, and keys rebuilds of dependencies off that
Packaging rules that depend on this will always be re-run.
![Page 32: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/32.jpg)
Language-Aware Smartspublic class Foo {
private boolean theCakeIsALie = true;
public void doSomething(int times) {
System.out.println("Wow! " + times);
}
}
Private field changed: ABI remains the same as before
![Page 33: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/33.jpg)
Language-Aware Smartspublic class Foo {
private boolean theCakeIsALie = true;
public void doSomething(int num) {
System.out.println("Wow! " + num);
}
}
Parameter name changed: ABI remains the same as before.
![Page 34: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/34.jpg)
Language-Aware Smartspublic class Foo {
private boolean theCakeIsALie = true;
public void doSomething(String num) {
System.out.println("Wow! " + num);
}
}
Parameter type changed: ABI changes too!
![Page 35: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/35.jpg)
Language-Aware Smartspublic class Foo {
private boolean theCakeIsALie = true;
public void doSomething(String num) {
System.err.println("Wow! " + num);
}
}
Body of method changed: ABI remains the same.
![Page 36: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/36.jpg)
Pervasive Caching• Only 3% of each build is unique.
• Local and distributed caches slash build times.
![Page 37: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/37.jpg)
Exopackage▪Exopackage is an optimization for building and installing development builds of Android apps.
![Page 38: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/38.jpg)
Buck Smarts
Build rule knows all of the inputs that can affect its output
Changes to dx and proguard for faster compilation
Java ABI smarts
Android resource smarts
Daemon to watch file changes
![Page 39: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/39.jpg)
Scalable Continuous Integration
![Page 40: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/40.jpg)
<10 MinutesFeedback to Developers
>1000test results per second
5 Yearsmachine work per day
![Page 41: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/41.jpg)
Life Cycle of a Diff
Write Code Test Locally Diff and review Sandcastle Land
![Page 42: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/42.jpg)
Sandcastle
RapidHigh-Signal Frequent
![Page 43: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/43.jpg)
Sandcastle Design
Single Master
Master
Worker Worker WorkerWorker
![Page 44: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/44.jpg)
Sandcastle Design
Single Master Distributed Queue
Master
Worker Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Distributed Queue
![Page 45: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/45.jpg)
![Page 46: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/46.jpg)
Test Runs
Flakiness SpeedCoverage
![Page 47: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/47.jpg)
Landcastle
Land
Target Determinator
Test Test Test
Push
![Page 48: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/48.jpg)
We don't want people waiting on computers
<
![Page 49: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/49.jpg)
We want computers waiting on people
<
![Page 50: ITT 2015 - Simon Stewart - Building Android Apps at Speed and Scale](https://reader031.vdocuments.us/reader031/viewer/2022022203/5878bbd21a28ab724c8b790d/html5/thumbnails/50.jpg)
Summary
Tool OSS? Link!
Buck Yes https://facebook.github.io/buck/
Mercurial Yes http://mercurial.selenic.com/
Watchman Yes https://facebook.github.io/watchman/
hgwatchman Yes https://bitbucket.org/facebook/hgwatchman
remote filelog Yes https://bitbucket.org/facebook/remotefilelog
Sandcastle No