qcon sf 2019 software supply chain observability in the · part pulling back the curtain part...
TRANSCRIPT
![Page 1: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/1.jpg)
Observability in theSoftware Supply Chain
Seeing into your build system
QCon SF 2019
![Page 3: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/3.jpg)
What’s this talk about
● Part story● Part pulling back the curtain● Part encouragement
![Page 4: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/4.jpg)
Part 1 - Story Time
![Page 5: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/5.jpg)
Part 1: Story Time
https://imgs.xkcd.com/comics/compiling.png
![Page 6: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/6.jpg)
Part 1: Story Time
https://imgs.xkcd.com/comics/compiling.png
● Our builds were slow● Well, they felt slow● Were they actually?● We had no idea
![Page 7: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/7.jpg)
Part 1: Story Time
https://imgs.xkcd.com/comics/compiling.png
… but we believe in
Observability!!!
![Page 8: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/8.jpg)
Part 1: Story Time
https://imgs.xkcd.com/comics/compiling.png
![Page 9: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/9.jpg)
Part 1: Story Time
https://imgs.xkcd.com/comics/compiling.png
![Page 10: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/10.jpg)
Part 1: Story Time
https://imgs.xkcd.com/comics/compiling.png
![Page 11: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/11.jpg)
Part 1: Story Time
https://imgs.xkcd.com/comics/compiling.png
![Page 12: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/12.jpg)
Part 1: Story Time
https://imgs.xkcd.com/comics/compiling.png
![Page 13: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/13.jpg)
Part 1: Story Time
https://imgs.xkcd.com/comics/compiling.png
![Page 14: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/14.jpg)
● Instrumentation○ Changed our model○ Guided work
Part 1: Story Time
![Page 15: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/15.jpg)
Part 1: Story Time
https://imgs.xkcd.com/comics/compiling.png
![Page 16: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/16.jpg)
Part 1: Story Time
https://imgs.xkcd.com/comics/compiling.png
![Page 17: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/17.jpg)
Part 1: Story Time
https://imgs.xkcd.com/comics/compiling.png
![Page 18: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/18.jpg)
Part 1: Story Time
https://imgs.xkcd.com/comics/compiling.png
![Page 19: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/19.jpg)
Part 1: Story Time
![Page 20: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/20.jpg)
● Instrumentation○ Changed our model○ Guided work○ Proved results○ Uncovered unknown variance
Part 1: Story Time
![Page 21: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/21.jpg)
● Using production-focused tools○ Developers unfamiliar with build
use the same tools they knowto understand a different service
Part 1: Story Time
![Page 22: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/22.jpg)
Part 2 - How does it work?
![Page 23: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/23.jpg)
● Go back to the beginning of the story○ Between “observability”
and “so we instrumented it”○ We need to define the problem
Part 2: How did we do it?
![Page 24: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/24.jpg)
● Vocab: Build System○ Really Build and Test○ On a trigger (usually a commit),
do some stuff○ Can Succeed or Fail○ Runs many shell commands
Part 2: How did we do it?
![Page 25: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/25.jpg)
● Core duties of a build system○ Set up an isolated environment○ Run a bunch of commands○ Stop when one fails○ Record the result
Part 2: How did we do it?
![Page 26: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/26.jpg)
● Core instrumentation for builds○ How long did it take○ What commands were run■ How long did each take
○ Did it succeed■ Did each command succeed
Part 2: How did we do it?
![Page 27: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/27.jpg)
● Core instrumentation for builds○ How long did it take○ What commands were run■ How long did each take
○ Did it succeed
Part 2: How did we do it?
![Page 28: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/28.jpg)
● Sample build config
Part 2: How did we do it?
![Page 29: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/29.jpg)
● Let’s write a small wrapper that○ Takes a name and a command○ Measures how long it takes to run○ Records its exit status■ and passes it along
○ Outputs the resulting data
Part 2: How did we do it?
![Page 30: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/30.jpg)
Part 2: How did we do it?
![Page 31: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/31.jpg)
Part 2: How did we do it?
![Page 32: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/32.jpg)
Part 2: How did we do it?
![Page 33: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/33.jpg)
● Our new build config
Part 2: How did we do it?
![Page 34: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/34.jpg)
● What gets this beyond prototype?○ Switch languages bash -> go○ Link commands together○ Improve our data model○ Collect additional context
Part 2: How did we do it?
![Page 35: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/35.jpg)
● Link our commands together○ Tie all the spans in to a trace○ Use the Build ID from the
build system
Part 2: How did we do it?
![Page 36: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/36.jpg)
Part 2: How did we do it?
![Page 37: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/37.jpg)
● Improve our data model○ Group commands into steps■ Time the whole group
○ build is the whole thing○ step is a group of commands○ cmd is one specific command
Part 2: How did we do it?
![Page 38: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/38.jpg)
Part 2: How did we do it?
![Page 39: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/39.jpg)
Part 2: How did we do it?
![Page 40: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/40.jpg)
Part 2: How did we do it?
![Page 41: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/41.jpg)
Part 2: How did we do it?
![Page 42: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/42.jpg)
● Collect additional context○ Branch name, PR#, CI system, etc.○ Custom fields from the developer■ Artifact size■ Build depth■ Links to other data sources■ Test results, etc.
Part 2: How did we do it?
![Page 43: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/43.jpg)
● Collect additional custom context○ Put name=value pairs in a file○ Put the name of that file in the env○ Buildevents includes those fields
Part 2: How did we do it?
![Page 44: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/44.jpg)
Part 2: How did we do it?
![Page 45: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/45.jpg)
● One more trick○ Interrogate the build system API○ CircleCI has live API endpoints○ Reveals the real start time○ Other juicy data
Part 2: How did we do it?
![Page 46: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/46.jpg)
● Speaking of open source …
github / honeycombio / buildevents
Part 2: How did we do it?
Travis-CI Google Cloud BuildGitLab JenkinsX CircleCI
![Page 47: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/47.jpg)
● Build systems run shell commands● By wrapping shell commands
we hook in instrumentation● Use the env or fs for IPC● Pull extra context from the env● Pull extra context from APIs● Send that data to visualization
Part 2: How did we do it?
![Page 48: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/48.jpg)
Part 3 - What’s next?
![Page 49: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/49.jpg)
● Build systems are part of the SSC○ and traces easily represent builds
● What other parts would benefit from new visualizations and tracking?○ Commit to deploy lifecycle○ ???
Part 3: What’s next?
![Page 50: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/50.jpg)
● Commit to deploy lifecycle○ Many commits to one PR○ Many review cycles before merge○ Many builds along the way○ Many environments in deploy○ Many PRs in one deploy?○ Reversions?
Part 3: What’s next?
![Page 51: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/51.jpg)
● Commit to deploy lifecycle challenge○ Represent individual run well■ Show commit-forward■ Show deploy-backward■ Represent cycles, delays
Part 3: What’s next?
![Page 52: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/52.jpg)
● Commit to deploy lifecycle challenge○ Represent runs in aggregate■ Overall lead time■ Trends in PR review delay■ Time from merge to deploy
Part 3: What’s next?
![Page 53: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/53.jpg)
● Commit to deploy lifecycle challenge○ Ease of integration into toolchain■ Source code repositories■ Build systems■ Deploy systems■ etc.
Part 3: What’s next?
![Page 54: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/54.jpg)
Part 3: What’s next?
Homework!!!!!
![Page 55: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/55.jpg)
● Instrumentation and visualizationlead to smarter work
● Build systems are aneasy first step
Summary
![Page 56: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/56.jpg)
● The tools we use for prodcan be used in the SSC
● The SSC is ripe fornew insights and visualizations
Summary
![Page 57: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/57.jpg)
● The ideas of observabilityapply outside of production
and bring value to the SSC
Summary
![Page 58: QCon SF 2019 Software Supply Chain Observability in the · Part pulling back the curtain Part encouragement. Part 1 - Story Time. Part 1: Story Time ... Go back to the beginning of](https://reader034.vdocuments.us/reader034/viewer/2022042302/5ecd44526a196229bd1c82d6/html5/thumbnails/58.jpg)
Ben Hartshornemaplebedhoneycomb.io
Thank you