empowering developers to take flight - …€¦ · jenkins pipeline job ... ... empowering...
TRANSCRIPT
Empowering Developers to Take FlightContinuous Integration at Gogo
June 26, 2017 • Room #304
Darin Pantley - Gogo
Introduction
The Problem
The Dream
Implementation
Lessons Learned
Gogo is the inflight internet company
At a glance:
• 17 commercial airline partners• 3,000+ commercial aircraft in-service• 1,500+ 2Ku backlog• 120M+ connectivity sessions to date• 85,000+ sessions/day• 8,000+ flights/day• Gogo equipped flight every 11 seconds• 1,300+ employees
About Gogo
Introduction
Gogo’s open architecture is designed to power passenger experience like no other in our industry
Flexiblepassengerexperience Entertainment
Entertainment/Messaging
PassengerConnectivity
ConnectedAircraftServices
Gogo Products and Services
Introduction
June 27, 2017 • Chicago Coder Conference 5
Introduction
Infrastructure Requirements§ Hardware:
§ airplanes, devices, WAPs, satellites, servers, etc.
§ Software:§ In the air, on the ground§ Local, off-site, outsourced
Introduction
The Problem
June 27, 2017 • Chicago Coder Conference 8
The Problem
June 27, 2017 • Chicago Coder Conference 9Disciconsfromwww.flaticon.comComputer iconmadebyPanierAvide
The Problem
Software Engineering: Before Improvements1. Drink coffee2. Write some code3. Compile your changes4. Run some “unit tests” (optional)5. Commit your changes6. Create a deployable artifact
June 27, 2017 10
The Problem
Software Engineering: Before Improvements7. Copy it onto a USB stick8. Use the elevator to get to the lab9. Prepare the lab machine
§ obtain a monitor, plug in cables10. Plug in your USB stick
June 27, 2017 11
The Problem
Software Engineering: Before Improvements11. Boot it up!12. Hold your breath13. Fix the environment so it looks pretty clean14. Start manually testing your changes
June 27, 2017 12
The Problem
1. Drink coffee – better coffee machines2. Write some code – send developers to conferences3. Compile your changes – faster laptops4. Run some “unit tests” (optional) – not optional5. Commit your changes – do this early in the day6. Create a deployable artifact – lunch time
Software Engineering: After Improvements1. Drink coffee2. Write some code3. Compile your changes4. Run some “unit tests” (optional)5. Commit your changes6. Create a deployable artifact
June 27, 2017 13
The Problem
7. Copy it onto a USB stick – USB hub8. Use the elevator to get to the lab – work from the lab9. Prepare the lab machine – guard it with your life
10. Plug in your USB stick – SCP onto a running system
Software Engineering: After Improvements7. Copy it onto a USB stick8. Use the elevator to get to the lab9. Prepare the lab machine
§ obtain a monitor, plug in cables10. Plug in your USB stick
June 27, 2017 14
The Problem
11. Boot it up! – keep it running12. Hold your breath – breathe deeply13. Fix the environment so it looks pretty clean – scripts14. Start manually testing your changes – hiring time
Software Engineering: After Improvements11. Boot it up!12. Hold your breath13. Fix the environment so it looks pretty clean14. Start manually testing your changes
June 27, 2017 15
The Problem
What’s Wrong?§ It takes too long to release new features§ Many bugs aren’t caught during code reviews§ It’s difficult to obtain a test environment§ Everything is a manual process§ Running automated tests is optional§ We discover bugs too late§ All of this is expensive
June 27, 2017 • Chicago Coder Conference 16
The Problem
June 27, 2017 17
Our Infrastructure• AWS• GitLab• Jenkins• Artifactory• Docker• Ansible• Vagrant• Redis• …
The Problem
The Dream
Replace Legacy Processes with Best Practices§ Distributed Version Control System§ Agile Development§ Robust Infrastructure
§ Immutable and scalable§ Monitoring, metrics, and alerts§ Version controlled
§ Continuous Integration
June 27, 2017 • Chicago Coder Conference 19
The Dream
June 27, 2017 20
The Dream
Continuous Integration§ Automatically triggered by merge requests§ Automated Building
§ Run static analysis§ Compile the code§ Run automated tests in the build environment
§ Automated Packaging§ Create a deliverable
§ Automated Deployments§ Run automated tests in the deployed environment
§ If everything worked, let the merge proceed
June 27, 2017 21
The Dream
Reality: Maybe we can do this later?
June 27, 2017 • Chicago Coder Conference 22
1. Our code is in SVN
2. We’ve never heard ofcoupling and cohesion
3. Our tests aren’t automated
4. Everyone is busy
The Dream
Reality: Maybe we can do this later?5. We just ordered more
lab equipment
6. Feature development cannot be interrupted
7. Our code must run on physical hardware
June 27, 2017 • Chicago Coder Conference 23
The Dream
We can make improvements1. Our code is in SVN
§ Hard cut-over to Git2. We’ve never heard of coupling and cohesion
§ Componentize what you can now, the rest can wait3. Our tests aren’t automated
§ Provide a way to run them, regardless of whether they exist4. Everyone is busy
§ Prioritize continuous integration
June 27, 2017 • Chicago Coder Conference 24
The Dream
We can make improvements5. We just ordered more lab equipment
§ Some tests will still require real hardware6. Feature development cannot be interrupted
§ Avoid major changes near releases7. Our code must run on physical hardware
§ Virtualize the hardware environment
June 27, 2017 • Chicago Coder Conference 25
The Dream
Sell the Dream§ Diagram out a continuous integration workflow
June 27, 2017 • Chicago Coder Conference 26
The Dream
Sell the Dream§ Diagram out a continuous integration workflow
June 27, 2017 • Chicago Coder Conference 27
The Dream
Sell the Dream§ Diagram out a continuous integration workflow§ Present your idea to developers and upper management§ Postpone solution space discussions§ Agree on the goals§ Agree on responsibilities
June 27, 2017 • Chicago Coder Conference 28
Engineering|DevOps
The Dream
Implementation
Overview from a DevOps perspective
1. JobDSL ↴2. Jenkins Pipeline Job ↴3. Jenkins Pipeline Library
§ → pre-defined “targets”§ → DevOps tools
June 27, 2017 • Chicago Coder Conference 30
Implementation
• runway/• static-analysis• build• unit-tests• integration-tests
• dsl.groovy• version
• smoke-tests• fast-functional-tests• slow-functional-tests• end-to-end-system-tests
Overview from a developer perspective
June 27, 2017 • Chicago Coder Conference 31
Implementation
• ci• nightly
• iteration• release
Pipeline Types
June 27, 2017 • Chicago Coder Conference 32
Implementation
GitLab Merge Requests
June 27, 2017 • Chicago Coder Conference 33
Implementation
Jenkins Pipeline
June 27, 2017 • Chicago Coder Conference 34
Implementation
runway / build#!/bin/bash -exexport SHORT_SHA=$(git rev-parse --short HEAD)export VERSION="$(cat runway/version).${BUILD_NUMBER}~$SHORT_SHA"export FULL_SHA=$(git rev-parse HEAD)export GIT_COMMIT="$FULL_SHA $(git log -1 --pretty=%B | head -1)"
docker login gogo-docker.jfrog.io -u builder -p hunter2docker run --rm -w /build -v $(pwd):/build/ gogo-docker.jfrog.io/
buildmachine:2.3.0.19 ./build.sh /build/runway/FS_ROOT $VERSION$GIT_COMMIT
June 27, 2017 • Chicago Coder Conference 35
https://www.docker.com
Implementation
runway / dsl.groovyimport utilities.ContinuousIntegrationimport utilities.NightlyBuild
REPO = '[email protected]:acpu/hal.git'
def ci = new ContinuousIntegration(job: pipelineJob('${SRC_JOB}_ci'))ci.createJenkinsPipeline()
def nightly = new NightlyBuild(job: pipelineJob('${SRC_JOB}_nightly'))nightly.createJenkinsPipeline(REPO)
June 27, 2017 • Chicago Coder Conference 36
https://github.com/jenkinsci/job-dsl-plugin
Implementation
ContinuousIntegration.groovythis.job.with {
description("Continuous Integration Pipeline")…
url('[email protected]:common/jenkins-pipeline.git')
…scriptPath('ContinuousIntegration.Jenkinsfile')
…}
June 27, 2017 • Chicago Coder Conference 37
https://github.com/jenkinsci/job-dsl-plugin
Implementation
Jenkins Pipeline Shared Libraries(root)+- src # Groovy source files | +- org | +- foo | +- Bar.groovy # for org.foo.Bar class+- vars | +- foo.groovy # for global 'foo' variable| +- foo.txt # help for 'foo' variable +- resources # resource files (external libraries only)| +- org | +- foo | +- bar.json # static helper data for org.foo.Bar
June 27, 2017 • Chicago Coder Conference 38
https://jenkins.io/doc/book/pipeline/shared-libraries/
Implementation
ContinuousIntegration.Jenkinsfilenode('ci') {
withEnv(["PIPELINE=ci"]) {… staticAnalysis()
buildComponent() unitTests() uploadComponent()
…
June 27, 2017 • Chicago Coder Conference 39
https://jenkins.io/doc/pipeline/
Implementation
vars / buildComponent.groovy...dir(COMPONENT_DIR) {
def target = 'runway/build'if (fileExists(target)) {
sh target}
}...
June 27, 2017 • Chicago Coder Conference 40
Implementation
• Artist• Written in Python• CLI tool
• Artifactory
Organizing the code
June 27, 2017 • Chicago Coder Conference 41
• Graffiti• Written in Python• CLI tool
• GitLab• Hosted Graphite
Implementation
Lessons Learned
Working with Developers§ Meet frequently with everyone involved§ Deliver features incrementally§ Spread access to tooling
§ Extend internal solutions to cover developer use cases§ Document answers to questions§ Agree on solutions to technical debt in existing code
June 27, 2017 • Chicago Coder Conference 43
Lessons Learned
Working without Developers§ Create alternatives to technical debt in existing code§ Create proof of concept projects
§ Choose between competing technologies§ Demo solutions to controversial problems§ Prove that something is possible
§ Working out-of-band allows constant refactoring
June 27, 2017 • Chicago Coder Conference 44
Lessons Learned
Hindsight is 20/20§ Explicitly define requirements up-front
§ General functionality, documentation, metrics, alerts, …§ Plan your current features with the next features in mind§ Find users quickly§ Eat your own dog food§ Feed the hungry§ Avoid bus errors
June 27, 2017 • Chicago Coder Conference 45
Lessons Learned
Overheard on Slack§ “I can show you how to use Artist to upload. It’s easy.”
§ “Should I merge? The button is red.”
June 27, 2017 • Chicago Coder Conference 46
Lessons Learned
June 27, 2017 47
Empowering Developers to Take FlightContinuous Integration at Gogo
Darin [email protected]
3:45pm – 4:45pm • Room #304
Thank You