running tests for every commit: gerrit, jenkins, docker, aws
TRANSCRIPT
![Page 1: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/1.jpg)
Running tests for every commitGerrit, Jenkins, Docker, AWS
Alexander AkbashevDevOps Specialist16/04/2016
![Page 2: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/2.jpg)
Nice to meet you
![Page 3: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/3.jpg)
HERE
• employs 6,500 people in 55 countries.
• produces maps for almost 200 countries.
• enables four out of five in-car nav systems in North America and Europe
• enables mobile, web, and enterprise solutions for global industry leaders.
HERE is a leading location cloud providing superior mapping, services and location intelligence across the automotive, enterprise and internet industries.
![Page 4: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/4.jpg)
![Page 5: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/5.jpg)
55
Stack
• CI: Jenkins
• Tests: RobotFramework/Cucumber, GoogleTest, etc
• Target platforms: Windows 32/64, Linux 32/64, QNX, Android/iOS
Copyright © HERE 2016
![Page 6: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/6.jpg)
66
Where we were
• Build every 40 minutes for several commits
• Review on-demand
• Each team could have own Jenkins
• Each team could have CI pipeline
Copyright © HERE 2016
![Page 7: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/7.jpg)
77
Desired
• Build on every commit
• Review for every commit
• One Jenkins
• Unified CI pipeline
Copyright © HERE 2016
![Page 8: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/8.jpg)
88
Numbers*
• 55 - 65k builds/day
• 5 - 6k builds/hour
• 1.8 - 2k commits/day
• 1100 -1200 executors
• 19 minutes
* - for the main productCopyright © HERE 2016
![Page 9: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/9.jpg)
</cut>
![Page 10: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/10.jpg)
1010
Big picture
Code Gerrit Jenkins Gerrit Repository
Copyright © HERE 2016
![Page 11: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/11.jpg)
1111
Gerrit
• Git
• Open source
• Plugins
• Scale
• JVM
Copyright © HERE 2016
![Page 12: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/12.jpg)
1212
Gerrit
• Code Review ( +2 )
• Test passed ( +1 )
Feedback time: 2 minutes for build fail, 19 minutes for success
Copyright © HERE 2016
![Page 13: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/13.jpg)
1313
Jenkins
• OpenSource
• Plugins
• Community
• JVM
Copyright © HERE 2016
![Page 14: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/14.jpg)
1414
AWS EC2 + EC2 Plugin
• Pool
• Maintenance
• Scalability
• Price
Copyright © HERE 2016
![Page 15: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/15.jpg)
1515
Docker
• DRY
• Isolation
• Limits
• Repeatability
Copyright © HERE 2016
![Page 16: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/16.jpg)
1616
Jenkins is not for
• Artifacts
• Statistics
• Managers
• Lazy people
Copyright © HERE 2016
![Page 17: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/17.jpg)
1717
Pipeline plugin*
• Reduce monkey work
• Review
• Configuration as code
* - by default in Jenkins 2.0Copyright © HERE 2016
![Page 18: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/18.jpg)
1818
In-house dashboard + Groovy Event Listener Plugin
• Nice UI
• Quick UI
• No action buttons
• Results for release scope
Copyright © HERE 2016
![Page 19: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/19.jpg)
1919
AWS S3 + S3 plugin
• Keep artifacts longer
• Save money
• Faster
Copyright © HERE 2016
![Page 20: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/20.jpg)
Room for improvements
![Page 21: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/21.jpg)
2121
Divide et empera
Copyright © HERE 2016
![Page 22: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/22.jpg)
2222
Divide et empera
Copyright © HERE 2016
![Page 23: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/23.jpg)
2323
Divide et empera
Copyright © HERE 2016
![Page 24: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/24.jpg)
2424
Profiling
• IO
• CPU
• Memory
• Time budget
• Cut explicitly
Copyright © HERE 2016
![Page 25: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/25.jpg)
2525
Jenkins
• Yourkit
• GC logs
• FlightRecorder
“All magic comes with a price”
Copyright © HERE 2016
![Page 26: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/26.jpg)
2626
Android
• TestDroid
• Battery Life
• OOM Killer
• Data updates
Copyright © HERE 2016
![Page 27: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/27.jpg)
Stability
![Page 28: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/28.jpg)
2828
Angry Devs
0
20
40
60
80
100
100 90 80 70 60 50 40 30 20 10 0
Copyright © HERE 2016
![Page 29: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/29.jpg)
2929
Verification jobs
• Same tests
• Schedule
• Pass rate - 100%
Copyright © HERE 2016
![Page 30: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/30.jpg)
3030
Dashboards
• Queue Size
• Chance to submit
• Feedback
• Executors
• Flaky tests/jobs
• etc
Copyright © HERE 2016
![Page 31: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/31.jpg)
3131
Automatization
• Sanity checks
• Dead nodes
• Configuration
Copyright © HERE 2016
![Page 32: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/32.jpg)
Summary
![Page 33: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/33.jpg)
3333
Plugins
• S3 Plugin
• EC2 Plugin
• Build Failure Analyzer Plugin
• Groovy Event Listener Plugin
• Hierarchy Killer Plugin
Copyright © HERE 2016
![Page 34: Running tests for every commit: Gerrit, Jenkins, Docker, AWS](https://reader030.vdocuments.us/reader030/viewer/2022021502/5883041e1a28abe70d8b60ff/html5/thumbnails/34.jpg)
3434
Summary
• Automate automatization
• Be strong
• Visibility
• Relax&enjoy
Copyright © HERE 2016