runs speeding up jenkins test - static.spiceworks.com€¦ · test:unit test:functional...
TRANSCRIPT
![Page 1: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/1.jpg)
Blake Lusenhop & Bill Creager
Speeding up Jenkins Test Runs
![Page 2: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/2.jpg)
Improving Test Automation
● Eventually codebase will become too large to run sequentially
● Releases/patches/hotfixes are all time sensitive
● Test has a greater responsibility to turn around finished code in a timely manner
● Must ensure that test results are accurate
![Page 3: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/3.jpg)
Common Problems● Slow execution time
○ ex: 120m + regressions
● Unreliable test results○ Timing issues can cause test results to be
inconsistent
![Page 4: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/4.jpg)
The Solution: Parallel Test Execution● Parallel test execution can decrease your
regression length drastically● Local parallelization vs Multi-System Setups
○ CPU core based vs Machine/VM based● Local Example: parallel_tests gem
○ https://github.com/grosser/parallel_tests○ Run separate process on each core: i5 could run up
to 4 simultaneous processes (2 physical and 2 virtual cores)
![Page 5: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/5.jpg)
Parallel Test Considerations● Each individual test case must be able to be
run independently of other tests○ Tests are run randomly○ No tests that require pre-reqs to be run
● Be conscious of what tests are hitting what areas of the application○ Possibility of multiple tests trying to modify the same
data at the same time○ If possible, group tests by feature of the app so only
one test will hit that area at a time
![Page 6: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/6.jpg)
Parallel Test Considerations Cont.● Do certain tests require test data to be
loaded before execution?● Do other tests require a clean DB install?● Think about what your individual tests
require and group them accordingly○ Example: testdb regression that uses a stock DB
when run vs non_testdb which uses a clean DB install
○ Use tags on your tests so that you can execute tests in groupings that you create
![Page 7: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/7.jpg)
Using Reruns for Better Test Results● Automation can be very inconsistent ● Consider automatically rerunning failed tests
multiple times before logging a failed result● Prevents having to run an entire regression
just because of a single timing error
![Page 8: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/8.jpg)
Log Individual Failures to Reduce Rerun Time
● Log each failing test● Gather the failures and pass back in as a
parameter for next test run● Re-run only failed examples
![Page 9: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/9.jpg)
![Page 10: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/10.jpg)
Test:Unit
Test:Functional
Test:Engines
Test:Integration
Jenkins: Slave
Jenk
ins:
Mas
ter
Problems● Physical constraints, always on● HUGE spinup time ● All or nothing ● Dirty data (database, indexes, file
system, cache)● Impossible to run locally (devs)
Traditional Jenkins Setup
![Page 11: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/11.jpg)
Docker
● Acts as a slave to Jenkins● Fully baked environment
○ Postgres, Redis, Elasticsearch○ Code, git, jruby○ EVERYTHING!!!
● Multiple (jailed) copies can be provisioned● Disposable, portable, consistent containers
![Page 12: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/12.jpg)
Dockerized Jenkins Setup
Test:Unit
Docker: Slave
Test:Functional
Docker: Slave
Test:Integration
Docker: Slave
Test:Engines
Docker: Slave
Jenk
ins:
Mas
ter
Docker solutions● Slaves spin up when needed, and
spin down when jobs complete● Short turnaround● Choose your own adventure● Run locally● CLEAN data
![Page 13: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/13.jpg)
![Page 14: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/14.jpg)
Quick Docker tips● VM vs Container● Script it, and zip it!
○ Dockerfile○ build.sh○ setup.sh
● Mount your data● Save your progress● Supervisord for multiple runnables● Clean up, and watch your cache!
![Page 15: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/15.jpg)
Examples - build.sh, supervisord, Dockerfile
![Page 16: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/16.jpg)
![Page 17: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/17.jpg)
Examples - clean_instances, clean_images
![Page 18: Runs Speeding up Jenkins Test - static.spiceworks.com€¦ · Test:Unit Test:Functional Test:Engines Test:Integration Jenkins: Slave Jenkins: Master Problems Physical constraints,](https://reader033.vdocuments.us/reader033/viewer/2022060212/5f0517857e708231d41138bc/html5/thumbnails/18.jpg)
Questions?