testing automaton - cfsummit 2016
TRANSCRIPT
Testing AutomatonLuis Majano @lmajano
Who am I
• Luis Majano
• Computer Engineer
• Imported from El Salvador
• CEO of Ortus Solutions
• Creator of all things Box
Why people don’t testWhy we need to test
How to testHow to automate
Where to start?
Write Tests
WHY PEOPLE DON’T TEST
COMFORT
WHY PEOPLE DON’T TEST
New MethodologyNew Learned Behavior
It is a leap….
BIGGEST LIE IN SOFTWARE DEV
Don’t worry, we will create the
tests and refactor it later!
Bugs cost money
Cost To Fix Time detected Requirements Design Building Testing Post-Release
Time Introduced
Requirements 1x 3x 5-10x 10x 10-100x
Design --- 1x 10x 15x 25-100x
Building -- -- 1x 10x 10-25x
^ Kaner, Cem; James Bach, Bret Pettichord (2001). Lessons Learned in Software Testing: A Context-Driven Approach. Wiley. p. 4. ISBN 0-471-08112-4.
• Just do it!
• You will get dirty
• It can hurt (a little)
• Learned behavior
NO MORE EXCUSES
IT WILL ACCELERATE YOUR DEVELOPMENT
TDD/BDD=ProcessContinuousIntegrationContinuousDelivery
ContinuousImprovement
AUTOMATION
TestBox : What you get
API Docs
Core
Test Browser
Test Harness
Test Runner
Samples
install testbox —saveDev
install testbox@be —saveDev
Test Harness
Automated test results!
xUnit/BDD Test Bundles
Harness bootstrap
HTML Runner
ANT Runner
Test Bundle CFCs• Inherits from testbox.system.BaseSpec
•TesBox will then execute all tests/specs/suites within 1 or more bundles
RUN TESTS
• Execute bundles via the URL• http://localhost/tests/bundle.cfc?method=runRemote
• SOAP Runner• HTTP/REST Runner• ANT Runner• NodeJS Runner• CommandBox Runner• Custom Runner
TESTBOX REPORTERS
•ANTJunit : Variant of JUnit for Jenkins•Codexwiki : Mediawiki syntax•Console : Server console•Doc : Semantic HTML•Dot : Awesome Dots•JSON•JUnit•Raw : CFML Struct•Simple : A basic HTML reporter•Text : Back to the 80’s, Text•MinText : Great for CommandBox Runners•XML •Tap : A test anything protocol reporter•Min : Minimalistic Report
MANUAL TESTING
BundleCFC Runner
CommandBox Runnerpackage set testbox.runner = "http://127.0.0.1:57538/tests/runner.cfm"
testbox run
1
2
{ "runner": "http://127.0.0.1:57538/tests/runner.cfm", "directory": "/tests/specs", "recurse": true}
NodeJS Test Runnernpm install -g testbox-runner
create .testbox-runnerrc
1
2
Run it : testbox-runner2
ANT TESTING
DEVELOPER AUTOMATION
1.Grunt Watchers + LiveReload2.Grunt Watchers + testbox-runner3.Grunt Watchers + commanbox4.ColdBox Elixir
CONTINOUS INTEGRATION
CI SERVERS
Jenkins + CommandBoxwget http://mirrors.jenkins-ci.org/war-stable/latest/jenkins.war
box server start warpath=jenkins.war
1
2
Travis CI - Sign Uphttps://travis-ci.org/
Sign Up
1
2
Accounts
Activate For Repositories
1
2
Create .travis.yml3
Commit to repository4
Travis CI - Getting Started
TRAVIS + ANTlanguage: javabranches: only: - developmentsudo: requireddist: trustybefore_install:- sudo apt-key adv --keyserver keys.gnupg.net --recv 6DA70622- sudo echo "deb http://downloads.ortussolutions.com/debs/noarch /" | sudo tee -a /etc/apt/sources.list.d/commandbox.listinstall:- sudo apt-get update && sudo apt-get --assume-yes install commandbox- box install- box server startbefore_script:# Startup the app- curl http://localhost:49616/script: - ant -f workbench/build.xml
TRAVIS + COMMANDBOXlanguage: javasudo: requireddist: trustybefore_install:- sudo apt-key adv --keyserver keys.gnupg.net --recv 6DA70622- sudo echo "deb http://downloads.ortussolutions.com/debs/noarch /" | sudo tee -a /etc/apt/sources.list.d/commandbox.listinstall:- sudo apt-get update && sudo apt-get --assume-yes install commandbox- cd tests- box install- box server startbefore_script:# Startup the app- curl http://localhost:49615/script: > testResults="echo $(box testbox run)"; echo "$testResults"; if grep -i "\[Failures: [1-9][0-9]\?[0-9]\?\]\|\[Errors: [1-9][0-9]\?[0-9]\?\]\|<t[^>]*>\|<b[^>]*>" <<< $testResults; then exit 1; fi
MULTIPLE VERSIONS
MULTIPLE ENGINES
TRAVIS + MULTIPLE ENGINESlanguage: javasudo: requiredjdk:- oraclejdk8env: matrix: - [email protected] - ENGINE=lucee@5 - ENGINE=adobe@2016 - ENGINE=adobe@11 - ENGINE=adobe@10before_install:- sudo apt-key adv --keyserver keys.gnupg.net --recv 6DA70622- sudo echo "deb http://downloads.ortussolutions.com/debs/noarch /" | sudo tee -a /etc/apt/sources.list.d/commandbox.listinstall:- sudo apt-get update && sudo apt-get --assume-yes install commandbox- box installbefore_script:- box server start cfengine=$ENGINE port=8500script: > testResults="$(box testbox run runner='http://127.0.0.1:8500/tests/runner.cfm' )"; echo "$testResults"; if grep -i "\[(Failures|Errors): [1-9][0-9]\?[0-9]\?\]\|\[Errors: [1-9][0-9]\?[0-9]\?\]\|<t[^>]*>\|<b[^>]*>" <<< $testResults; then exit 1; finotifications: email: false
Resources
• TestBox : ortussolutions.com/products/testbox
• CommandBox: ortussolutions.com/products/commandbox
• Slack: boxteam.herokuapp.com
• CFML Slack: #box-products
• travis-ci.org
• jenkins.io
Thank you!