[xp2013] narrow down what to test
TRANSCRIPT
![Page 1: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/1.jpg)
Not sure if I have to check out this problem
Or just wait for the customer feedback
How to Narrow Down What to Test
@ZsoltFabokhttp://zsoltfabok.com/
#xp2013http://xp2013.org/
byZsolt Fabok2013-06-05
![Page 2: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/2.jpg)
@ZsoltFabokor
#xp2013
![Page 3: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/3.jpg)
Exercise 0: Setup
![Page 4: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/4.jpg)
Download putty.exe if you are using windows:
http://<host>:8080/xp2013/putty.exe
Log in (user, userpass):
ssh user@<host> -p 3022
Create your environment:
$ cd xp2013$ mkdir <yourname>$ cp -r arithmetic.expression.evaluator <yourname>$ cd <yourname>/arithmetic.expression.evaluator$ ant init
![Page 5: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/5.jpg)
Check the source directory in your browser:
http://<host>:8080/xp2013/<yourname>
![Page 6: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/6.jpg)
“I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence (I suspect this level of confidence is high compared to industry standards, but that could just be hubris). If I don’t typically make a kind of mistake (like setting the wrong variables in a constructor), I don’t test for it. I do tend to make sense of test errors, so I’m extra careful when I have logic with complicated conditionals. When coding on a team, I modify my strategy to carefully test code that we, collectively, tend to get wrong.”
Kent Beck - September 10, 2010
quote: http://stackoverflow.com/questions/153234/how-deep-are-your-unit-tests/153565#153565
![Page 7: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/7.jpg)
I’d like to [re]start working on this legacy application
![Page 8: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/8.jpg)
My “where to start” list
![Page 9: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/9.jpg)
#1 Determine which parts of the code are really used
![Page 10: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/10.jpg)
The good old Standish Group Study
7%
13%
16%19%
45%AlwaysOftenSometimesRarelyNever
![Page 11: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/11.jpg)
The goal is to find those features which are always or often used.
By studying coverage, access logs, traces, web analytics, heat maps, etc.
![Page 12: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/12.jpg)
Let’s have a look at the coverage (using instrumented class files):
% cp jetty/cobertura.ser web.ser% cp uploader/cobertura.ser ant.ser% ant usage_coverage
usage_coverage:[cobertura-merge] Cobertura: Loaded information on 12 classes.[cobertura-merge] Cobertura: Loaded information on 11 classes.[cobertura-merge] Cobertura: Saved information on 16 classes.[cobertura-report] Cobertura: Loaded information on 16 classes.[cobertura-report] Report time: 600ms
BUILD SUCCESSFULTotal time: 2 seconds
![Page 13: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/13.jpg)
Example #1: overview
![Page 14: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/14.jpg)
Example #2: execution
not even executed
![Page 15: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/15.jpg)
Example #3: number of execution
.NET wins
![Page 16: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/16.jpg)
FileBasedMetadata (usage)
FileHelper (usage)
![Page 17: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/17.jpg)
Exercise 1: Usage
![Page 18: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/18.jpg)
Run the application:$ run.sh “4+3”
Turn on data collection:
$ ant instrument$ vi run.sh
Run the application a couple of times:
$ run.sh “4+3”
Generate report:
$ ant usage_report
![Page 19: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/19.jpg)
#2 Find out which parts of the code change often
![Page 20: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/20.jpg)
By checking how many times a file has been committed into VCS:
% ./git_stat.sh
14, VerifierTask.java13, index.jsp11, FileBasedUserHome.java11, FileBasedUser.java11, FileBasedContentTracker.java 8, IntegrityCheckTask.java 7, MailSender.java
![Page 21: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/21.jpg)
FileBasedMetadata (usage)
FileHelper (usage)
VerifierTask (changes)
index.jsp (changes)
FileBasedUserHome (changes)
![Page 22: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/22.jpg)
Exercise 2: Have a beer (or tea of coffee) :-)
![Page 23: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/23.jpg)
#3 Determine which part of the code changes data
![Page 24: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/24.jpg)
Code review“You have exactly 1 minute to explain to me what that method does!”
![Page 25: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/25.jpg)
FileBasedMetadata (usage)
FileHelper (usage, review)
VerifierTask (changes)
index.jsp (changes)
FileBasedUserHome (changes, review)
FileBasedContentTracker (review)
![Page 26: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/26.jpg)
Exercise 3: Code Review
![Page 27: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/27.jpg)
Check the code in the reports directory in your browser:http://<host>:8080/xp2013/<yourname>/.../target/reports/src
![Page 28: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/28.jpg)
#4 Determine where the code is most likely going to fail(e.g. with static code checkers)
![Page 29: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/29.jpg)
![Page 30: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/30.jpg)
PMD is not helpful at the moment, but good to know about it
![Page 31: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/31.jpg)
![Page 32: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/32.jpg)
FileBasedMetadata (usage)
FileHelper (usage, review, bugs)
VerifierTask (changes)
index.jsp (changes)
FileBasedUserHome (changes, review)
FileBasedContentTracker (review, bugs)
HarversterTask (bugs)
FileBasedContentTracker.fsck() (crap4j)
FileBasedContentTracker.gc() (crap4j)
VerifierTask.execute() (crap4j)
![Page 33: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/33.jpg)
Let’s order our list and we are done!
![Page 34: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/34.jpg)
FileBasedMetadata (usage)
FileHelper (usage, review, bugs)
VerifierTask (changes)
index.jsp (changes)
FileBasedUserHome (changes, review)
FileBasedContentTracker (review, bugs)
HarversterTask (bugs)
FileBasedContentTracker.fsck() (crap4j)
FileBasedContentTracker.gc() (crap4j)
VerifierTask.execute() (crap4j)
![Page 35: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/35.jpg)
Exercise 4: Code Checkers
![Page 36: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/36.jpg)
Check the reports directory in your browser:
http://<host>/xp2013/<yourname>/.../target/reports
Generate reports:$ ant reports
![Page 37: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/37.jpg)
Now we know where to start, and now let’s talk about how to start.
![Page 38: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/38.jpg)
Gaining 30% coverage in 2 minutes:
public class CheaterTest { @Test public void shouldIncreaseTheCoverage() { HarvesterTask harvester = new HarvesterTask(); Project project = new Project(); project.setBaseDir(new File(".")); harvester.setProject(project); harvester.setRepository("../repository"); harvester.setHistory("history"); harvester.setTemplate("templates"); harvester.execute(); }}
Covered code != Tested code
![Page 39: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/39.jpg)
So, you start with an assertion:
public class FileHelperTest { @Test public void shouldReturnTheContentOfAFile() throws IOException { assertEquals("", FileHelper.getFileContent(null)); }}
➡ The ‘assertEquals’ makes sure that your test actually does something
➡ The ‘null’ parameter - along with the NullPointerException - will show you where to continue
![Page 40: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/40.jpg)
First test case is done:
public class FileHelperTest { @Test public void shouldReturnTheContentOfAFile() throws IOException { File input = File.createTempFile("foo", "bar"); assertEquals("", FileHelper.getFileContent(input)); }}
➡ Now the test is green, let’s continue with a more meaningful test case
![Page 41: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/41.jpg)
Now we have two test cases:
public class FileHelperTest { @Test public void shouldReturnTheContentOfAFile() throws IOException { File input = File.createTempFile("foo", "bar"); assertEquals("", FileHelper.getFileContent(input)); }
@Test public void shouldReturnTheContentOfAFile() throws IOException { File input = File.createTempFile("foo", "bar"); new FileOutputStream(input).write("something".getBytes()); assertEquals("something", FileHelper.getFileContent(input)); }}
➡ Test method names remains the same until the body is filled properly
![Page 42: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/42.jpg)
And we are done (assertion + coverage):public class FileHelperTest { private File input;
@Before public void setUp() throws IOException { input = File.createTempFile("foo", "bar"); }
@Test public void shouldReturnAnEmptyStringForAnEmptyFile() throws IOException { assertEquals("", FileHelper.getFileContent(input)); }
@Test public void shouldReturnTheContentOfAFile() throws IOException { setInputFileContent("something"); assertEquals("something", FileHelper.getFileContent(input)); }
private void setInputFileContent(String content) throws IOException { new FileOutputStream(input).write("something".getBytes()); }}
![Page 43: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/43.jpg)
Exercise 5: Writing tests
![Page 44: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/44.jpg)
Get test coverage (also runs tests):
$ ant cobertura
Run tests:$ ant test
If you don’t want to write tests, you can find examples in this folder:
samples
![Page 45: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/45.jpg)
Well tested code
Well tested code everywhere
![Page 46: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/46.jpg)
What about web applications?(I’ll use a ruby on rails example, but the principles apply to other frameworks as well)
![Page 47: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/47.jpg)
#2 Find out which parts of the code change often (a.k.a VCS statistics)
#3 Determine which part of the code changes data (a.k.a code review)
Points
are just the same.
and
![Page 48: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/48.jpg)
A large variety of tools are available for:
#4 Determine where the code is most likely going to fail (a.k.a static code checkers)
![Page 49: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/49.jpg)
% gem install rails_best_practices% rails_best_practices -html .
![Page 50: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/50.jpg)
![Page 51: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/51.jpg)
![Page 52: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/52.jpg)
Everything is nice and straightforward until now, but the last remaining point
is tricky:
#1 Determine which parts of the code are really used (a.k.a. coverage)
![Page 53: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/53.jpg)
We can have coverage data in Ruby on Rails, too:~/Temp/repos/sample_app % gem install simplecov~/Temp/repos/sample_app % cat script/rails #!/usr/bin/env ruby
require 'simplecov'SimpleCov.start do add_group "Models", "app/models" add_group "Controllers", "app/controllers"end
APP_PATH = File.expand_path('../../config/application', __FILE__)
# rest of the script/rails script
![Page 54: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/54.jpg)
![Page 55: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/55.jpg)
There is only one problem: the application must be stopped in order to get the report, which is not really efficient and user friendly.
![Page 56: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/56.jpg)
Fortunately, we can use a metric called ‘funnel’:
![Page 57: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/57.jpg)
![Page 58: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/58.jpg)
1. This is a huge lost in visitors, would be good to know why.
2. Users visit this page more often than the other page.
![Page 59: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/59.jpg)
Slides: http://zsoltfabok.com/speaking/
Code: https://github.com/ZsoltFabok/arithmetic.expression.evaluator/tree/xp2013/
![Page 60: [xp2013] Narrow Down What to Test](https://reader035.vdocuments.us/reader035/viewer/2022062903/589c7c051a28abc2258b5469/html5/thumbnails/60.jpg)
Thank you very much for your attention!
http://zsoltfabok.com/ @ZsoltFabok