plugin testing

Plugin Testing

Thursday, October 22, 2009

Plugin TestingPlugins & other stu

ff


Why Test?

• Correctness

• Performance

• Knowing Your Code

• Documentation


Cost of Change

v1.0 v2.0 v3.0 v4.0

Tim

e

Development Feature Testing Regression Testing


Feedback Loops

Development

Testing

Testing

JIRA


Feedback LoopsDevelopment

Testing

Testing

JIRA


Cost of Change

v1.0 v2.0 v3.0 v4.0

Tim

e

Development Writing Tests Executing Tests


Eat Your VegetablesThursday, October 22, 2009

Variety

http://www.flickr.com/photos/sharontroy/3606495880/Thursday, October 22, 2009

http://www.flickr.com/photos/sharontroy/3606495880/


Ingredients Matter




Ingredients + Technique = Art




No Regretshttp://www.redflagdeals.com/forums/costco-west-deals-dove-nailer-lays-tuna-gain-pantene-734644/Thursday, October 22, 2009

http://www.redflagdeals.com/forums/costco-west-deals-dove-nailer-lays-tuna-gain-pantene-734644/

http://www.redflagdeals.com/forums/costco-west-deals-dove-nailer-lays-tuna-gain-pantene-734644/

Test-Driven Developmenthttp://www.agiledata.org/essays/tdd.htmlThursday, October 22, 2009

http://www.agiledata.org/essays/tdd.html

http://www.agiledata.org/essays/tdd.html

Types of Tests

Unit Integration


Unit Tests


What is Unit Testing?

• Tests individual components

• Avoids state & external systems

• Runs in development environment


Why Unit Test?

• Correctness

• Improve Design

• Design by Contract


How to Unit Test

• Set up

• Execute

• Verify

• Clean up


How to Unit Test an Atlassian Plugin


Unit Testing Tools


JUnit

public class UriTest { @Test public void absoluteUriParamIsReturned() { URI absoluteUri = URI.create("http://www.example.com"); URI resolvedUri = resolveUriAgainstBase( "http://localhost:8080", absoluteUri) assertEquals(absoluteUri, resolvedUri); }}


http://www.example.com

http://www.example.com

http://localhost:8080

http://localhost:8080

JUnit

@Test(expected= GadgetSpecUriNotAllowedException.class)public void blankLocationThrowsException() { urlBuilder.parseGadgetSpecUrl( BASE_GADGET_SPEC_PATH + PLUGIN_KEY + "/" + "");}


MockitoDashboard dashboard = mock(Dashboard.class);DashboardStore store = mock(DashboardStore.class);when(dashboard.getId()) .thenReturn(DASHBOARD_ID);when(store.update(dashboard)) .thenReturn(DashboardStore.SUCCESS);

repository.save(dashboard);

verify(store) .update(DASHBOARD_ID, dashboard);verifyNoMoreInteractions(store);


Hamcrest

Iterable<ColumnIndex> range = ColumnIndex.range(ColumnIndex.ZERO, ColumnIndex.ONE);Iterator<ColumnIndex> columnIt = range.iterator();assertThat(columnIt.next(), is(equalTo(ColumnIndex.ZERO)));assertThat(columnIt.next(), is(equalTo(ColumnIndex.ONE)));assertFalse(columnIt.hasNext());


Hamcrest

assertThat(iterator.next().getState(), is(sameInstance(gadget.getState())));

assertThat(store.entries(), hasItem(GADGET_SPEC_URI));

assertThat(staticContentTab.asText(), containsString("Static Content"));

assertThat(changesList.get(1), is(deeplyEqualTo( new RemoveGadgetChange(gadgetId))));


Clover


Unit Testing Traps

• False Positives

• False Negatives

• Flapping


Non-Test

• Doesn’t test what it appears to test

• No assertion

• Passes when code is broken

• Solutions:

• Test-Driven Development

• IDE inspections, PMD, FindBugs


Missing Test

• Doesn’t test full range of inputs

• Boundary conditions, special cases


• Solutions:

• Clover

• JUnit Theories


Coverage as a Crutch

• Over-reliance on Clover

• Misses code paths, boundary cases


• Solutions:

• Your brain

• Peer review


Side-Effects

• Produces or relies on side-effects

• Maintains state between runs

• Fails inconsistently

• Solutions:

• Mock objects

• @Before/@After


Over Constraint

• Tests implementation details

• Overuse of mocks

• Fails when code is not broken

• Solutions:

• Stubs/fakes

• Refactoring


Unit Testing Traps

• False Positives

• False Negatives

• Flapping

←More tests

←Reduce coupling

←Isolate


Integration Tests


What is Integration Testing?

• Tests full system

• Tests each supported configuration

• Runs in simulated production environment

• Many types: functional, load, stress, etc.


Why Integration Test?

• Correctness

• Robustness

• Performance


• Set up

• Execute

• Verify

• Clean up

How to Integration Test


How to Integration Test a Web Application

• Deploy your plugin or application

• Load data

• Simulate or script a web browser

• Parse the HTML to verify results

• Clean up


How to Integration Test anAtlassian Plugin


Integration Testing Tools


JWebUnit

beginAt("/home");clickLink("login");assertTitleEquals("Login");setTextField("username", "test");setTextField("password", "test123");submit();assertTitleEquals("Welcome, test!");


HtmlUnit

WebClient webClient = new WebClient();HtmlPage page = webClient.getPage("http://example.net");HtmlDivision div = page.getHtmlElementById("some_div_id");HtmlAnchor anchor = page.getAnchorByName("anchor_name");List<?> divs = page.getByXPath("//div");HtmlDivision div = page.getByXPath("//div[@name='John']") .get(0);


http://htmlunit.sourceforge.net

http://htmlunit.sourceforge.net

JIRA/Confluence Test Frameworks

PageHelper helper = getPageHelper();

helper.setSpaceKey(spaceKey);helper.setParentId(parentId);helper.setTitle(title);helper.setContent(content);helper.setCreationDate(new Date());helper.setLabels(labels);assertTrue(helper.create());


JIRA/Confluence Test Frameworks

addUser("john.doe");addUserToGroup("john.doe", JIRA_DEV);activateTimeTracking();String issueKey = addIssue(TEST_PROJECT_NAME, TEST_PROJECT_KEY, "Bug", "First Bug", "Major", null, null, null, ADMIN_USERNAME, null, null, "1w", null, null);logWorkOnIssue(issueKey,"1d");gotoProjectBrowse(TEST_PROJECT_KEY);


Atlassian Plugin SDK

$ atlas-unit-test$ atlas-clover$ atlas-integration-test--version--container--product


Selenium


Integration Testing Traps

• False Positives

• False Negatives

• Flapping


Incomplete• Doesn’t test the full system

• JavaScript disabled, ignores browsers or app servers


• Solutions:

• HtmlUnit

• Selenium

• Cargo

• atlas-integration-test --container


Entangled• Each test covers too much functionality

• Requires lots of setup to test each feature

• Bugs cause failures in tests for other features

• Solutions:

• Data loaders

• Remote APIs

• More specific test methods

• Domain-driven test frameworks


Fragile

• Tests break due to UI changes

• Screen-scraping, complicated XPath

• Fails when not broken

• Solutions:

• Semantic markup (class and id attributes)

• Domain-driven test frameworks


Timing or State Dependencies

• Inconsistent results from one test run to another

• Concurrency bugs, no clean up, environment issues

• Solutions:

• @Before/@After

• @BeforeClass/@AfterClass

• Continuous Integration

• Blood, sweat, and tears


Slow

• Tests take too long to run

• Increases length of feedback loop

• Solutions:

• Clover test optimization

• Maven profiles

• Bamboo build trees


Build TreesClover-Optimized

Unit Tests

All Unit TestsJava 5

IntegrationJava 5/Tomcat 6

IntegrationJava 5JBoss

All Unit TestsJava 6

IntegrationJava 5

Tomcat 5.5

IntegrationJava 6/Tomcat 6

IntegrationJava 6JBoss

IntegrationJava 6

Tomcat 5.5

Faster

Slower


• False Positives

• False Negatives

• Flapping

←Test more

←Framework

←Get to work

Integration Testing Traps


Resourceshttp://j.mp/plugin-testing


http://j.mp/plugin-testing

http://j.mp/plugin-testing

plugin testing

Technology