plugin testing

Plugin Testing

Thursday, October 22, 2009

Plugin Testing

Plugin TestingPlugins & other stu

Why Test?

• Correctness

• Performance

• Knowing Your Code

• Documentation

Cost of Change

v1.0 v2.0 v3.0 v4.0

Development Feature Testing Regression Testing

Feedback Loops

Development

Testing

Feedback LoopsDevelopment

Testing

Feedback LoopsDevelopment

Testing

Cost of Change

v1.0 v2.0 v3.0 v4.0

Development Writing Tests Executing Tests

Eat Your VegetablesThursday, October 22, 2009

Variety

http://www.flickr.com/photos/sharontroy/3606495880/Thursday, October 22, 2009

Ingredients Matter

Ingredients + Technique = Art

No Regretshttp://www.redflagdeals.com/forums/costco-west-deals-dove-nailer-lays-tuna-gain-pantene-734644/Thursday, October 22, 2009

Test-Driven Developmenthttp://www.agiledata.org/essays/tdd.htmlThursday, October 22, 2009

Types of Tests

Unit Integration

Unit Tests

What is Unit Testing?

• Tests individual components

• Avoids state & external systems

• Runs in development environment

Why Unit Test?

• Correctness

• Improve Design

• Design by Contract

How to Unit Test

• Set up

• Execute

• Verify

• Clean up

How to Unit Test an Atlassian Plugin

Unit Testing Tools

public class UriTest { @Test public void absoluteUriParamIsReturned() { URI absoluteUri = URI.create("http://www.example.com"); URI resolvedUri = resolveUriAgainstBase( "http://localhost:8080", absoluteUri) assertEquals(absoluteUri, resolvedUri); }}

@Test(expected= GadgetSpecUriNotAllowedException.class)public void blankLocationThrowsException() { urlBuilder.parseGadgetSpecUrl( BASE_GADGET_SPEC_PATH + PLUGIN_KEY + "/" + "");}

MockitoDashboard dashboard = mock(Dashboard.class);DashboardStore store = mock(DashboardStore.class);when(dashboard.getId()) .thenReturn(DASHBOARD_ID);when(store.update(dashboard)) .thenReturn(DashboardStore.SUCCESS);

repository.save(dashboard);

verify(store) .update(DASHBOARD_ID, dashboard);verifyNoMoreInteractions(store);

Hamcrest

Iterable<ColumnIndex> range = ColumnIndex.range(ColumnIndex.ZERO, ColumnIndex.ONE);Iterator<ColumnIndex> columnIt = range.iterator();assertThat(columnIt.next(), is(equalTo(ColumnIndex.ZERO)));assertThat(columnIt.next(), is(equalTo(ColumnIndex.ONE)));assertFalse(columnIt.hasNext());

Hamcrest

assertThat(iterator.next().getState(), is(sameInstance(gadget.getState())));

assertThat(store.entries(), hasItem(GADGET_SPEC_URI));

assertThat(staticContentTab.asText(), containsString("Static Content"));

assertThat(changesList.get(1), is(deeplyEqualTo( new RemoveGadgetChange(gadgetId))));

Clover

Unit Testing Traps

• False Positives

• False Negatives

• Flapping

Non-Test

• Doesn’t test what it appears to test

• No assertion

• Passes when code is broken

• Solutions:

• Test-Driven Development

• IDE inspections, PMD, FindBugs

Missing Test

• Doesn’t test full range of inputs

• Boundary conditions, special cases

• Solutions:

• Clover

• JUnit Theories

Coverage as a Crutch

• Over-reliance on Clover

• Misses code paths, boundary cases

• Solutions:

• Your brain

• Peer review

Side-Effects

• Produces or relies on side-effects

• Maintains state between runs

• Fails inconsistently

• Solutions:

• Mock objects

• @Before/@After

Over Constraint

• Tests implementation details

• Overuse of mocks

• Fails when code is not broken

• Solutions:

• Stubs/fakes

• Refactoring

Unit Testing Traps

• False Positives

• False Negatives

• Flapping

←More tests

←Reduce coupling

←Isolate

Integration Tests

What is Integration Testing?

• Tests full system

• Tests each supported configuration

• Runs in simulated production environment

• Many types: functional, load, stress, etc.

Why Integration Test?

• Correctness

• Robustness

• Performance

• Set up

• Execute

• Verify

• Clean up

How to Integration Test

How to Integration Test a Web Application

• Deploy your plugin or application

• Load data

• Simulate or script a web browser

• Parse the HTML to verify results

• Clean up

How to Integration Test anAtlassian Plugin

Integration Testing Tools

JWebUnit

beginAt("/home");clickLink("login");assertTitleEquals("Login");setTextField("username", "test");setTextField("password", "test123");submit();assertTitleEquals("Welcome, test!");

HtmlUnit

WebClient webClient = new WebClient();HtmlPage page = webClient.getPage("http://example.net");HtmlDivision div = page.getHtmlElementById("some_div_id");HtmlAnchor anchor = page.getAnchorByName("anchor_name");List<?> divs = page.getByXPath("//div");HtmlDivision div = page.getByXPath("//div[@name='John']") .get(0);

JIRA/Confluence Test Frameworks

PageHelper helper = getPageHelper();

helper.setSpaceKey(spaceKey);helper.setParentId(parentId);helper.setTitle(title);helper.setContent(content);helper.setCreationDate(new Date());helper.setLabels(labels);assertTrue(helper.create());

JIRA/Confluence Test Frameworks

addUser("john.doe");addUserToGroup("john.doe", JIRA_DEV);activateTimeTracking();String issueKey = addIssue(TEST_PROJECT_NAME, TEST_PROJECT_KEY, "Bug", "First Bug", "Major", null, null, null, ADMIN_USERNAME, null, null, "1w", null, null);logWorkOnIssue(issueKey,"1d");gotoProjectBrowse(TEST_PROJECT_KEY);

Atlassian Plugin SDK

$ atlas-unit-test$ atlas-clover$ atlas-integration-test--version--container--product

Selenium

Integration Testing Traps

• False Positives

• False Negatives

• Flapping

Incomplete• Doesn’t test the full system

• JavaScript disabled, ignores browsers or app servers

• Solutions:

• HtmlUnit

• Selenium

• Cargo

• atlas-integration-test --container

Entangled• Each test covers too much functionality

• Requires lots of setup to test each feature

• Bugs cause failures in tests for other features

• Solutions:

• Data loaders

• Remote APIs

• More specific test methods

• Domain-driven test frameworks

Fragile

• Tests break due to UI changes

• Screen-scraping, complicated XPath

• Fails when not broken

• Solutions:

• Semantic markup (class and id attributes)

• Domain-driven test frameworks

Timing or State Dependencies

• Inconsistent results from one test run to another

• Concurrency bugs, no clean up, environment issues

• Solutions:

• @Before/@After

• @BeforeClass/@AfterClass

• Continuous Integration

• Blood, sweat, and tears

• Tests take too long to run

• Increases length of feedback loop

• Solutions:

• Clover test optimization

• Maven profiles

• Bamboo build trees

Build TreesClover-Optimized

Unit Tests

All Unit TestsJava 5

IntegrationJava 5/Tomcat 6

IntegrationJava 5JBoss

All Unit TestsJava 6

IntegrationJava 5

Tomcat 5.5

IntegrationJava 6/Tomcat 6

IntegrationJava 6JBoss

IntegrationJava 6

Tomcat 5.5

Faster

Slower

• False Positives

• False Negatives

• Flapping

←Test more

←Framework

←Get to work

Integration Testing Traps

Resourceshttp://j.mp/plugin-testing

plugin testing

Technology

wordpress plugin development plugin development (with...

shift4 plugin for magento 2x magento payment module user...

sig talk: quality & testing - testing tools...current...

in your ci api testing pipeline - continuous lifecycle …...

ironjacamar 1.2 developer's guide · • eclipse...

3.06 logo logon plugin exercisen plugin exercise

cypress visual testing - international javascript...

software engineering for android – security and user...

snakes in a plugin - wordpress plugin security

accelerating permutation testing in voxel-wise analysis...

fuel vmware dvs plugin testing documentation vmware dvs...

plugin safety check - how to ensure your plugin gets...

accelerating ui development · storyboard engine: task...

cad - dl.sariasan.comdl.sariasan.com/new softwares/dialux 1...

facebook regisrtration plugin (plugin)

blender plugin installation guide - amazon s3 · blender...

building and testing an automotive platform · ci builds -...

resources plugin

easyazon plugin - easyazon plugin review, easyazon plugin...

20180929 middleware performance testing final€¦ ·...