Discussion on Testing

Simantics Technical Board Meeting30.1.2012

Organizing Test Suites

• Suggestion: new plug-in org.simantics.tests– Contains test suites for different testing activities:• Regression• TDD• Performance• Stress

– Composes test suites for the SDK from around the codebase, doesn’t contain test cases

– Most suites are automatically tested via Jenkins

Testing Activities

• Regression– First step is to move all currently existing and passing tests into

regression tests• TDD

– Development-time, moved into regression when development is complete, manually executed

• Performance/Scalability– Focus on performance-critical parts of the SDK right now:

databoard, DB, graphics, history• Stress

– For uncovering hidden unstabilities, mainly relevant for DB

Tools to Incorporate

• Squish (finally)• EMMA (code coverage for unit tests)• Possibly static analysis tools

Static Analysis

• Several (free) tools available for Java [1] that have Eclipse & Jenkins plug-ins :– PMD, FindBugs, Checkstyle

• AFAIK, none of us has been actively using any of these with Simantics

• Relatively cheap to automate• Need to test the tools first to say anything

definitive of usefulness

What to Test?

• org.simantics.tests: everything that is part of the platform – not products built on top of it (sysdyn, modelica, etc.)

• A separate test suite for every product• UI testing – Difficult without a Workbench product– Use movie tutorial ?– Perhaps create the planned diagram editor for it ?

Reporting

• Jenkins provides unit/squish test reports• Performance testing needs different logging– Instrument all tested code with necessary

performance logging (See Eclipse tracing facility)– See about using Apache JMeter (Jenkins plug-in)

• For stress tests it is important to have proper logging enabled and relevant data kept for failed tests (databases, etc.)

Epics (prioritized)

• Create and automate org.simantics.tests• Sysdyn product test suite with UI tests• List most important testing topics• Coverage/Static analysis tools into use• Performance testing plan– How to measure? Relativity of results? Reporting/tools?

• Prevent publishing of builds with broken tests & ensure email is delivered about breakage– Try to force developers to react to acute problems

Working with tests and failure

• Regression tests divided into three suites:– 1st: only intended to contain tests that work. It is

important to know when something breaks, and it is easiest to discover this when builds turn from green to red/yellow or vice versa.

– 2nd: broken tests that have been moved from 1st if fixing the bug that caused the regression is not possible immediately.

– 3rd: broken tests that have no incoming fix for a long time. 2nd->3rd moves done only at the turns of sprints.

References• [1] http://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis#Java