Download - Interoperability Testing
Interoperability Testing
Work done so far
WSDL subgroup Generated Web Service Description with
aim for maximum interoperability between various SOAP stacks
WSDL tested against tooling supplied by JAX-RPC RI AXIS .NET
Basic and informal interop tests .NET <-> AXIS
WSDL subgroup (cont’d)
Released a subgroup report (Andre) Philosophy followed
e.g. choosing document/literal style Extension mechanism Found problems/concerns Raised bug reports against the stacks
AXIS begun work on all of them .NET reported fixes were in progress JAX-RPC RI has not responded yet
Fixes/workarounds reflected in the WSDL and interfaces
Interoperabilty
WSDL subgroup work Allows various stacks to generate
code from WSDL definitions Ensures interoperability on SOAP level
Need for interoperability on WSRP protocol level cross-vendor testing Compliance testing ?
Cross-vendor testing
Release open source project which provides sample implementation
Allows vendors to test their Producers and Consumers against this implementation
Does not guerantee complianceVendors willing to make their Producers accessible to others for interop tests?
Allows others to test their Consumers In turn vendors return feedback on their Producer
implementations Requires similar (spec) level of implementation
Should be easy once version 1 is released
Still no compliance testing in a formal sense
Cross-vendor Testing - Matrix
Consumer/Producer
Vendor1
Vendor2
Vendor3
Vendor4
Vendor1
Vendor2
Vendor3
Vendor4
Compliance Testing
Testing a particular implementation for compliance with the specification.Compliance testing is strictly “black box” testing
Compliance Testing - Tasks
Develop a set of testable assertions which represent the specification
Derive testable assertions from the specification Determine test cases implied by each assertion
Develop a Compliance Test Suite available for public download so that vendors and customers can verify that their implementations are compliant with the WSRP 1.0 Version of the spec
Design, develop, and test the tests for each assertion Create a test framework to run, evaluate, and report
on the tests
Deriving Testable Assertions
Significant Effort to recast the spec as assertionsRequires committee involvement
Test Suite Framework
A technical framework for running compliance tests.The framework shall be used to run, evaluate, and report test cases.Should be highly scriptable and configurable so test cases may be added easilyThis may be a significant effort by itself, depending on our ability to leverage existing frameworks
Testing Producers
Develop a Consumer which executes test cases against a Producer and reports the results.Test cases are configured as XML inputReports are generated as XML output
Testing Consumers
Develop a RI Producer conforms to the spec (I.e. passes all
the compliance tests) produces reports on the behavior of
the Consumer, if in error.
For example, validating the state that was supposed to be kept and returned by the Consumer
Testing - Roadmap
Form interop test subgroup Start interop tests February, 15th? Establish interoperability April, 15th
Form compliance test subgroup Draft assertion suite March, 15th Finalize assertion suite May, 1st
Interop
Mike AlanRichardAndreNigelSubbuGil(?)
Compliance
RossDanRich
Backup
Existing Examples
Java Community Process Calls for developing a Technology Compatibility Kit
(TCK) side by side with Reference Implementation Well developed process and tools
W3C The W3C standardization process only produces
specification documents (Recommendations, etc), and does not formally address compliance testing.
Apache Each project defines its own testing practices and
standards
WS-I Testing Working Group2
Deliverables The Test Tools Development Working Group
will produce the following deliverables: 1. WS-I Test Methodology White paper 2. WS-I Test Tools Specification 3. Two or more implementations of Test Tools and
supporting documentation 4. Assertions and Test Conditions used as input to
the Test Tools 5. WS-I Tool Configuration Template 6. WS-I Experience Report: Tool Development as a
component of WS-I Profile Development
2WS-I Testing Working Group Charter, http://www.ws-i.org/docs/charters/Test_Charter.pdf
Scope of the Task1
Derive testable assertions from the specificationDetermine test cases implied by each assertionDesign, develop, and test the tests for each assertionCreate a test framework to run, evaluate, and report on the tests
1TCK Project Planning and Development Guide, Sun Microsystems, Inc
Existing Frameworks
JUnit -- open source projectBaRT – Batch generator for Regression Tests -- IBM open source tool used for testing JVMs. Sun’s Compatibility Test Toolkit – licensed, limited to JCP projects
Testing Optional Functionality
Separate, or at least configurable, test suites for discrete levels of functionThe discrete levels of function currently mentioned are Simple and SophisticatedFiner distinctions may have to be madePerhaps the test suite can determine at runtime which level of function the Producer supports and be able to apply the correct tests.