bringing collaborative test to life an example of community effort

1

Bringing collaborative test to life:An example of community effort

Fred BlesserTechnology Manager – Symbian Foundation

Mariusz LasekBusiness Development Director – Comarch

Collaborative test

Collaborative Test consists of having the community contributing to the

validation and testing of the Symbian Foundation asset, with specific

focus on:

Feature development

Hardening

These tests

Shall be run on vanilla kits delivered by the Symbian Foundation to the

community

No differentiation

Shall be easily reproducible, without the need of specific equipment/tools,

except the ones available through the Symbian Foundation or open source

community

No discrimination

Test results shall be available for all to see and compare

2

Test working group update

Key current responsibilities:

Define the Symbian^3 test plan

Define the Symbian^4 test plan

Help the Release Council define the milestone acceptance criteria (proposals, etc)

Feature Complete

Stable

Participate in the validation and testing of Symbian^3

New members

Atelier

Symbian DevCo

Comarch

3

4

Comarch Introduction

5

Comarch established in 1993

Founder & CEO Prof. Janusz Filipiak

Comarch is the largest Polish Software House

Quoted on the Warsaw Stock Exchange (Public company)

since 1999

Headquartered in Kraków, Poland

Well diversified in order to embed business stability and security

Developing products and rendering services for

Telecommunications, Banking & Finance, Industry &Utilities, Public

Sector, Trade and Small & Medium Enterprises

Comarch Intro

6

Independent software vendor selling

proprietary products, solutions and services

on the global market

Strong direction towards matured markets of

Western Europe and the USA

Offices located in Europe, Americas and

APAC

Market Capitalization – 225 M€

Customers on 5 continents in more than 30 countries,

with over 3000 successfully completed projects

Comarch Today

7

Customer Mobile Services Application Development

Platform Development

Customization and Variant Creation

User Experience Design

Customer Mobile Applications Comarch Drafter

Comarch VOM (Targeted Adds)

Comarch Mobile Insurance

Comarch Mobile Banking

Customers OEM

ODM

Telco operators

Banking

Public Sector

Retailers

8

Comarch Contribution

Events and Marketing

World Mobile Congress in Barcelona

Towel Day 2010 in Turku, Helsinki

Educating

Training Contribution

Bug Fixes

Code Contribution

Headset Simulator

Collaborative Test

Database

9

Bluetooth Headsets Simulator

Business case

Existing client – Mobile Services for OEMs

Code contribution

Services for new clients based on contributed solution

Consulting

Customization

Simulator controlled remotely by API and UI

Test automation support

Modular architecture

New profiles support

New platforms support

10

Controlclient

Test ModulePlug-in

Controlserver


11


Headsets Simulator

Package: /btservices/bluetoothengine/

/*

* Component Name: Headset Simulator

* Author: Comarch S.A.

* Version: 1.0

* Copyright (c) 2010 Comarch S.A.

* /

Code size: 30 kloc

Effort: 9 Man-Months

Current Status: Architectural Review

12

SYMBIAN^3 TEST PLAN

Symbian^3 system test plan

System test cases have been contributed by:

These test cases have now been prioritized: Priority 1: 32 test cases

Non-concurrent test cases; to be run with each kit

Priority 2: 152 test cases

All priority 1 test cases, plus some more, providing sufficient feature coverage

Will be part of FC criteria

Priority 3: 468 test cases

All non-concurrent test cases, plus some key concurrent ones

Will be part of Stable criteria

Priority 4: remaining test cases

To be run ad-hoc

13

Proposal for Symbian^3 stable milestone

Take the latest Symbian^3 PDK (3.0.3)

The community will run the tests on this vanilla PDK according to the test plan defined in previous slide

All priority 2 system tests cases

These results will constitute an intermediate milestone between the Feature Complete milestone and the Stable milestone

The Test Working Group will then have to analyze these results, and propose to the Release Council one of the acceptance criteria for the Stable milestone

All test results will be stored on the new tool contributed by Comarch: the Collaborative Test Database (CTD)

14

15

Collaborative Test Database (CTD) goes live during SEE’2010 event

New Symbian Package under Tools domain

Grzegorz Wachocki – Package Owner from Comarch

Project is hosted at Symbian.org

Source Code available

Contribution welcome

Defects managed by Symbian’s Bug Tracking

Manual available at Symbian’s Wiki

http://developer.symbian.org/wiki/CTD_User_Guide

Collaborative Test Database

16

CTD has been established to record and present the Symbian quality

Various PDK releases

Various test targets

CTD is to be used by community members

Community verifies and correct test results

Check-in / Check-out mechanism in order to improve community efficiency

Integration with Bug Tracking system


17

Every test result is confirmed by at least two members (users)

Single authority is not possible – cross-checking improves reliability of results

Once the test is check-out for the testing, the whole community is aware of that fact

Avoid overlapping in testing effort

Check-in of the final result doesn’t mean that it’s fixed for ever

Any user can change the verdict at any time – results always up to date & reflect the actual platform quality

Results and summaries available for everybody


18

Similarities to Test Management Tools

Recording Test Specifications– list of Test Cases

Grouping Test Cases based on the Domain & Sub-domain

Multimedia

Telephony

Kernel…

Verifying status of the each Test Case

Draft

Published

Un-published


19

What makes the CTD Different?

Designed for the purpose of collaboration -> use by the ecosystem in contradiction to regular test management tools

Build from scratches to reflect certain workflow typical for the Symbian community

Ad-hoc testing instead of test plans and sessions – test when you need and when you can

Open for everybody

The tool is open source

Results are freely available


20

Collaborative Test Database Demo

21

THE FUTURE OF COLLABORATIVE TEST

“1 person-day per month” Each test working group member should if possible allow one

person to run tests for one day, each month, in order to gather results

Not mandatory

... but each member can offer to contribute more

Effort project-managed through the Foundation, then transition to CTD for:

Organize the test case list

Dispatch test cases to avoid redundant results

Follow-up on execution and bugs raised

Gather and collate test results

Follow-up with the bug squad

This is an effort that is being asked jointly and regardless of company size

But this effort is small when compared to the benefit it can bring to the Symbian platform and the Symbian Community

22

23

CTD will continue to evolve with new features in the pipeline

Collaborative test is a concept that can be applied to any open source community

Open information sharing

Provides standard test specifications and tools

CTD offers a centralized collection point for results

Hopefully this is bound to continue in the new world organization as announced this week

WHAT NEXT???

24

Thank You

Mariusz [email protected]

Fred [email protected]