20080622 bupt qa_opensource

Quality Assurance at Novell for Linux Products

Oliver Riesories@novell.comDirector OPS Quality Assurance

© Novell Inc. All rights reserved

2

Agenda

• How Open Source QA is different

• Aspects of OpenSource QA at Novell– Test Approach

– Test Environments

– Test Automation

– Maintenance Testing

• openSUSE & QA– future plans

© Novell Inc. All rights reserved

3

Software Testing as in the Books

Unit Testing

Integration Testing

System Testing

Component Testing

Stress Testing

Regression Testing

Acceptance Testing

Code

Product

© Novell Inc. All rights reserved

4

Multi Phase Test Approach

UNIT TEST

Autobuild

-Package dependencies-Correct libraries- Integrated tests and sanity checks

COMPONENT TEST

TIDB/Cerberus Harness

- Functional tests for all packages- Test Cases from PDB- Priority Based (CORE->ADDON)

FEATURE TEST

Code

Package

Feature Doc TIDB

- Manual test of all features includedin the feature document

SYSTEM TEST

Installed Systems Cerberus

- Stress and reliability tests- Industry test suites- Certification tests

REGRESSIONTESTS

TIDB/Cerberus

- Regression of all automated tests- Validation of fixed bugs- Regression of failed feature tests

automated manual/interactive

© Novell Inc. All rights reserved

5

Multi Phase Test Approach II

Beta 1

Beta 2

Beta 3

Beta 4

RC 1

RC 2

Unit Test ComponentTest

Packages

ComponentTest

SpecialFocus:

- Kernel- Network- SysMgmt- GUI

SystemTest

Stress - Reliability - Stability - Scalability

System test

Cert Tests

Feature Test

RegressionTest

Test Environment

© Novell Inc. All rights reserved

7

Overview of the Test Environment

PDB

Feature Document

Autobuild

TIDB /TESTLINK

EngineerTest Machines

KOTD(kernel of the day)test machines

i386 x86-64 IA64 s390PPC

i386 x86-64

QA LABNetwork

QA Kernel Laband ISV Test(Oracle, Veritas,...)

Reference Hosts

SLES9 SLES8 NLD SLOX

Test Automation

© Novell Inc. All rights reserved

9

Automation Architecture

CTCS

Apache testscript

Samba testscript

Bind testscript

Mount testscript

... testscript

Startssl

Httpd Smbd

Nmbd

Smbclient

Smbmount

named Mount /dev/nnn

mount -t nfs

App 1

App 2

App 3

Start Results

Application Layer

Test Control Files

"PASSED""FAILED"

start check and update

TXT, HTML, CSV, SQL, XML

create reports

Create reports

SQL db

...

Maintenance Update Testing

© Novell Inc. All rights reserved

11

MainCoord initiates SWAMPincl. Prio, etc.Patch is build and pkg. available

QA checks onhttp://w2d.suse.de/abuildstat/patch-statusneeded arch and products

MainCoord initiates SWAMPincl. Prio, etc.Patch is build and pkg. available

SLES 8i386

SLES 8s390

SLES 9x86_64

OESi386

SLES N<ARCH>

...

Update Test

Bug Fix Test

Feature Test

Update Test

Bug Fix Test

Feature Test

Update Test

Bug Fix Test

Feature Test

Update Test Update Test

Bug Fix Test

Feature Test

Collect ResultsTests passed ?

Approve Send report toqa@suse.de

Reject

ISV Tests ISV Tests ISV Tests ISV Tests

Alert TPMs, Sec Team,Support (NTS), TAMs, ISVs

NO

YES

Feature Testfor Type A/B

Feature Testby OES team for Type C

Bug Fix Test

?

openSUSE & QA

© Novell Inc. All rights reserved

13

How can the community contribute?

• bugzilla / testopia– main interface and used by Novell internally as well

• Novell is opensourcing test tools / infrastructure / documentation

– QA code will be available in the Build Service

– Build Service will be able to run tests after a build job is done

– easier integration for Software maintainers

• building a QA community– awards & prizes for the best QA contributors (reports, test

cases, ...)

• better cooperation between the community and Novell– QA community manager

Testopia (off-line) Demo

© Novell Inc. All rights reserved

15

© Novell Inc. All rights reserved

16

© Novell Inc. All rights reserved

17

© Novell Inc. All rights reserved

18

© Novell Inc. All rights reserved

19

© Novell Inc. All rights reserved

20

© Novell Inc. All rights reserved

21

© Novell Inc. All rights reserved

22

© Novell Inc. All rights reserved

23

© Novell Inc. All rights reserved

24

Questions, Feedback, Discussions

© Novell Inc. All rights reserved

26

Thank you for your attention

Unpublished Work of Novell, Inc. All Rights Reserved.This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.

General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for Novell products remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.