qa with canonical - ubuntuodm.ubuntu.com/uhs/2011/oem qa with canonical.pdf · 2011-12-12 · 3...
TRANSCRIPT
QA with CanonicalDecember 8th 2011
Ethan Chang
QA Engineer, PES
2 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
Agenda
● Overview
● Process Flow
● Tools Overview
3 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
QA Team Overview
Ubuntu QA
● Driven by a core of QA Professionals in Canonical
● Community supported with a network of thousands
● Beta user testing begins as soon as the repos are created
PES QA
● Designed around more formal SQA common practices
● Globally distributed with members in USA, Spain, France, Taiwan, and
Mainland China to maximize “up time”
Tools
● All tools used by Canonical QA and Community QA are open source
and all test cases are public
● Automated testing combined with manual repeatable tests and Beta
usability testing
4 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
Quality Advantages
Four QA passes
● Upstream QA
● Ubuntu QA
● PES QA
● Hardware Certification
Many “Eyes” on Ubuntu
● Over 100 bugs a day submitted against Ubuntu
● Over 1300 unique Ubuntu mailing list posts per day
● Over 300,000 bugs logged to date
● Over 100,000 bugs fixed released
5 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
Actual Ubuntu Testing Workflow
Spec Dev QA IntegrateBetaTest QA
Upstream
No
Yes
Ubuntu
Yes
No
CustomizeHardware
Enable ReleaseRelease Release
No
PES QA
Bug Fixes
Bug Fixes
6 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
PES QA Workflow
RequirementGartering
QA Test PlanTest CaseCreation
Test CaseExecution
Final TestReport
Review
Project WikiCertification
Bug Stats
Bug CreationBug Triage
7 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
PES Documentation
PES QA has produced a series of documents that provide more detailed
information on our process
● Test PlanningThe test plan will be used to verify and ensure the system meets its design specifications and other requirements
● Test ReportingThe test report will record all the test results and provide online review and office CSV file
● Test CasesPES QA uses Checkbox-editor to create test cases and save test cases on Launchpad via the Bzr tool
● Testing SOPAccording to Customer's requirement, we will provide SOP document for ODM/OEM QA to perform test and Canonical Tools
8 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
Customer QA
Customer QA
Customer QA
TL/PM
TL/PM
NewIncomplete Invalid
TL/PM
CQA
Confirmed
Fix Released
Fix Committed
Dev
In Progress
Won't Fix
Triaged
CQACQA
TL/PM
TL/PM
CQA
CQA
Tag: CQA verified Tag: Verified
CQA
Incomplete
Invalid
Bug Workflow
9 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
Tools Overview
Design Organize
Checkbox Editor GTK application for creating, editing and managing test cases for Checkbox framework. Developed and maintained by PES QA
Checkbox Testing framework for executing manual or automated tests and reporting their results. PES QA maintains its own branch
Lazybones installerApplet designed to make installation of OEM test frameworks easierCombines public and private PPAs and their dependent packagesDeveloped and maintained by PES QA
Install Execute
10 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
Tools Overview
Log Review Analyse
OEM ReportsCollection of advanced reports designed from customer requirementsGenerated automatically from OEM test frameworksDeveloped and maintained by PES QA
Bug Time TrackerApplication which collects defect information from LaunchpadGenerates trending reports based on collected dataDeveloped and maintained by PES QA
Report
OEM BugGTK applet for reporting defects to LaunchpadIncludes automatically collected data on the system under test Developed and maintained by PES QA
11 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
Checkbox Editor
Developed by PES QA GTK front endCreating, Edit, Organize test cases
Enforce standard templatesProduce customer facing documents BZR integration
Design
12 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
Checkbox Editor
All generated files are tracked by BZR Easily export test run configurations Tracks test dependencies and statistics
Executes tests, provides debug info More information on the QA WikiOrganize
13 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
PES QA Case Structure
Req Based Device Spec Customer
0
200
400
600
800
1000
1200
# of cases
Creation cases based on customer requirements
Creation of device specific cases
Integration of cases provided by customer
14 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
Lazybones
Installs all required packages Assistant interface guides the user Configures public and private PPAs Auto accepts PPA invitations Package list can be configured
Install
15 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
Checkbox
● Guided test runs● Automated test support● Manual test support● Pluggable Framework● Config file version control● Launchpad Lib support● GTK, CLI, and Urwid interface● HTML and Excel reports● Localized cases in Mandarin
Execute
16 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
OEM Bug
Log
● Uses apport.hookutils for automatic collection of system information● Uses assistant interface to guide the user through a decision tree● Developer feedback determines additional information to be collected● Automatically populates hardware information● Integrated into Checkbox and Stand-alone application
17 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
OEM Reports
Automatically Generated by Checkbox on the SUT
Tabbed report provides system specific information, test statistics, individual test case result detail, and logs Report
18 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
OEM Reports
Easily converted to common formats such as XLS, ODT, and CSV
Formatted to match the reporting requirements of our largest OEM partnersReview
19 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
Bug Time Tracker
Analyse
● Collects the history of every bug for a given project ● Displays bug statistics in table form● Allows the user to filter data on the same criteria● Plots filtered data● Figures can be saved to disk● Data can be exported to csv
20 Canonical Confidential Ubuntu Hardware Summit. December 8th 2011
The Future
Enhance
● Checkbox and Checkbox Editor merging to produce a single more flexible tool for managing test cases, runs, and results.
● Server based automated trend reporting for all projects using Launchpad Lib, and Certification.Canonical.com APIs
● Remote test execution with the integration of Jenkins into Checkbox
● Consolidated test result reporting with Platform QA for better integration of test efforts
● Full tool and test case support across all relevant architectures (x86, x86-64, ARM)
Thank you
Questions ?