wojciech sliwinski [email protected] beams department, controls group cern

Wojciech Sliwinski [email protected] Beams Department, Controls Group CERN

Outline Defining Software Quality Continuous Improvement Context Accelerator Controls Codebase Applying QA & CI for the Controls Codebase Conclusions 2Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN24th October 2011

Software Quality (SWQ) We know its a good thing People think I know it when I see it I know the value of it But what are the actual characteristics of SWQ? Do we have a common understanding of SWQ? The lack of it normally appears as consequences 424th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

1985: Therac-25 Software issues Radiation Deaths linked to Software Errors Killed 2 people Injured dozens of others http://www.ccnr.org/fatal_dose.html 524th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

1996: Ariane 5 floating point conversion error 624th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

2007: Airbus A340 Software issues 724th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

80s The Kano Model (Marketing) Customer Satisfaction Product Features Basic Needs Met Linear Performance Delight, Surprise, Innovate 24th October 20118Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN Satisfied Dissatisfied Needs well fulfilled Needs not fulfilled

Early and Other Quality Definitions Zen and the art of motorcycle maintenance, Robert Pirsig, 1974: If quality exists in an object, then you must explain why scientific instruments are unable to detect it On the other hand, if quality is subjective, [existing only in the eye of the observer] then this Quality is just a fancy name for whatever you'd like. Quality is not objective. It doesn't reside in the material world [...] Quality is not subjective. It doesn't reside merely in the mind. McCall et al, 1977 and Boehm et al, 1978 First elaborate studies on the notion of 'software quality Quality factors only indirectly measurable like reliability Quality criteria objective aspects that support the factors, like uptime. 924th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

IEEE SWQ Standards IEEE Glossary of Software Engineering Terminology defines it as the degree to which a system, component, or process meets customer or user needs or expectations IEEE Std 1061-1992 IEEE Standard for a Software Quality Metrics Methodology Does not actually prescribe any metrics! Defines method to evaluate metrics that could be applicable to a particular case Plenty more eg. IEEE Std 730 and friends 1024th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

ISO Quality Standards ISO 9000 series of standards for quality management systems: Process based the degree to which a set of inherent characteristics fulfils the requirements ISO 9126 Software Quality Characteristics and Metrics, 2001 Comprehensive but large & complex 6 key quality attributes: Functionality, Reliability, Usability Efficiency, Maintainability, Portability 1124th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

SWQ Huge Set of Characteristics 12 Only a subset applies to all projects Choose those that suit the project goals 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

McCalls categories of SWQ factors Source: J.A. McCall, P.K. Richards and G.F. Walters, Factors in Software Quality, RADC-TR-77-369, 1977, US Dep. of Commerce. 13 Operation: CorrectnessDoes it do what I want? ReliabilityDoes it do it accurately all of the time? EfficiencyWill it run on my hardware as well as it can? IntegrityIs it secure? UsabilityCan I run it? Revision: MaintainabilityCan I fix it? TestabilityCan I test it? FlexibilityCan I change it? Transition: PortabilityWill I be able to use it on another machine? ReusabilityWill I be able to reuse some of the software? InteroperabilityWill I be able to interface it with another system? 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

Internal & External Quality Attributes 14 Important Primarily to UsersImportant Primarily to Developers AvailabilityMaintainability EfficiencyPortability FlexibilityReusability IntegrityTestability Interoperability Reliability Robustness Usability 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

The SW Quality Iceberg 1524th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

The Maintainability Index (MI) ParameterNameMeasures aveVAverage Halstead complexityComputational density aveV (g)Average extended cyclomatic complexityLogical complexity aveLOCAverage count of lines of codeCode size PerCMAverage percent of lines of commentsHuman insight 16 MI = 171 5.2 x ln(aveV) - 0.23 x aveV(g) - 16.2 x ln(aveLOC) + 50 x sin 2.4PerCM 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

MI applied to FreeBSD codebase 1724th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN Kernel CodebaseUser Programs Modules with MI < 40 are rejected !

Applying Quality Computers dont care about quality! Metrics are tests for quality characteristics Quality clusters Testing is ensuring a quality of conformance Bugs occur in clusters too! Like testing, if quality is low, builds should fail Quality characteristics must be part of the whole development lifecycle Quality products only come from quality requirements, quality design, quality code, 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN18

Applying Quality (Assurance) Software Quality consists of: 1. Engineering Methods (proper analysis, design, ) 2. Standards and Procedures (common to all) 3. Technical Reviews (eg. code reviews) 4. Configuration management (repositories, versioning) 5. Measurement (code, metrics) 6. Testing (unit, system, integration, acceptance) 7. Improvement (of the items above) 1924th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN Quality is a way of life If you think Quality is expensive, try an accident

Capability Maturity Model (CMM) Levels * 1. Initial (chaotic, ad hoc, individual heroics) the starting point for use of a new process. 2. Repeatable some processes are repeatable, possibly with consistent results 3. Defined the process is defined/confirmed as a standard business process, and decomposed to levels 1 and 2 4. Managed (quantatively) using process metrics, management can identify ways to adjust and adapt the process to particular projects 5. Optimized process management includes deliberate process optimization/improvement. * Capability Maturity Model from SE Institute. 2024th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

Continuous Improvement (CI) Origins Demings Plan-Do-Check-Act (PDCA) The Toyota Production System Stop the line quality control About eliminating waste (obstacles) A learning, non-judgmental whole team approach Adopted by the Agile movement for SWD Caveat: Development is an empirical process Introduced when an organization recognizes it has arrived at some impasse 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN22

CI - How it works People (at all levels) freely discuss the information, issues, ideas, evaluate them, choose, plan and execute improvements Focuses on how things get done: Standardized (and ing) common tasks Only fixing possible issues e.g.: repeated mistakes, time sinks, quality holes Very suitable where tools, technology, external outcomes and requirements change often 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN23

CI - What it relies on Objective information: To evaluate current situation and processes Assess applied improvements Judge outcome as worthwhile, or retry Demonstrate added value of CI against overhead Improvement culture: Free speech everyones ideas are welcome A real desire to try it and improve Accepting there will not be total agreement Replacing skeptical inaction with real experimentation 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN24

CI It can fail! No one owns responsibility for implementation No charter to make changes No time allocated for improvement No follow up by team or sponsors 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN25

Improvement Practices Change Agents AKA: Coaches, Monitors, Consultants, Champions One or more floating people searching for improvements Kaizen time aka renovation days Team plans some improvements All actions executed by all at same time Reflection workshops Regular meetings to decide on improvements Apply one or more ideas Keep what works 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN26

CI - Summary Team driven incremental changes Team evolves and improves together Drives increase in quality, efficiency and satisfaction Not a tool or technique it is an attitude Not leader oriented, instead peer based consensus Keeps us ahead in best practice and professionalism 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN27

Bibliography Software Requirements K. Wiegers, Microsoft Press, 2003 Code Complete: A Practical Handbook of Software Construction S. McConnell, Microsoft Press, 2004 Software Engineering: Principles and Practice H. van Vliet, John Wiley & Sons, 2008 Wikipedia page on SWQ ISO standard 9126. 2824th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

Bibliography The Toyota Way, J. K. Liker, 2004 Agile and Iterative Development, C. Larman, 2004 Sustainable Software Development, Kevin Tate, 2006 Implementing Lean Software Development, M. & T. Poppendick, 2007 Rapid Development, S. McConnell, 1996 The Enterprise and Scrum, K. Schwaber, 2007 Collaboration Explained, J. Tabaka, 2006 Peopleware Papers: The Human side of Software, L. Constantine, 2001 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN29

Context - CERN Accelerator Complex 24th October 201131Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

Context - The CERN Control Centre 24th October 201132Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

Controls Software Infrastructure Client tier Server tier Resource tier CTRL DB Business Layer Device Layer Presentation Layer CMW A 3-tier architecture Presentation Layer Graphical interactive applications Fixed Displays Business Layer General purpose and specific Application servers Device Layer Front End Device servers Communication to the equipment goes through Controls MiddleWare CMW 24th October 201133Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

Controls Software Codebase - Situation Complex Accelerator domain ~6 million LOC Java/J2EE ~200 developers (CERN + other labs) ~1000 projects/modules C/C++ ~35 developers (CERN + other labs) +20 major projects Oracle database Eclipse IDE + plugins SVN + Maven + CommonBuild( Atlassian Suite (JIRA, Fisheye, Crucible, etc.) Nowadays encourage Scrum Focused on Software Quality 34Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN24th October 2011

Controls Software Codebase - Situation Lots of Code ~6 million lines, 20000+ classes. and still growing 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN35

Situation Self Assessment and Outlook Unknown quality Quality unchecked No mentoring or guidance? Limited standards High Staff turnover Students, Project Associates, Fellows More projects coming (PS renovation, ) More service provision (looking after running services) Will this cause future problems like maintenance? 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN36

Objectives improve Software Quality Introduce quality improvement as an integral part of the everyday development work Leverage tools to automate the process as much as possible Establish guidelines and metrics to measure progress Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN3824th October 2011

Quality in the Development Cycle For each stage Agreed mandatory activities and deliverables Tools identified and integrated with IDE (Eclipse), giving immediate feedback Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN39 Design Implement, Test, Document Deploy, Maintain 24th October 2011

Design Reviews Design meetings with external members To verify the soundness of design in an early stage To identify overlapping functionality Promote a set of common components and libraries Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN40 At the level of sub- components, main classes and design patterns UML: class and sequence diagrams Design Implement, Test, Document Deploy, Maintain 24th October 2011

Code Reviews & Code Analysis Goal: Identify defects Ensure maintainable code Verify conventions are followed Static code analysis tools to identify common mistakes and bug patterns: FindBugs Checkstyle Eclipse warnings Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN41 Person-to-person time consuming Only for critical code Mentoring of junior developers Light-weight person-to- person code reviews with FishEye + Crucible Design Implement, Test, Document Deploy, Maintain 24th October 2011

Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN42 Files being reviewed Author and reviewers Comments inline 24th October 2011

Reviewing Code (Crucible) 43

Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN44 A list of bugs The bug line indicated Bug explained 24th October 2011

Testing & Continuous Integration General agreement on benefits of testing and CI To increase level of testing, unit tests mandatory deliverable of each project >30% test coverage for non-trivial classes, measured with Clover Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN45 To discover inter-project issues early, Continuous Builds with Bamboo: CI server from Atlassian Compiles and runs unit tests Builds fail on: Low test coverage Basic PMD rules Design Implement, Test, Document Deploy, Maintain 24th October 2011

Continuous Integration and Test (Bamboo) 46 Test coverage for project Code metrics Classes with highest risk

Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN47 Red = not covered Green= covered 24th October 2011

Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN48 Triggered by changes in a dependency accsoft- commons- core accsoft- common s-util accsoft- commons- concentration accsoft- commons- io Top/flop list generated (Work in progress) 24th October 2011

Integration and System Testing with CO Testbed CO Testbed - mini-accelerator lab Hardware and Software Core components from the Accelerators Controls duplicated into this Test Environment System and Integration tests are executed by Bamboo CI server Automatic execution of tests with real system Automatic deployment of Release Candidates 4924th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

TIMING FEC03 FEC01 FEC02 FEC04FEC05 SERVER06 SERVER07 5024th October 2011 Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

Integration Tests done through Client APIs 51 Bamboo CI Server CMW Proxy CMW Directory Svc RBAC A1 Service Config DB Client APIs (JAPC, CMW, RBAC) Timing CMW, RBAC A2, FESA PPC/LynxOSLinux/i386 TESTS Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN JMS 24th October 2011

CO Testbed controlled by Bamboo CI server 24th October 201152Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

53 1. 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

54 1. 2. 24th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

5524th October 2011Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN

Challenges Think of and organize us as one big team, not many small ones The code belongs to the group, not to a project or an individual developer The guidelines and standards established and agreed by everyone Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN56 Challenges Agree on standards and configurationsAgree on standards and configurations 24th October 2011

Encourage developers to invest time on quality Quality-related tasks part of the yearly objectives for a project Progress measured against the level a project adheres to guidelines top/flop lists Focus the effort where the effect is highest, rely on tools as much as possible CI days common days with a common goal Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN57 Challenges 24th October 2011

Conclusions Integrated quality improvement in the development cycle Introduced guidelines/standards and supporting tools Increased developer motivation through Immediate feedback when developing Official tracking of progress and top/flop lists Increased awareness of benefits, application in individual projects and confidence when making changes Agile approach reflect, identify waste and improve Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN5824th October 2011

wojciech sliwinski [email protected] beams department, controls group cern

Documents