Download - [EN] Club Automation presentation "Quality Model for Industrial Automation", Nov. 22nd 2011
1
Quality Model for Industrial Automation
Safe design of control applications
Tuesday, November 22nd, 2011
Thierry COQ [email protected]
System and Software Reliability
Principal Consultant
Denis CHALON [email protected]
Technical Director
2 All rights reserved
Content
Software - apprehension or apprehension
Software quality in traditional computing
Application to automation
Real Case study – DNV Audit
Quality model’s thresholds for automation
Conclusions
3 All rights reserved
Software is everywhere
Increasingly complex applications - More variables, more I/Os, more treatments - Applications distributed over several PLCs
Replacing hardware functions by software features (more flexible, cheaper)
The development is mostly subcontracted
Re-use of already developed libraries
4 All rights reserved
Apprehension of software
Where is software? How is its integrity managed over its life span?
What is the quality delivered by our suppliers? How to ensure that suppliers are qualified?
What are the causes of software errors? How can we trust software corrections later in the project? And during the operation?
How to prevent delays?
5 All rights reserved
Methods Project Manager Automation Engineer
Stakeholders around software
Different jobs
Very different environments and tools
Knowledge hardly shared
Software is more difficult to grasp than mechanical and electrical controls
Client
Different levels of focus required:
400mm 120mm 14mm
6 All rights reserved
Methods
Projet Manager Automation Engineer
How is software perceived ?
Client
400mm fix
Quick, quick, Done before,
copy/paste
Unreadable, no test,
late
?
PLC 1 PLC 2 PLC 3
PLC 4 PLC 5 PLC 6
PLC 7 PLC 8 PLC 9 Always down, maintenance complains,
poor performance ?
Unfinished, complicated,
important
?
7 All rights reserved
Quick, quick, Done before, copy/paste...
Unfinished, complicated, Important…
Unreadable, no test,
late
Always down, maintenance complains,
average performance
Need to make software visible
Methods
Projet Manager
Automation Engineer
Client
Software must be measurable: - objectively - repeatedly
This measure must be shared by all stakeholders
8 All rights reserved
Software quality in traditional computing
What do computer specialists do to make software more visible? How to define software quality? How can we measure it? Does measuring really make sense?
9 All rights reserved
A brief history of software quality
1970's - Theory formalized by Mac Cabe
1980's - Available tools (eg. Logiscope)
1990's - Tools are mainly used for critical software
2000's - Democratization of the monitoring methods:
Automating data generation from source code
Simplifying the use of quality measurement tools (no need for specialist anymore)
Ergonomic user interfaces, tailored to different stakeholders
Standardization of concepts (ISO9126)
"If you can not measure it, you can not improve it."
(Lord Kelvin)
10 All rights reserved
Principle of a quality model
CMMI ISO9126
code complexity
reusability
testability
reliability
scalability
efficiency
maintainability
coupling
exceptions handling
fault tolerance
comprehensibility
readability
ergonomics
performance
operational reliability
features
bug detection rate
maintenance cost
architecture INTERNAL
EXTERNAL
11
Methods
Automation engineer
1
Client
Develops Development workshop
2'
Controls
Source code
Analysis tool
Automatic operation
Analysis results Projet manager
Dashboards
Automatic operation Analysis model
Quality monitoring cycle
4
Decides
3
Follows
2 Corrects
12 All rights reserved
Readability
Comprehensibility
Testability
Reliability
Scalability
Effectiveness
Maintainability
Reusability No GOTO
Comment ratio
…
Attributes of the program Sub-attributes Measure and
verification points
Methods
Project Manager
Automation Engineer
Client
Quality model
13 All rights reserved
Analysis model
Attribute 1
Attribute 2
Attribute 3
Attribute 4
Attribute 5
Attribute N
Measure 1
Measure 2
Measure 3
Measure N
Verification 1
Verification N
Verification 2
Analysis model
...
...
...
The “fctn_vannes” function has 67
lines of code The program has a good testability
The “cbfe_34” variable has no comment
14 All rights reserved
Characteristics of the SQALE method
The SQALE method takes into account the entire life cycle of the software, including maintenance, renovation and reuse.
The program features are hierarchical:
Who wants to reuse a non reliable program?
Who can demonstrate the reliability of a non testable program?
The result is a measurement of a technical debt:
How much does it cost to have a quality program from the
current situation?
The problems to be solved are counted once on the attribute
with the highest priority
- The quality properties are regarded as independent
The methods tells you where to start
SQALE is independent from a language and from a specific technology
- The results are directly comparable from one program to another
Contrary to ISO9126, SQALE applies directly, does not require to be interpreted
SQALE is automated and economical to implement. It is standardized.
http://www.sqale.org/
15 All rights reserved
Application to industrial automation
What software analysis should be used? - cross-PLC brands - 5 languages of IEC-61131 Which quality model should be implemented? - transposition of the quality models of traditional computing - specificities of industrial automation Which tools for stakeholders? - how to cope with the diversity of stakeholders? - how to manage outsourcing?
16 All rights reserved
Workshops
Software analysis tool
Quality model
Dashboard
One solution
The black box is open to all stakeholders…
5 IEC languages
17 All rights reserved
Methods
Project Manager
Automation Engineer
Control engineer and software
Client
Solved problems:
Make objective the non-functional evaluation of the program
Positive feedback on the programming practices
Higher level view than just the application under development
18 All rights reserved
Methods
Project Manager
Automation Engineer
Project Manager and software
Client
Solved problems:
Quality monitoring
Monitoring the progress of the project
Benchmarking
The dashboard allows navigation from overall
vision to detail
It also allows a temporal
monitoring of the project’s
progress
80-400mm
19 All rights reserved
Methods
Project Manager
Automation Engineer
Methods and software
Client
Solved problems:
Taking into account existing data
Verify that specifications and code match
Formalization and sharing of development methods
Transversal software indicators
24-120mm
20 All rights reserved
Methods
Project Manager
Automation Engineer
The end client and software
Client
Solved problems:
Simplify decision making on the means to assign, according to an objective view
Possible correlation with other sources of information available in the plant
PLC 1 PLC 2 PLC 3
PLC 4 PLC 5 PLC 6
PLC 7 PLC 8 PLC 9 10-24mm
21 All rights reserved
Real case study – The DNV Audit
Is it usable in real life? How to implement it? How much time is saved?
22 All rights reserved
PLC program audit
Need: risk management
Client: DNV, Malmö, Sweden
Function: PLC in charge of controlling the lay tower on a boat
PLC: Rockwell RSLogix 5000
Analysis tool: PLC Checker
Method: SQALE
Unrepresentative image of the boat in question
23 All rights reserved
Code audit : the need
Objectives:
- Identification of key risks associated with software
Scope:
- Software for control systems of the lay tower
- Reliability, maintainability and dependability of the tower
Actions: manual and automated code review
- Functional analysis of the software application
- Analysis of the development process
- Specifications, design, coding, unit testing, integration testing and acceptance testing
- Analysis of the internal quality of the software application: SQALE
24 All rights reserved
Code Audit : SQALE analysis
LADDER code, about 7000 code locations
Normalized index figures
Most common problems:
- Testability: variables written in several places, dead code, important complexity, code in comments
- Reliability: variables are read before being written
- Maintainability: uncommented code
25 All rights reserved
Code Audit : the results
Consistent with other SQALE results for other languages (traditional computing)
The results are better than what is usually observed
Consistent with manual code reviews and “top ten” verifications
Some persistent difficulties with the tools have to be solved
- Interaction between the program and the HMI may not be identified automatically
- Copy / paste of code not yet detected
Final comment of the Swedish client:
« The SQALE analysis provided a very valuable complement to the manual part of the software review »
« While the manual review focused on thoroughly checking selects parts of the code, the SQALE analysis measured defined quality characteristics of the complete code »
26 All rights reserved
How to validate that the thresholds of the quality model are suitable for automation?
Why would a traditional computing quality model suit the IEC languages? How to tune the model to ensure a good match between the ratings and the actual quality?
27 All rights reserved
A study based on real life programs
Step 1: Formalization of measurements to be made on the programs
Step 2: Creation of a client program database
- No test program
- Multi-PLC (Schneider Electric PL7 Pro and Unity Pro, Siemens Step7, Rockwell RSLogix5000)
Step 3: Running the analyzer (PLC Checker) on each program with formalized measures
Step 4: Analysis of results
- Results per PLC
- Results of all PLCs combined
- Definition of thresholds
28 All rights reserved
A few figures
~25 measures
~300 PLC codes Step7, Unity Pro, PL7 Pro and RSLogix5000
~180 000 Program Organisation Units (POUs)
~2 500 000 instructions
~2 000 000 variables
55 hours of calculation
Results: 112MB of raw data to analyse
29 All rights reserved
The quality model is not elitist, it must correspond to the actual use:
50%: A
75%: A or B
90%: A, B or C
95%: A, B, C or D
97,5%: A, B, C, D or E
99%: A, B, C, D, E or F
Definition of thresholds
Acceptance criterion
30 All rights reserved
Number of lines of code
APPLICATIONS
No quality criteria on the size of the application, just an information
Analysed applications are up to 60,000 lines of code
50% <10 000 lines
PROGRAM ORGANIZATION UNITS
Ensure that each POU has a reasonable size
90% of POUs have less than 100 lines of code
The threshold is comparable to what is recommended in traditional computing
31 All rights reserved
Complexity of the codes
Are programs easy to understand?
Two different complexities:
- cyclomatic complexity, essential complexity
- in both cases, the thresholds are in line with the levels seen in traditional computing:
eV(G) < 5
V(G) < 15
Most automation engineers already program correctly Beware! The cyclomatic complexity is not available as such on all languages (limitations on graphical languages : SFC, FBD and LD).
Acceptance criterion
Acceptance criterion
32 All rights reserved
Level of comments of codes
Acceptance criterion
Are applications well commented?
Elements within the program:
50% of all applications have a comment ratio greater than 85%
75% have a ratio greater than 70%
90% have a ratio greater than 60%
Check on size of comments
Density of comments in the code:
50% of all applications have a density of comments greater than 67%
75% have a density greater than 57%
90% have a density greater than 52%
33 All rights reserved
Conclusion
Low dispersion on very general measurements
PLC programs are comparable with one another regardless of their functionality
The quality model used in the experiment is sound
The complexity thresholds used in traditional computing can also be used in automation, with the following restrictions: - have to be tuned for graphical languages (SFC, FBD and LD) - detection limits on copy/pasted codes - has to take into account typical malpractices
34 All rights reserved
Conclusion
How to participate?
How to use?
I have a use case, what to do?
Can I adapt all of this to my needs?
For more information
35 All rights reserved
Key Takeaways
The late discovery of bugs and low quality is costly. The monitoring of quality during the life cycle prevents it:
Thanks to the democratization of quality control, with the following parts…
- Dashboards that allow navigation between high and low level view - Analysis tools automatically generating data - Quality models implementing ISO 9126
…That are applicable and suitable for automation:
- Cross-PLC brands support - Support of all languages of the IEC-61131
Calling all end users and
integrators
Invitation to participate in the development of a
Quality Model adapted to PLCs
Please contact DNV or IAS
To go further, we are looking for: motivated industrial end
users and system integrators, willing to participate in the
improvement of a quality model suitable for automation
36 All rights reserved
Want to know more?
Software quality on Wikipédia -http://en.wikipedia.org/wiki/Software_quality
SQALE website - http://www.sqale.org/
Der Norske Veritas - http://www.dnv.com/
Itris Automation Square – http://www.automationsquare.com/plc-checker.html
SQUORING - http://www.squoring.com/en
Inspearit - http://www.inspearit.com/en/
Thierry COQ [email protected]
Principal consultant
Denis CHALON [email protected]
Technical Director