on the complexity measurement of …1113035/fulltext01.pdf · 4.4 beremiz v1.2 [1] - an open source...

Malardalen UniversitySchool of Innovation Design and Engineering

Vasteras, Sweden

Thesis for the Degree of Master of Science in Computer Science -Software Engineering 15.0 credits

ON THE COMPLEXITYMEASUREMENT OF INDUSTRIAL

CONTROL SOFTWARE

Adnan [email protected]

Examiner: Professor Daniel SundmarkMalardalen University, Vasteras, Sweden

Supervisor: Dr. Eduard Paul EnoiuMalardalen University, Vasteras, Sweden

May 17, 2017

Malardalen University On the Complexity Measurement of Industrial Control Software

Abstract

Embedded systems are becoming a dominant part of the computer systems that we use both in ourevery day life and in the industry. For domain specific and industrial use, specialized embeddedsystems, called Programmable Logic Controllers, are used to run software which provides supervi-sory control. Such software, also known as industrial control software, is programmed using oneof the standardized IEC 61131-3 programming languages. As the size of the industrial controlsoftware increases and given that more requirements are imposed on such software, the effort andcomplexity of engineering software also increases. In the past decades much effort has been putinto researching software complexity measures and their impact on software quality attributes suchas maintainability, testability, understandability and test effort.

In this thesis we present how software complexity can be measured on industrial control softwarewritten in IEC 61131-3 Function Block Diagram (FBD) graphical programming languages. Weshow how software complexity can be represented on this type of software, what factors it caninfluence and how it can be directly measured. A set of software complexity metrics is selectedand the measurement techniques adapted for the graphical syntax of FBD. These techniques areimplemented in a open-source tool for software complexity measurements of FBD programs. A casestudy was performed using several industrial control programs from a train control managementsystem. We have investigated the relationship between test effort (e.g., in terms of test suite sizeand test execution time of manually created tests) and different FBD software complexity measures.

We found that there is a low but consistent correlation between software complexity measuresand test effort. In particular, the FBD equivalent of Source Lines of Code metric had the highestcorrelation coefficient (i.e., 0.368) when compared to the test effort. Also, our results suggestthat Cyclomatic complexity might not be useful for measuring the structural dimension of an FBDprogram and that a new structural software complexity metric might be required. The results ofthe correlation analysis was also used to construct a linear regression model. This model can beused as a rough estimation approach based on the software complexity measurement results. Theconstructed regression model might be improved by taking into account other software attributes thatinfluence the test effort such as the program specification used for manually testing the software.

1

Table of Contents

1 Introduction 6

2 Background 82.1 Software Complexity Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.2 PLCs and Embedded Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.3 Software Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3 Related Work 12

4 Method and Proposed Measurement Solutions 134.1 Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.2 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.3 State of the Art in Software Complexity Metrics . . . . . . . . . . . . . . . . . . . 144.4 A Method for Measuring Software Complexity of FBD Programs . . . . . . . . . . 15

4.4.1 Measurement Approach and Set of Complexity Metrics . . . . . . . . . . . 154.4.2 Source Lines of Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.4.3 Cyclomatic Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.4.4 Halstead Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184.4.5 Information Flow Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . 20

5 Tool Supported Complexity Measurement 225.1 The Measurement Approach Applied on an Example . . . . . . . . . . . . . . . . . 24

6 Case Study on the Complexity Measurement of FBD Industrial Control Soft-ware 286.1 TCMS FBD Programs Selection and Test Effort Measurement . . . . . . . . . . . 286.2 Overall Complexity Measurement Results of TCMS . . . . . . . . . . . . . . . . . . 296.3 Is Software Complexity Correlated With Test Effort? . . . . . . . . . . . . . . . . . 326.4 Predicting Test Effort using Software Complexity Scores . . . . . . . . . . . . . . . 336.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346.6 Threats to Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

7 Conclusion 36References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Appendix A Test Effort 40

Appendix B Software Complexity Measurement Results 42

2

List of Figures

2.1 Two equivalent source codes written in two different program languages and paradigms.On the left is a Java example, on the right an FBD example. . . . . . . . . . . . . 9

2.2 A complete and timed testing phase cycle. . . . . . . . . . . . . . . . . . . . . . . . 11

4.1 The process that was used to collect relevant articles and papers. . . . . . . . . . . 144.2 An FBD function that outputs the square of the bigger input parameter. . . . . . 164.3 Control flow graph of the squareOfBigger Java function. . . . . . . . . . . . . . . . 174.4 Beremiz v1.2 [1] - an open source IEC 61131-3 IDE with the FBD implementation

of the squareOfBigger problem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194.5 A high-level architecture view of an examined software module D, which is used by

A, B, C and is using modules E, F. . . . . . . . . . . . . . . . . . . . . . . . . . . 20

5.1 Initial architecture and functional parts of the FBD software complexity tool - Tiqva. 225.2 A high-level view of the input processing layer of the tool. . . . . . . . . . . . . . . 235.3 Graphical representation of the stereo cassette recorder. . . . . . . . . . . . . . . . 245.4 FBD graphical program code of the Norm function. . . . . . . . . . . . . . . . . . . 255.5 FBD graphical program code of the Volume function block. . . . . . . . . . . . . . 25

6.1 Number of test steps for individual test case (left) and the execution time of indi-vidual test case (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

6.2 Normalized values for the Number of elements complexity (left) and Cyclomaticcomplexity(right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

6.3 A normalized area chart of two Halstead metrics (Volume and Difficulty), showingthe similar distribution of said metrics across the TCSM. . . . . . . . . . . . . . . 31

6.4 Normalized values for the Information flow metric (left) and Halstead effort com-plexity(right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

6.5 Visualized predictions of the trained linear regression model. The two plots showthe predictions of the model for the test effort proxy measures (number of test steps- left, test case execution time - right). . . . . . . . . . . . . . . . . . . . . . . . . . 34

3

List of Tables

4.1 Halstead complexity measurement equations [2] and measurement results for square-OfBigger function. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

4.2 Halstead complexity measurement results of the squareOfBigger FBD implementation. 20

5.1 Defined variables for the Norm function POU. . . . . . . . . . . . . . . . . . . . . . 255.2 Defined variables for the Volume function block POU. . . . . . . . . . . . . . . . . 255.3 Software complexity measurements of the stereo cassette recorder POUs. . . . . . . 265.4 Software complexity measurements of the stereo cassette recorder FBD project. . . 27

6.1 Test effort measure characteristics. . . . . . . . . . . . . . . . . . . . . . . . . . . . 296.2 Software complexity measures and the descriptive statistics of the results. . . . . . 306.3 Kendall correlation coefficient (τ) and Kendall p-value between software complexity

metrics and test effort. The test effort was expressed by two proxy values, test caseexecution time (E) and the number of test steps in a test case (N). . . . . . . . . . 32

6.4 Software complexity weights, mean square error of predictions made based on saidweights and the variance score of the predictions. . . . . . . . . . . . . . . . . . . . 34

A.1 Train Control Management System Test Effort expressed via Number of Test Stepsand Test Case Execution Time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

B.4 Software Complexity Measurement Results of the Train Control Management Sys-tem (TCMS). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

4

Listings

2.1 A simple Java function that outputs 10 lines of code. Physical SLoC is 8 while thelogical SLoC is 7 (blank line not counted). . . . . . . . . . . . . . . . . . . . . . . . 8

4.1 A Java function that returns the square value of the bigger input parameter. . . . 165.1 Snippet of a Tiqva method which measures the NoE metric . . . . . . . . . . . . . 23

5

Chapter 1

Introduction

Compared to the physical counterparts that execute it, software is very abstract, and its inter-pretation and understanding can vary from person to person. While it is easier to grasp the sizeof a building, or the different gears of a vehicle, fully understanding what software is and whatit is made of, presents a difficult task [3]. To measure software, or to measure the complexity ofsoftware, is an engineering research field that has been studied for over four decades [4] with thegoal to define what software complexity is, how it can be measured and what knowledge it provides.Software complexity metrics could potentially be used to determine non-functional properties ofthe software. In addition, it can be used to predict other software characteristics, like the requiredtesting effort based on the structural complexity of the software [5] or the ease of maintainabilitybased on the size of the software [6]

Software has increased its presence in our everyday lives by being used in television remotes,smartphones, as well as in the industrial sectors. Running on computer systems called embeddedsystems, software is now responsible for managing the lights of a transport vehicle, opening andclosing of train doors, or performing complex process controls of a power plant. One type ofembedded systems used in the industrial domain are Programmable Logic Controllers (PLCs), usedto execute safety-critical and operation-critical software. The domain-specific software executed onPLCs is sometimes called industrial control software. PLCs are usually programmed in one of IEC61131-3 [7] standardized programming languages. One of the IEC 61131-3 programming languagesis the Function Block Diagram (FBD) graphical programming language, which uses blocks andconnections to realize a required functionality.

Software complexity metrics can be used on industrial control software. However, this has notbeen thoroughly researched in the scientific literature. We are not aware of any existing softwarecomplexity metrics that are used for measuring software complexity of domain-specific softwarewritten in the IEC 61131-3 FBD graphical programming language. There is a need to investigatehow existing software complexity metrics can be applied on FBD programs and what changes haveto be made in order to tune the measurement techniques to the FBD graphical syntax. Naturally,engineers would like to use these complexity metrics to estimate different development efficiencyand effectiveness metrics. This is intuitively appealing, as these relations can be used to find arelationship between software complexity and effort needed to test the software.

In this thesis we present the adaptations needed to apply existing software complexity metricsto the FBD syntax. In addition, we present the results of a case study in which we investigatedhow software complexity measured can be used to estimate the test effort needed to manually testindustrial control software. This software was provided by Bombardier Transportation Sweden ABand it is used for performing a wide variety of operations when controlling the train functionality.

In Chapter 2 we provide the theoretical background on the concepts of software complexitymeasurements, IEC 61131-3 programming language standard and software testing. Related work ispresented in Chapter 3. In Chapter 4 we define the problem, present the set of software complexitymetrics and software complexity measurement techniques for the FBD programming language. Atool for automatic measurement of FBD software complexity is presented in Chapter 5. Theperformed case study is described in Chapter 6, where we analyze and discuss the correlationresults between software complexity and the test effort using several industrial control programs.

6


In Chapter 7 we present our conclusion and possible future work.

7

Chapter 2

Background

In this chapter, we explain the concepts necessary for understanding this thesis. The backgroundcovers three main areas of software engineering, software complexity metrics, PLCs and embeddedsoftware and software testing. The work presented in this thesis focuses on the measurement ofFBD software complexity and its use as a means to estimate the necessary needed test effort.

2.1 Software Complexity Metrics

Software complexity metric is a quantitative value that describes a dimension of a software, de-pending on the type of the artifact used for measurement [3]. This statement points out twocharacteristics of every software complexity metric:

• A software complexity metric measures a complexity dimension of software.

• The software complexity measurement process has to be defined for a particular softwareartifact.

We are not aware of any universal software complexity measure that can determine all thedifferent complexity dimensions of a software nor can it be applied to all possible software artifacts(such as the program source code or the software architecture). However, there are a number ofsoftware complexity metrics that have been used successfully in practice [8]. As shown in Listing2.1, a simple program can be measured in terms of lines of code.

1 pub l i c void foo ( ) {2 St r ing s = ”Number : ” ;3 i n t l im i t = 10 ;45 f o r ( i n t i = 0 ; i < l im i t ; i++) {6 System . out . p r i n t l n ( s + i ) ;7 }8 }

Listing 2.1: A simple Java function that outputs 10 lines of code. Physical SLoC is 8 while thelogical SLoC is 7 (blank line not counted).

8


Source Lines of Code (SLoC) is a simple size metric that measures the logical (statements)and the physical (lines) size of a source file. It is a size metric, since it can only describe thesize dimension of a software artifact (e.g. the program source code). The SLoC example abovedemonstrates how an artifact (program source code written in Java) is measured and how the resultsare evaluated. Some research has been performed on formal specification of software complexitymetrics and properties that describe a software complexity metric [9].

Although the motivation for measuring a specific dimension of a software may vary [10], re-search done in this area showed [6, 11] that some complexity metrics may be good indicators ofsome quality and development effort properties. Results from the application of these softwarecomplexity measurements [4] can be used to indicate the amount of test effort required for thatparticular artifact or can be used to show that a software architecture has high levels of coupling[12].

In Chapter 4 we examine some of these metrics in more detail as well as discuss the implicationsof the results and how they might be of use for domain-specific programming languages as the onesused in this thesis.

2.2 PLCs and Embedded Software

Embedded software’s presence has become very dominant in most electronic devices that are usedfor personal and industrial use. The software running on such systems (also known as embed-ded software), can be programmed in a wide variety of programming languages. Self-containedcomputer systems with a CPU, RAM and IO ports called microcontrollers can be programmedusing general-purpose programming languages such as the C programming language. However,for domain-specific domains (e.g. transportation, chemical industry, aerospace, automotive etc.),embedded systems called Programmable Logic Controllers (PLCs) are used to provide supervisorycontrol [13]. In a complex system this supervisory control can be used for opening and closingdoors or controlling the temperature in a furnace.

PLCs differentiate from their general-purpose counterparts in the way that they are pro-grammed, specifically the language used for programming this kind of software. While mostembedded systems can be programmed using general-purpose programming languages, PLC pro-grams are usually created in one of the IEC 61131-3 programming languages [7]. IEC 61131-3 is ainternational standard that describes the properties and requirements for a programming languageused for creating PLC programs [7]. IEC 61131-3 has a number of programming language im-plementations: Structure Text (ST), Instruction List (IL), Ladder Diagram (LD), Function BlockDiagram (FBD). Two of these languages, FBD and LD, are graphical programming languages,thus they do not use the textual source code notation but rather a graphical representation (seeFigure 2.1).

Since the IEC 61131-3 programming languages are used in a domain-specific application, theselanguages have their own paradigms and best practices when it comes to embedded softwaredevelopment. FBD programs, which are IEC 61131-3 compatible programs, on an organizationallevel operate using Program Organization Units (POUs) [7]:

• Function: procedure-like program code

• Function Block: similar to Function POU with the addition of status information memory

• Program: top-level program code that has access to IO ports

i f ( a > b) {sum = a + c ;

} e l s e {sum = b + c ;

}

Figure 2.1: Two equivalent source codes written in two different program languages andparadigms. On the left is a Java example, on the right an FBD example.

9


Structured Text, the more traditional IEC 61131-3 programming language, resembles a Pascal-like program code, which results in an easy adoption of established practices and guidelines forgeneral-purpose programming languages (e.g. Pascal best practices and guidelines). FunctionBlock Diagrams, originally used in signal processing, are based on networks which realize the soft-ware functionality. FBD uses blocks to represent variables, operations, functions, function blocksand other IEC 61131-3 defined operators and operands. To provide a function or an operation withinput parameters (variables), FBD uses connections. Connections transfer the value of a variable,a previously executed operation or function to the next block. When an FBD program is compiledto machine code, the program is evaluated from it’s exit point, meaning that the blocks in a blockdiagram will be evaluated based on their input connections. This means that programs will resolveevery block until it reaches the input variable blocks.

Most of the properties that make up a programming language (variables, data types, functions)are also present in FBD. However, conditional statements and loops have a different approachin FBD. A Java example source code and the corresponding FBD source code shown in figure2.1 demonstrate that difference. Although the result is same, the notion of an IF statement isencapsulated in the MAX block, and using it the FBD program performs the same function as theST program.

Variables in FBD are initialized in a separate part of the IEC 61131-3 Integrated DevelopmentEnvironment (IDE). Compared to software development in Java, where all types of statements(variable declarations, function invocation and similar) can be contained in a single file and accessedin a single IDE window, FBD IDEs use the mentioned Variable window for variable declarationsand a graphical programming window for the programming (as seen in Figure 2.1.

Although FBD is a graphical programming language, the program content, the source code, issaved in an XML file. The XML file follows the IEC 61131-3 schema [7] for FBD programs, and isinterpreted and visually displayed in an IEC 61131-3 IDE. This type of source code format enableseasier parsing of FBD programs as well as extending the schema with IDE specific features.

2.3 Software Testing

An important phase of every software development process is the verification phase. Verificationis the process of determining whether or not the current state of a software product fulfills theestablished requirements of a previous development phase [14]. One method of software verificationis software testing, which is used to check if the behavior of the software is as intended (thusverifying it) and to expose faults in the software.

Software testing can be performed at different levels during software development. The softwareunder test varies based on the level of testing: unit testing, integration testing and system testing.Unit testing is performed on a specific function in isolation. Typically, unit testing occurs directlyusing the code being tested and with the support of debugging tools. This type of testing might in-volve the programmers who wrote the code. Nowadays, there are many different testing techniquesthat are used in industry. Testing techniques can be grouped in the following main categories: ex-ploratory testing based on the software engineers intuition and experience, specification-basedtesting [15], Code coverage-based testing [4] and Fault-based testing. These techniques can beused in combination. For example, specification-based testing and code coverage-based testing arecomplementary to each other mainly because these techniques are using different test goals andsources of information.

Although these testing techniques are different, all of them employ the concept of a test case,a structured documentation of a test. A test case is defined in the IEEE Standard [16] as a set oftest inputs, execution conditions, and expected results developed for a particular objective, suchas to exercise a particular program path or to verify compliance with a specific requirement. Atest case provides means for a software developer or software tester to create, perform, exerciseand verify tests. A set of test cases, grouped together, is called a test suite and is used to test acomplete feature of a software.

The testing phase can be similar for all testing levels and testing techniques, and starts with thetest design step (shown in Figure 2.2). This step requires collection of intended behavior require-ments, based on other software artifacts such as the documentation of the software, specification

10


Figure 2.2: A complete and timed testing phase cycle.

for the artifact that is tested or similar. The effort to verify the expected behavior of a softwarevaries, and can be influenced by different properties of a software. Test effort, a measure thatassigns a weight factor to the testing phase of a software, can be represented by the effort putinto the individual steps of the test cycle as seen in Figure 2.2 (in this case, the effort is a timevariable). In other words, test effort represents the value of the resources put into the softwaretesting phase.

11

Chapter 3

Related Work

Although the literature on both software complexity metrics and industrial control software writtenin IEC 61131-3 FBD programming language is scarce in western research databases, softwarecomplexity measurements have been researched for other graphical programming languages. Thearticle by Olszewska, Marta and Dajsuren on tailoring software complexity metrics to Simulinkmodels [17] describes the research on measurement of Simulink programs software complexity.The paper shows how to tailor the measurement techniques of software complexity measures to theblock syntax of Simulink models and how to perform a correlation analysis between the complexityresults and fault data from a Car Fuel program created using Simulink. A positive correlation wasdetected between components with high complexity and the number of faults. Knowledge of threedomain experts has been used to validate the results, as their understanding of complexity directlyrelates to their results.

The lack of support for static analysis in modern IEC 61131-3 Integrated Development Envi-ronment (IDE) has been tackled by Prahofer and Ramler in their paper on the implementationof a static analysis tool for IEC 61131-3 Structured Text [18]. This paper introduces six commonissues (code fault, bad smell, correct writing conventions) in software development that are alsopresent in Structured Text. These are detected by a set of rules implemented in an extendableJava tool for automated static analysis of programs written in Structured Text. Integrating thistool with Sonar, a popular framework for static analysis of Java code, six programs are analyzedfor faults in a continuous integration process.

Software complexity measurements have often been used for quantifying software quality at-tributes. Kushwaha, Dharmender Singh designed [19] a software complexity metric that wouldbetter describe testability, a common software quality attribute. This paper provided argumen-tation and evidence on how Cognitive Information Complexity Measure (CICM) captures thetestability of software. CICM is explained and different properties of the metric are analyzed.CICM is used in a case study to measure complexity of a procedural C program code. A com-parison between CICM and Cyclomatic Complexity [4] is made, which points out the strengths ofCICM for estimating test effort.

12

Chapter 4

Method and ProposedMeasurement Solutions

This chapter will serve as a roadmap through the different phases of the proposed measurementmethod. We will present the problem definition, the research questions based on the tackling ofthese problems, and the methods used to give answers to these research questions.

4.1 Problem Definition

A large number of complexity measures have been discussed in the literature for general-purposeprogramming languages [8] (i.e., C and Java). Different complexity and architectural componentbased metrics [8] (e.g., Lines of Code, McCabes Cyclomatic Metric, Halstead Metric, InformationFlow Metric, McLure Complexity, Card and Glass Complexity) have been proposed to be used inpractice for different programming languages. However, for domain specific programming languagesused in embedded software development, contributions in the area of software complexity arelimited. Based on the theoretical background from Chapter 2, it is obvious that applying softwarecomplexity measurement techniques to the IEC 61131-3 FBD graphical programming language isnot directly straightforward.

In addition, testing software systems is becoming increasingly difficult. Therefore, the studyand development of static measurement techniques and tools to better predict the amount of effortneeded to test those systems is becoming a very important field of research.

• There is a need for investigating how complexity measures can be applied toFBD software.

• There is a need to investigate how software complexity measurements can beused for estimating the test effort.

4.2 Research Questions

To summarize the defined problems and to present the problems in a structured way, the followingresearch questions have been constructed:

• RQ 1: Which are the most relevant software complexity metrics found in the scientific liter-ature review?

• RQ 2: How to measure software complexity on FBD programs?

• RQ 3: Can software complexity metrics be used for estimating the effort needed for manuallytesting the FBD software?

13


4.3 State of the Art in Software Complexity Metrics

We reviewed papers and articles that focus on software complexity metrics and other softwareaspects that might be affected by higher or lower software complexity measures such as the testingeffort, maintainability and understandability. Figure 4.1 displays how we used the Google Scholarsearch engine to filter out papers that could be relevant to RQ 1-3.

Figure 4.1: The process that was used to collect relevant articles and papers.

We initially started with papers on the general theory of software complexity which helpedin capturing the details of each individual software complexity metric. One of the first paperson software complexity metrics was the original McCabe paper[4]. McCabe, using graph theoryto represent program code, designed rules that are used to calculate the Cyclomatic complexitymetric. This metric measures the amount of decisions that are made in the program code. Anothersoftware complexity metric that was researched during the last few decades is the Information Flowmetric [12]. Kafura builds upon Hennry’s model of Information Flow theory to define a softwarestructure complexity metric. Specific metrics are defined for procedures, module complexity andmodule coupling. This paper shows that this metric is appropriate for large scale systems becauseit is applicable at the early design phases of the system and it reveals properties regarding thesystem’s connections.

To formally define software complexity metrics, Weyuker [9] discussed different properties ofsoftware complexity measures and reviews popular complexity metrics based on formal definitionsof software complexity properties. The paper provides proofs and examples on why some softwarecomplexity metrics describe the same properties although they measure different aspects of asoftware artifact. In addition, Weyuker shows some complexity metrics that do not satisfy someof these formal properties.

We reviewed papers looking in the software complexity metrics and their analysis on a higherlevel. Mens [3] reviewed different research trends on software complexity and previously gatheredknowledge in this domain. Another case study paper [11] on the empirical study of measuredsoftware complexity metrics of two program versions gives insight into the possible expected resultsin this thesis. Based on the measured complexity of the programs, existing problem reports (bugs,faults, errors) as well as program reliability levels, a correlation test was performed to determinethe impact of software complexity on different non-functional properties.

A similar study on the impact of software complexity measures on the maintainability of asoftware was performed in Kafura et al. [6] paper. Kafura analyzes the evolution of a DatabaseManagement (DBM) system developed by master level computer science students and measuresits complexity and detected faults for each version of the DBM. Software complexity metrics thatwere measured have been grouped into code metrics (McCabe, Halstead [2]) and structure metrics(Information Flow, McClure [20]). The paper summarizes the reasons why a particular softwareartifact has a higher software complexity measurement results.

We used the knowledge gained from this literature review to construct a set of software com-plexity metrics that have been thoroughly researched and used in studies that focused on thecorrelation between software complexity measurements and other software characteristics.

14


4.4 A Method for Measuring Software Complexity of FBDPrograms

4.4.1 Measurement Approach and Set of Complexity Metrics

During our literature review, a significant number of software complexity metrics were discovered[4, 2, 12, 20, 21]. However, some of these metrics did not yet pass the test of time, as they wereeither very young or were not researched enough to be considered in papers looking at indicators ofsoftware quality and effort. We mentioned here that we did not find any relevent research done onsoftware complexity metrics for FBD programs in the Google Scholar scientific research database.This implied that the software complexity metric set to be used in this thesis had to containsoftware complexity metrics that were researched in more depth by the scientific community andthat some empirical evidence of their impact in practice had to exist. With this in mind, weselected a set of four metrics:

1. Source Lines of Code (SLoC)

2. Cyclomatic complexity (CC)

3. Halstead complexity (HC)

4. Information Flow complexity (IFC)

Although these metrics are somewhat conservative (since they were designed in the 70s and 80sof the 20th century), they were the subject of in-depth research and are among the most popularsoftware complexity metrics used in the industry as well as in research [3].

Both SLoC and HC are code size metrics and can abstract the size or the length of a programartifact. The HC metric also provides information about the possible existing bugs in a softwareartifact or the required testing time [2] (both indicators of the maintainability of the software).

CC is a direct measure of the amount of decisions that are made in a program [4] and is oftenused for determining the required amount of tests for achieving basic path coverage, which can beused as a test effort measure [5].

The IFC metric proposed my Henry and Kafura is mainly used in software architecture design,since the metric computes the amount of coupling and cohesion between different software modules[12]. Although the research that we are conducting is on a unit level and not on an architecturelevel, FBD programs are by default very similar to other architectural models, as both architecturemodeling and FBD programming is done using blocks and connections [7].

4.4.2 Source Lines of Code

We already discussed the SLoC metric as a basic example of a software complexity measure thatdescribes the length of a software artifact, most often being a program source code. The SLoCmetric is easily computed for most program source codes, as it measures a very basic property ofa program - the number of program lines. However, results of SLoC measurement of two programsource codes written in two different program languages cannot be simply compared, as they canbe written in two different programming paradigms (procedural, functional, object-oriented).

15


1 pub l i c void squareOfBigger ( i n t a , i n t b) {2 i n t c = 0 ;34 i f ( a > b) {5 c = a ;6 } e l s e {7 c = b ;8 }9

10 re turn c ∗ c ;11 }

Listing 4.1: A Java function that returns the square value of the bigger input parameter.

The results of SLoC measurements require some context to provide useful information, suchas the reason for the measurement, the program language of the measured artifact, the expectedlength of the program and similar characteristics.

The Java function in Listing 4.1 is a very simple Java function that outputs the square value ofthe bigger input parameter (no specific scenario is provided when the input parameters are equal).The physical SLoC is 11, while the logical SLoC is 9 (not counting empty lines). The equivalentsquareOfBigger FBD program is shown in figure 4.2. It can be noticed that the technique used formeasuring number of source lines from the previous Java example cannot be directly used on theFBD program.

Figure 4.2: An FBD function that outputs the square of the bigger input parameter.

In the IEC 61131-3 FBD programming language, the notion of a program statement is verydifferent compared to other general-purpose programming languages. One of the reason why thisis the case is that the notion of a program statement, strictly speaking, does not exist in FBD.While in Java, a line of code can be a function call, in FBD, functions are encapsulated insideblocks. The question of how to map the SLoC metric to FBD is very similar to the questions of”What are lines of code in an FBD program”?

If the function calls and other program statements are abstracted via blocks, and the order oftheir execution controlled via connections, then we can assume that SLoC metric for FBD wouldmeasure the number of blocks and connections in an FBD program. Considering that blocks andconnections are graphical elements of the FBD language, and that there are other elements suchas variables and data types [7] controlled by an IEC 61131-3 IDE, we could think of this metricas Number of Elements (NOE), rather than Source Lines of Code, in the context of the FBDprogramming language.

We propose the following steps for calculating the NoE metric for an FBD program:

1. Measure the number of variables

2. Measure the number of data types

3. Measure the number of connections

4. Measure the number of blocks

5. Sum the measured values

FBD variables can have a base type (Integer, Double, Boolean etc.) or be used as input, outputor local variables, and in general, can have the same properties variables have in most programminglanguages (e.g. initial values, constants). FBD data types are custom data structures and havea standard FBD variable as a base structure with additional constraints [7], and can be used

16


as a base type of a variable. When initialized in the graphical programming environment, FBDvariables and data types are represented as blocks (variable blocks), and are used as inputs oroutputs for function blocks. When measuring the number of variables and data types of an FBDprogram, all types of variables and data types have to be included (input, output, local variables).However, their block representation in the graphical programming environment representation hasto be included into the number of blocks as well.

The final result of the NoE metric is equal to the sum of individual measurements. Usingprevious steps we can measure the NoE metric of the FBD program shown in Figure 4.2:

• Variables: 3

• Blocks: 5 (2 function blocks, 3 variable blocks)

• Connections: 5

The final NoE value is 13 (sum of all FBD elements).

4.4.3 Cyclomatic Complexity

In the original paper [4], McCabe proposed a software measurement technique that measures thenumber if linearly independent paths through a program code. This structure metric is foundupon graph theory and can be applied to a wide specter of software artifacts (from simple programfunctions to architectures [22]). To apply rules from graph theory, artifacts measured for CC haveto be abstracted via control flow graphs.

Figure 4.3: Control flow graph of the squareOfBigger Java function.

As most graphs, control flow graphs are represented via nodes and edges. A program code canbe abstracted using nodes to represent program statements and edges to show the execution orderof the statements, and conditional statements are presented via branching nodes. Figure 4.3 showsthe control flog graph for the squareOfBigger 4.1 Java function. The yellow node denotes the ifstatement, the blue and red nodes are the two bodies of the if-else block.

The Cyclomatic complexity is defined as:

M = E −N + 2P, (4.1)

where M is the CC score, E is the number of Edges in the control flow graph, N is the number ofnodes while P is the number of strongly connected components. The P usually denotes the numberof exit points from a program code [4]. Using this formula and the control flow graph shown inFigure 4.3, we get a CC of 2 for the squareOfBigger function. The CC metric can be measuredusing an alternate equation [4]:

M = Π− S + 2, (4.2)

17


where Π is the number of decision points of a program and S is the number of exit points. Equation4.2 points out that the CC score is shaped by the number of conditional statements in the program.

The FBD variant 4.2 of the squareOfBigger program uses the MAX block to determine thebigger of the two input parameters. The IEC 61131-3 programming language standard defines anumber of different blocks that can act similarly to an if statement [7].

Since CC is influenced by the decision points of a program, the measuring technique used onprograms written in general-purpose programming languages can be directly translated to the FBDprogramming language. Using equation 4.2 and the FBD example 4.2 we can compute aCC of 2 aswell. This implies that the CC metric is less influenced by the choice of a programming languageand more by the way a program is written.

4.4.4 Halstead Complexity

Compared to SLoC or CC, the HC metric [2] defines ways to compute multiple software dimensionsbased on the measurement of operands and operators. We assume that a set of operators for a pro-gramming language can be all the different mathematical and logical operations and programminglanguage functions and syntax, while the set of operands of a programming language are variablesand values used in the operations.

We measure the number of unique operators and operands as well as the total number of oper-ators and operands of a program to compute the proposed HC measures. For the squareOfBiggerJava function 4.1, following unique operators and operands occur in the function:

• Unique operators (η1): public, void, squareOfBigger, (), int, {}, =, ;, , , if, >, else, return,*

• Unique operands (η2): a, b, c, 0

The following measurements are defined for the HC metric: Program vocabulary, Programlength, Calculated program length, Volume, Difficulty, Effort, Time, Delivered bugs. Some ofthese measurements are computed based on the unique number and the total number of operatorsand operands, and the rest of the measurements are built upon them. Table 4.1 contains theequations for calculating individual HCy measures as well as the result of those measurements onthe squareOfBigger Java function.

Halstead measurements Equation Result

Number of unique operators η1 14Number of unique operands η2 4Total number of operators N1 24Total number of operands N2 12Program vocabulary η = η1 + η2 18Program length N = N1 +N2 28

Calculated program length N = η1 log2 η1 + η2 log2 η2 61.30Volume V = N × log2 η 116.75Difficulty D = η1

2 ×N2

η221

Effort E = D × V 2451.91Time T = E

S s, S = 18 136.21 s

Delivered bugs B = E23

3000 0.06

Table 4.1: Halstead complexity measurement equations [2] and measurement results forsquareOfBigger function.

The HC metric aim to describe a quantitative relationship between software properties whichcan be measured, thus, measurements such as Volume or Difficulty describe a relationship betweenthe measured values of operands and operators. Similar to McCabe’s Cyclomatic complexity, weobserved that HC is more influenced by the syntax of the programming language.

18


IEC 61131-3 textual languages (Structured Text, Instruction List) resemble some general-purpose programming languages (Structured text having a similar syntax to Pascal) and HCmeasurement techniques are defined for such languages. Considering that the FBD language,in it’s essence, is a different approach to writing IEC 61131-3 programs, we can assume that thetechniques used to measure operands and operators of a Java program can also be used to mea-sure operands and operators of FBD programs. However, FBD program variables, values and thecode/algorithm itself - different operators and operands, are created in different parts of an IEC61131-3 IDE, compared to a C program, where all the required operators and operands can be ain a single textual source code file.

Figure 4.4: Beremiz v1.2 [1] - an open source IEC 61131-3 IDE with the FBD implementation ofthe squareOfBigger problem.

Figure 4.4 shows the open-source IEC 6113-13 IDE Beremiz with the FBD equivalent of theJava function 4.1 open in it. This figure demonstrates the mentioned difference between general-purpose programming language IDE and an IDE with FBD support. POUs and DataTypes aredeclared in the Project frame, and the Variables and the FBD function in the main frame inthe middle. The technique of measuring operators and operands can be directly applied to theVariables section of the main frame and to the DataTypes (custom data structures [7]) in theProject section. Operators like INT, Input and operands a, b can be directly measured from thisIDE section, and POU and DataType operators from the Project section.

HC techniques for operators and operands is not directly defined for FBD connections andblocks, which make up the most of the FBD program and have to be measured as they directlyimpact the size of the program (which has been discussed in Section 4.4.2).

Functions and function blocks are the different types of FBD blocks that can be found in anFBD program code. Observing the FBD implementation of the squareOfBigger function, we candeduce that operations such as comparison and multiplication are done via blocks. This impliesthat the different types of blocks examined in an FBD program increase the number of operations(unique and total).

Further, FBD connections do not have a direct translation such as block = function, operation,but the answer can be found if the role of connections in an FBD program is revised: the valueof variables and block outputs are propagated through the FBD algorithm using connections. Wealready defined operands as values provided to operations and functions, thus we can concludethat connections influence the amount of operands in a program.

Using these rules, we can measure the following number of unique operations and operands:

• Unique operators (η1): POU, Input, INT, MAX, MUL, function

• Unique operands (η2): a, b, a-MAX, b-MAX, MAX-MUL, MAX-square, square (functionname/output)

19


where the a-MAX and similar operands are unique connections between blocks. In this measure-ment we can also observe that the square POU function displayed in figure 4.4 has a default outputvariable with the same name, which is a characteristic of FBD POUs. The rest of the computedmeasurements are shown in table 4.2.

Halstead measurements Result

Number of unique operators 6Number of unique operands 7Total number of operators 8Total number of operands 11Program vocabulary 13Program length 19Calculated program length 35.16Volume 70.31Difficulty 4.71Effort 331.45Testing time 18.41 sDelivered bugs 0.02

Table 4.2: Halstead complexity measurement results of the squareOfBigger FBD implementation.

4.4.5 Information Flow Complexity

The software complexity metrics that we examined in the previous sections are designed to measurethe software complexity of a program on a unit level, to measure the complexity of functions,classes, data structures. However, other software artifacts, like software architecture and softwaresystem design also present a point of software complexity. Henry and Kafura designed a softwarecomplexity metric [12] that could be applied at earlier stages of software development, during thesoftware architecture design.

Figure 4.5: A high-level architecture view of an examined software module D, which is used byA, B, C and is using modules E, F.

IFC of a software module provides information about the amount of coupling between themodule and rest of the software system. As shown in Figure 4.5, a software module (module D)may depend on other software modules and be a dependency to others. In this example, threemodules are using the examined modules, while two modules are being used by it. IFC computesa value based on those two numbers (fan-in for the number of modules using a specific moduleand fan-out for the number of used modules by the examined module):

c = SLOC × (fan-in× fan-out)2 (4.3)

IFC is designed to be used at a system level, but it may be used to measure the information flowbetween procedures or functions of a single program unit. Although IEC 61131-3 POUs (functionsand function blocks) are used as program units, they can be represented as software modules inthe overall software architecture of a PLC software system. Especially in FBD, a function is usedas a block, thus graphically implying it’s module-like properties in an FBD program.

20


IFC measurement of FBD units can be simplified by measuring number of defined inputs andoutputs of an FBD unit (function or function block). This provides a baseline IFC score of anFBD POU, and that value can only increase when it is used in other FBD programs.

If we examine the FBD implementation of the squareOfBigger problem in Figure 3.2, we cancount two input parameters (a,b) and one output parameter (result). Considering the earlierdefined properties of fan-in and fan-out numbers, we concluded that fan-in will be the numberof output parameters and fan-out the number of input parameters. Thus, we can calculate theinformation flow complexity of squareOfBigger FBD function:

c = SLOC × (fan-in× fan-out)2 = 13× (1× 2)2 = 52 (4.4)

The SLoC value was measured with the technique defined in Section 4.4.2. As with the rest ofthe software complexity metrics that we have examined, the measurement result requires a contextto provide any useful information. Thus, if compared with other FBD POUs, a POU with a highinformation flow complexity may be harder to maintain [11].

21

Chapter 5

Tool Supported ComplexityMeasurement

Integrated Development Environments (IDEs) such as Eclipse [23] or IntelliJ [24] provide featuresfor statical analysis of the software being developed. This static analysis of the code might provideinformation about different software complexity metrics of the given program code (e.g., SLoC be-ing a common example). Those metrics can be very useful to the developer during the developmentand testing phase, depending on the context in which they are evaluated.

During the work on this thesis, we used two IEC 61131-3 IDEs: the open-source Beremiz IDE[1] and the enterprise IDE Multiprog [25]. Beremiz was already shown in Section 4.4.4, with allthe support for the IEC 61131-3 standard and available languages of that standard. These toolsare lacking any type of software complexity measurements that some general-purpose IDEs usuallyoffer by default.

This gave us an opportunity to design and develop a tool that would provideautomatic software complexity measurements of FBD programs and report the resultsin a readable file format.

Figure 5.1: Initial architecture and functional parts of the FBD software complexity tool - Tiqva.

Tiqva [26] was developed, during this thesis, as an effort create an automatic software complex-ity measurement tool for FBD programs. We decided to develop the tool in Java, as we wantedan OS-agnostic tool that could be easily used without manual configuration.

In Figure, 5.1 a high-level architecture view of Tiqva is shown containing three essential layers:

• Processing input files: reading XML files containing FBD programs and generating amodel.

• Software complexity metrics: measuring different software complexity metrics using theearlier defined techniques and generated models.

22


• Results writer: Reporting the results in a machine readable file format.

Our goal is to develop a tool that could be easily added to either a continuous integrationprocess or used independently by a developer, engineer or manager in charge of testing (thus, therequirements for a standardized input format (XML) and output format (CSV)).

Using the constructed techniques for measuring software complexity of FBD programs shownin Section 4.4, we devised a set of rules for applying these techniques to an FBD file. However,applying these techniques to the XML contents of an FBD program is not convenient, as it resultsin complicated program code that has to deal with the XML syntax. We resolved this issue bydesigning an FBD data structure model that is similar to the Abstract Syntax Trees (AST) usedfor parsing and processing source code. The data model had to be organized hierarchically and hadto be iterable as most of the techniques required iterations through an organized FBD programcode.

Figure 5.2: A high-level view of the input processing layer of the tool.

We mentioned in Chapter 2 that the IEC 61131-3 programming language standard providesan XSD Schema which defines the structure of an FBD XML file. The use of FBD XDS Schemarules in creation of the data model results (shown in Figure 5.2) in a data model that contains allrelevant program information from the FBD XML file and maintains the hierarchy and relationshipsbetween FBD elements.

The top piece of our hierarchy is the FBD Project object. A Project may have two child models,DataTypes and POUs. POUs may then have Variables, Blocks and Connections. All of the datamodels contain relevant information from the XML file, like the type of the object (input variable,MAX block), names, parameters and similar.

1 @Override2 pub l i c HashMap<Str ing , Double> measureProjectComplexity ( Pro j e c t p r o j e c t ) {3 HashMap<Str ing , Double> metr ic = th i s . addKeysToMap ( ) ;4 metr ic . put ( ”DataTypes” , ( double ) p r o j e c t . getDataTypes ( ) . s i z e ( ) ) ;5 f o r (POU pou : p r o j e c t . getPOUs ( ) ) {6 metr ic . put ( ” Var iab l e s ” , metr ic . get ( ” Var iab l e s ” ) + pou . g e tVar i ab l e s ( ) . s i z e ( ) ) ;7 metr ic . put ( ”Blocks ” , metr ic . get ( ”Blocks ” ) + pou . getBlocks ( ) . s i z e ( ) ) ;8 metr ic . put ( ”Connections ” , metr ic . get ( ”Connections ” ) + pou . getConnect ions ( ) . s i z e ( ) ) ;9 }

10 re turn metr ic ;11 }

Listing 5.1: Snippet of a Tiqva method which measures the NoE metric

To increase modularity and reusability of our tool, we created a Java interface, ComplexityMet-ric, that defines methods for measuring software complexity of an FBD program. The interfacedictates two methods, one for measuring complexity of a single POU, while the other measuresthe complexity of the FBD project, both taking the mentioned data model objects as parametersand performing measurements on them. In Listing 5.1 shows a snippet of the method used formeasuring NoE scores. The snippet shows an implementation of a ComplexityMetric interfacemethod, which performs a number of operations on the FBD data model.

23


Once the software complexity measurements are finished, the collected results have to be re-ported. The SoftwareComplexity interface methods return HashMap objects which contain themeasurement results, with the name of the measurement being the map key. This makes for a con-venient and fast access of the results. For the initial version of Tiqva, we reported the measurementresults in a CSV file.

Maven [27] was used for building and handling the dependencies of the tool. Tiqva dependsstandard Java packages found on the Maven repository. We used DOM4J framework to do theinitial XML parsing, and many features of the tool depend on this framework. Since XML parsingis slow, compared to other operations that are present in Tiqva, we had to use multithreading forcertain parts of the tool to compensate for the overhead introduced by the parsing. If multipleXML files are provided to the tool, each will have an individual thread for XML parsing, modelbuilding, complexity measurement. Up to 100 FBD XML projects can be parsed at a given time.Once all XML files have been measured for software complexity, the results are collected andreported using the CSV Writer.

5.1 The Measurement Approach Applied on an Example

To validate the measurement techniques that we adapted for the FBD programming language andto verify the tool for automatic analysis of FBD programs, we analyzed the stereo cassette recorderexercise and solution that was given as an example exercise in the introductory book for the IEC61131-3 programming language [7]. The FBD solution for this problem embodies most of the thedomain-specific elements of the FBD language, such as different functions, function blocks anddata types.

The given requirements for the control software of the stereo cassette recorder [7] are:

1. Adjustment of the two speakers depending on the current balance control setting(say an integer value between 5 and +5) and the volume control setting (say aninteger value between 0 and +10). The amplifier output needs a REAL data type.

2. Volume control. If the volume exceeds a pre-defined constant value for some time,a warning LED must be turned on. Additionally a warning message is sent to thecalling program.

3. There are two models of this recorder with different limits of loudness.

The graphical representation of the stereo cassette recorder is shown in Figure 5.3 and displaysthe hardware view of the recorder consisting of inputs (volume and balance) and outputs (left andright speaker):

Figure 5.3: Graphical representation of the stereo cassette recorder.

The problem solution contains two POUs: a function Norm and a function block Volume. TheNorm function is used for normalizing volume levels, as requested in the requirements, and theVolume function block realizes the rest of the requirements. We will not go into details about theprogram, however we will use the FBD solution provided in the IEC 61131-3 book for softwarecomplexity measurements.

The FBD source code of the Norm function contains variables declaration and the graphicalprogram code of the function. The variables used in this solution are shown in Table 5.1 and theprogram code is shown in Figure 5.4.

24


ID Name Class Type Initial value1 BIK Input SINT2 LCtrlK Input SINT3 MType Input BOOL4 Calib Local CalType5 Type1 Local REAL 6.06 Type2 Local REAL 4.0

Table 5.1: Defined variables for the Norm function POU.

Figure 5.4: FBD graphical program code of the Norm function.

One non-standard variable type is used in this POU, called CalType. This variable type is acustom data structure (data type) and was created as part of the cassette recorder project, andis in essence a data structure with a REAL value as it’s base type. The Norm function POU isused as a block in the Volume function block POU for normalizing input volume levels. Similarto Norm, Volume function block consists of variable declarations in Table 5.2 and the graphicalprogram code in Figure 5.5.

ID Name Class Type Initial value Option1 BalControl Input SINT2 BalFactor Input SINT3 VolControl Input SINT4 ModelType Input BOOL5 LED Output BOOL6 Critical InOut BOOL7 MaxVolume Local REAL 26.0 Constant8 Timeout Local TIME 2.09 HeatTime Local TON10 Overdrive Local BOOL

Table 5.2: Defined variables for the Volume function block POU.

Figure 5.5: FBD graphical program code of the Volume function block.

25


We implemented these programs in Beremiz and saved the project solution in the appropriateFBD XML format (the v1.01 IEC 61131-3 FBD XML structure).

Before starting software complexity measurements, we had to perform a domain-knowledgeanalysis of the solution, which meant understanding the standard blocks and variables as well assolution-specific blocks. Besides the CalType variable, the Norm function is used in the VolumePOU, and has to be considered when measuring the CC metric, since it contains decision blocks,thus we assigned a cyclomatic weight to the block itself (similar to other decision blocks like MAXor SEL).

The initial step was the manual software complexity measurement of previously defined metrics:NoE, CC, HC and IFC. The manual measurements were verified against the measurements reportedby Tiqva, with the following results reported by the tool:

Name Volume NormVariables 10 6

Blocks 17 11Connections 18 10

Information flow metric 4500 243Cyclomatic complexity 10 2

Unique operators 17 11Unique operands 32 19Total operators 30 18Total operands 41 26

Program vocabulary 49 30Program length 71 44

Calculated program length 229.49 118.76Difficulty 10.89 7.53Volume 398.64 215.9Effort 4341.49 1624.96

Testing time 241.19 90.28Delivered bugs 0.09 0.05

Table 5.3: Software complexity measurements of the stereo cassette recorder POUs.

From the measurement results shown in Table 5.3 it can be concluded that the Volume functionblock has higher software complexity scores compared to the Norm function. In terms of programsize, this conclusion could have been made after examining the graphical program codes shown inFigure 5.5 and Figure 5.4. However, these are complexity measurements of individual POUs (twoPOUs of the Cassette Recorder FBD program), and not the complete FBD program. Comparedto measuring a POU, measuring the complete FBD programs results in the aggregation of somemeasurement results (i.e. NoE measurements of individual POUs will be combined for the completeproject). There is no obvious reason to aggregate also the CC or IFC scores, therefore we onlyreported the highest scores of individual POUs. The software complexity measurements for thecomplete FBD program are shown in Table 5.4.

Some of the results reported in Table 5.4 can be computed by adding together the individualresults of the projects POUs, however, measurements like Unique operators and Unique operandsreported in the table 5.4 also take into account the operators and operands occurrences found inthe CalType data type.

26


Name Stereo cassette recorderVariables 16

DataTypes 1Blocks 28

Connections 28Information flow metric 4500Cyclomatic complexity 10

Unique operators 22Unique operands 52Total operators 50Total operands 69

Program vocabulary 74Program length 119

Calculated program length 394.53Difficulty 14.6Volume 738.92Effort 10785.46Time 599.19

Delivered bugs 0.16

Table 5.4: Software complexity measurements of the stereo cassette recorder FBD project.

Tiqva proved to be helpful addition to the measuring process, especially for very specific detailsof a software program (unique operators and operands). With a textbook example successfullymeasured for software complexity scores using the FBD techniques from Section 4.4, we assumedfurther reported results by Tiqva as valid for further research.

27

Chapter 6

Case Study on the ComplexityMeasurement of FBD IndustrialControl Software

In Chapter 2 we defined software complexity metrics as values that provide information about aproperty of a system depending on the context, which for an instance, can be the size of a softwareprogram or its structure. We developed a set of techniques for measuring software complexityof FBD programs, and used the Tiqva tool to measure the software complexity of a textbookFBD program. However, to gather more representative results, a case study on the complexitymeasurement of an existing industrial control software was performed.

We researched software complexity metrics on domain-specific embedded systems and software- PLCs and FBD. We used this type of software in an industrial case study to investigate therelation between software complexity metrics and test effort attributes. The safety-critical indus-trial control software we used in this case study is part of the Train Control Management System(TCMS), a system developed and used by Bombardier Transportation Sweden AB, a leading com-pany that specializes in building and manufacturing of trains and train equipment. TCMS is anembedded software running on PLCs and used for handling a wide variety of operation-critical andsafety-critical functionalities of a train. TCMS is programmed in IEC 61131-3 FBD programminglanguage, making it viable for our case study, and uses a combination the IEC 61131-3 standardfunction and function blocks, and in-house built function blocks.

6.1 TCMS FBD Programs Selection and Test Effort Mea-surement

Bombardier Transportation Sweden AB provided us with 122 FBD programs from TCMS. Theseprograms perform different operations and are in essence developed independently of each other.The measurement process would be very similar to the automatic software complexity measurementthat we have performed in Section 5.1. In addition, we also wanted to investigate the correlationbetween software complexity measures and the effort needed to test these programs.

We had access to test cases for individual TCMS FBD programs, which have been manuallycreated and used for unit testing. These test cases are used to perform a correlation analysisbetween the test effort and the reported software complexity scores. This correlation could beuseful to estimate to estimate the test effort.

The test effort was not measure directly and we used a proxy measure of effort. This was per-formed before proceeding with the software complexity measurements and correlation analysis. Asmentioned before, test cases were designed for individual FBD programs and consisted of severaltest steps. We also had access to the execution time of these test cases. This implied that we couldexpress test effort for a FBD program as:

28


1. Number of test steps for each program

2. Execution time of the test case for each program.

We note here, that the second test measure contains information about the first; the executiontime of a test case is the execution time of individual test steps. However, we kept both of thesemeasures because we wanted to explore any correlation and relation between test effort and FBDsoftware complexity metrics.

From all the programs provided, 82 had test cases. This resulted in the removal of some ofTCMS FBD programs from our correlation study, as they did not contained a manually createdtest case. We calculated the Kendall’s rank correlation coefficient as a measure of correlationbetween the software complexity measurements and the test effort measures. We discussed theresults in Section 6.3.

20 40 60 80 100

20

40

Test Case ID

Nu

mb

erof

test

step

sin

test

case

20 40 60 80 100

100

200

300

Test Case ID

Tes

tca

seex

ecu

tion

tim

e[s

ec]

Figure 6.1: Number of test steps for individual test case (left) and the execution time ofindividual test case (right).

We separated the test data into two sets for each test effort measure, making the Kendallcorrelation easier to perform based on individual data sets. A graphical representation of the setsis shown in Figure 6.1, each circle representing the test effort measure for that test case. Test effortmeasure data outliers can be identified in Table 6.1, such as the 900 seconds execution time of atest case (not shown in Figure 6.1) or a test case with only 1 test step.

Test effort measure Min Max Average Median Standard Distribution

Number of test steps 1 31 8.5 6 6.236Test case execution 1 900 32.2804 10 105.206

Table 6.1: Test effort measure characteristics.

6.2 Overall Complexity Measurement Results of TCMS

We measured different software complexity metrics of the FBD program set provided by Bom-bardier Transportation Sweden AB using the Tiqva tool that we have developed for this particularscenario. This work can be a tedious and mistake-prone job of measuring software complexity ifmanually performed. Although we were provided with more than a hundred programs, we selectedonly those who had a paired test case, as those could be used for the correlation analysis.

We discussed how similar measurements techniques and tools do not exist for the IEC 61131-3graphical programming language FBD, resulting in a lack of tools that perform FBD softwarecomplexity measurements. Therefore, we could not use other measurement tools to verify ourresults. In addition to our own measurement results, we had access to the file properties of the

29


FBD programs used in the case study (e.g. the memory size of the FBD program, number of XMLlines in a file).

The software complexity measurement techniques for FBD programs discussed in Section 4.4can be broken down into smaller individual measurements, such as the measurement of blocks andconnections for the Number of Elements metric. This resulted in a total of 15 measurements forall exercised software complexity measures (NoE, CC, HC, IFC). The results reported by Tiqvaconsisted of these 15 measurements for the selected 82 programs (shown in Appendix B).

Software complexity measure Xmin Xmax X X std(X)

Variables 4 85 22 18.5 14.7Connections 3 216 33.8 25.5 32.5

Blocks 5 228 39 29.5 35.1Number of elements 12 483 94.8 74.5 80

Cyclomatic complexity 1 133 18.9 13 21.8Information flow complexity 12 57065472 2690651 68506 8478907.8

Unique operators 8 19 11.4 11 2.64Unique operands 9 262 59.5 47.5 47.06Total operators 12 229 59.1 49.5 39.88Total operands 12 402 86.3 69 71.18

Program vocabulary 17 278 71 60 48.59Program length 24 599 145.5 117 110.13

Calculated pogram length 52.5 2168.7 413.2 309.1 385.6Volume 98.1 4844.3 942.5 691.5 886.8

Difficulty 5.3 14.1 8.1 7.9 1.8Effort 523.2 59747.4 8756.5 5470.9 10925.8Time 29 3319.3 486.4 303.9 606.9

Delivere bugs 0.02 0.5 0.1 0.1 0.09

Table 6.2: Software complexity measures and the descriptive statistics of the results.

The results shown in Table 6.2 helped us to develop a notion of a complex FBD program andhow different complexity metrics represent an industrial control software. The different softwarecomplexity measures cannot be directly compared, but most of the values are in the same order ofmagnitude. However, the maximal complexity value for the IFC stands out as the highest valueand is one of the outliers from of the TCMS software complexity measurement. We traced thesehigh complexity values shown in Table 6.2 to their corresponding programs, and gathered thenames of programs that were the cause for these outliers.

Four FBD programs from TCMS had high software complexity scores: Program 9, Program60, Program 32 and Program 55. In particular, Program 32 had some of the highest scores fora number of software complexity measures (Number of Elements, CC, and HC), Program 55 hadthe highest Number of Elements score, Program 60 had the highest IFC score and Program 9the highest Halstead Difficulty score. Upon closer inspection, Program 32 had one of the highestnumber of inputs/outputs (almost as high as Program 60) and also a high Number of Elements(also close to program 55). We could identify Program 32 as the most complex program in theTCMS, but this information on itself does not indicate good or bad software qualities of TCMS.

Since we did not have any other software complexity measurements besides the ones providedby Tiqva, we did not have the information about which software complexity scores implied that aprogram is complex (in respect to that software complexity). We normalized the reported values toget a better look on how individual software complexity metrics are distributed over the completeTCSM program suite. Normalized value 0 represent a very low complexity while a normalizedvalue of 1 was the highest complexity for a particular complexity measure of the TCMS.

The scatter graphs in Figure 6.2 shows the distribution of one size metric (Number of elements)and one structure metric (CC). The outliers can be easily detected (Programs 32 and Program55), but the rest of the programs have half the complexity scores the outliers have. This trendcan also be seen with IFC and Halstead Difficulty (this HC metric is a function of the rest of

30


20 40 60 80

0.2

0.4

0.6

0.8

1

Program ID

NoE

com

ple

xit

y

20 40 60 800

0.2

0.4

0.6

0.8

1

Program ID

Cycl

om

atic

Com

ple

xit

y

Figure 6.2: Normalized values for the Number of elements complexity (left) and Cyclomaticcomplexity(right).

HC measurements). IFC shown in Figure 6.4 is certainly the most polarized, due the enormousdifference between the median value of that complexity and the outlier. Figure 6.3 shows the areachart of two HCs, Halstead Difficulty and Halstead Volume, which are functions of basic Halsteadmeasurements metrics and are used to construct the rest of the metrics (Effort, Testing Time andDelivered bugs). Both of these metrics shown in Figure 6.3 have a very similar shape, which couldimply that HCs will have a similar distribution through out the whole industrial control software.

10 20 30 40 50 60 70 80

0.5

1

1.5

2

Program ID

Hal

stea

dM

etri

cs

VolumeDifficulty

Figure 6.3: A normalized area chart of two Halstead metrics (Volume and Difficulty), showingthe similar distribution of said metrics across the TCSM.

A similar distribution to CC and Number of Elements can also be observed for the HalsteadEffort metric. The outliers are again programs 61 and 32. The similar measurement results andthe repeating outliers suggest that these complexity metrics measure the same software dimension(size) by measuring different properties.

The set of used software complexity measurements, consisting of Number of Elements, HalsteadMetric, CC and Information Flow metric, contained two size metrics (NoE, Halstead), one structuremetric (CC) and one architectural metric (IFC). However, we got very similar measurement resultsfor size metrics and structure metrics. This could be an indicator that the way we measured thestructural complexity of FBD programs (the proposed cyclomatic complexity technique) does notcorrectly capture the structure of the program, or that FBD program structure is a different typeof property compared to structures of general purpose programming languages.

31


20 40 60 800

0.2

0.4

0.6

0.8

1

Program ID

Info

rmat

ion

Flo

wM

etri

c

20 40 60 80

0.2

0.4

0.6

0.8

1

Program ID

Hal

stea

deff

ort

com

ple

xit

y

Figure 6.4: Normalized values for the Information flow metric (left) and Halstead effortcomplexity(right).

6.3 Is Software Complexity Correlated With Test Effort?

The descriptive statistics of the software complexity measurement results helped in identifyingoutliers and obtaining the notion of what program properties could have an impact on softwarecomplexity and whether or not software complexity metrics are similar in nature. Number ofElements, CC and Halstead Effort had a similar distribution across the analyzed industrial controlsoftware.

To provide answers for the third research question (RQ3), a correlation analysis between themeasured software complexity scores and test effort proxy measures was performed to determinethe possible statistical relationship between the complexity of an industrial control software andthe test effort attribute. We used Kendall’s [28] rank correlation coefficient to calculate the sta-tistical relationship between the scores, with a significance level of 0.05 for the significance test.We performed the correlation analysis with every defined complexity measure, as well as basiccomplexity measure units (number of operators, number of connections and similar).

Software complexity metrics τE p-valueE τN p-valueN

Number of Elements 0.342 8.192e−6 0.368 2.315e−6

Cycloamtic Complexity 0.225 0.003 0.252 0.001358Information Flow Metric 0.264 0.0005 0.345 9.116e−06

Halstead Volume 0.328 1.878e−5 0.351 6.25e−6

Halstead Difficulty 0.208 0.006 0.125 0.1061Halstead Effort 0.32 2.882e−5 0.32 3.876e−5

Table 6.3: Kendall correlation coefficient (τ) and Kendall p-value between software complexitymetrics and test effort. The test effort was expressed by two proxy values, test case execution

time (E) and the number of test steps in a test case (N).

The two test effort proxy measures required separate correlation tests, which were done inR programming language [29] and are reported in Table 6.3. This table shows τ and p-valuesfor the two proxy measures (E - Test case execution time measure, N - Number of test stepsin a test case measure). A positive correlation can be observed for four software complexitymetrics (HC is shown as three separate complexity measures - Difficulty, Volume and Effort).However, the Kendall p-valueN for Halstead Difficulty is bigger than the significance level, thusnot rejecting the null hypothesis (no statistical relationship between the said metric and the testeffort proxy measure). The rest of the correlation results reject the null hypothesis, ensuring astatistical relationship between software complexity measures and the test effort. Overall, theKendall correlation coefficient τ shows a week correlation, with τN being the biggest with a scoreof 0.368 for the number of elements metric.

32


These results show that there is a statistical relationship between the defined software com-plexity measures and test effort measures. Nevertheless, high software complexity scores do notnecessary imply that the test effort will also be high. The Number of Elements has the highestcorrelation scores and is the metric that captures the size of an FBD program. This implies thattest effort is influenced by the size of the program that is being tested. On other hand, the Cy-clomatic complexity metric, a structure metric, has a lower correlation than Number of Elementssize metric. This can be taken as an argument in favor of not measuring the structure of FBDprograms in this way, as these programs do not branch in the same way that programs written ina general-purpose programming language do.

6.4 Predicting Test Effort using Software Complexity Scores

In the previous section we concluded that there is a weak yet significant correlation between themeasured software complexity metrics and the proxy measures of test effort. Ideally, the test effortcould be estimated based on the computed software complexity scores. In statistical analysis,predicting a dependent variable using correlated independent variables can be done using linearregression [30]. In our case, we used multiple linear regression (multiple independent variable).

RQ3 Answer: To predict test effort using software complexity scores, a statistical rela-tionship between software complexity metrics and test effort measures can exist. A positivecorrelation was measured (in Section 6.3), implying that statistical analysis methods such aslinear regression can be used to predict test effort using software complexity metrics scores.

If we want to predict the amount of test effort required for adequate testing of an FBD program,we would have to form a linear regression model of the test effort measure using the softwarecomplexity measures:

β1C1 + β2C2 + · · ·+ βn−1Cn−1 + βnCn = Tmeasure, (6.1)

where the test effort measure Tmeasure is a linear function of different weighted software com-plexity scores Cns. The linear regression model shown in Equation 6.1 could be used to predict thetest effort for an industrial control software after determining the weight values (βs), and althoughit requires an existing set of software complexity measurements and test effort to be solved usinga learning algorithm, a model like this could be used for creating a rough test effort estimator fornew FBD programs that will be added to an industrial control software.

Using the previously measured software complexity values as the input data set, and the twotest effort proxy measures as the output dataset, we determined weights using a Python machinelearning library [31]. We used the LinearRegression module to determine the weights as well as toassign a variance score (the amount of correct predictions of the model with 1.0 being the highestscore). Two regression models have been formed based on the previous equation (Equation 6.1)and our translated software complexity metrics for FBD programs:

βNoECNoE + βCCCCC + βHECCHEC + βIFMCIFC = TN (6.2)

βNoECNoE + βCCCCC + βHECCHEC + βIFMCIFC = TE, (6.3)

following the same software complexity notation from the statistical analysis: NoE - Numberof Elements, CC - Cyclomatic Complexity, HEC - Halstead Effort Complexity, IFC - InformationFlow Complexity, N and T - Number of Test Steps and Test Case Execution time test effortmeasures. Only Effort has been taken into account from the different HC metrics.

We assumed that the linear regression model will not have a high prediction percentage consid-ering the weak correlation between software complexity metrics and the same could be concludedafter examining the trained linear regression model. The full data set of 82 units has been usedfor training and for testing the model, however the results in Table 6.4 show that only half ofthe test effort predictions were correct. The mean squared value was lower when the model tried

33


to predict the number of test steps proxy measure and significantly bigger for the execution timeproxy measure. This can be traced back to the non-linear and irregular shape of the executiontime test effort measure shown in Figure 6.1. Another characteristic of the model is the higher βweight value for Number of Elements in both models, which strengthens the argument that FBDtest effort for TCMS investigated in this empirical study was mostly influenced by the size of theFBD programs.

Test Effort Measure βNoE βCC βHEC βIFM MSE ScoreNumber of Test Steps 1.118e−7 4.69e−2 −6.528e−4 1.36e−7 17.31 0.55

Test Case Execution Time 1.92 −1.4 −3.82e−3 −5.56e−6 5077.68 0.54

Table 6.4: Software complexity weights, mean square error of predictions made based on saidweights and the variance score of the predictions.

0 5 10 15 20 25 30Predicted Number of Test Steps

0

5

10

15

20

25

30

Mea

sure

d Nu

mbe

r of T

est S

teps

Measured Test EffortPredicted Test Effort

0 200 400 600 800Predicted Test Case Execution Time

200

0

200

400

600

800

Mea

sure

d Te

st C

ase

Exec

utio

n Ti

me

Measured Test EffortPredicted Test Effort

Figure 6.5: Visualized predictions of the trained linear regression model. The two plots show thepredictions of the model for the test effort proxy measures (number of test steps - left, test case

execution time - right).

The plots in Figure 6.5 visualize the predictions (blue scatter points) of the test effort in contrastto the measured test effort (black line). Although the predictions of the test execution time arealmost as good as the predictions for the number of test steps, they are clustered around onearea (left plot, lower test execution time is mostly predicted), while the number of test steps isdistributed across the complete test effort spectrum).

The attempt to find a good prediction model of FBD test effort measure shows that an estima-tion can be made using the software complexity measurement scores. In Section 2.3 we emphasizedthat test cases are created based on different software requirements (the expected behavior). Thesetest cases are designed using different information sources; like the software specification or humandomain knowledge. In the linear regression model (Equation 6.1) for predicting test effort of TCMSFBD programs, we did not include other sources that contribute to the overall test effort (e.g.,FBD program specification).

6.5 Summary

In the effort to estimate IEC 61131-3 FBD software complexity measurements with certain testeffort attributes, a correlation analysis between complexity measurements and proxy test effortmeasures for an existing industrial control software has been performed.

We examined a train control software - TCMS, which is a valid and representative case for IEC61131-3 FBD software used in the industry. The set of FBD programs and their corresponding tests,although relatively small in numbers, were used for measuring software complexity and analyzingthese results. Complex programs of TCMS have been successfully identified (programs 32 and59). These results also helped in indicating that FBD programs cannot be directly measured forstructural complexity as program written in a general-purpose programming language can.

34


By using the measurement results and the known test effort (i.e., the number of tests stepsof a test case and the execution time of that test case), the correlation analysis showed that theset of software complexity metrics adapted for the FBD graphical programming language do havea statistical relationship with the test effort measures. However, the correlation is weak (0.368Kendall’s Tau coefficient being the highest correlation). There are indications that higher FBDsoftware complexity does not imply a higher test effort. We developed a test effort predictionmodel, designing a linear regression model of the test effort using the software complexity metrics,The results are promising but but we only achieved a rough estimator of test effort. However,this was expected as other software artifacts (e.g., specification, domain-knowledge) are heavilyinfluencing the overall test effort of the software.

6.6 Threats to Validity

Industrial control software used in PLCs can be programmed in a wide variety of programminglanguages, such as IEC 61131-3 FBD and ST languages. In this thesis, we examined how soft-ware complexity metrics tie in together with industrial control software developed in FBD by onecompany (Bombardier Transportation Sweden AB), thus narrowing the scope of the case study.However, the examined software (TCMS) shows general characteristics of the safety-critical indus-trial domain.

The set of software complexity metrics chosen to be viable for FBD programs is a subset ofsoftware code complexity metrics and does not contain other complexity metrics that are proposedin computer science literature, such as entropy [32], Kolmogorov complexity [33] and similar. Thepurpose of this thesis is to investigate a general applicability of thoroughly researched softwarecomplexity metrics on a domain-specific graphical programming language, as the first step in aneffort to quantify and describe software complexity of industrial control software.

In the process of adapting software complexity measurement techniques to the specific syntaxof the IEC 61131-3 graphical programming language FBD, number of possibly false assumptionshave been made for FBD syntax elements (blocks, connections, POUs). To verify the correctnessof these assumptions, textbook examples of IEC 61131-3 programs have been used in the feasibilitystudy, which meant measuring the software complexity of both, FBD and ST implementations ofsuch programs. Cross-validating the results helped in removing false assumptions.

One of the research goals of the thesis was the examination of how FBD software complexityinfluences certain software qualities or characteristics. Since there are numerous software quality,we chose one quality that could be empirically measured - the test effort. Although other qualitiesexist, such as the maintainability, readability, adaptability, modularity, we researched only testeffort as it was the only quality for which we had data usable in experiments. The findings of thisthesis can be very well used for additional research of the relationship between the complexity ofFBD programs and other software qualities.

The test effort measure is a complex function that requires knowledge of multiple phases ofsoftware testing to be fully able to describe the testability of an FBD program. Although test effortmeasure has many measurable and objective parts, a software tester’s perception of the programalso influences the overall testability of the software. In the correlation analysis, we used test effortmeasures that can be measured and are not a subject of software tester’s vision of the program,thus leaving out the input of software testers.

35

Chapter 7

Conclusion

As embedded systems become more dominant in both every day life, and in most industry sectors,it is required to investigate how established software engineering methods and techniques canbe applied to embedded software. In our effort to explore the quantitative nature of embeddedsoftware used in the industry - industrial control software, we propose different ways to measurethe software complexity of such software. In this thesis we established two objectives, to see howsoftware complexity metrics can be applied to industrial control software written in the IEC 61131-3 FBD graphical programming language, and the impact of software complexity on the softwarequality of FBD programs.

Our first effort was to analyze how existing and well researched software complexity metrics -Source Lines of Code, Cyclomatic complexity, Halstead complexity, Information Flow complexity,can be applied to the graphical syntax of the FBD programming language. We developed a tool,Tiqva, to streamline the measurement process of multiple FBD programs.

To see how FBD software complexity impacts a software quality, we performed a case studywhere we examined the Train Control Management System (TCMS), an industrial control softwaredeveloped by Bombardier Transportation AB and used in their trains. We analyzed the influenceof software complexity metrics on the testability of the TCMS in a correlation analysis. Our resultsshow that size of a program, thus size software complexity metrics, have the highest impact on therequired test effort of a TCMS FBD program. Additionally, we created a prediction model basedon the measurement results and the test effort with relatively promising results.

Our work presents an initial exploration on how software complexity measurement can be usedfor domain-specific software written in the FBD graphical programming language. We used four,well known, software complexity metrics in our study, as the first stepping stone in this researchdirection. It is still required to see how other software complexity metrics [20, 21] can be applied toFBD programs and if they can capture the appropriate complexity dimension of FBD programs.

The results of this thesis show that for a particular industrial control software, the size of anFBD program has the highest influence on the FBD software complexity. The software complexitymeasurement results from the case study show that four programs fro TCMS are outliers, havinga higher than average software complexity scores from all the programs (NOE - 483, CC - 133,Halstead Effort - 532.2, IFC - 57065472), as well as being the largest programs in terms of FBDXML lines.

The correlation analysis shows that the Number of Elements metric has the highest significantKendall correlation coefficient for the two test effort proxy measures, 0.342 for Test Case Executionmeasure and 0.368 for Number of Test Steps in a Test Case measure. On the other hand, HCDifficulty software measurement has not been successfully correlated with Number of Test Stepstest effort measure, failing the significance test with a score of 0.1061.

The results from the case study also indicate that a new software complexity metric might berequired to capture the software complexity dimensions of FBD programs (function block rela-tionships, block coupling, timing) that are not represented in programs written in general-purposeprogramming languages.

Other future research might explore how the software complexity metrics analyzed in thisthesis can be used in other industrial control software, like aircrafts, power-plants, chemical plants

36


or similar, and if the results of those measurements correlate to software test effort or qualitycharacteristics.

Embedded systems, and domain-specific software require additional attention when it comes tomeasuring their attribute. As mentioned, this thesis presents an initial exploration that has beenused in software engineering research in general for several years. This line of research has notbeen used widely for domain-specific software.

37

Bibliography

[1] “Home page of beremiz,” http://www.beremiz.org/, (Accessed on 05/15/2017).

[2] M. H. Halstead, Elements of software science. Elsevier New York, 1977, vol. 7.

[3] T. Mens, “Research trends in structural software complexity,” arXiv preprintarXiv:1608.01533, 2016.

[4] T. J. McCabe, “A complexity measure,” IEEE Transactions on software Engineering, no. 4,pp. 308–320, 1976.

[5] A. H. Watson, D. R. Wallace, and T. J. McCabe, Structured testing: A testing methodologyusing the cyclomatic complexity metric. US Department of Commerce, Technology Admin-istration, National Institute of Standards and Technology, 1996, vol. 500, no. 235.

[6] D. Kafura and G. R. Reddy, “The use of software complexity metrics in software maintenance,”IEEE Transactions on Software Engineering, no. 3, pp. 335–343, 1987.

[7] K.-H. John and M. Tiegelkamp, IEC 61131-3: programming industrial automation systems:concepts and programming languages, requirements for programming systems, decision-makingaids. Springer Science & Business Media, 2010.

[8] A. Abran, Software metrics and software metrology. John Wiley & Sons, 2010.

[9] E. J. Weyuker, “Evaluating software complexity measures,” IEEE transactions on SoftwareEngineering, vol. 14, no. 9, pp. 1357–1365, 1988.

[10] M. Shepperd, “A critique of cyclomatic complexity as a software metric,” Software EngineeringJournal, vol. 3, no. 2, pp. 30–36, 1988.

[11] K. Kevrekidis, S. Albers, P. J. Sonnemans, and G. M. Stollman, “Software complexity andtesting effectiveness: An empirical study,” in Reliability and Maintainability Symposium, 2009.RAMS 2009. Annual. IEEE, 2009, pp. 539–543.

[12] S. Henry and D. Kafura, “Software structure metrics based on information flow,” IEEE trans-actions on Software Engineering, no. 5, pp. 510–518, 1981.

[13] W. Bolton, Programmable logic controllers. Newnes, 2015.

[14] P. Ammann and J. Offutt, Introduction to software testing. Cambridge University Press,2016.

[15] P. Stocks and D. Carrington, “A framework for specification-based testing,” IEEE Transac-tions on software Engineering, vol. 22, no. 11, pp. 777–793, 1996.

[16] “Ieee standard for software and system test documentation,” IEEE Std 829-2008, pp. 1–150,July 2008.

[17] M. Olszewska, Y. Dajsuren, H. Altinger, A. Serebrenik, M. Walden, and M. G. van den Brand,“Tailoring complexity metrics for simulink models,” in Proccedings of the 10th European Con-ference on Software Architecture Workshops. ACM, 2016, p. 5.

38

http://www.beremiz.org/


[18] H. Prahofer, F. Angerer, R. Ramler, H. Lacheiner, and F. Grillenberger, “Opportunities andchallenges of static code analysis of iec 61131-3 programs,” in Emerging Technologies & FactoryAutomation (ETFA), 2012 IEEE 17th Conference on. IEEE, 2012, pp. 1–8.

[19] D. S. Kushwaha and A. K. Misra, “Software test effort estimation,” ACM SIGSOFT SoftwareEngineering Notes, vol. 33, no. 3, p. 6, 2008.

[20] C. L. McClure, “A model for program complexity analysis,” in Proceedings of the 3rd inter-national conference on Software engineering. IEEE Press, 1978, pp. 149–157.

[21] D. N. Card and R. L. Glass, Measuring software design quality. Prentice-Hall, Inc., 1990.

[22] T. J. McCabe and C. W. Butler, “Design complexity measurement and testing,” Communi-cations of the ACM, vol. 32, no. 12, pp. 1415–1425, 1989.

[23] “Eclipse - the eclipse foundation open source community website.” https://eclipse.org/, (Ac-cessed on 05/15/2017).

[24] “Intellij idea the java ide,” https://www.jetbrains.com/idea/, (Accessed on 05/15/2017).

[25] “Index - kw-software — en,” http://www.kw-software.com/en/, (Accessed on 05/15/2017).

[26] A. Muslija, “amuslija/fbd-complexity-tool 0.0.2,” May 2017. [Online]. Available: https://doi.org/10.5281/zenodo.580762

[27] “Maven welcome to apache maven,” https://maven.apache.org/, (Accessed on 05/15/2017).

[28] M. G. Kendall, “Rank correlation methods.” 1948.

[29] “Kendall rank coefficient — r tutorial,” http://www.r-tutor.com/gpu-computing/correlation/kendall-rank-coefficient, (Accessed on 04/27/2017).

[30] D. C. Montgomery, E. A. Peck, and G. G. Vining, Introduction to linear regression analysis.John Wiley & Sons, 2015.

[31] “scikit-learn: machine learning in python scikit-learn 0.18.1 documentation,” http://scikit-learn.org/stable/index.html, (Accessed on 04/29/2017).

[32] C. E. Shannon, “Prediction and entropy of printed english,” Bell Labs Technical Journal,vol. 30, no. 1, pp. 50–64, 1951.

[33] A. N. Kolmogorov, “On tables of random numbers,” Sankhya: The Indian Journal of Statis-tics, Series A, pp. 369–376, 1963.

39

https://eclipse.org/

https://www.jetbrains.com/idea/

http://www.kw-software.com/en/

https://doi.org/10.5281/zenodo.580762

https://doi.org/10.5281/zenodo.580762

https://maven.apache.org/

http://www.r-tutor.com/gpu-computing/correlation/kendall-rank-coefficient

http://www.r-tutor.com/gpu-computing/correlation/kendall-rank-coefficient

http://scikit-learn.org/stable/index.html

http://scikit-learn.org/stable/index.html

Appendix A

Test Effort

ID Number of Test Steps Test Case Execution Time [sec]1 15 162 4 103 19 2654 30 305 11 116 10 107 4 48 7 79 11 1710 4 411 5 512 6 613 12 11814 10 6815 4 1616 14 18017 7 1318 6 2219 3 320 6 621 5 822 2 223 4 724 15 1525 3 326 6 627 3 328 4 429 7 2730 7 831 6 732 17 7533 11 1134 7 735 2 336 3 337 6 638 8 8

40


39 6 640 4 441 9 942 4 443 10 1044 3 645 4 1046 10 1047 1 148 16 1649 14 4450 9 951 24 2452 9 12653 6 654 21 2155 31 90056 18 3657 17 1658 5 1059 3 360 24 2461 13 2662 12 1263 5 564 9 965 18 1866 4 467 5 568 5 569 9 970 4 471 6 672 4 1273 4 1574 5 1575 4 1076 4 1077 6 1578 5 13079 11 1180 4 1081 11 1182 11 11

Table A.1: Train Control Management System Test Effort expressed via Number of Test Stepsand Test Case Execution Time.

41

Appendix B

Software ComplexityMeasurement Results

ID UniqueOperators Difficulty Effort Time1 8 5.6 962.61 53.482 12 8.57 5031.42 279.523 13 10.43 25604.37 1422.474 10 7.26 16831.62 935.095 11 7.7 7697.45 427.646 15 10.45 15193.44 844.087 11 7.82 5269.46 292.758 14 9 2169.69 120.549 19 14.1 49087.5 2727.0810 8 5.43 919.92 51.1111 8 5.5 1815 100.8312 10 7.19 2945.55 163.6413 12 8.25 6579.03 365.514 11 8.25 5629.69 312.7615 11 7.89 2248.22 124.916 17 12.57 35788.94 1988.2717 13 8.88 13377.13 743.1718 10 6.75 1622.95 90.1619 8 5.33 523.2 29.0720 8 5.5 1815 100.8321 16 11.41 11963.41 664.6322 8 5.94 6122.14 340.1223 15 11.8 20358.83 1131.0524 10 7.56 9771.96 542.8925 9 6.43 2344.25 130.2426 12 8.56 6671 370.6127 11 7.89 5477.46 304.328 12 7.94 3100.32 172.2429 9 6.35 4840.87 268.9430 13 9.54 6651.28 369.5231 13 9.54 6651.28 369.5232 17 12.33 59747.48 3319.333 16 10.56 10431.98 579.5534 10 7.08 2234.24 124.1235 10 7.25 1849.9 102.7736 8 5.67 783.71 43.54

42


37 13 8.79 6753.88 375.2238 13 8.81 12254.42 680.839 12 8.72 6824.59 379.1440 14 10.14 12701.88 705.6641 15 10.87 11811.89 656.2242 12 8.81 13810.37 767.2443 13 9.08 10447.26 580.444 14 10.03 3998.67 222.1545 14 9.64 7193.81 399.6646 13 8.79 11401.43 633.4147 9 6.62 3565.59 198.0948 9 6.82 2723.2 151.2949 12 8.37 4486.87 249.2750 12 8.43 4899.05 272.1751 12 8.57 12838.08 713.2352 13 8.86 3965.67 220.3153 8 5.87 1034.99 57.554 8 5.95 9449.72 524.9855 16 12.27 56805.32 3155.8556 10 7.92 13315.68 739.7657 13 9.88 17525.33 973.6358 8 5.83 1895.83 105.3259 10 7.11 3571.44 198.4160 8 5.96 11031 612.8361 9 7.34 16450.19 913.962 8 5.92 4968.15 276.0163 11 8.17 3971.91 220.6664 10 7.05 1902.27 105.6865 11 7.98 5464.43 303.5866 13 9.24 4453.6 247.4267 9 6.68 2954.5 164.1468 9 6.52 1584.92 88.0569 10 6.83 2654.76 147.4970 8 5.88 2854.76 158.671 10 6.67 1831.49 101.7572 10 6.96 1965.12 109.1773 11 7.22 2859.47 158.8674 14 8.97 3864.08 214.6775 11 7.97 7013.33 389.6376 14 9.02 6214.8 345.2777 14 9.36 10481.1 582.2878 18 12.3 13297.23 738.7479 11 7.72 8051.13 447.2980 10 7.25 4050.88 225.0581 9 6.64 4257.97 236.5582 12 8.54 2912.6 161.81

43


ID CalculatedProgramLength UniqueOperands TotalOperators InformationFlow1 82.6 15 17 3522 269.5 42 42 368643 994.96 134 126 99507204 971.57 133 131 64739845 429.51 65 69 5184006 634.94 89 93 8640007 285.19 45 52 1199528 145.54 21 20 2709 1555.09 194 162 1094529710 77.3 14 19 32011 134.04 24 33 921612 193.22 32 30 313613 368.23 56 54 10886414 306.13 48 44 3120015 142.1 23 23 126016 1249.76 161 143 613267217 656.25 93 97 173793618 119.66 20 22 44819 52.53 9 12 1220 134.04 24 33 921621 477.95 68 67 17463622 422.93 66 68 8820023 690.76 96 103 38234724 523.48 78 82 72926725 163.14 28 30 420026 353.78 54 52 6123627 292.14 46 53 42688828 196.6 31 31 70429 317.82 51 57 20000030 309.17 47 49 7577631 309.17 47 49 7577632 2108.05 255 229 2837315633 508.23 72 58 4360034 143.26 24 28 1231235 119.66 20 23 432036 67.02 12 15 28837 337.4 51 59 23833638 632.37 90 86 64749639 346.6 53 53 30240040 543.56 78 79 43920041 480.09 69 70 25930842 675.18 96 91 110073643 499.96 73 77 57024044 200.51 30 30 1080045 356.88 53 50 13120046 592.9 85 81 50803247 214.65 36 45 24576048 182.11 31 28 999649 242.44 38 42 1408050 269.5 42 42 248451 635.23 91 94 130867252 214.57 33 36 1800

44


53 82.6 15 17 82854 592.43 88 110 2936663155 2168.76 262 168 9466856 665.38 96 98 1988388057 696.35 98 112 1055129658 134.04 24 30 2437559 232.64 38 36 486060 688.39 100 125 5706547261 899.25 125 113 2593338862 320.42 52 65 182476863 217.58 35 36 2200064 131.33 22 23 198465 327.35 51 41 550466 247.53 38 31 396867 194.99 33 33 510068 114.97 20 21 204869 180.43 30 32 922570 196.97 34 40 13500071 143.26 24 22 90072 137.26 23 24 211273 198.05 32 31 122574 213.3 32 37 235275 392.47 60 56 5702476 349.73 52 47 1687577 535.85 77 69 3855678 511.69 71 70 9540079 444.48 67 72 152266480 246.1 40 41 3049281 255.01 42 51 53433682 165.23 26 28 9472

45


ID Variables NumberOfElements TotalOperands ProgramVocabulary1 6 22 21 232 16 64 60 543 50 230 215 1474 50 209 193 1435 27 100 91 766 36 135 124 1047 21 68 64 568 6 30 27 359 58 313 288 21310 7 20 19 2211 13 36 33 3212 11 49 46 4213 20 84 77 6814 16 78 72 5915 8 35 33 3416 54 252 238 17817 35 149 127 10618 8 28 27 3019 4 12 12 1720 13 36 33 3221 26 99 97 8422 32 98 98 7423 41 147 151 11124 33 123 118 8825 12 42 40 3726 19 84 77 6627 21 72 66 5728 11 44 41 4329 22 80 72 6030 19 74 69 6031 19 74 69 6032 85 401 370 27233 18 109 95 8834 10 38 34 3435 9 30 29 3036 5 18 17 2037 23 76 69 6438 33 136 122 10339 20 84 77 6540 30 122 113 9241 27 108 100 8442 33 156 141 10843 32 110 102 8644 10 48 43 4445 17 82 73 6746 31 128 115 9847 17 60 53 4548 10 51 47 4049 17 55 53 5050 14 69 59 5451 37 142 130 10352 13 50 45 46

46


53 6 23 22 2354 43 151 131 9655 39 483 402 27856 38 170 152 10657 45 161 149 11158 11 39 35 3259 12 60 54 4860 49 172 149 10861 42 227 204 13462 25 88 77 6063 14 55 52 4664 9 31 31 3265 12 86 74 6266 9 62 54 5167 13 51 49 4268 7 32 29 2969 13 41 41 4070 16 54 50 4271 7 36 32 3472 9 33 32 3373 10 49 42 4374 13 48 41 4675 20 99 87 7176 16 75 67 6677 23 119 103 9178 26 106 97 8979 27 104 94 7880 16 63 58 5081 20 69 62 5182 11 37 37 38

47


ID Volume CyclomaticNumber DataTypes1 171.9 3 02 587 5 03 2455.09 56 04 2319.8 73 05 999.67 21 06 1454 43 07 673.65 28 08 241.08 4 09 3480.62 57 010 169.46 11 011 330 21 012 409.82 5 013 797.46 22 014 682.39 16 015 284.9 8 016 2848.25 63 017 1507.05 46 018 240.44 8 019 98.1 6 020 330 21 021 1048.34 24 022 1030.77 2 023 1725.78 4 024 1291.89 42 025 364.66 3 026 779.73 24 027 694.11 8 028 390.69 15 029 761.99 26 030 697.01 16 031 697.01 16 032 4844.39 133 033 988.29 36 034 315.42 4 035 255.16 2 036 138.3 1 037 768 29 038 1390.79 26 039 782.91 11 040 1252.52 41 041 1086.69 34 042 1567.13 47 043 1150.3 14 044 398.54 4 045 746.13 9 046 1296.48 25 047 538.2 1 048 399.14 6 049 536.17 14 050 581.24 12 051 1497.78 33 052 447.41 12 0

48


53 176.42 3 054 1586.98 1 055 4627.8 92 056 1681.98 2 057 1773.34 19 058 325 1 059 502.65 10 060 1850.84 1 061 2239.95 27 062 838.78 1 063 486.07 17 064 270 7 065 684.73 19 066 482.16 15 067 442.17 5 068 242.9 5 069 388.5 12 070 485.31 6 071 274.72 6 072 282.49 7 073 396.12 9 074 430.84 9 075 879.41 14 076 689.06 17 077 1119.34 21 078 1081.45 24 079 1043.38 6 080 558.74 6 081 640.98 1 082 341.12 4 0

Table B.4: Software Complexity Measurement Results of the Train Control Management System(TCMS).

49

on the complexity measurement of …1113035/fulltext01.pdf · 4.4 beremiz v1.2 [1] - an open source...

Documents