1997 supporting inspections with an electronic meeting system

8/8/2019 1997 Supporting Inspections With an Electronic Meeting System

1/7

Supporting Inspections With an Electronic Meeting SystemMichiel van Genuchten, S&P Consulting/ Blenks GroupwareVan Meursstraat 33-5615 JN Eindhoven, The Netherlandse-mail: [email protected]

Wleger CornelissenPhilips Medical SystemsBuilding QP-2 - Postbox 10000-5680 DA Best, The Netherlands

Cor van DijkBaan CompanyPost Box 143-3770 AC Barneveld, The NetherlandsTel. 0342427231e-mail: [email protected]

Abstract

Fagan inspections are a structured review ofdevelopment documents that consists of individualpreparation, a meeting and rework by the author of thedocument. The meeting is used to log the defects found inpreparation and to search for more defects. Theeffectiveness and eficiency of the meeting is typically lowas compared to that of the preparation. This paperdescribes the use of an EMS to support the loggingmeeting of a total of 14 electronic inspections in PhilipsMedical Systems and Baan Company. The results indicatethat the electronic logging meeting contributed muchmore to the overall result of the inspection than was thecase in a traditional inspection. The results haveimplications, both for the softiare inspections and ELLS.The implications for EMS as they are discussed in thispaper are the increased use of meeting metrics, the use ofinspections as a guinea pig and the benefits of jixedformat input in an EMS.

1. INTRODUCTIONSoftware engineering is not known for its predictability

and quality. One could argue that the software industry isa problem ahead and not a solution behind if it iscompared to other disciplines. Constructing a high quality,million line program is a compelling task for a group ofengineers. There is no question that software engineeringneeds to improve considerably and that it can use manymethods and techniques originally developed in otherdisciplines. Howeven other disciplines can also learn from

the methods and techniques that software engineersdevised to solve part of their problems.One example of such a technique is a Fagan inspectionwhich is a structured review aimed to detectdefects in development documents or code. Faganinspections have proven to be one of the most effectiveways to improve software quality [3,6,5,11]. It consists ofindividual preparation, a meeting in which the defects arelogged and the group searches for more defects, followedby reworking the defects by the author of the document.The effectiveness and efficiency of the logging meeting istypically low ascompared to that of the preparation. Thereis an ongoing discussion among software engineers andresearchers [10, 12] whether an inspection needs ameeting.One can also approach this from another angle: can themeeting be improved to the extent that it can contribute toeven more effective and efficient inspections. Improvingmeetings by the use of IT has been the main goal of theresearchers and practitioners that have been working onelectronic meeting systems (EMS) over the last 20 years[9]. This paper describes the use of an EMS to support thelogging meeting of a total of 14 electronic inspections inPhilips Medical Systems and Baan Company.

2. INSPECTIONSAn inspection aims to detect the defects in development

documents such as specifications design and code.Inspections have been carried out by engineers around theworld for more than twenty years. Key characteristics ofinspections are individual preparation, data collection anda fixed syntax to report defects. More information oninspections is provided in numerous sources [3,6,5].

1060-3425/97 $10.00 (c) 1997 IEEE


2/7

a fixed syntax to report defects. More information oninspections is provided in numerous sources [3,6,5].

As any process, inspections can be improved. The focusof the effort described in this paper has been to improvethe effectiveness and efficiency of the logging meeting. Alogging meeting has two purposes:- consolidating the defects found by the inspectors into onelist- look for more defects during the logging meeting. Thedefects reported by the other inspectors fimction as atrigger to detect more defects.

During the logging meeting the participants go throughthe document and state the defects out loud. The defectsmay have been found in preparation or during the loggingmeeting. The moderator leads the meeting and the scriberecords all the defects. The logging meeting is key to aninspection. Experienced groups claim to find many defectsduring the logging meeting.

One could say that it is quite a primitive way toconsolidate the lists of defects of the individual inspectorsinto one list for the group. The first and second goal of thelogging meeting often turn out to be conflicting: inspectorsget distracted by the other inspectors stating a defect. Themoderator is usually busy to prevent discussion and tocontrol the logging rate. Inspectors should pick up thepointer to find new defects but they are often alsodistracted by another minor reported. The logging meetingin an inspection was supported with an Electronic MeetingSystem in an attempt to overcome some of these problems.3. EMS SUPPORT FOR INSPECTIONS

The original inspection process was followed in theelectronic inspections. Instead of stating the defects outloud the inspectors submit the defects into a networkedcomputer. Defects appear on the screen of all the otherparticipants. As such the defects can still fimction as atrigger for the other inspectors. The first electronicinspection all the inspectors came in with their defectswritten down on a piece of paper as is usually the case inan inspection. It took considerable time to type in all thedefects. From the second inspection on the defects thatwere found in preparation were mailed on beforehand andconsolidated in one list before the logging meeting started.The list was imported into the meeting system so that theonly keyboarding that was required was for the defectsfound during the meeting. During the first meeting it alsobecame clear that the author wanted an explanation forsome of the defects reported. It was decided to allow ashort electronic discussion between the author and oneinspector. The discussion could only be provoked by theauthor and was not allowed to involve more than fourcomments. The advantage is that the discussion does not

distract the others while they can see the result if theywant to.

The EMS used was GroupSystems [9,8]. The tool usedwas categorize. This tool is often applied in structuredbrainstorms. It allows to build a group list of the remarksof the individuals of the group. It further allows to put theideas in different categories. The categories were used tolist the defects found on separate pages. This increases theoverview over the defects for the inspectors which shouldhelp them to find new defects.

One characteristic of an electronic inspection is thesilence in the meeting room. This may be new forsoftware engineers in inspections but it is common duringparts of many electronic meetings. Another characterist icis that it is not necessary to be at the same paragraph whileinspecting. Some inspectors may go ahead or staybehind. One can at all times read what the others havereported. This is again well known from all kind ofelectronic meetings: people can be doing different thingsat the same time and still contribute to the successof themeeting.

Electronic support for inspections has been applied inPhilips Medical Systems and Baan Development. Theresults of these inspections will be discussed in twoseparate sections. Both companies have considerableexperience with inspections. The electronic inspections inboth companies were almost identical. We howeverchoose to stay on the tie side and to present the resultsseparately because of major differences in the softwareunder development the nature of the documents underinspection and the fairly limited number of electronicinspections undertaken.

4. CASE 1: EMS SUPPORTEDINSPECTIONS IN PHILIPS MEDICALSYSTEMS4.1. Characteristics of the inspections

Philips is one of the world market leaders in providingmedical electronic equipment such asX-ray and MagneticResonance scanners. The amount of software in medicalsystems varies from 100,000 lines of code to multimill ionline programmed.

A total of seven electronic inspections were executed.The documents inspected were specification, design andcode. The size of the inspected documents was around 10pages and the number of inspectors was four (includingthe author) on average. The inspectors had, on average,participated in six inspections (ranging iiom zero to 20)before they joined the electronic inspections. Theengineers had on average nine years of experience(ranging from 0.5 to 14 years).

1060-3425/97 $10.00 (c) 1997 IEEE


3/7

4.2. ResultsThe results of inspections are often measured in terms ofeffectiveness, efficiency and yield.- Effectiveness is defined as the number of defects foundper inspected page. A distinction is made betweenpreparation effectiveness and meeting effectiveness.Preparation effectiveness is calculated as the number ofdefects found per page in preparation whereas meetingeffectiveness is the number of defects found per pageduring the meeting.- Eficiency is defined as the number of defects found perperson hour invested in the inspection [5]. Again, thepreparation and meeting eff iciency are distinguished. Thepreparation and meeting hours are calculated. The reworkhours are not calculated.- Yield is defined as the percentage of the defects foundduring the inspection versus the percentage that slippedthrough and was detected during later inspections, in testor in the field [7]. It was not possible to calculate the yieldof the electronics inspections yet for several reasons, one isthe fact that the software that was inspected is not in thefield yet. Another reason is the fact that the sofiware thatwas inspected was an add-on to a multimillion line system.The calculation of the yield is not trivial. It is clearhowever that the yield is a very important pefiormanceindicator for inspections and future inspections shouldprovide the metrics that will allow to calculate the yield ofour (electronic) inspections.

Before the graphs are shown and the results arediscussed it is important to acknowledge the fact otherfactors than the electronic support also have an impact theeffectiveness and efticieney of the inspection. Examples ofother factors are the type and the quality of the documentaswell asthe preparation time. The number of inspectionsand the number of factors involved does not allow tocalculate the contribution made by each of these factors.

The four figures in which the results are presentedcompare the preparation effectiveness and efficiency to themeeting effectiveness an eftlciency. Figure 1 contains theeffectiveness if only the Major defects are counted. Majorsare defined in all inspections as a defect that would haveresulted in a test or field defect if it would not have beendetected in the inspection.

non EMS EMS

I S preparation effectiveness II umeeting effectiveness IFigure 1 Effectiveness(Majors only)

The effectiveness of the EMS supported inspections isconsiderably higher. The preparation effectiveness is alsohigher, despite the fact that the only change to thepreparation process was the fact that a defect list had to beprovided the day before the meeting. Metrics revealed thatthe preparation effort for the electronic inspections washigher than that for the traditional inspections. The factthat the engineers had to submit their defect on beforehandapparently motivated them to put in more time. This wasconfirmed during the evaluation of the inspections.

The ratio between the preparation and meetingeffectiveness for the traditional inspections is 13while it is3 for the EMS supported inspections. This is remarkablegiven the fact that the preparation takes place first and asa result, typically catches the more obvious defects. Thefact that the obvious defects are already found isapparently compensated to a large extent by the synergy ofthe group in the EMS supported inspections. One of thegoals of electronic meeting systems is to do work inmeetings instead of in between meetings. The ratios foundhere indicate that for electronic inspections the EMSs areat least approaching that goal.Figure 2 compares the effectiveness if only the minors areconsidered.

1060-3425/97 $10.00 (c) 1997 IEEE


4/7

non EMSEMSI upreparation effectivenessWrnaetina effectiveness I

Figure 2 Effectiveness(Minors only)This figure indicates that hardly any minors were

reported during the traditional logging meeting. This canbe explained by the fact that engineers do not want toreport too many minors in a traditional logging meetingbecause that may be perceived as splitting hairs. This isconsistent with the results found by Porter ea. [10] whoreported meeting lossesthat exceeded the meeting gains.

In an electronic meeting engineers are not restrained toreport minors because they know that this will not distractthe meeting.Figure 3 discusses the efficiency limited to Majors onlywhereas figure 4 pictures the efficiency with regard tominors.

32.62

1.61

0.60

non EMSEMS

I u preparation efficiencyumeeting efficiency IFigure 3Efficiency (Majors only)

non EMSEMS

=Figure4 Efficiency (Minors only)

The ratio between the preparation and meetingefficiency for majors for the traditional inspections is 9while it is 2 for the EMS supported inspections. The ratiobetween the preparation and meeting etliciency for minorsfor the traditional inspections is 13 while it is 3 for theEMS supported inspections.4.3. The opinion of the participants

The opinion of the participants was evaluated by meansof a questionnaire at the end of each inspection and bysoliciting comments on the electronic inspections. Thequestionnaire asked for the opinion on eight statements ona five point scale: Strongly agree (5 points), Agree (4),Neutral (3), Disagree (2), Strongly Disagree (l). Thestatements are numbered 1 to 8:1 The computer-aided process is better than the manualprocess.2 The computer aided process helps the group toconcentrate on defect detection during the meeting.3 The computer aided process helps the group to focus onMajor over minor defects.4 The computer aided process helps the group to achieveits goals.5 The groups problem solving process was fair.6 The groups problem solving was efficient.7 I am satisfied with the computer-aided process.8 It is worth to come back.Most of the statements are derived from a standardevaluation of electronic meetings. Statements 2 and 3 werespecitlc to the electronic inspections. The opinions of over20 participants to EMS supported inspections are given inTable 1.

1060-3425/97 $10.00 (c) 1997 IEEE


5/7

Table 1Opinionsof the participants in Philipsstate- SA A N D SD mean Nment (5) (4) (3) (2) (1)no.1 10 48 33 10 0 3.57 212 29 33 13 25 0 3.67 243 5 27 23 41 5 2.86 224 0 65 30 4 0 3.61 235 8 67 21 4 0 3.79 246 4 42 38 17 0 3.33 247 13 54 25 8 0 3.71 248 17 70 13 0 0 4.04 23The opinion of the participants was favorable towards allbut one statement. The participants disagreed with thestatement that the electronic inspections helped to focus onMajor over minor defects. Apparently EMS support aloneis not sutlicient. A combination of a scenario detectionmethod [10] in combination with EMS support may assistthe inspection team in focusing on majors over minors.

The qualitative evaluation by the participants showedthe following results:- The meeting was considered less stressfid than atraditional logging meeting. This was stated by both theauthor of the document and the inspectors.- Some inspectors still doubted the usefidness of theinspection meeting, despite data that indicated thebenefits. The evaluation of the inspection metrics by theparticipants showed that their intuition did not match themetrics in all aspects.- Several suggestions for improvements of the inspectionprocess and electronic meeting system were put forward.One suggestion was to provide the author of the documentwith the combined defect list half a day before themeeting. This would allow him to study the defect listbefore the meeting, which would make him an even moreeffective inspector.- The experience with electronic inspections inspired oneof the engineers to replicate some of the group supportfimctionality on their workstation based developmentplatform. This prototype allowed to share the commentsthat were found before the meeting and those that weregenerated within the meeting. The inspections werecarried out a room where enough workstations wereavailable. The software group executed another seveninspections using this system. The results indicate that theinspections done on this platform were more effective andefficient than the tmditional inspections, but not aseffective and efficient asthe EMS supported inspections.

5. CASE 2: EMS SUPPORTEDINSPECTIONS IN BAAN DEVELOPMENT5.1. Characteristics of the inspections

Baan is one of the world market leaders in EnterpriseResource Planning soilware. Its products are known underthe names Triton and Baan. A total of seven electronicinspections have been carried out. All concernedinspections of source code that is part of a multimillionsotlware package. The inspectors had on average,participated in 33 inspections (ranging from 5 to 100)before they joined the electronic inspections. Theengineers had on average four years of experience(ranging from 0.25 to 22 years). Most engineersparticipated in one electronic inspection.5.2. Results

The results of the electronic inspections will becompared to the result of another 100 traditionalinspections of source code as they were executed in thesame development organization in the previous quarter.The results can be compared in this way because allinspections concerned are similar code inspections. Theresults are given in Table 2.Table2 A comparisonof traditional andelectronic inspections

Number of inspectionsMajors per inspectionMinors per inspectionDefects per inspectionMajors in meetingMinors in meeting%0Majors in meeting/0 Minors in meetingYO Defects in meetingLogging rate

r.raditiond1001.521

22.50.250.75

1745

36

EMS SUppOrted73

5760

0.578

17161660

The following observations can be made:- The absolute number of defects found in the inspectionsincreases considerably, both the majors and the minors.The percentage found during the meeting does not changefor the majors but increases significantly for the minors. Infact in the traditional meetings there were hardly anyminors found during the meeting.- The ratio between the preparation and meeting eff iciencyfor majors is 5 for both the traditional inspections and theEMS supported inspections. The ratio between thepreparation and meeting efficiency for minors is 25 for thetraditional inspections and 5 for the EMS supported

1060-3425/97 $10.00 (c) 1997 IEEE


6/7

inspections. The gains in this case are in catching moreminors during the meeting.- The logging rate (defined as the number of defectsreported per hour in the logging meeting) is considerablyhigher for the electronic inspections.5.3. Opinion of the participantsThe opinions of the participants was evaluated by meansof a the same questionnaire that was used in PhilipsMedical Systems. The statements are listed in section 3.3.The results for Baan are in table 3.Table 3 Opinions of the participants in Baanstate- SA A N D SD Mean Nrnent (5) (4) (3) (2) (1)no.1 4 57 17 22 0 3.43 232 4 48 24 16 8 3.24 253 4 16 28 48 4 2.68 254 9 52 26 9 4 3.52 235 5 57 24 14 0 3.52 216 8 46 29 17 0 3.46 247 16 60 12 12 0 3.80 258 25 46 21 8 0 3.88 24

The opinion of the participants in Baan is similar to thosein Philips. Comparing the results of these studies withearlier studies such as [9] indicate a less favorablejudgment of the electronic support. Two possibleexplanations can be given. The first is that softwareengineers are very demanding users of informationtechnology. A second explanation may be the fact that amanual inspection is already a very mature meeting withextensive preparation and a very clear structure. Theintroduction of EMS often brings improved preparationand structure to meetings which is appreciated by theparticipants. In the case of inspections, this benefit is notexperienced by the participants because the preparationand structure were already there before the EMS supportwas brought in.6. IMPLICATIONS FOR INSPECTIONSThe following implications for inspections are foreseen.More effective and ej?cient inspections process.This is a significant improvement for the overalldevelopment process, given the fact that inspections haveproven to be the most effective way to detect defects. Ontop of that, mature software groups spend 10 to 20 percentof their t ime in inspections.

Defect lists available in electronic formThe defect lists are available in electronic form. Thisfacil itates the rework for the author. This is also beneficialfor the use of the data in later test or for regulatorypurposes.Abandon sequential nature ofpreparation and loggingTmditional inspections clearly distinguish individualpreparation Ilom the logging meeting. It maybe possibleto combine the two with the same or better results in anelectronic meeting. The triggering of one inspector bydefects found by one of his colleagues now only takesplace during the logging meeting. It may work as wellduring preparation. Experimentation and analysis of theinspection metrics should tell the impact.Distributed inspectionsDistributed inspections are feasible. Experiences inelectronic meetings have shown that it is not alwaysnecessary to be at the same time at the same place. Thismay also hold for electronic inspections. It may be verybeneficial to be able to allow engineers in dit l?erent placesto participate in an inspection. One of the companiesinvolved intends to do distributed inspections withengineers from sites in Europe and India.7. IMPLICATIONS FOR EMSMany implications for EMS can be envisioned. Three arediscussed.Meeting metricsInspections are the only meeting in the world, to ourknowledge, that are managed in quantitative terms. Itmust be possible to exploit the benefits of metrics in othermeetings as well. It must for example be possible to planand track household meetings based on metrics such asthenumber of action items generated and dealt with. Thebenefits are huge if one realizes the improvements in otherdisciplines after quantitative understanding wasaccumulated and put to use. One example is themanufacturing industry. A more recent example is thesoftware industry that is trying to get a grip on theirproblems by means of metrics [6,1,4].

A next generation of EMS should provide more tools toplan and track meetings in quantitative terms. This shouldgo much tirther than logging the number of comments perminute. It includes norms, analysis of deviations andquantitative insight into optimal numbers of participantsand amount of work to be done.Inspections as guinea pigInspections could be a benchmark meeting to study theimpact of independent variables such as remote meetings,

1060-3425/97 $10.00 (c) 1997 IEEE


7/7

multiple languages or cultures on the result of electronicmeetings. The impact of these can be measured andmetrics can be compared. As such, inspections can be aguinea pig.Fixed format input in EMSThe input into most current EMSs is free format. Aninspection is a structured meeting that could benefit fromfixed format input. For example: if one is sure that thesecond position always contains the defect code, it ispossible to sort the defects during the meeting and put theinsight gained to use in the current meeting. There aremore meetings that could gain from fixed format input.Two examples:1) when generating action items it is useful to fix theformat and ascertain that the first position always containsthe person who will take the action and the last willalways indicate the estimated effort required to implementthe action. This would allow to calculate the estimatedtotal effort for one person and use this data in the meetingwhile distributing the rest of the work.2) even brainstorms can benefit from fixed format input.Experienced facilitators often fix the format by requiring asentence to start with a verb (e.g. increase the marketshare to 30 percent). Enforcing such a formatelectronically does facilitate to sorting of ideas later on[13].

8. CONCLUSIONSBased on 14 EMS supported inspections we conclude thatelectronic support of the logging meeting can improve theeffectiveness and etllciency of the inspections. Thisconclusion is based on the data collected and the opinionsof the participants. This conclusion is confkmed by thefact that both companies involved have implementedelectronic inspections as part of their normal operation.We also conclude that electronic inspections have manyimplications for both the software engineering and theEMS world.

It is clear that more empirical data is required to fhrtherimprove the electronic support of inspections. We willcontinue electronic inspections in industry and collect dataas presented here as part of our industrial work. We wouldlike to encourage scientific studies of questions such as:- What is the quantitative effect on inspections that aredistributed in time ador place? What is the quantitativeimpact of inspections that are distributed over differenttime zones? What does this learn about other distributedmeetings?- What will be the effect if the document that is to beinspected is presented electronically?- How can soflware tools as artificial inspectors contributeto the effectiveness of inspections?

We are convinced that many will benefit from these kindof studies, both in the software engineering and EMScommunities.9. REFERENCES[1] Comelissen,W, Klaassen,A., Matsinger, A., Van Wee, G.,How to make intuitive tesing more systematic,IEEE Software,September1995[2] Dean, Orwig, Vogel, Facilitation methods to enable rapiddevelopment of high quality business processmodels, HICSS1996.[3] Fagan,Advances in software inspections, IEEE TSE, July1986[4] Genuchten,M., Why is software late, an empirical study ofreasonsfor delayin sotlsvaredevelopment,IEEE TransactionsonSotlwareEngineering,July 1991[5] Gilb, T., Graham,D., Softwareinspections,AddisonWesley,1993.[6] Humphrey,Managingthe softwareprocess,Addison Wesley,1989[7] Humphrey, A discipline for software engineering, AddisonWesley, 1994[8] Nunamaker, J.F., Briggs, R.O., Mittleman, D.D., ElectronicMeetings Systemsten yearsof lessonslearned, HICCS 1996.[9] Nunamaker, Dennis, Valacich, Vogel, George, Electronicmeeting to support group work, Communicationsf the ACM,July, 1991[10] Porter, Votta, Basili, Comparing detection methods forsoftware requirements inspections: a replicated experiment,IEEETransactionsonSoftwareEngineeringJune1995.[11] Rooijmans, Aerts, Van (knuchten, Sotlware quality inconsumer electronic products, IEEE Software, January 1996[12] Votta, Does every inspection need a meeting? Proceedingsof the ACM SIGSOFT 1993 Symposium on foundations ofsoftware engineering.[13] Weatherall, Nunamaker, An introduction to ElectronicMeeting Systems, Electronic meetings services ltd., Hampshire,UK, 1995.

1997 supporting inspections with an electronic meeting system

Documents