pqalk - department of computer science (idi) - ntnu in accordance with the cmm framework, gave...

�� !��"��$#%��&��#'��"�� (��)��*��

+-,).0/214365278.01�/)9:1)/ ;=<2>@?)ACB�DFE2G2H�I8A0J�K2L2MON�P�QRM2K2SUT�V2L2SWK2M2QYX

Z\[^]`_ba cedgfhdgi�jlkmi&noi�cWpqjWa rsit_suokwvWi�xì�[ousj�vWi�xì�pqusy{zouY|h]}kmfoj8iU~Rv:vWz\�4noi��i�xùhdgi�n^fo[sn�a yldgx}i�y�i�[o]}i�ndbj�ubpqi�c8c�i�c��oj8uox}i�clfo[sn�]}uousx`cl]ù^fgp�_ga i��itj8i��oc8ituY|ti�c�]qa y�fo]qa us[�fs[on^j�a c��&y�fo[ofs�oi�y�i�[o]�i��dgi�j�a i�[bpqi��ba ��i��usj��sfo[ga ��fs]Ya uo[sfox�x}i�foj8[ga [o�s�gvW_sitj8i�c��sx`]}c�|hj8uoy{]}_oi�pqfsc�i&c�]}�on��a [gpqx}�onsi)�• �}�o�&�o��}��b��o��s�o�¡ �s�`¢8�o�o£b¤q�q �s��Y¥t�o�^�q¦��b��¢W ��g¤q�&�o�s�`�s§o�s¨�� s�`��©o¢��s�`��ª4 �}��}�o�&¨��Y¥h�}ªm�o¢8��o��`�h�g��s��g¢��b¤q��¨�¨l«��Y¥`¥��¢W �s©�¢8��}��s�o��q¦��g��¢� ��g¤��tªm�s��^�o��o��¬m ®�£o¨8�2 �^�Y ��2 ¯• °q±�²o³l´bµ`°�¶ ·Y¸º¹ »�´g°�¶l·\¸t°�±�´g°�¼�½ °�¾b¿q°&Ào¶�°Á¸hÀsµ�¸h·s¼ ¶8·\¸�¹`Âm²s¼�°&Ão°�Ä�°�µ}·�´b°�¼�¶• Å�Æ�ÇqÈsÉ�É�Æ�ÊoËsÌoÍqÎ ÈsÊoÏlÈoÊ^ÐoÈsÑÒÍ`È¡ÇqÈsÓ`Ó}Æ)Ç�Í`ÔhÕgÌbÇqÖ�Ìs×oÆ&ÌoÊsË�ËbÎ Ï8Í`ÅWÎ ØsÙoÍ}ÆtÆ�Ú�ÕgÆ�Å�Î Æ�ÊbÇqÆ• ÛqÜ�ÝgÛ�Þ�ß Û�àgá�Ûtâsà�Þ8âoã}Û�älåoàsæçÝbÞ�âgá�Û�ä8ä è}â�ä8é�ÝsÝgâoÞ8è�Þ�Û�ésä�Û&â\êtä�âYêhè}ëmåoÞ8ÛtæsÛ�ì�Û�ã}â�Ýgí�Û�àoè�ÛqÜ�ÝgÛ�Þ�ß Û�àgá�Ûî�ï�ð�ñ4òoó�ôeõ

: Experience database, software improvement, organizational learning

öø÷8ùUú:ûWü�ý$þUÿ�ú�� üºù

The reported case study on reuse of software development experience was carried out in1997-1998, supported by the national research project SPIQ (Software ProcessImprovement for better Quality). The case study was, among others, motivated by thefollowing challenges:

1) How can software development experience be efficiently shared between differentdevelopment teams? ("Many-to-many learning")

2) What types of experience are worth reusing?

3) What is the role of reuse of "local" (context-dependent) experience compared with more"global" (best practice) experience?

Our approach and results to help meeting these challenges, we believe, can be useful forother organizations facing similar challenges.

The remainder of the paper is organized as follows. Section 1.1 describes the researchproject SPIQ. Section 1.2 describes the organization studied. Section 2 describes and arguesfor the approach chosen. Section 3 describes the results. Section 4 describes related work.Section 5 concludes, summarizes and suggests further work.

1 ex-Telenor, now at Storebrand and University of Oslo, e-mail: [email protected] University of Oslo, e-mail: [email protected] The Norwegian University of Science and Technology, e-mail: [email protected]

�� "!�� #��$��&%��$� ��&')(��*,+��-/. �&��')0

In April 1997, following a pre-project in 1996, the software process improvement projectSPIQ was started. The program is sponsored by the Research Council of Norway (NFR) forat least three years. Its main goal is to: 132,46587�98:<;�9>=�?69@58A6BDCE98=,2�=�2,F898469G;H;I:646J�CE7�ALKM2,=�:6N62,O�2�= PQALKR A67�S&9UT�2�:64>VXW�Y 2�46J6Z<;�=,7[PQ=�?67HA6Z<T�?>; P\;�=,98B]:6=,2�5@:646J^5<A646=�2�46Z69U;_CE7�A6589G;H;I2�BDCE7�A6F898B`9846=bacacacaH1<a TheSPIQ project is based on the software process improvement principles of “Total QualityManagement” , see for example (Deming 1986), and the “Quality Improvement Paradigm”,see for example (Basili 1985). An important aspect of SPIQ is that it provides a means forthe academia and the software industry to meet and discuss software improvementexperiences and research results. See http://www.fw.no/spiq/ for more information aboutSPIQ. The work described in this paper has benefited from SPIQ in at least three ways:

1) The experience database design and results were discussed at the SPIQ meetings.

2) SPIQ has provided valuable research support.

3) SPIQ has financed parts of the Telenor Telecom Software’s (TTS's) internal work onreuse of experience.

d�e�f�g�h�i�j�k�l�m�n�o�p6m�q$o,j�n

TTS is split into five geographical locations and has more than 400 employees, most ofthem software developers. In other words, reuse of software development experience is animportant, but not trivial task. In 1995-1996 the company went through a “Business ProcessReengineering” , see (Hammer 1996), resulting in a well documented, standardized softwaredevelopment processes. The process descriptions and documents are available to allemployees through the Intranet using an Internet browser.

The software development process used by the developers is called “solution delivery” andis based on incremental delivery of software functionality in so called “ time-boxes” . Each“ time-box” lasts 3-6 month, which provides good conditions for experience reuse, at leastcompared with organizations with a waterfall development model leading to projects withcycles of 1-2 years. The organization includes several support teams (development toolsupport team, measurement and estimation support team, test support team, quality team,etc.) for the development and maintenance processes. These teams turned out to be veryimportant in the implementation of the process changes and collecting experience. A recent,informal, in-house assessment (carried out by one of the authors of this paper) of thecompany, in accordance with the CMM framework, gave maturity levels on different keyprocess areas between 2 and 4, i.e. TTS is a reasonable mature software developmentorganization.

The company’s software development process prescribes several steps motivated by theneed for reuse of development experience: Each project should, for example, be measuredaccording to a measurement model and deliver an experience reports when completed. The"Measurement and Estimation Team” is allocated to carry out the measurement and the"Quality Team" is the receiver of the experience reports. We found that the projectmeasurement and the experience reporting were to some extent carried out. However, there

were not much systematic use of the information to improve the process. This observationwas a major motive for our focus on reuse of experience in TTS.

rts)uwvtx`yIyIz�{Ix}|ù

Our approach can be characterized as action science (Argyris et al, 1985), which is a typicalresearch method when studying industrial software development. Action science has bothadvantages and disadvantages. Advantages are, for example, that action science may be themost efficient way to get:

• In-depth knowledge about software development organizations. This belief is amongothers supported by the learning model of (Dreyfus and Dreyfus 1986), which focuseson the role of collecting concrete and context-dependent experience to support thelearning process. According to this learning model only the "lower levels" of manytypes of knowledge is context-independent and rule-based. In order to achieve higherlevels of knowledge (being an expert) lots of context-dependent experience (localexperience) have to be collected. ~`�6�w�6�<��8��<�6��6�<�E��U�6�E�6��,�6��I�,�8�6��6�,�<��]�6�6�8�,�<�&�6��G�\�6�D�E��U��&�6�,��@�,�6�G�G�E�8��,�8�6�8�8��E�H�L�M�8�8�b��8�6�6�<� �I�<��8�<��M�6�w��6�,�@�6�<��8�^�`�8�,�6�6�<�I��G��6��6�,�<��@�`�6�6�<��8�]�<�6��`�6��@�G�G�E�<��8�6�<�8��E��L�M�<�8��,�8�6�6�8� �I��8��>�]�6��>��6��<��G��,�8�^��^�6�6��6��6�8��E��L��8�8�� 6�6�^�8�6��,�8�^�6�6��,�6�8��w�H��@�`�6�6�<��8�`�8�6��6�<��<��,�G��

• Representative and realistic information on how terms and models important formeaningful reuse of experience are used. ¡&¢6£w¤U¥\¦6§©Ëª,¤G«�¬6¤8®^¬&¤>¯8¢6¢UË¤8£�¦6°,¤8±^¬&²,°�^°,6¤Ë£�¢L³�¤8¯8°�¤<±^ª�¤8¦6±6¤8£µÍ¢6®^¤G´�°,²�§`¦6°�²�¢6®^¢L¶]¤·¶�¶�¢6£�°�«�¬¤�¶�¢6¸6®6±^¦^¹8¦6£�²�¤<° ºQ¢L¶]²�®6°�¤<£[Ë£�¤8°,¦6°�²�¢6®<´ ¢L¶»°�6¤°�¤<£�§½¼3¤·¶,¶M¢6£�°b¤G´�°,²�§`¦6°�¤_¾3¿�À�6²�Í¹<¦6£�²�¤8° ºQ¯8ª,¤8¦6£�ª º�£�¤8±6¸6¯8¤8±^°,6¤�Ë¢6°�¤<®6°�²�¦6ª ¶M¢6£w£H¤8¸<´�¤@¢L¶]°�6¤@¤·¶,¶M¢6£�°¤G´�°,²�§`¦6°�²,¢6®^¤G¥GË¤8£�²,¤8®6¯8¤@¦6®6±^±6¦6°,¦6¿�À�6£�¤8¤@§`¦L³M¢6£w° º8Ë¤GÍ¢L¶]²�®6°,¤8£[Ë£�¤8°,¦6°�²,¢6®<Í¬&¤<£�¤�¶M¢6¸6®6±6ÁÂ ´�°�²,§]¦6°,¤8±^¤·¶,¶M¢6£�°b§]¤8¦6®<´ ¦UÃ�¼3§`¢<´�°�ª,²�Ä8¤8ª ºQ¤·¶,¶M¢6£�°Å¾U«�ÆUÃÇ¼3°�6¤@¤È¶�¶M¢6£H°�¬²�°�^°,6¤�Ë£�¢6Æ6¦6Æ6²,ª�²�° ºQ¢L¶É6Ê6Ë ®6¢6°b°�¢^¤G¥6¯8¤8¤8±E¾ÍÌc§]¤8±6²,¦6®UÃÎ¢6£Ï¯ÐÃ�¼3°,6¤@§]¢<´Ñ°�ª�²,Ä8¤8ª ºQ¤·¶�¶�¢6£�°ÓÒÔ¦�Ì Ë£�¢L³M¤8¯<°�±6¤ÐË¤8®6±6¤<®6° Ã £�²�´�ÄÆ6¸L¶�¶�¤8£b¾3¿

Disadvantages of action science are, on the other hand, that:

• Action science studies are not carried out as strict experiments with control of thevariables. Thus, a formal cause-effect relationship between the actions and the resultscannot be established. In particular, the mixing of the participation and observer rolemakes objective analyses difficult. In addition, it is unlikely that anyone will (be able to)repeat the study to validate our observations.

• There is no available observational language or theory to remove subjectivity and biasin the description of the observations. See for example the discussion of how theexpectations impact the observational language in (Goodman 1951) — i.e. there is adanger of “ theory loaded observations” .

It is important to be aware of these disadvantages, but it should not stop anyone fromcarrying out studies like ours. Currently, action science (or similar methods) seems to be theonly practical way of achieving in-depth “ real-world” results about software improvement.We believe, however, that more quantitative and experimental research on software

processes should be the long-term goal of the software improvement research, leading tomore general and objective knowledge. A more general discussion and comparison ofresearch methods, particularly the role of case studies, can be found in (Flyvebjerg 1991).

Stimulated by the work at NASA-Software Engineering Laboratory on Experience Factory,see for example (Basili et al. 1992) and the opportunities we had at TTS, we started asearch for “pilots” where reuse of experience would improve the development process.Based on an informal analysis of the availability of information, availability of resources,time, probability of success, estimated cost and benefit we decided to focus on thefollowing two topics within the software development process:

• estimation of software development effort

• risk management

A brief analysis gave that in order to support reuse of estimation and risk managementexperience, there was a need for:

• an experience reuse process, including new or modified role descriptions

• a supporting tool (the experience database)

• allocated experience reuse resources, both for implementing the experience reuseprocesses and for administrating the experience database

Õ Ö)×wØÚÙ�Ø}Û`ÜIÝ�Þ�Û

This section describes the work and some of the results achieved in the period Spring 1997-Spring 1998. The organization continues to focus on experience reuse, i.e. the results andproducts are to some extent preliminary.

ß6àcá^âäã6å�æ�ç$è8é�ê$ã6ê æ�ë6åìë6ç©è8í6î�è8ï<æ�è8å�ð8è

During the requirement analysis we soon discovered that the manifestation of experiencecan and should take many forms to be useful to the developers, such as:

• quantitative and qualitative information that can be stored in traditional databases.

• general tools implementing or based on “best practice” within the organization

• rule based systems (expert systems) reflecting expert experience and knowledge

• pointers to people with useful experience (this may be the only way of “ representing”experience that cannot be articulated, i.e. tacit knowledge)

• process descriptions on different levels and with different degrees of contextdependence

In addition, it was considered important that the experience database (the tool enabling theaccess to the stored experience) was available to all the developers at a low cost, integratedwith the quality system, easy to use and easy to maintain.

ñ6òcó^ôõ8ö8÷�ø�ù�ö8ú6ûbü�û,ú6ý$þ ÿ ��

The technical platform chosen to meet these requirements was based on:

• The organization’s own Intranet. This made the experience database available to all thedevelopers and well integrated with the organization’s quality system.

• A user interface based on a web-browser with links to experience of different types.This removed the need for local installation.

• An “experience database” based on tables of data, spreadsheets, documents and rulesimplemented in executable programs, i.e. no traditional database.

Further, we decided to integrate the experience reuse support with the organization’sprocess descriptions, i.e. from the relevant steps in the process descriptions we had links touseful information and tools in the experience database. The idea was to offer usefulexperience when needed (“ just in time”), instead of, for example, writing a reportcontaining useful experience.

�� !" #�$� � �� !�%

The effort estimation experience we offered was of the following types (linked to therelevant process steps):

&�'�&�'�()�* +�* ,�-�.�/0*1+�2�*�3�4�4�, 5�4�, . 3�+�*�* 6�+�. -73�+�. 5�/"-75�8�* 9:3�/�8"4�, 5�; * 6�6<'An “expert system” recommending estimation models were developed based on thecollection and analysis of the experience of the organization’s estimation experts.Following an analysis of whether formalized effort estimation is recommended or not, theexpert system asks the user to answers nine questions. A simplified description of thequestions and the implications of different answers is given in Table 1 and 2. Theestimation models are briefly described in Section 3.3.2. This expert system uses, inaddition to the answers from the users, empirical data from TTS on the accuracy of thedifferent estimation models, see Table 3, and the quality of the relevant historical data, i.e. ahigh degree of organizational dependent experience.

=?>�@0A�B�CDFE�GIHKJMLON:P�H?Q�RSGIH?NUT�VWNUXZY[P�H]\^GUT_H�`acb�d:egf h h�iZjlknm�kpoSkpqrjlf s�jut0kns�m�knkpvxw�f y w�m�q�zMiZm�{�|xi�{�m�k}t0kx~Uknh vn�S�}kny iSqSylt0�Kv:m�|nvn�}�Sh kx��qSh s�vnm�f i�jl��z<�a��Id�� z[iZjlkp�Sm�vx��kn|xiS|nv:y�i�kx��ilz�f s�ylf w�f |nqSy i�h ��tSf w�w�knm�kny ilw�m�v:��0m�kx~Uf vn{�z[�Z�Z��Z�0m�vx��k:|xi z<�a��Id:�1m�kp�cvIz�iSvxw�i�jlk}m�kn�0{�f m�kn�ckny i�zWtSkxz<|nm�f oSk:tS�

a��Id�� zWqrtSq�iZqr�}vnt0knh�ql~Uq0f h q0oSh k}v:m�|nq0yuknq�z�f h �uo0k}t0kx~Uknh vn�0kntS�a��Id��vnkIz[iZjlk}t0knh f ~Uknm ��|nvny z<f zMi zWvIw��cq0y ��z<�cq0h h � ylvxiSh vns�f |nq0h h �u|:vnyly¡k:|xiZkntr|:j¡q0yls�kxz��¢�}vntS{�h kxz<�a�£Id:egf h h�iZjlkpkxw�w�v:m i¡i�v}|:vn�c�0h kIiZkFiZjlk}�0m�vx��kn|xiS�0m�vnoSq0o0h ��oSkp�cv:m�k¤i�j¡q0y�z<f ��cvny i�j�z<�a�¥�d�� z[iZjlkp�Sm�vx��kn|xil¦§f h h f ylsîZvFz��0knyltgb�� §�}qSy ��tSq�� zWvxw�kIw¨w�vnm iSvny�kxz�i�f �}qli�f v:y�w�vnm©z<�cq0h h��0m�vx��kn|xiSª«h kIz�z[i�j¡q0yb��§�cq0y �Z�}vny iZj z�d�qSylt�� §�}qSy ��tSq�� z[w�vnm�h q0m�s0k:m��0m�vx��k:|xi z<�a�¬�d:egf h h�tSkI~:k:h vn�0knm z[¦§f iZj�kx�I�Sk:m�f knyl|nkFw�m�vn�z�f �cf h qSmx�Sm�vx��kn|xi�zWoSkpq�~:q0f h q0oSh kF¦§jlknyukxzMiZf �cq�iZf ylsîZjlkpkxw�w�v:m iZ�a�®Id:egf h h�iZjlknm�kpoSkp�cv:m�k¤i�j¡q0y�w�f ~:kpt0knh f ~Uknm�f kxz[z�f �cf h qSm�iZvFiZjlf zWv:y¡k:�¯?°�±0²�³�´µ?¶�·�¸�¹�°�·�¸ º�»"¹�º�¼�³ ² ½^³ ¾�º�¹�¹7³ »�¼�°�·�¸ º�»"¿ À�²�³§Á�Â�Ã:Ä�Å�ÆFÇ Ä�È�É Å�Ê�Ë�È§ÌÎÍuÏ�ÐÒÑ?Ä�È ÓÇ Ä�È�Ë�Æ�È Ô

Å�Ó�Æ Ñ�È�Ô�Æ}Õ�ÐWÖ7×�ÓØ�Ë�ÈKÆ Ç Ã�×�ÓÂ^Ã_Ù0Å�Ó�ÚÇ Ä�È�É Å�Ê�Ë�È�ÛÝÜÝÞ0Ö�Ðàß0Ã�É È ÓÇ Ä�È1Å�Ó�Æ Ñ�È�Ôáuâ�ã�äOåæÍ1ÃUÃçÆ¤Å�Æ<Æ Ãçß0Ó�È ÚÇ�Ä�È�É Å�Ê�Ë�È§ÌÎÍÏ�ÐÝÙ0Ç Ä�È Óè�×�Ú�È Ê Ð1ÃUÃçÆÔ�È é ×�è�è^È�Ó�Ú�È Ú�ã

E1) FPA R1 = not Q1 ê�ë^ì not Q2 í^î�ï Q3 ð�ñ^ò not Q5 ó�ô^õ Q6 ö�÷�ø Q7

E2) ROPD R2 = not Q2 ù�ú^û Q8

E3) FPA simplified R3 = not Q1 ü�ý^þ not Q2 ÿ�� Q4 �� not Q5 �� not Q7

E4) Tailor madeestimation model

R4 = Q2 �� Q9

E5) No modelrecommended

R5 = not (R1 �� R2 �� R3 �� R4)

When more than one model is recommended we give the estimation model with the lowestindex the highest recommendation and presents the other recommended models asalternatives.

�� HKJML"!cY�JMLONUP#!}N%$[G%& '('()�*�L H]JMNUT�L"+UY,&�Y,+-+:E�TŸ,+/.}N/0-!cN-$[G%&ÎQ�Y21UG:TŸ,3[G:`4 {�h h65pq0m87©� � 4 {�yl|xi�f v:y:9�vnf y i¡��ylq0h � z<f z b��<;ª«�ck:qSyu�}qSs�ylf iZ{�t0k}vxw[m�knh q�iZf ~Uk}k:m�m�vnm d�xf �}�Sh f w�f knt 4 {�yl|xi�f vny#9�vnf y�il��ylqSh � z<f z �<=<;ª«�ck:qSyu�}qSs�ylf iZ{�t0k}vxw[m�knh q�iZf ~Uk}k:m�m�vnm d>�? 9�� <=<;ª«�ck:qSyu�}qSs�ylf iZ{�t0k}vxw[m�knh q�iZf ~Uk}k:m�m�vnm d@�A8@�A8BDC�EGFIH�JLK�FNM�MPOQOQR�STFDepending on estimation model, different types of experience data are available. Among

others, the following estimation models and planning tools were supported by theexperience database:

U�VXWYU�ZP[]\G\_^a`]b]cPdIe�f�bhgaf�e�b]dXi�b�U�j�k�lGemlonIWp[]\G\_^agai�V , see (Symons 1993). We improved andextended an existing spreadsheet implementing the MkII FPA estimation model. Thisestimation model takes as main input the size of the functionality to be developed measuredin function points. Earlier we had analyzed data from more than 30 software developmentprojects regarding how different variables, such as use of CASE tool, had had an impact onthe development productivity, see (Jørgensen 1995). The productivity data indicated thatthe choice of development environment explained very much of the productivity variance.q rts�uovxwzy�{X|o}mvx~�y��Dv��2s�uG�-vx�<��m{�y��m�ms��D{�s��vP}N�2s�u_��s��s�}%y��t��s��vPuG��m}��vPuG� |ouGs��2vP�T��G~��yT�<vT�Ds��}msT��N}m��vPy�u_uGvx��u�vx�G�<��s��Ds��D�m��v��Ps�}�}mvP�T�mvP�D��y��yT~��m��P}��m�T�hs��} �h�m��v��<��<v�s��m��v��yT�<��y��D�m��v��vP�PvT}ms�|o{�vP��%��s�s�}��}�vP�PvP}��,yT��v�|ovP��vP��-�Py�uG��y��}mvx�_��y��Tv�yt�� ¢¡�£#¤T¥§¦�¨�© ª

We provided the estimator with historical data on previous projects similar to the currentproject. Table 4 shows some of the historical information that the estimator could make useof. The productivity is measured as UFP/w-h, unadjusted function points per work hour.Notice that the estimator has to predict a productivity category for his project, i.e. expertknowledge is still required.

«�¬��®�¯�°±�¬�²I³P´]µI¶]¯x·�¯T®m¸�¹]ºL¯T»�² ¼�¸T½¾¹]¿P¸�¶]À Áp¯T¶�Â�Ã]ºÄ¹]¿P¸�¶]ÀÆÅ�Â�Ç�´È¹�¿T¸�¶�À «�Ã]¿P]¸D¹]¿P¸�¶]À

Cobol - environment 0.05 UFP/w-h 0.10 UFP/w-h 0.20 UFP/w-h 0.30 FP/w-h

Powerbuilder - environment 0.15 UFP/w-h 0.25 UFP/w-h 0.50 UFP/w-h 0.70 UFP/w-h

É�Ê]ËNÌmÍmÊ]Î�Ï]ÎxÐ�ÎTÌmÑ�Ò]ÓLÎTÊ�Ô Õ�ÑTÖ¾Ò]×PÑ�Ï]Ø ÙpÎTÏ�Í�Ú]ÓÄÒ]×PÑ�Ï]ØÆÛ�Í�Ü�ÝÈÒ�×TÑ�Ï�Ø Þ�Ú]×Pß]ÑDÒ]×PÑ�Ï]Ø

Cobol - environment 0.07 UFP/w-h 0.15 UFP/w-h 0.20 UFP/w-h 0.30 UFP/w-h

Powerbuilder - environment 0.20 UFP/w-h 0.35 UFP/w-h 0.70 UFP/w-h 1.00 UFP/w-h

à]á A bottom up, task and risk based estimation model was developed. This estimation

model was supported with experience in the form of lists of “ tasks to remember” andsuggestions on the effort distribution between the phases. Currently, there is ongoing workon how to improve the collection and reuse of historical data to support this bottom-up, taskand risk based estimation model, see (Schrader 1998). We labeled this model â�ã�äaå�æ (TheNorwegian acronym for Risk Based Division into Sub-tasks.)

çPè A risk analysis tool integrated in the estimation tools (or to be used separately) wasdeveloped. The risk analysis tool contains risk models, textual advise and guidelines basedon previous experience. The content varies from a simple (but useful) checklist of tasks andrisk factors to more sophisticated probability (beta-distribution) based risk models.Typically, the content was based on general frameworks and models, then adapted to theorganization's needs according to expert knowledge and experience. This tool resulted in aprobability based effort estimate and predictions such as "there is an 80% probability of not

exceeding 3000 w-h of effort". It turned out that this type of probability based predictionswere essential to introduce the distinction between planned and estimated effort in theorganization. Similar to the results in (Conolly and Dean 1997) we believe that probabilitybased estimation had a positive impact on the realism in the effort estimates.

é]ê Finally, pointers to the human estimation experts were provided.

ë�ì8íDî�ïPð]ñGï�ò�ó¢ôPõ�ñGöÈ÷Lø�ù]ø�ú�ïP÷hïPù]ûXïTü�ý�ïTôPõmïPù]þPï

Similar to the estimation support we linked our experience database to the risk managementprocess. The experience database offers support through several tools to identify, analyzeand manage software project risks. We interviewed several experienced project leaders inthe organization to get the most relevant risk factors and the most relevant methods toreduce and control the risks. In addition, data from quality revisions were used to tailor therisk management support. Based on the collected information we developed:

• a “TTS best practice” risk management process (extensions to the existing developmentprocess)

• a tool to identify, assess and store risk factors, and suggestions on how to reduce orcontrol the risks

• a tool to visualize the risk exposure over time

In many ways, what we did was to collect only a small fraction of the organization'sknowledge about risk management. To become a learning organization the organizationwill need to continuously collect and distribute experience, i.e. new roles and a changedprocess is needed. Since systematic experience reuse in risk management has a short historyin TTS, we found that we needed to start small in order to understand what sort of riskexperience would be useful to collect.

ÿ��

The studies and results described earlier in this paper resulted in the identification of needsfor new roles and an increased focus on the implementation of the development process.

��

• An “ �� !�"#��$�%��'&�(�)*(�+�(�,-�'(�&�./"$�",�)*!�(�)10�! ” (a “gardener”) responsible for theavailability and usability of the experience to be reused. This role may be split into tworoles dividing the responsibility into a technical administrator and a contentadministrator. We suggest that the “gardener” should be a part of the software processimprovement team of the organization.

• Several “ 2�3�4�576�8-8�9�:�9�;<�8�=*8 ” responsible for analysis of information from each sub-process, such as the estimating process, the project management process or the testingprocess. The “process analysts” is responsible for collecting and analyzing relevant

information from completed projects and to generalize, tailor and package the usefulexperience.

• A network of “ >-?�@�@BA�C7DED*F7G�HI> ” teaching and guiding the project leaders and membershow to properly reuse the experience within each sub-process/topic.

• A JBK7L�M�N�O-O�L7PRQ�N�K for the experience reuse process.

Notice the distinction between role and person. In a small organization a small team or (atleast in theory) one single person may fill all these roles. Based on our experience at TTS, acritical minimum central effort to enable substantial reuse of estimation and riskmanagement experience seems to be 2-3 man-years to for an organization of TTS's size.

SUT�V�W�X7Y-Y�Z

When we started our study, the organization did collect project data and it was mandatory towrite experience reports, i.e. the process description had elements of experience reuse.However, the collected information was not systematically used to improve the processes.In other words, the process (or even more, the implementation of the process) had not hadenough focus on the use of the collected information. Looking at other case studies ofsoftware process improvement, see for example (Cusumano and Selby 1996), this seems tobe a typical problem leading to graveyards of data and unused documents. In our opinion,this is a situation even worse than the situation where no data is collected and no reportswritten, and there is probably no more efficient way of destroying the respect for ameasurement and experience report.

We found that TTS's current process description was sufficient to enable experience reuse,given sufficient resources to fill the experience reuse roles described earlier. For a moregeneral experience reuse process and organization, see (Basili et al. 1992).

[�\�]�^`_�a�_�bdce1f

An underlying initial hypothesis on experience reuse, is, of course, that it has a long termbenefit higher than the costs. Currently, we are not in the situation to decide whether this istrue or not. We cannot validate the hypothesis, partly because it is too early, and partlybecause it is difficult to isolate the impact of our work from the impact of other parallelprocess improvement initiatives. However, even without a formal impact study we believeto see the following results of the experience reuse work:

• Improved estimation accuracy and more widespread use of the estimation models.(Earlier, we have shown that the the use of estimation tools seems to have improves theestimation accuracy in TTS compared to pure expert estimates, see (Jørgensen 1997).)

• An increased focus on experience based risk management in the projects.

• An acceptance in the organization for the need to collect and share experience

In addition, we have made a number of interesting observations increasing the probabilityof successful reuse of experience in TTS, such as:

• Currently, the experience reports written by the projects were of little use to otherprojects. This may indicate that without a clear model on how the experience will bereused, there is a great danger of reporting and collecting useless information.

• The mere focus on reuse of experience had a positive impact on the “ improvementculture” in the organization. It would have been very interesting to carry out controlledexperiments on how different actions impacts the software improvement culture. Anexperimental design similar to the one described in “Goals and performance incomputer programming” (Weinberg 1974) may be appropriate.

gihkj�lnmpo-jpqsrutwv�x

The Experience Factory or EF (Basili 1993, Basili et al. 1994) is a framework for reuse ofsoftware life cycle experiences and products. EF relies on the Quality ImprovementParadigm (Basili and Rombach 1988) for continuous and goal-oriented processimprovement, resembling the Shewhart/Deming Plan-Do-Check-Act cycle (Deming 1982).The EF framework prescribes an improvement organization inside a company, a kind of"extended quality department". This implies the " yz7{B|#}�~�y��~��-~��|#z��z��U��-z��}��y#z�� z��/��-z��}��1{B~��|��~��|#z�� -z��¡�#��1� ��~��|#}'y��~��|�7{/~��¢��~�}�£�~7{B|#�7{/z��-��¤7��~��y�'�¦¥��-|�7��}��¦� � �� z��-��7��I��§¨¥¦��7�-|��}7��©�~�}��z��ª�7� " (Basili et al. 1994). ThePERFECT EF framework extends this model by adding a third organizational component:the Sponsoring Organization, which uses the EF for strategic purposes (PERFECT 1996).Within the EF framework, the NASA-Software Engineering Laboratory with its 275developers has collected information about 150 projects in the period 1976-1996. Thepurpose is to record the effects of various software technologies (methods, tools,programming languages, QA techniques, etc.). However, NASA represents a special kind ofstable and resourceful organization. It is a challenge to apply the EF ideas outside ofNASA, i.e. to downscale it to companies with typically 10-30 developers, and where the EFroles are partly being played by the developers themselves. More applications of the EFframework in other contexts are therefore needed, see e.g. (PERFECT 1996).

«¬¯®±°�²�³1´pµ�¶ª®w°pµ

We believe to have contributed to the answers regarding the challenges we described inSection 1 through an in-depth example of how the questions/challenges were approached byTTS. TTS has introduced a standardized development process documented on the web andmade the processes available for all the software developers through the organization’sIntranet. In many ways, this opens new possibilities for software developmentorganizations. We have found that software development experience efficiently can belinked to the process steps and made available to all the developers in a very flexible way.However, the main challenges regarding becoming a learning organization and reusingexperience is not the technology. We found that a lot of “ trial and error” and pragmatism isneeded to find the useful experience and ways to formulate and spread this experience.

We found it useful to be very pragmatic regarding the manifestation of experience. Forexample, a very useful information in our experience database was the links to the expertshaving the required experience. Regarding the role of local (organization dependent)experience vs. best practice experience we found that the local experience made the bestpractice processes significantly more useful. In other words, optimal use of best practiceprocesses seems to require collection and reuse of more local experience.

Achieving a learning organization is a formidable task. Senge claims that the following fivedisciplines are essential to creating learning organizations: ·�¸7¹1º�»�¼�½�¾�¿�½7º�À#¸�¹ªÁ ÂB¿�¸�¼�À½�¾¿�»�Ã�¸�¾Äº-Â�º�Å�½�¹-¸�Ã�Æ�Ç�º�Ç»�¼7º-ÂBÀ#¸�½�¿È¾¸�½�¹-¼�Ç#¼7É and Ê1Ë Ê�ÌÍ�ÎpÊÏÌÐ�Ñ#Ò�Ó�ÑÒ7Ô (Senge 1995). An experiencedatabase like the one we have designed and implemented in TTS can serve as a basis foractivities involved in all five disciplines. An experience database is also a useful means toagree on a common understanding of the current situation. “ Õ�Ö�×�Ø�Ø�Ù�Ú�×�ÛÜ¦ÝBÞ#Ö7ß�ÞÄàBá�Û â�Ù�ã�ä7ÞÜ�åçæ�âØ�Ù�Ú-Ú�Ü�Ö�Û�Ú-Ü7×�ãÞÛ èIÞÄßé×7ßÏÞêEë�æ�Ú-Û×�Ö�Û�×7ßÏ×�Ø�ã#Ü�×�Ú�ä�Þ�ß�Þæ�Ö ” (Senge 1995).

Future work will address the major issue of how projects (contexts) should be characterizedso that experiences collected in one project (context) are applicable to another project(context). How can we judge whether a project is sufficiently similar to (a subset of) theprojects for which we have experience? The approaches described in (Basili ì�í�î�ï#ð�ñ�ò�ò�ó�ômay be taken as a starting point.

õ÷ö�ø�ù¨úüûþý#ÿ�� ÿ�ù��

The authors wish to thank Geir Kirkebøen, University of Oslo and the TTS employees PålWoje, Geir Ove Espås, Majeed Hosseiney, Oddmar Aasebø and Tor Larsen for theircontribution to the work described in this paper.

� ��

C. Argyris et al, 1985, �� !��"#�$�%"�&('��$��"*)+� ,.-0/1"��20��30,+40�03��$5��6�6 ,070��8+9:"#,;"<408;��2:40�03= �$��"#8.>?"#�!�� . San Francisco: Joosey-Bass.

V. R. Basili, 1985, Quantitative evaluation of software engineering methodology, @�A;BDC�E�E#F$GIH0J0KLB�MN�O EQP:GIA.K N @�R0H�@�R$C%G M0G�CTS�B0U!V+W N E#ALS�B�H�M0E#A;E<H$C�E , Melbourne, Australia.

V R. Basili and H. D. Rombach, 1988, The TAME Project: Towards improvement--orientedsoftware environments, XZY�Y�Y\[^]._0`Da._$b�c�d�e�`0aLe�`�f$e�g0cIh�_0];iQY�`0j!d `$i�i#]kd `0j , vol. SE-14, pp. 758–773.

V. R. Basili et al. 1992, The software engineering laboratory — An operational software

experience factory. In: l�m;n0o%p�p#q$r sDtun�v w�x0pTy0z0{ |~} �!��#�.�0�!��}��0�!��%��0�#�k�#�$��T} ��;��D� ��0�;�T�#�D�$} �$��<�;} �0� ,Melbourne, pp. 370-381.

V. R. Basili, 1993, The experience Factory and its Relationship to Other Improvement Paradigms,

pp. 68-83, in I. Sommerville and M. Paul (eds), ��;�D��;�D�� 0�:�� D��$�0� � �¢¡�£;¤�¥+¦<§0¨�©$¤�ª0« ¬�§0£;¦��¨0!® ¨$¦�¦#£k® ¨0°¯�¤�¨�ªD¦#£;¦#¨$±%¦ , Garmisch-Partenkirchen, Germany, September 1993, Springer-Verlag,Lecture Notes in Computer Science 717.

V. Basili, L. Briand, and W. Thomas, 1994, Domain Analysis for the Reuse of SoftwareDevelopment Experiences, In ²�³k´0µ�¶0´�·�¸�¹Dº¢»D¼0¸�¹�½:¾0¾$¿�À$Á;Â$´�·0¸ Ã�À0³;ºQÄ�¾0Å$ÆI¾$º�º#³;ÆI¾0ÅÈÇ ´�³;É#Ê;¹D´�Ë ,NASA/GSFC, Greenbelt, MD..

T. Conolly and D. Dean, 1997, Decomposed versus holistic estimates of effort required forsoftware writing tasks, ÌÎÍDÏ0Í0Ð$Ñ�ÒÓÑ#Ï$Ô;Õ$Ö�×�Ñ<Ï$Ö�Ñ , vol. 43, no 7, 1029-1045.

M. A. Cusumano and R. W. Selby, 1996, Ø1Ù�Ú#Û;Ü�ÝkÜ�Þ0ß;à$á�Ú#Ûká�ß Ý , Harper Collins Business, ISBN0006387780.

W. E. Deming, 1982, âLã�ä!å�æ�ç è?é�ê$ë;ì�í$ãDî�ç�æ ï?æ�ç è?é0ä0ðDíuî%ì0ñ$ê+ò�ç�æ�ç�æ ï?òóê+ì�ô;æ�ç�æ�ì�ð . Massachusetts Institute ofTechnology Center for Advanced Engineering Study, Cambridge, Mass.

W. E. Deming, 1986, õLö0÷�ø�ù�÷�úDû¢ü<ý;þ ÿ;þ ÿ . MIT Center for Advanced Engineering Study, MIT Press,Cambridge, MA.

H. Dreyfus and S. Dreyfus. 1986, �� !��"#�� $ "#� $ � ��!��#�%'& �� ($)� *(� ��$ ��+ �,�%�)�-$)��+��"#$ � , Free Press, New York.

B. Flyvebjerg, 1991, .0/�1 2 34�/�5 2 1 6�1738�9�/#8�1 :�;�6�1=<�34�< >(6�1 6 ?�@�2 ;�6'4�?(< /ACBED�2F4�;!GIH , Akademisk Forlag.

N. Goodman, 1951, J�K�LNM�O PRQ�S�O QPRLUT)VXWZY�Y�L [�P,[�\�S�L , Cambridge, Mass.

M. Hammer, 1996, ]_^a`cbd#e-fR^�^ d�g�hFd�^�^ f(hFd�g , Harper Collins, New York.

M. Jørgensen, 1995, Empirical evaluation of CASE Tool efficiency. i0jRk�l�mn�oFp�q r_sut�q m#vwktyx�m#ktz{�{}| o l z q)o kt�~�k x_n�k x#q � z j(�N�� z ~(�jR�� t�q , Orlando, 207-230.

M. Jørgensen, 1997, An empirical evaluation of the MkII FPA estimation model, �0�(��'�� y�#��(��)� �+�w��y��'�(� �� , Voss, 7-18.

G. Morgan, 1993, �� )��! ¡� ¢�£N��¤(�7� ¥!¦ ¤(£'�� §�£+��#��£��¨£ �� , SAGE Publications, London,1993, ISBN 0-8039-5299-6.

M. Paulk et al, 1995, ©�ª�«+¬c®��¯�° ± ° ² ³µ´¶�² ·¸R° ² ³¹´»º¼�«�± ½�¾�·#° ¼�«�± °F¿�« À�Á�º¸�° Â�®�¸(ºÃ�° ¿#Ä%²)ª�«NÀ(º Á�² ÅÆ�¸(«®�¸(º#Ç�« À�À . Software Engineering Institute, ISBN 0-201-54664-7.

PERFECT Consortium, 1996, È0ÉËÊÍÌZÎ Ï�Ð Ñ(Ò)Ð Ó�Ô�ÐNÕ0Ö�Ô�× ØÑÚÙ�Û�Ü�Ý�ÐNÈ0Ì0ÕßÞ»Ø�à�Ð�á , ESPRIT Project 9090, D-BL-PEF-2-PERFECT9090.

E. H. Schein, 1987, â0ãRä�å�æ ç�ç�å�ä�è�ç(é�ê ëFì�ë í äè�îðï�ä#ê é�ñ�æNòuòIó,ô Addison-Wesley, ISBN 0 201 06744 7.

T. Schrader, 1998, õ÷ö�ø#ù ù ø�ú�ûýüyþ!þ�ÿ(ø��ù��ø��(ù��(ù� ú ��ù�)ø��ú��ù��ø��%ü�� Rù øÿ��ù ��(ù ��ÿ�� øÿ��Uö�ÿ��ø�� (ù ÿ(ü��ù üÿ�� , Project Report, The Norwegian University of Science andThechnology.

P. M. Senge, 1995, !#"�$�%'& (�)�"'*+& ,�-�& . /�& 0�$213!#"�$#4'5�)�6�0�7�895:6�-2)�&�-�$<;�(=)�"�$�>�$6�5�0�& 0�?�@ 5�?�6�0�& A�6�)�&�;�0 ,Currency/Doubleday.

C. R. Symons, 1993. B�C�D�EGF H�I�J�K�L M�L N�O�H�N�P�JK�E�L�Q H�E�L�C�N�R�SUTVWV3X9Y�Z , New York: John Wiley and Sons.

G. Weinberg and E. Shulman, 1974, Goals and performace in computer programming. []\�^ _�`a _�b�c�d�e�f , vol 16.

pqalk - department of computer science (idi) - ntnu in accordance with the cmm framework, gave...

Documents