from agile software product line engineering towards software … · at sintef, and other places....
TRANSCRIPT
From Agile Software Product Line Engineering Towards Software
Ecosystems
Geir Kjetil Hanssen
Doctoral Thesis
Submitted for the Partial Fulfillment of the Requirements for the Degree of
Philosophiae Doctor
Department of Computer and Information Science Faculty of Information Technology, Mathematics and Electrical Engineering Norwegian University of Science and Technology
Trondheim, September 10, 2010
Copyright © 2010 Geir Kjetil Hanssen
ISBN printed - ISBN 978-82-471-2425-3
ISBN electronic - ISBN 978-82-471-2426-0
ISSN - 1503-8181
NTNU - 2010:219
Printed in Norway by NTNU Trykk, Trondheim
Alla dessa dagar som kom och gick, inte visste jag att det var livet
- Stig Johansson
- i -
Abstract Development and evolution of software products is a challenging endeavor and a significant sub-
field of software engineering. One of the commonly applied approaches to control and manage this process is software product line engineering (SPLE). There exist a few process frameworks where the development of lines of related software products is basically a sum of two processes: the development of reusable assets and the rapid construction of software applications using pre-developed assets. Agile software development (ASD) is another major paradigm, which also has been widely adopted by the industry over the past decade.
SPLE and ASD seek to achieve the same goal i.e. rapid and efficient construction of software. However, the former emphasizes extensive up-front investment in the development of assets for later re-use in contrast to ASD, which emphasizes a reactive approach, avoiding up-front planning and development
Even though these two approaches may seem to oppose each other, the industry has lately showed great interest in combining them both, aiming to cover the weaknesses of the one with the strengths of the other. In combination with the overall shift in the software industry from closed systems mindset towards open systems, the uptake of some ASD principles (for example active customer engagement, incremental and iterative development, and open information flows) in product line organizations may contribute to the emergence of more flexible software ecosystems.
This thesis presents a longitudinal study of a software product line organization, which has adopted an adapted ASD methodology in an SPLE context and to a large extent has successful in gaining benefits from both approaches, developing towards more open yet controlled processes. Data have been collected over a period of approximately five years following the progression from a strictly waterfall oriented approach, via the adoption of the agile method Evo, up to the current combined agile software product line engineering approach.
The following research questions have been addressed in this thesis: RQ1: How can software product line engineering and agile software development be combined? RQ2: How does a software ecosystem shape? The main contributions of this work are: C 1. Through a longitudinal study of a software product line organization we provide detailed
insight into an industrial case and how they have changed over time. C 2. We have illustrated some of the details of how SPLE and ASD can be combined in practical
terms. We describe the current organization, their product line and their agile software product line engineering process.
C 3. We have illustrated how the incorporation of some of the central agile principles has enabled a closer cooperation with external actors.
C 4. We have explained the emergence and mode of operation of a software ecosystem, and provided a conceptual model of software ecosystems based on our findings.
C 5. We have proposed a theory of software ecosystems, rooted in socio-technical theory and the concept of organizational ecology.
- ii -
Preface This thesis is submitted to the Norwegian University of Science and Technology
(NTNU) for partial fulfillment of the requirements for the degree of Philosophiae Doctor.
The work leading to this thesis has been performed at the Department of Computer and Information Science, NTNU, Trondheim, with Professor Reidar Conradi (NTNU) as the main supervisor, and Chief Scientist Tore Dybå (SINTEF ICT) and Professor Eric Monteiro (NTNU) as co-supervisors.
This work was conducted as part of the EVISOFT research project, supported by the Research Council of Norway under grant 156701/220.
- iii -
Acknowledgements There are many persons that have encouraged, helped, and supported my work on
this thesis in various ways over the last four years. First, I would like to thank my supervisors Reidar Conradi (NTNU), Tore Dybå (SINTEF), and Eric Monteiro (NTNU) for their timely advice and kind pressure whenever needed. During my studies I have continued to work at SINTEF ICT but with a part-time position. It has been of great importance to me to have this contact with my colleagues throughout the study, both for professional and social reasons. Thanks to Tore Dybå, Tor Erlend Fægri, Torgeir Dingsøyr, Nils Brede Moe, Børge Haugset, and all the other great people at SINTEF ICT. I would also like to thank my manager, Eldfrid Øfsti Øvstedal for giving me this opportunity; I hope your investment will pay off. I am also grateful to Pekka Abrahamsson that has provided valuable advice through burning discussions during his time as my colleague at SINTEF. I am also grateful to the Research Council of Norway that have supported my work through grant number 156701/220, funding the EVISOFT research project which was managed by Tor Ulsund. Thank you Tor! I also need to thank the good people at my case company CSoft for their support over many years. My non-disclosure agreement with them prohibits me from mentioning names, but you know who you are – thank you.
During the time of this study I have been a member of a group of PhD students and the like, that – despite the slightly humorous name “The Scientist Factory” – provided a highly valuable arena for discussing good and bad ideas, learning from failures and successes of others, getting nice and nasty feedback, and the sharing of frustrations. Thanks to Eric Monteiro and Torgeir Dingsøyr for managing our meetings, and thanks to my peers; Børge Haugset, Gasparas Jarulaitis, Gro Alice Hamre, Kirsti Elisabeth Berntsen, Ole Andreas Alsos, Øyvind Hauge, Thomas Østerlie, Tor Erlend Fægri, Torstein Hjelle, and Vigdis Heimly. I have also benefited immensely from numerous conversations with a lot of other great people at the university, with industrial partners, at SINTEF, and other places. Some of them are: Anita Gupta, Sven Ziemer, Knut Harald Hassel Nielsen, Tor Stålhane, Maria Letizia Jaccheri, Anders Kofod-Petersen, Sverre Tinnen, Haakon Fougner Spilde, Hans Westerheim, Øyvind Pettersen, Frank Bertheussen, Darja Šmite, mom, and dad. I also need to thank my co-authors at the University of Oslo and the SIMULA research laboratory, Aiko Fallas Yamashita and Leon Moonen. Finally I want to thank my wife Randi for her unfaltering support, and my children Mikael, Silje, and Marius for accepting that at evenings, weekends, and holidays their dad put priority on work instead of spending time with them. Well guys – now I’m done!
NTNU, Trondheim, September 10, 2010 Geir Kjetil Hanssen
- iv -
Contents
Abstract........................................................................................................................ i
Preface......................................................................................................................... ii
Acknowledgements ................................................................................................... iii
Contents ..................................................................................................................... iv
List of Figures........................................................................................................... vii
List of Tables ........................................................................................................... viii
Abbreviations ............................................................................................................ ix
1 Introduction.......................................................................................................... 3
1.1 Problem Outline 3 1.2 Research Context 4 1.3 Research Questions 4 1.4 Research Design 6 1.5 Publications 7 1.6 Contributions 9 1.7 Thesis Structure 13
2 Background ........................................................................................................ 15
2.1 State of the art and research 15 2.1.1 Software product line engineering......................................................... 15 2.1.2 Agile software development .................................................................. 18 2.1.3 Concepts combined: Agile software product line engineering.............. 20 2.1.4 Software Ecosystems ............................................................................. 23 2.1.5 Organizational change ........................................................................... 26
2.2 The case 29 2.2.1 Organization........................................................................................... 29 2.2.2 Product line ............................................................................................ 31 2.2.3 Processes ................................................................................................ 33 2.2.4 A retrospective of the development of the organization........................ 36
- v -
2.3 A theoretical framework: organizational ecology 38
3 Research Approach............................................................................................ 42
3.1 Studying software engineering 42 3.2 The applied research process of the thesis 48
3.2.1 The selection of the case organization................................................... 51 3.2.2 Data sources ........................................................................................... 52 3.2.3 Data analysis .......................................................................................... 54
4 Results ................................................................................................................. 60
4.1 Study 1: Adoption of Evo 60 4.1.1 Paper 1 - Agile Customer Engagement - a Longitudinal Qualitative Case
Study (Hanssen and Fægri, 2006)........................................................................... 60 4.1.2 Paper 2 - Collaboration and process fragility in evolutionarily product
development............................................................................................................ 62 4.2 Study 2 – Combining agile software development and software product line
engineering 63 4.2.1 Paper 3 - Process fusion - an industrial case study on agile software
product line engineering ......................................................................................... 63 4.3 Study 3 – Process agility and software entropy 64
4.3.1 Paper 4 - Software entropy in agile product evolution .......................... 65 4.4 Study 4 – Software ecosystem emergence 67
4.4.1 Paper 5 – Opening up product line engineering .................................... 68 4.4.2 Paper 6 - A Longitudinal Case Study of an Emerging Software
Ecosystem: Implications for Practice and Theory.................................................. 69
5 Discussion and implications .............................................................................. 76
5.1 An agile software product line engineering process 76 5.1.1 Actions taken to combine the processes ................................................ 76 5.1.2 Effects of the combined process ............................................................ 80 5.1.3 Contextual factors .................................................................................. 84
5.2 An emerging software ecosystem 85 5.2.1 Changing from agile software product line engineering to a software
ecosystem ............................................................................................................... 85 5.2.2 A case of improvisational change.......................................................... 86 5.2.3 A conceptual model of a software ecosystem........................................ 87
5.3 Implications for theory 91 5.4 Implications for practice 95 5.5 Limitations 95
- vi -
6 Conclusions and further work .......................................................................... 97
6.1 Answering the research questions 97 6.2 Directions for future work 99
References............................................................................................................... 101
Appendix A............................................................................................................. 108
Study 1 – Adoption of Evo 108 Data #1: Group interview (postmortem analysis) with 6 developers .............. 108 Data #2: Guide for three semi structured interviews with participating customers
.............................................................................................................................. 108 Data #3: Guide for five semi structured interviews with PMT members ........ 109
Study 2 – Agile SPL 110 Data #4: Guide for one semi structured interview with the CTO.................... 110 Data #5: Guide for one semi structured interview with PMT manager........... 111
Study 3 – Software entropy 113 Data #8: Guide for one semi-structured interview with one of the team leaders
.............................................................................................................................. 113 Data #9: Guide for one semi structured interview with one of the developers 113
Study 4 – Software ecosystems 113 Data #11: Guide for one interview with the R&D manager ............................ 113 Data #12: Guide for one interview with the manager of professional services114 Data #13: Guide for one interview with the PSG manager ............................. 114 Data #14: Guide for three interviews with PSG members............................... 115 Data #15: Guide for one interview with a Technical Account manager ......... 115 Data #17: Observation of the 2008 product conference .................................. 115 Data #18: Observation of review meeting ....................................................... 115
- vii -
List of Figures Figure 1 - Study design..................................................................................................... 6 Figure 2 - Evolution of research ....................................................................................... 7 Figure 3 - High-level SPLE processes............................................................................ 16 Figure 4 - A software product line engineering framework ........................................... 17 Figure 5 - Agile software development .......................................................................... 19 Figure 6 - Orlikowski’s and Hofman’s model of improvisational change ..................... 27 Figure 7 - The Shewhart/Deming improvement cycle ................................................... 28 Figure 8 - High-level product line overview .................................................................. 31 Figure 9 - An external-actor centric development process............................................. 33 Figure 10 - Timeline of the development of the case organization ................................ 36 Figure 11 - The socio-technical system and the organizational ecosystem.................... 39 Figure 12 - Types of organizational environments......................................................... 40 Figure 13 - Empirical software engineering ................................................................... 43 Figure 14 - Open innovation........................................................................................... 89 Figure 15 - A conceptual model of a software ecosystem.............................................. 90
- viii -
List of Tables Table 1 - Focus of the studies........................................................................................... 6 Table 2 - ASD and SPLE compared............................................................................... 22 Table 3 - Software Ecosystem Taxonomy...................................................................... 24 Table 4 - OSS and closed-source ecosystems - similarities and differences.................. 26 Table 5 - The iteration week plan................................................................................... 35 Table 6 - The application of the seven principles of interpretive field studies .............. 49 Table 7 - Data sources .................................................................................................... 53 Table 8 - Example coding nodes .................................................................................... 57 Table 9 - Values from being a network actor ................................................................. 88
- ix -
Abbreviations ASD Agile Software Development ASP Application Service Provider ASPLE Agile Software Product Line Engineering CEO Chief Executive Officer CI Continuous Integration CSO Chief Strategy Officer CTO Chief Technical Officer EPG Electronic Process Guide ICT Information and Communication Technology IET Impact Estimation Table IPR Intellectual Property Rights KLOC K (1000) Lines of Code NTNU Norwegian University of Science and Technology OSS Open Source Software PDCA Plan Do Check Act PMA Post-mortem analysis PMT Product Management Team PSG Product Strategy Group (succeeding the PMT from approx. 2006) QA Quality Assurance R&D Research & development (also the name for the development
department) SE Software Engineering SECI Socialization Externalization Combination Internalization SEI Software Engineering Institute SOAP Simple Object Access Protocol SPLE Software Product Line Engineering TAM Technical Account Manager
- 1 -
PART I - Summary of studies
Introduction
- 3 -
1 Introduction
1.1 Problem Outline
Development of software products as well as the maintenance and evolution of these products over time is a complex and challenging task for most software product organizations (Adler, 2005). To support this effort, a large section of the product oriented software industry applies variants of software product line engineering (SPLE) (Clements and Northrop, 2002; Pohl, Böckle et al., 2005). SPLE is an extensive approach to organizing the continuous development of software products, where the main property is the planned, prepared, and anticipated reuse of a set of domain artifacts for later fast and efficient composition of applications. This is in principle a better approach than developing each product from scratch each time (Böckle, Clements et al., 2004), a principle which have been supported by industrial case studies, for example (Buhrdorf, Churchett et al., 2004; Kiesgen and Verlage, 2005), and more recently (Babar, Ihme et al., 2009). Several rather comprehensive SPLE methods or frameworks exist, and have been adopted and practiced in the industry for more than a decade. These frameworks are based on the development of assets for later (anticipated) use, an approach, which may seem to conflict with another major recent methodological development in the software industry: agile software development (ASD). Recently, there has been a lively interest in combining these two approaches as they both promise to improve the efficiency of the software product development process and the responsiveness towards an increasingly volatile market (Tian and Cooper, 2006).
The first challenge this creates is how to practically combine the two methods: SPLE imposing up-front investments and a pro-active approach, and ASD emphasizing flexibility and a fundamentally re-active approach. The second challenge, more relevant to this thesis, is to understand the rationale of such a combination and how it affects the product line organizations ability to produce quality software solutions.
This combination of processes is related to a change in the industry and its market, from traditional closed processes towards open development processes crossing organizational borders (Moore, 1993), a trend that may lead towards a new entity – software ecosystems (Bosch, 2009; Messerschmitt and Szyperski, 2003). This change seems inevitable, and it is important to understand its development and how to respond to it.
Introduction
- 4 -
1.2 Research Context
This thesis summarizes about five years of studies of a software product line organization called CSoft (a pseudonym1), and shows how they have progressed from a plan-based and closed process approach towards a more open process, where feedback and collaboration with various external actors are fundamental drivers. This change in process has affected how the organization relates to its external environment – which we describe as a software ecosystem. The adoption of Evo (Gilb, 2005), an agile method, in 2004 has been an important enabler for this change. During the period from 2004 to 2010, four studies have been conducted, starting with an investigation of the early adoption of Evo and its preliminary effects, and then the later studies looking into how the organization collaborates with its external environment in both planning and development. Thus, the object of study in this thesis is twofold. Firstly, we describe and investigate the present organization, its processes, and its product line. This is the result of fourteen years of development since the establishment of CSoft, and is an example of an agile software product line engineering process. Secondly, we study this change that has taken place and how the situation has progressed towards a software ecosystem.
1.3 Research Questions
What are we studying? -‐ We have conducted four successive studies of a medium-sized software
product line organization. The first study started in 2004 and the last one was completed in 2010, with completion of data collection in 2009 – thus covering almost five years of development. The focus of study has been 1) the longitudinal development of this organization, and 2) the resulting (present) organizational set-up, its planning and development processes, its product line, and its relations with its organizational environment.
Why are we interested in it? -‐ The investigated organization has adopted an agile development method in a
product line environment, and matured this combination over a period of five years in order to improve software process performance in general and the relationship with actors external to the organization in particular.
-‐ The organizational development being described in this thesis – a movement towards an open product line engineering process – is strongly related to an
1 Due to a non-disclosure agreement with CSoft; company, product, and domain-specific details are
omitted to maintain the anonymity of the case organization.
Introduction
- 5 -
increasing focus on the concept of software ecosystems. This concept is still poorly understood within software engineering, and no explicit coupling to relevant and well-established theories on organizational ecology has been developed.
Why should this be interesting to others? -‐ The company’s position as the market leader is a strong indication of a well
functioning organization (including its processes and products). Generalized descriptions of their agile software product line engineering approach, founded on data and related to theory, have a great potential of revealing valuable implications for practice. Likewise, we also propose implications for theory as a contribution to the research community and continued research on software ecosystems.
-‐ Based on the four studies and the resulting six publications, this thesis seeks to answer the following two research questions:
RQ1: How can software product line engineering and agile software development be combined?
RQ2: How does a software ecosystem shape?
Introduction
- 6 -
1.4 Research Design
The complete study of how CSoft has developed over time and their resulting approach to evolving their product line builds on four recent studies, reported in six publications. Studies 1-3 are used to answer research question 1, and study 4 is used to answer research question 2. In sum, these studies constitute a longitudinal study and each of the underlying studies focuses on topics related to the research questions of this thesis. This longitudinal study has been exploratory, and the overall objective has been to collect data to describe the ongoing development of CSoft as well as the resulting process and organization. This means that the study did not start out with a hypothesis or theory to be tested but rather that it is a theory building type of study. For example implications for theory being discussed in section 5.4 have emerged throughout the study.
Figure 1 - Study design The four studies covered the following topics: Table 1 - Focus of the studies
Study Focus Paper Study 1 The initial adoption of the agile method Evo in a waterfall-‐
oriented product line organization, with an emphasis on how developers relate to customers and the fragility of the agile approach.
P1, P2
Study 2 The combination of ASD and SPLE and the coordination between the strategic, tactical, and operational processes.
P3
Study 3 The emerging problem of software entropy, the potential reinforcing effect from the agile development process, and potential solutions to improve the situation.
P4
Study 4 The increasing openness of information sharing and cross-‐organizational collaboration.
P5, P6
Introduction
- 7 -
In addition to the six publications we have also made use of two earlier background studies (by other authors) for details related to the early development of the case organization (Johansen, 2005; Moe, Dingsøyr et al., 2002).
All these studies, conducted sequentially in time, build on the results from previous studies. The initial focus was the adoption of agile software development in a software product line engineering context (discussed in section 5.1). This is followed by a discussion of the emergence of a software ecosystem (section 5.2).
Figure 2 - Evolution of research
1.5 Publications
P1: G.K. Hanssen, T.E. Fægri, Agile Customer Engagement: a Longitudinal Qualitative Case Study, in: Proceedings of the 5th International Symposium on Empirical Software Engineering (ISESE'06), IEEE Computer Society, Rio de Janeiro, Brazil, 2006, pp. 164-173.
Relevance to this thesis: This paper presents the results of a study of the change from a waterfall-like approach to the agile Evo development process, and how it has affected the way the development organization relates to the customers in their release projects. This study was conducted shortly after the adoption of Evo, and presents the initial effects.
My contribution: As the principal author I initiated and conducted the study with the case organization. The analysis of data was done collaboratively.
P2: T.E. Fægri, G.K. Hanssen, Collaboration and process fragility in evolutionarily
product development, IEEE Software 24 (3) (2007) 96-104. Relevance to this thesis: This paper reports on a study of the transition to Evo,
and explains both positive and negative effects that were experienced shortly after the process adoption.
My contribution: I participated in the whole process, from planning of the study to data collection, analysis, and reporting. The first author had the overall responsibility, and made the final decisions on form and language.
Introduction
- 8 -
P3: G.K. Hanssen, T.E. Fægri, Process Fusion - Agile Product Line Engineering: an
Industrial Case Study, Journal of Systems and Software 81(2008) 843-854. Relevance to this thesis: This paper reports on the study of the same case, but
builds on new data collected. The paper has a holistic focus on the development and evolution of the product line, and explains how the long-term strategic planning process relates to the tactical processes of developing software and the day-to-day operational processes. The data in this study were collected some years after the initial adoption of Evo (reported in P1 and P2), and thus represent a more mature organization.
My contribution: As the principal author I had the overall responsibility, and made the final decisions on form and language. The whole process, from planning of the study to data collection, analysis, and reporting was done collaboratively.
P4: G.K. Hanssen, A.F. Yamashita, R. Conradi, L. Moonen, Software entropy in
agile product evolution, in: Proceedings of the 43rd Hawaiian International Conference on System Sciences (HICSS'10), IEEE Computer Society, Hawaii, USA, 2010.
Relevance to this thesis: This paper goes deeply into one of the most severe problems in the case organization and one of the main impediments in the agile software development process, namely software entropy. Based on a thorough overview of relevant empirical studies on code-smell2 detection and the resulting code refactoring as well as new data collected, this paper gives an insight into the causes behind the problem and the negative effects it has on the agile workflow. Potential solutions are discussed.
My contribution: As the principal author I was in charge of the study design, data collection, and analysis. The second author was in charge of the inherent literature review, but with my contribution for some of the summary of results. The second author also contributed significantly to the discussion of the results, both from the case study and the literature review.
P5: G. K. Hanssen, Opening up Software Product Line Engineering, In proceedings
of the 1st International Workshop on Product Line Approaches in Software Engineering, in conjunction with the 32'nd International Conference on Software Engineering (ICSE). 2010. Cape Town: ACM.
Relevance to this thesis: This workshop paper builds on the same data as reported in P6 but elaborates the topic of open processes and cross-organizational
2 “Code-smells” is a term close to “anti patterns” and refers to symptomatic flaws in code
structures.
Introduction
- 9 -
collaboration. Doing so, relevant literature is referred to and discussed in relation to the study of CSoft.
My contribution: All work was done by the sole author. P6: G. K. Hanssen, An Emerging Software Ecosystem: A Longitudinal Case Study,
submitted to the Journal of Systems and Software, special call on “Software Ecosystems”, guest editor is Jan Bosch. See: http://eventseer.net/e/12074/.
Relevance to this thesis: This paper is the main contribution in the series of studies of CSoft as it summarizes the organizational development over time. It also identifies a relevant theoretical foundation (organizational ecology), to which the results are related.
My contribution: All work was done by the sole author. (Note: the pseudonym of the case organization were CompNN in papers 1-3 and CSoft
in papers 4-6 as well as in this thesis. These names refer to the same organization. The shift of name is only due to aesthetical reasons.)
1.6 Contributions
This thesis contributes to ongoing research on (1) how agile methods can be applied in software product line engineering (Babar, Ihme et al., 2009; Carbon, Lindvall et al., 2006; Ghanam and Maurer, 2009; Ghanam and Maurer, 2010; Hanssen and Fægri, 2008; Mohan, Ramesh et al., 2010; Noor, Rabiser et al., 2008), and (2) how software ecosystems shape and function (Bosch, 2009; Bosch and Bosch-Sijtsema, 2009; Jansen, Brinkkemper et al., 2009; Jansen, Finkelstein et al., 2009; Messerschmitt and Szyperski, 2003).
Through a longitudinal study of a software product line organization and its change, we have studied the practicalities and outcome of such a process combination. Through this study, we have discovered how some of the fundamental agile principles have improved collaboration with the external environment of the product line organization, consisting of customers and external organizations relating to the product line. Based on these insights we propose a theory on software ecosystems, illustrated by our studies.
In more detail, this thesis has five main contributions. C1-C3 gives insight into the combination of ASD and SPLE (based on the discussions in section 5.1). C4-C5 relates to the development of the concept– and a theory on software ecosystems (based on the discussions in section 5).
C 1. Through a longitudinal study of a software product line organization we provide
detailed insight into an industrial case and how they have changed over time.
Introduction
- 10 -
C 2. We have illustrated some of the details of how SPLE and ASD can be combined in practical terms. We describe the current case organization, their product line and their agile software product line engineering process.
C 3. We have illustrated how the incorporation of some of the central agile principles has enabled a closer cooperation with external actors.
C 4. We have explained the emergence and mode of operation of a software ecosystem, and provided a conceptual model of software ecosystems based on our findings.
C 5. We have proposed a theory of software ecosystems, rooted in socio-technical theory and the concept of organizational ecology.
The following list summarizes contributions reported in the selected papers
supporting this thesis: P1: Agile Customer Engagement: a Longitudinal Qualitative Case Study (ISESE
Conference, 2006) -‐ Active engagement of external stakeholders in the development process requires
continuous proactive management. An external stakeholders’ motivation to contribute relies on immediate benefits, in return active stakeholders positively affect developers’ motivation and confidence in the result.
-‐ The adoption of agile principles within a software product line context requires a high degree of discipline to coordinate activities. Also, a technical infrastructure automating repetitive tasks improves the effectiveness of agile development practices.
-‐ Applying agile principles in a product development context improves process visibility both internally in a development organization as well as externally among stakeholders.
P2: Collaboration and process fragility in evolutionary product development
(IEEE Software, 2006) -‐ Maintaining a watchful, vigorous stakeholder management capability is
paramount to successful application of core agile principles in a product development context. Rapid feedback on developed code, appropriate metrics, and efficient decision-making reduce uncertainty and thus improve motivation.
-‐ To reduce the risk of architectural erosion, involvement of external stakeholders should be balanced with internals, with the focus on the product as an engineering artifact. (Relates to P4).
Introduction
- 11 -
-‐ An agile process with a high iteration frequency increases vulnerability to irregularities. Unresponsive stakeholders expose this fragility. Maintaining a high iteration frequency is expensive and requires a high degree of discipline at many organizational levels.
P3: Process fusion: An industrial case study on agile software product line
engineering (Journal of Systems and Software, 2008) -‐ SPLE naturally supports a long-term strategic process, and ASD naturally
supports a short-term tactical process. The day-to-day operational process connects these two. This three-process focus enables exploitation of long-term ambitions for innovation as well as small-scale tactical innovations, respectively radical and incremental innovations.
-‐ Running an explicit strategic process and working closely with external stakeholders poses additional overhead but promotes valuable creativity.
P4: Software entropy in agile product evolution (HICSS Conference, January
2010) -‐ Complexity of the software product hampers productivity and quality of the agile
development process. The focus on high speed in short iterations may not leave enough time to resolve problems related to system entropy.
-‐ In order to manage system entropy, agile development processes need additional support for understanding, planning, and evaluating the impact of changes. Code smell analysis and refactoring is a viable solution.
P5: Opening Up Software Product Line Engineering (PLEASE Workshop, May
2010) -‐ The orientation of the software industry towards software ecosystems can be
supported by an opening of 1) information flow, 2) innovation processes, and 3) technical interfaces.
P6: A Longitudinal Case Study of an Emerging Software Ecosystem:
Implications for Practice and Theory (Submitted to the Journal of Systems and Software, April 2010)
-‐ The software (product) industry is going through a change where the traditional closed system mindset is being replaced by more open collaboration between the software provider and external actors.
-‐ Agile software development principles, in particular iterative/incremental development and active stakeholder participation, may enable a product line organization to manage and to benefit from the change in the industry.
Introduction
- 12 -
-‐ We propose a conceptual model of software ecosystems. -‐ The theory on organizational ecology is used to 1) explain results of the case
study, and 2) establish a common platform for future research.
Introduction
- 13 -
1.7 Thesis Structure
This remainder of the thesis consists of two parts.
PART I – Summary of studies
Chapter Content
2 – Background Presents the background for the analysis, discussion, and the conclusions of this thesis. The chapter covers three areas. First it gives insight into the present case organization and its development over time, in particular for the last five years. Second, it presents an overview of the state of the art, and the state of research on software product line engineering, agile software development, the combination of these two approaches, and the emerging concept of software ecosystems. Third, the background chapter describes a theoretical framework based on socio-‐technical theory and organizational ecology.
3 – Research Approach
States the research goals and the related research questions. It also motivates for and explains the research approach. The chapter gives details on how data have been collected and analyzed.
4 – Results Presents the synopsis of the results of the four studies.
5 – Discussion and Implications
Discusses the main concepts being investigated and developed, which is agile software product line engineering, the opening of the development process, and how this leads towards an ecosystem. Further, the chapter explains the implications for theory and practice.
6 – Conclusions Uses the discussion of results to answer the research questions and to clarify the practical and theoretical contributions of the thesis. We also provide directions for future research.
Introduction
- 14 -
PART II – selected publications
1. Hanssen, G.K. and Fægri, T.E., Agile Customer Engagement: a Longitudinal Qualitative Case Study. In proceedings of 5th International Symposium on Empirical Software Engineering (ISESE'06), 2006, Rio de Janeiro, Brazil. ACM: p.164-173.
2. Fægri, T.E. and Hanssen, G.K., Collaboration and Process Fragility in Evolutionarily Product Development. IEEE Software, 2007. 24(3): p. 96-104.
3. Hanssen, G.K. and Fægri, T.E., Process Fusion - Agile Product Line Engineering: an Industrial Case Study. Journal of Systems and Software, 2008. 81: p. 843-854.
4. Hanssen, G.K., Yamashita, A.F, Conradi, R., and Moonen, L., Software Entropy in Agile Product Evolution. In proceedings of 43rd Hawaiian International Conference on System Sciences (HICSS'10). 2010. Hawaii, USA: IEEE Computer Society. p. 1-10.
5. Hanssen, G.K. Opening up Software Product Line Engineering. In proceedings of 1st International Workshop on Product Line Approaches in Software Engineering, in conjunction with the 32nd International Conference on Software Engineering (ICSE). 2010. Cape Town, South Africa. p. 1-7.
6. Hanssen, G.K., A Longitudinal Case Study of an Emerging Software Ecosystem: Implications for Practice and Theory, submitted to Journal of Systems and Software, under review.
Statement of authorship of joint publications from Tor Erlend Fægri Statement of authorship of joint publications from Aiko Fallas Yamashita Statement of authorship of joint publications from Reidar Conradi Statement of authorship of joint publications from Leon Moonen Listing of all publications
Background
- 15 -
2 Background This chapter is organized in three main sections. First, we give an overview of a set
of relevant topics, which forms a basis for the later discussion of our findings. Second, we present and explain the details of the case organization we have studied: the organization, their product line, their processes, and a retrospective of how this organization has changed over the past years. Third, we present a theoretical framework explaining the concepts of socio-technical theory and organizational ecology.
2.1 State of the art and research
The topic of this thesis relates to several areas of Software Engineering, with a particular emphasis on three of them: 1) Software Product Line Engineering, 2) Agile Software Development, and 3) Software Ecosystems. The former two are relatively well-established concepts as they both have been practiced in industry and researched by academia for 10-15 years – a long time span in software engineering. This chapter explains concepts and ideas, and gives an overview of relevant research. Lately, the idea of combining SPLE and ASD, seemingly conflicting methodologies, has emerged to take advantage of their respective strengths. This chapter covers this recent development. The latter concept, Software Ecosystems, is currently emerging, enriched by ideas from several related fields, thus it is as yet “unpolished” as a sub-discipline. However, importantly to the discussion and conclusions of this thesis, the substance of this concept is presented together with some viewpoints on the probable development in the near future.
2.1.1 Software product line engineering
SPLE is an extensive and organization-wide approach to developing and evolving lines of software products. In this context, a product line is a set of related software products that have some common parts, for example software components, architectural design, data structures, but that still show distinct and different characteristics. The main concept of SPLE is to prepare for and support this variability in the characteristics of the products in the product line, in order to prepare for reuse, to reduce time-to-market, to reduce development costs, to reduce maintenance effort, to improve the quality of software products and to cope with complexity (Pohl, Böckle et al., 2005) p.9. For example, industry reports a reduction of time-to-market less than 50% (Philips Medical Systems) (ibid., p.233) and a reduction of resource consumption by 20-30% (Bosch Gasoline Systems) (ibid., p.133).
A similar term, product families (Linden, 2002), is also frequently used, but for simplicity the product line term will be used throughout this text.
Background
- 16 -
Various comprehensive and detailed descriptions of SPLE practices and principles are available, for example, in the frequently quoted books by Clements and Northrop (2002), Pohl et al. (2005) and van der Linden et al. (2007). These descriptions vary somewhat in their use of terminology. At a conceptual level, they all describe SPLE as two main complementary processes. As Figure 3 (derived from (Clements and Northrop, 2002), p.30) illustrates, the first process deals with the development and maintenance of a reusable core or domain assets, typically code organized as components or modules. The second process deals with the development of software products, or applications, using these core assets for rapid and (cost) efficient composition of software products. Variants of this fundamental process are the pro-active approach where core assets are developed up-front due to an anticipated need in the near future and the re-active approach where re-usable assets are derived or harvested from products and stored for later re-use in other applications (McGregor, 2008).
Figure 3 - High-level SPLE processes Beyond the central dual process of domain asset development and product
development, various other supportive processes and techniques are described, for example management processes and software architecture practices. Pohl et al. (2005) (p. 22) has developed a software product line engineering framework that defines and relates these practices (Figure 4). Another extensive SPLE process description is the guidelines provided by the Software Engineering Institute, which describes 29 practice areas in detail (Clements and Northrop, 2002)
Background
- 17 -
This approach to product development and evolution of products over time is very much inspired by practices in other industries. For example, the idea is comparable to how consumer electronics products are made today. A supplier of home appliances offers lines of e.g. washing machines having variable characteristics such as number of programs, motor size etc., but that still share many common parts or components, for example the door locking mechanism and the water pump. This gives the customer the opportunity to find a product matching the intended use at a suitable price level. These two concepts of reuse and variability are equally essential when it comes to software product lines.
The fundamental principle of SPLE is that extensive initial investment should be made in the development of a platform of reusable domain-generic assets for anticipated later use in products. Following this principle presents an opportunity for the efficient construction of new products, but carries with it a risk of not getting an economically viable return from the investment, if pre-developed assets are not sufficiently reused for any reason (McGregor, 2008). This real risk is caused by the time span between development (investment) and use (pay-off).
SPLE is a holistic approach to software engineering, affecting aspects from process to technology. van der Linden et al. (2007) has defined the BAPO-model which identifies four interrelated aspects; Business, Architecture, Process and Organization. In order to describe and practice SPLE, these four concerns must be addressed. The description of the case organization in section 2.1 addresses these concerns as well as providing an overview of their historical development.
Figure 4 - A software product line engineering framework
Background
- 18 -
2.1.2 Agile software development
Agile software development (ASD) is a way of organizing the software development process, emphasizing direct and frequent communication – preferably face-to-face, frequent deliveries of working software increments, short iterations, active customer engagement throughout the whole development life-cycle, and responsiveness to change rather than change prevention. This can be seen to be in contrast with plan-based processes, which emphasize thorough and detailed planning and design upfront followed by consequent plan conformance. Over the past ten years or so, agile methods have gained lively interest and great popularity as they promise to avoid schedule overruns, development of wrong features, excessive overhead, heavy documentation, excessive formalism, costly re-planning, and extensive management. A relatively large number of (more or less) empirical studies on the use and effects of agile methods were reported in the past years, yet the sum of these studies provide little qualified knowledge about agile software development (Dybå and Dingsøyr, 2008). The most thoroughly investigated agile techniques or practices so far are pair programming (Arisholm, Gallis et al., 2007), test-driven development (Erdogmus and Morisio, 2005; Müller and Hagner, 2002) and the customer on-site practice (part of XP) (Martin, 2009). Other, more complex techniques and aspects of ASD are still not sufficiently investigated and understood.
There exists a wealth of introductions to the basic concepts of agile software development, one of the more fundamental and most referred to is the so called agile manifesto (http://www.agilemanifesto.org) which defines four fundamental values and twelve principles. Several methods based on these ideas are in use, all sharing the common set of values and principles. The best known and the most used agile methods are Extreme Programming (XP) (Beck, 1999) and Scrum (Schwaber, 2001). The ideas behind agile software development are not new (Merisalo-Rantanen H., 2005), as they were clearly inspired by agile and lean manufacturing which has been in use in many types of industries for decades. The radical innovations in the Japanese post-war industry is probably the best known example (Takeuchi and Nonaka, 1986). Yet, some important changes need to be made to make these ideas fit software development, which unlike other industries does not include the manufacturing and handling of physical goods (Poppendieck and Poppendieck, 2003). The most fundamental principle of lean development is that of waste reduction, i.e. all work and its products not directly contributing to the development of software should be considered as waste and thus avoided or minimized. Since the first book on Extreme Programming was published in 1999 (Beck), the interest and industrial use grew surprisingly fast (Abrahamsson, Salo et al., 2002). The enormous interest seen in industry stems mostly from the grass-root, that is the developers, and can be explained by the simple and human-centric values inherent to agile methods, which may be appealing to practitioners but perhaps
Background
- 19 -
threatening to management du to the de-emphasizing of traditional command and control mechanisms. The basic principles are easy to grasp, and seem to address the most fundamental problems concerning developers. However, among this interest and willingness to radically change the development process, several critical voices have emerged and many reports indicate that it is not straight-forward, and in most cases it is an act of balancing agility and discipline (Boehm and Turner, 2004).
Available agile methods such as XP, Scrum and others vary in detail and focus level. Scrum for example, can be seen as a management framework whilst XP is a collection of practices and techniques. It is quite common to combine elements from various agile methods (Kniberg, 2007) leading to a well of variants. However, in principle all agile projects share the same structure, summarized in Figure 5.
Figure 5 - Agile software development
An agile project typically has a relatively short start-up phase without extensive
engineering of requirements specifications, domain models, architecture definitions etc. The major part of the project consists of a series of iterations, each lasting a few weeks normally. Each iteration starts out by evaluating – in collaboration with the customer side – the outcome of the previous iterations to verify whether the resulting product increment meets the requirements or not. Then, requirements for the next iteration are prioritized. Requirements are by principle expressed as simple and short stories. The team of developers defines estimates per requirement or feature and the customer representative does the prioritization. This forms an iteration backlog of stories to implement. Then, the development develops the next product increment, which is released – preferably as working software – to the customer representative for demonstration and evaluation. In this way, the whole development project consists of a series of iterations, each resulting in increments that in sum forms the resulting product by the end of the project (when time or money runs out).
Background
- 20 -
2.1.3 Concepts combined: Agile software product line engineering
SPLE methods aim for large-scale development of product lines that serve large and diverse markets over long periods of time, whereas ASD methods address small-scale development from scratch for a single, well-defined customer. These two approaches have different home grounds. Typically when a new paradigm or field of thought is introduced and especially when it promises commercial success, researchers and industry respond by experimenting with the new ideas. Such experimentation results in these new ideas being developed in new directions. One such recent and promising development is the potential combination of SPLE and ASD (Tian and Cooper, 2006). Although relatively new, there are already a few empirical studies (Babar, Ihme et al., 2009; Carbon, Lindvall et al., 2006; Ghanam and Maurer, 2009; Ghanam and Maurer, 2010; Hanssen and Fægri, 2008; Mohan, Ramesh et al., 2010; Noor, Rabiser et al., 2008).
Carbon et al. (2006) did a simple experiment, where students were given development tasks following an agile SPLE approach based on Fraunhofers PuLSE-I framework (Bayer, Gacek et al., 2000), being extended with the practice of incremental design (which is fundamental in all agile methods), and one other practice that is specific to XP called the “planning game” (a planning meeting held in each iteration). Results showed that practices such as the planning game, for instance, includes a great potential to increase the agility of a product line organization, but also other practices like continuous integration or automated regression testing can contribute.
Noor et al. (2008) performed a small-scale industrial case study. They found that it is feasible to combine ASD and SPLE and that it gave a collaborative process relying mostly on face-to-face conversation as a means of communication, but that it requires involvement of both business and technical people to work. The authors derived three success factors from their case study. (i) It is important that participants are familiar with fundamental concepts of product line planning to allow a smooth start. (ii) Preparation is required on part of the process facilitators but also on part of the industry partner. (iii) The willingness to collaborate in a team is also essential.
Babar et al. (2009) report on a case study of a Finnish software product line organization that has been using XP and Scrum for years. Two important lessons can be learned from this study. Firstly, product development projects need documented background information about the system architecture of the product line and a project team that has tacit knowledge pertaining to and experience of constructing the system architecture. Secondly, considerable proactive exploratory work has to be done on the agile development projects before development begins. Related to these findings, the authors also observed that the studied company introduced two new roles to communicate architectural decisions and information. Hanssen at al. (2010) also
Background
- 21 -
observed a similar development where a software architecture team was established to serve the agile product line organization.
Ghanam and Maurer (2009),(2010) have, through two case studies found that besides being practically feasible, the combination of some XP practices and SPLE reduced rework and the cost of producing customized solutions and that it was feasible to target customers with diverse needs without having to disturb the agility of their practices. The proposed approach was found to have a potential to substantially reduce the adoption barriers of a product line practice through its incremental and non-burdening nature. The agile SPLE approach that was tested gave customer involvement a special treatment by enabling customers to pick from variants and contribute to the variability model when available variants are not satisfactory.
In a recent study, Mohan et al. (2010) applied the theory of complex adaptive systems (CAS) to define a set of practices to balance the long-term strategic objectives of SPLE and the short-term tactical objectives of ASD. Together, these practices aim to prepare the product line organization to be flexible and able to handle changing environmental conditions.
Finally, in this overview of empirical studies on the combination of SPLE and ASD we see that our own studies (Fægri and Hanssen, 2007; Hanssen and Fægri, 2006; Hanssen and Fægri, 2008) support and complement the research being discussed here. We have also, as others, concluded that it is practically feasible to implement such a combination of processes, which is supporting strategic and tactical objectives, and that an operational (day-to-day) process constitutes the experience-bearing link from the tactical to the strategic process.
To summarize, the collective impression from these relatively few existing empirical studies of the combination of ASD and SPLE is mainly that these two approaches may – in practical terms – very well be combined. On the one hand, some of the central agile practices may increase flexibility and customer collaboration. On the other hand, the concepts of SPLE are needed in order to manage the diversity of products, the larger customer base, and the long-term perspective, that are characteristics of managing and developing a product line over time.
Building on this overview of research, we take a step back and compare the key agile practices with SPLE as described in various source (Clements and Northrop, 2002; Linden, Schmid et al., 2007; Pohl, Böckle et al., 2005).
Background
- 22 -
Table 2 - ASD and SPLE compared Agile principles (http://www.agilemanifesto.org/principles.html)
Corresponding SPLE practices
1) Priority on early and continuous delivery of valuable software.
Dependent upon SPLE approach. A reactive model supports early delivery of valuable software (no particular bias with respect to continuous delivery).
2) Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.
Foreseen changes covered by platform and variability models are cheap. Other changes become increasingly expensive if they affect domain-‐engineering artifacts, which may be components of multiple products, and thus increase the overall cost of maintenance.
3) Deliver working software frequently, from a couple of weeks to a couple of months, with a preference for the shorter timescale.
An SPLE approach does not reject or inhibit this principle. Product delivery is, nevertheless, somewhat slowed down compared to agile development, due to the overhead of main-‐taining integrity with the product platform.
4) Business people and developers must work together daily throughout the project.
An SPLE approach neither rejects nor inhibits this practice.
5) Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
At the product level, an SPLE approach neither rejects nor inhibits this practice. At the platform level, however, more formalism is required.
6) The most efficient and effective method of conveying information to and within a development team is face-‐to-‐face conversation.
SPLE, due to increased technical and/or organizational complexity, introduces the need for more explicit, formal, and disciplined communication.
7) Working software is the primary measure of progress.
SPLE incorporates significant value in the platform that is more difficult to measure. However, for product development, the SPLE approach neither rejects nor inhibits this principle.
8) Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
SPLE promotes sustainable development at a larger scale.
9) Continuous attention to technical excellence and good design enhances agility.
SPLE platforms embody technical excellence and design to support rapid product derivation.
10) Simplicity – the art of maximizing the amount of work not done – is essential.
The main motivation for SPLE is the same – but by exploiting reuse to eliminate work. In a reactive SPLE approach, the risk of unnecessary work is reduced.
Background
- 23 -
Agile principles (http://www.agilemanifesto.org/principles.html)
Corresponding SPLE practices
11) The best architectures, requirements, and designs emerge from self-‐organizing teams.
The assumption of SPLE is that the investment in requirements, architectures, and designs can be reused successfully across multiple products.
12) At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
An SPLE approach neither rejects nor inhibits this practice. In fact, due to investment in the platform, most SPLE organizations promote experience gathering and learning.
This comparison shows that ASD and SPLE have few principal conflicts according
to our interpretation of these software engineering paradigms. They are virtually independent of each other. On most counts, they possess complimentary properties (Carbon, Lindvall et al., 2006; Tian and Cooper, 2006). Whereas SPLE attempts to introduce large-scale cross-product economic benefits through reuse, ASD focuses on delivering single products with the best possible quality for the customer. Thus, there is, prima facie, no reason why these two approaches should not be combined. On the contrary, there is a clear motivation to combine them.
2.1.4 Software Ecosystems
Software ecosystems is a very recent term, referring to a networked community of organizations, which base their relations to each other on a common interest in a central software technology. Some other definitions of this emerging concept have been proposed, for example by Jansen et al. (2009): “a set of businesses functioning as a unit and interacting with a shared market for software and services, together with the relationships among them” (p. 2). Another definition by Bosch (2009), focusing more on the common interest in the software and its use, is: “the set of software solutions that enable, support and automate the activities and transactions by the actors in the associated social or business ecosystem and the organizations that provide these solutions” (p. 2).
Well known examples of communities that may be seen as software ecosystems are Apples iPhone/Appstore platform and the open-source development environment Eclipse. The first is an example of a partially closed and controlled ecosystem, and the latter is an example of an open ecosystem allowing more flexibility in use and development. This simply illustrates that the ecosystem concept may refer to a wide range of configurations. Yet, they all involve two fundamental concepts: 1) a network of organizations or actors, and 2) a common interest in the development and use of a central software technology. These organizations may have different relations to the central software technology, and for this reason, different roles in the ecosystem. In our definition of the concept, there are at least three key role types. First, one organization
Background
- 24 -
(or a small group) acts as the keystone organization, and is in some way leading the development of the central software technology. The second key organizational role is the end-users of the central technology, who need it as a tool to carry out their business, whatever that might be. The third key role is third party organizations that use the central technology as a platform for producing related solutions or services. In addition to these key roles, various other related roles might be part of the ecosystem, for example standardization organizations, resellers, and operators (Jansen, Brinkkemper et al., 2009).
A fundamental property of the central software technology is that it is extensible beyond the keystone organization (Alspaugh, Hazeline et al., 2009). This extensibility can be achieved in various ways, for example by providing an application programming interface (API) or a software development kit (SDK), by supporting exchange of open data formats, or by offering parts of the technology as open source. Opening up the technology in these, and potentially other ways, enables external organizations to use the central software technology as a platform where existing services or data can be used and extended. Bosch (2009) (p. 112) proposed a Software Ecosystem Taxonomy that identifies nine potential classes of the central software technology, according to classification within two dimensions. The first one is the category dimension, which ranges from operating systems to applications, and to end-user programming. The second one is the platform dimension, ranging from desktop to web, and to mobile. The case discussed in this thesis is an application-web category. Accordingly, implications for practice and theory that are developed later in the thesis relate to this category.
Table 3 - Software Ecosystem Taxonomy
A keystone organization has a special position in an ecosystem as it controls, strictly
or loosely, the evolution of the central software technology. This may include various
Background
- 25 -
responsibilities, for example certain software product development activities such as strategic planning, R&D, and operational support. These responsibilities come in addition to activities specific to ecosystems, such as enabling efficient external extensibility, provision of insight into planning and development, and supporting ecosystem partners in various other ways.
One potential benefit of being a member of a software ecosystem is the opportunity to exploit open innovation (Chesbrough, 2006), an approach derived from open source software (OSS) processes where actors openly collaborate to achieve local and global benefits. External actors and the effort they put into the ecosystem may result in innovations being beneficial not only to themselves (and their customers) but also to the keystone organization, as this may be a very efficient way of extending and improving the central software technology as well as increasing the number of users. Closer relationships between the keystone organization and the other actors may drive both an outside-in process as well as an inside-out process, as described by Enkel et al. (Enkel, Gassman et al., 2009). Also, the proximity between the organizations in an ecosystem may enable and improve active engagement of various stakeholders in the development of the central software technology (Hanssen and Fægri, 2008).
When explaining the concept of software ecosystems it is also necessary to address how software ecosystems relate to the development of open source software (Fitzgerald, 2006). There are clear similarities between these two concepts, but also several differences, which justify the definition of software ecosystems as a unique concept. The main difference between these two relates to the underlying business model. Brown and Booch (2002) explain the open-source business model as follows:
“The basic premise of an open-source approach is that by “giving away” part of the company’s intellectual property, you receive the benefits of access to a much larger market. These users then become the source of additions and enhancements to the product to increase its value, and become the target for a range of revenue-generating products and services associated with the product.”
In a (closed) software ecosystem – like the case being studied in this thesis – the intellectual property (the code) is not shared in any way. This leads to some fundamental differences, listed in Table 4. (When using the term ‘ecosystem’ in this thesis, we implicitly refer to the development of closed-source systems.)
Background
- 26 -
Table 4 - OSS and closed-source ecosystems - similarities and differences A community of organizations and actors. A shared interest in the development, evolution, and use of a software product. Independent actors collaborate and contribute to development. Open innovation. Si
mila
ritie
s
New business models as compared to traditional licensed software. OSS (Close source) ecosystems Open source code. Closed source code. Ownership is shared. Ownership and control lies with the
keystone organization. Free use (with options for paying for specializations and related services)
Pay for use. Diff
eren
ces
Extensibility through open source code. Extensibility through controlled interfaces. The ultimate objective for investing in and working towards an ecosystem is that all
members will derive more benefits from participating, as compared to the more traditional approach to software product development with segregated roles, a low level of collaboration, and closed processes. A well functioning ecosystem is, in summary, a complex configuration with collaboration across traditionally closed organizational borders (Bosch, 2009). Such multi-organizations are most likely not established as a deliberate planned effort. Rather, they emerge as a result of many congruent factors such as domain and business development, technology development, globalization, new collaborative patterns, and customers becoming more and more accustomed to participating in the shaping of the technology they use.
2.1.5 Organizational change
An important aspect of understanding an organization and its functions is to understand how it changes. Kurt Lewin, one of the original contributors to the study of organizational change even stated: “You cannot understand a system until you try to change it” (Lewin, 1951). In relation to this, he also coined the term and the concept of action research (Lewin, 1946), which is the repeated process of evaluating a situation, planning, and executing a change, evaluating the effects, and implementing a permanent change. A key feature of this model is that change is not done in a laboratory-like setting but in a real context, for instance in industry and in production. This principle is also fundamental to Lewin’s original model of organizational change. According to this model, change is performed through three steps: unfreeze – change – freeze. This concept has been formative to the wide understanding of how organizations change. Unfreeze refers to an organization that exits a stable phase, an equilibrium, so the established structures and processes become open to change. Change refers to the actual change of the organization, and freeze refers to the establishment of the new
Background
- 27 -
organization or processes as the new standard, and the new equilibrium. This model represents a view of organizational change as planned or episodic (Weick and Quinn, 1999). A premise for such an approach is stability (in the business domain, technology, market etc.) – which again is a premise for planning the change and foreseeing its effects.
Stability is, however, not the case for many organizations (Emery and Trist, 1965), in particular software organizations (Baskerville, Ramesh et al., 2003), which work with rapidly developing technologies and with business domains that change in line with technology. Consequently, developing plans of change up-front, based on the knowledge of a changing and turbulent environment may not be a good idea as such plans may be obsolete, wrong, or even harmful at the time of their execution. Orlikowski and Hofman (1997) address this problem and suggest a model of improvisational change. Their model contrasts the idea of change as punctuated equilibrium (Weick and Quinn, 1999) with change as an ongoing process and a result of a series of events. They define three types of change: anticipated, emergent, and opportunity-based. Anticipated changes “are planned ahead of time and occur as intended”. Emergent changes are “changes that arise spontaneously out of local innovation, which are not originally anticipated or intended”. Opportunity based changes are “changes that are not anticipated ahead of time but are introduced purposefully and intentionally during the change process in response to an unexpected opportunity, event, or breakdown.” (ibid., p.4). In practice, the total change consists of an arbitrary series of these types of changes, which can be understood as responses to the internal events in the organization as well as the events in the external environment.
Figure 6 - Orlikowski’s and Hofman’s model of improvisational change A change of an organization usually stems from the need to improve this
organization; to improve a process, restructure the organization, reduce schedule, or costs etc. In line with the concept of improvisational change, and as an instrument to manage it, W. E. Deming (2000) formulated principles for transformation and the Plan-
Background
- 28 -
Do-Study-Act improvement cycle. This is well known as the Deming cycle, or as the Shewhart cycle (ibid., p.88) as Deming himself named it (after W. A. Shewhart).
Figure 7 - The Shewhart/Deming improvement cycle
Plan – is to analyze the present situation, to observe or experience a potential for a
change, and to suggest how to perform the change. Do – is to implement the change. Study – is to evaluate the change, to create an understanding of the effect and causality of the change. Act – is to make the change permanent, assuming that the effect of the change creates a better situation than before the change. Over time, these cycles represent a change, and an improvement. Change can be of any of the types defined by Orlikowski and Hofman, meaning that planning can vary from explicit planning to more improvisational responses to emergent and opportunity-based events.
Background
- 29 -
2.2 The case
This section describes the present case organization, their product line and their development processes. We also give a retrospective of the past organizational change that have led to the present state.
2.2.1 Organization
CSoft is a medium-sized Norwegian software company that develops, maintains, and markets a product line under the same name. They serve the high-end segment of the market and have a wide international customer base, mainly consisting of Global 50003 companies. Products are marketed and sold by several resellers throughout the world. Despite facing considerable challenges in the first years, CSoft is now one of the market leaders. The company was established in 1996, and has grown continuously since then with a major increase in size through an acquisition of a former competitor in 2006. Currently, CSoft employs more than 200 people, including over 60 developers.
CSoft has several development locations across Europe and Eurasia, as well as sales and support departments in U.K. and USA. The main office, including the main section of R&D, is located in Norway.
The most relevant (to this study) roles and organizational functions are: The product management team (PMT) is a group of five product managers, each
being responsible of one of the main modules in the product line. The product managers are responsible of developing the product roadmaps, one per module, in collaboration with externals and internals. In the development projects, the product managers are the connection between the development teams, other internal roles and external actors, typically customers which act as stakeholders. Thus, this role resembles the Scrum-master role in Scrum projects, but with additional responsibilities of developing the product strategy as well as supporting development.
The development teams. Each module of the product line is developed and evolved by a fixed team of developers (no job rotation). Each team has a team leader, which is highly experienced and has the most insight into the solution and the business domain.
The Chief Technical Officer (CTO) has a special role as he has worked on the solution from the very start and thus possesses the most insight into the product line. He participates in strategic planning and plays a guru-role in development.
3 http://theglobal5000.com/
Background
- 30 -
The R&D manager plays an important role with respect to the development processes and was the initiator of the change from the plan-driven process to the agile development.
The architecture team consists of four experienced developers and has the responsibility of improving the structure/architecture of the product line and improving the technical infrastructure being used by the R&D organization. This team was formed in 2007 to counteract increasing difficulties with the growing technical complexity of the product line and the development of it.
Professional services provide support, training and adaptation of the solution to make the product work optimally for customers requesting such services. This department knows the clients, the business domain and the market very well and can be considered to be experts on the use of the product line.
Technical account management (TAM) is the customers primary contact in case of support needs or if customers want to provide input, needs or ideas. TAM communicates this type of input to the PSG, which takes it into account when developing product roadmaps.
It is important to realize that the SPLE process that is being used at CSoft is not the
result of an immediate, deliberate adoption and introduction of one of the commonly used frameworks; rather it is the result of years of development and adjustments. This transition will be discussed in more detail later on.
Background
- 31 -
2.2.2 Product line
The CSoft software is a highly modular product line that allows many configurations and variations in use. It contains five main modules that serve various business-related activities. In addition, it offers an Application Programming Interface (API). This is implemented as open standard web services. The composition and use of the modules varies according to customer and case. Some modules can be used in any configuration, while the use of others depends on the situation. CSoft software comes with a set of predefined configurations for the most common usage scenarios, and it provides built-in support for detailed customization to create more variants.
Figure 8 - High-level product line overview
The product line, or family, consists of several modules that together support a value chain of planning, data collection, analysis, and reporting of results. The composition of a specific product based on the product line can be varied in several ways:
Variability in task: Data can be collected in various ways using different modules.
For example, there is one module for collecting data via telephone and one for collecting data via the Internet. All variants of the product use the same core component for storing and analyzing data, and have the same functionality for reporting. It is possible to select dynamically which features to include according to the license, in order to satisfy the greatly varying needs of the customers.
Variability in application domain: The product line is being used in three different but strongly related domains.
Variability in feature richness: The product comes in two main variants, the full professional suite and a simple version. The simple version offers only the basic features, is low in cost, and is not supported.
Background
- 32 -
Variability in operation: The product is a three-tier application with a web-browser front-end only. This allows for a dual operational model. A customer may buy the product entirely as a service provided by CSoft, meaning that the software runs within a remote hosted environment together with all data, and all access is done through an ordinary web browser (software as a service - SAAS). Alternatively the total solution can be hosted locally. This is more suitable for customers who use the product extensively and need to retain maximum operational control. However, the largest clients only use this variant.
A common commercial model for a software product line supplier is to either sell
licenses to use a product (a derivation of the line), or as a commodity for local deployment or, increasingly, as access to the software through a service. CSoft apply a different model. The principle is that cost of use is based on the extent of use; customers pay a certain amount of money per completed transaction, that is, cost is related to the actual generated business value to the customer.
Background
- 33 -
2.2.3 Processes
Taking a process view of CSoft shows that the organization’s development-related activities can be grouped into one of three types of processes: a long-term strategic process, an intermediate-term tactical process, and a short-term operational process. The common denominator of these three processes is their interplay with external actors (customers and third-parties that extend the product line through the API’s).
Figure 9 - An external-actor centric development process
Background
- 34 -
Strategic (long-term, SPLE): Evolution at the SPLE level addresses the need to
implement long-term strategic plans. SPLE supports CSoft’s need to introduce tactical innovation, which is fuelled, for example, by innovations in technology, changes in the business models, or new business opportunities. CSoft is a product platform that contains a set of building blocks that can be tailored to provide different product offerings. This platform can be considered as the vehicle used by the Chief Technical Officer (CTO) and the Product Management Team (PMT) to implement long-term vision. More often than not, these innovations are costly in terms of effort and risk. Thus, they should be planned and introduced with great care, in order to exploit the organization’s development capability. SPLE-style development inherently supports this ambition.
Tactical (medium-term, ASD): Evolution at the project level (note that a release typically includes one project per module) addresses the customer’s detailed requirements and needs. Close cooperation in planning and evaluation of the two-week development iterations ensures that customers’ ideas and experience with the software is translated into what we may call tactical innovation, i.e. reactive innovation that seeks to polish, improve, adjust, or otherwise make small to moderate adjustments to the product. According to the ASD method Evo, Customers’ requirements are defined as product qualities and documented in an impact estimation table (IET). This is a simple list of prioritized (by a customer stakeholder) product qualities of the product under development with an estimate (by the development team) of the anticipated effort needed to implement the quality. An example of such a quality is “Usability.Productivity” for a given task, e.g. “Define a simple report”. This product quality can be defined as number of seconds needed by a novice user to complete the task. The IET also contains the goal value of the quality and the present value. Thus, the quality is improved until it has reached its goal value performance. (See Part II, paper 2, for a more detailed example). The IET correspond to the more known concept of product and sprint backlogs which are important components of the more known and practiced agile method Scrum (Kniberg, 2007).
Background
- 35 -
Each development project follows a common process template of successive fortnightly iterations:
Table 5 - The iteration week plan
One of the first days
Meeting with external stakeholder Iteration postmortem Technical meeting 1 Monday
Development iteration kickoff
2 Tuesday 3 Wednesday 4 Thursday
Wee
k 1
5 Friday 1 Monday 2 Tuesday
Development
Internal demo (prepare for next meeting) 3 Wednesday
Evo iteration planning and review meeting
4 Thursday Fix change requests Retrospective
Wee
k 2
5 Friday Project Status Meeting
Operational (short-term, day-to-day operations): Customers rely on the software to help them implement vital business functions in their own organizations. Thus, it is of great importance that CSoft is able to sustain a good level of customer satisfaction with the software in its day-to-day use. During operation, the products are exposed to more users, and this exposure increases the potential amount of feedback to the innovations implemented at the strategic and tactical levels.
Background
- 36 -
2.2.4 A retrospective of the development of the organization
CSoft was established in 1996 and has grown continuously since then, with more than 200 employees in 2010. Their development went through three phases and has now entered a fourth. This section presents a summary of these phases of development and indicates some important milestones in the development of the organization. The timeline in Figure 10 shows main events in the development of the organization and the growth in staff. We have identified four phases of the organizational development.
Figure 10 -‐ Timeline of the development of the case organization
1996-1999: “Creative chaos” The company initially grew out of a small business providing manual services to
very few customers. A simple homemade supportive software tool developed into a solution that could be sold as a stand-alone software product. The main focus of the company changed, and the development of this product became the main objective. In the beginning, in 1996, there were only a few employees providing this product to a handful of customers. The process can best be described as ad-hoc as the main driver was feedback from customers obtained in almost daily interactions. A customer request was literally routed directly to the developers. This start-up phase was a creative chaos – that is, it had nearly no plans and no control but was undoubtedly extremely creative and productive. The product grew rapidly, not only in terms of features and functionality but also in terms of defects and complexity. Work became stressful, with little control and a lot of overtime work.
Background
- 37 -
1999-2003: Waterfall and software product line engineering As the number of customers grew, the organization formalized the development
process according to the principles of the waterfall model, famously described by W. W. Royce (1970). This somewhat disreputable approach (Boehm, 2006; Takeuchi and Nonaka, 1986) to organizing software development emphasizes upfront detailed planning of requirements, design, and development. The development is divided into consecutive phases, where requirements are developed into a design, and the design is developed into a software system, which is tested close to deployment.
Prior to this, the R&D department was extended with a QA-manager and a few other supportive roles. The waterfall-like process was also documented as an electronic process guide (EPG), being available to all developers as a tool to spread knowledge and control process compliance (Moe, Dingsøyr et al., 2002). This structured approach established a certain level of control and helped the organization in the continuing development of their product, which grew alongside the customer base. After a few years, several problems arose clearly related to the waterfall approach (Johansen, 2005); testing and verification was postponed until late stages leading to late identification of problems which in turn caused much rework. Also, requirements were nearly solely focused on functionality, leaving out the quality perspective. The product expanded and eventually became a product line, being capable of serving various usage scenarios. To manage this increasing complexity a Product Management Team (PMT) was formed – a group of experienced employees with other responsibilities that were supposed to spend part of their time in strategic product planning. In addition various specialized functions, beyond software development, were introduced such as Technical Account Managers (TAM), the operations department, training services etc.
2004 – approx. 2008: Agile product line engineering Due to a critical decline in process performance, management of R&D was looking
for a way to improve the situation. At a software engineering conference, a few representatives from the company learned about the evolutionary development and the Evo method (Gilb, 2005). As it seemed to address some of their concerns they initiated a three-month test period to try out this radically different development approach in a real release project. Instead of an extensively prepared process adoption, they started out with a few principles, focusing on requirements management, where functional requirements were replaced by explicit expression and evaluation of product qualities, preferably stated by customers involved in the development process (Hanssen and Fægri, 2006). Early experience showed that the amount of issues near release was reduced, and that the delivered product matched customer expectations better than before. After this initial process trial, Evo was adopted on a permanent basis (Fægri and Hanssen, 2007; Hanssen and Fægri, 2008). Alongside the growth in organization and
Background
- 38 -
product line the PMT group was re-established as a full-time Product Strategy Group (PSG) with a dedicated management by a Chief Strategy Officer (CSO). Another supportive service, the architecture team, was established, originally with three full time members. Their task was to handle the excessively high level of system complexity, and to support R&D in architecture-related issues. In 2006 CSoft acquired a former competitor and boosted the number of employees up to more than 200. Adding new offices, for both R&D and marketing, was a considerable challenge. Through extensive internal training in the following year, the new organization was using Evo almost exclusively as the development process, and was following the release cycles.
Approx. 2008 →: Towards a software ecosystem Since the first use of Evo, the actual use of it has further matured and was adapted to
the organization’s needs. During the past few years an external community has emerged, using the CSoft product line as a platform for value-adding services and products. CSoft supports the operation of this community through the offering of a set of API’s for extending the product line. Also, during the most recent years, collaboration with external stakeholders, first of all customers, has been professionalized, and the first product conference was held in 2008 with extensive participation from employees, customers, prospects, and external partners. In sum – the organization is opening up and is increasing collaboration with external actors. We see this as a transition towards a software ecosystem.
2.3 A theoretical framework: organizational ecology
In search for a foundation to understand and communicate our findings from the study of CSoft we look to the pioneering work done by Eric Trist, Fred Emery et al. from the Tavistock Institute of Human Relations4. Their studies of the development and behavior of work groups and organizations have lead to theories on socio-psychological, socio-technical and socio-ecological theories whereof we look into the two latter ones.
Socio-technical theory is a view of the organization as the sum of and interplay between a social system and a technical system. That is 1) the people, their relations, their knowledge, and how they work together as a whole, and 2) the tools and techniques being used to perform the work. The fundamental principle of socio-technical theory is the interdependence between these two systems, meaning that to improve the performance of the organization (productivity, quality of work etc.) both subsystems have to be considered at the same time. Changing one affects the other. This
4 http://www.moderntimesworkplace.com/archives/archives.html
Background
- 39 -
principle was first drafted by Trist and Bamforth in 1951, based on their studies of changes of work processes in coalmines (Trist and Bamforth, 1951).
Through studies that followed, the theory has been developed to consider how organizations relate to their external environment (Emery and Trist, 1965). This implies that an organization has to be understood as both 1) the internal interplay between a social subsystem and a technical subsystem (the socio-technical system), and 2) the interplay between the organization and its external environment.
Figure 11 - The socio-technical system and the organizational ecosystem Emery and Trist (ibid.) developed a simple classification system of four types of
organizational environments – forming a series, in which the degree of causal texturing is increased. Thus, understanding and ordering the types of environments is useful in understanding socio-technical systems beyond the limits of a single organization.
1) The first and simplest is the placid randomized environment where “goods” and “bads” are unchanging, and are randomly distributed in the environment (ibid., p.7). The optimal strategy is to do one’s best on a purely local basis – there is no difference between strategy (planning) and tactics (execution).
2) The second type is the placid clustered environment where “goods” and “bads” are not randomly distributed but band together in certain ways. Strategy is different from tactics, and survival becomes critically linked with what an organization knows about its environment. Organizations in this environment tend to become hierarchical, with a tendency towards centralized control and coordination (ibid., p.8).
3) The third type, the disturbed reactive environment, is an environment where there is more than one organization of the same kind. The existence of a number of similar organizations becomes the dominant characteristic of the environmental field. These organizations compete, and their tactics, operations, and strategies are clearly distinguished. The flexibility encourages a certain decentralization, and it also puts a premium on quality and speed of decisions at various peripheral points (ibid., p.9).
Background
- 40 -
4) The fourth, and the most recent type is the turbulent fields. This type implies that significant variances arise from the field itself, not simply from the interaction of the component organizations. Three trends contribute to the emergence of these dynamic field forces: i) the growth to meet type-three conditions, ii) the deepening interdependence between the economic and other facets of the society, and iii) the increasing reliance on research and development to achieve the capacity to meet competitive challenge. A change gradient is continuously present in the field (ibid., p.10).
Figure 12 - Types of organizational environments This interplay, inherent in turbulent organizational environments, has been further
studied, leading to the development of the concept of organizational ecology (Trist, 1977). It is particularly relevant to organizations operating in complex and unstable domains. Viewing the organizational environment as an ecosystem means that it is considered to be an open system as opposed to a closed one. Organizational borders are permeable, and organizations relate dynamically to other organizations in the same field.
Developing the ecology concept further, Trist (ibid.) describes three classes of organizations within the turbulent environment. In a Class 1 system, member organizations are linked to a key organization among them. The key organization acts as a central referent organization, doing so even though many of the member organizations are only partially under its control, or linked to it only through interface relations. Interface relations are as basic to systems of organizational ecology as superior-subordinate relations are to bureaucratic organizations. Interface relations require negotiation as distinct from compliance – a basic difference between the two types of systems. In a Class 2 system, the referent organization is of a different kind. It is a new organization brought into being and controlled by the member organizations, rather than being one of the key constituents. A Class 3 system has no referent organization at all.
Background
- 41 -
Technocratic bureaucracies have been the natural organizational form for disturbed-reactive environments (up to the 1960’s or so), and this is a form that has been applied to many software engineering organizations. However, this type of system fails to adapt to conditions of persistent and pervasive environmental turbulence, mostly because it is constructed and optimized to work well in stable environments. This leads to the emergence of the new ecologically oriented systems, which show clear differences in that they promote self-regulation (as opposed to centralized control), and that they have a networked character (as opposed to segregated organizations). According to Trist (ibid., p.172), such systems, lacking formal structure, exist through the use of technology. Further, they also need shared values. Trist used the example of the 60/70’s youth-culture that had a set of distinct (political) values. A more appropriate example from a business perspective is a shared value in growth and profit. An even more low-level example may be the software product as a shared value.
Unlike the micro-level systems (the single organization) and the macro-level systems (society), the intermediate level systems (organizational ecosystems) are hard to observe, understand, and describe due to their weak structuring. They are also the most recent type, so there is less experience with them. This relates especially to software engineering ecosystems, which is a new but rapidly advancing concept (Bosch, 2009; Messerschmitt and Szyperski, 2003). This approach is driven by the Internet as a rich and fast collaborative platform (the technology), and a common interest in the product line (the shared value).
Research approach
- 42 -
3 Research Approach This section will explain the chosen research approach. We begin with a general
discussion of the variety of approaches to research in software engineering. This overview forms a background, and it is used to argue for the approaches we have chosen and applied, and it justifies why we disregarded others. We then continue by explaining the components of the applied research design of this thesis: how we selected the case, how data were collected, and how data were analyzed. Finally we discuss issues related to the validity of the results.
3.1 Studying software engineering
Software engineering (Vliet, 2002) is the discipline of specifying, designing, developing, deploying, and maintaining software systems. This includes many types of interdependent tasks such as management, budgeting, requirements management, application design, architectural design, coding, testing, maintenance, deployment and several other high- and low-level tasks. Most commonly, software in an industrial setting is being developed within the project format, which includes one or more suppliers and one or more contractors. However there are many variations, from developing tailor-made solutions for a well-defined customer or group of customers, to the production of standard commercial products being offered to a large and diverse market. Regardless of the variant, the basic task is to capture and understand requirements, needs, and ideas – tacit and explicit – to engineer a solution that meets requirements, within limits of time and cost. This can be a challenge as requirements are evolving or may be unclear, thus requirements engineering is an important practice within software engineering. Accordingly, testing is also a challenge; the engineered solution must be validated to ensure that it meets the requirements.
Software engineering is both a technical process and a human-intensive process. To understand this diverse practice we thus need to understand the interplay between technology and humans, who are typically organized in teams.
The term empirical software engineering (Basili, 1996) describes the discipline of applying empirical methods to measure, evaluate, and thus understand the practice of software engineering in general, and tools, techniques, and practices in detail. Such an understanding is needed by the industry to select an appropriate approach. The term empirical study (Wohlin, Runeson et al., 2000) refers to the practice of observing an object of study to collect experience-based data, and describing the object through its features, effects, and outcomes. Data are analyzed in order to understand how the object works and evolves within a given context. Typically within software engineering, the studied objects are software development methods and guidelines in general, as well as
Research approach
- 43 -
design techniques, tools, programming languages etc. Data of interest are mostly in some way connected to costs and gains, for example the level of code quality related to a given testing technique. In its most general form, empiricism is the process of learning from experience, thus being able either to validate theory or build new empirically qualified theory. This process should be transparent, explaining how data were collected, how they were analyzed, and how they relate to the existing research. Within the software engineering field, the terminology is somewhat diffuse: besides the term empirical software engineering we can find other similar terms such as evidence-based software engineering (Kitchenham, Dybå et al., 2004), experimental software engineering (Basili, 1996), and other variants. However, they all refer to the basic concept of systematizing experience in order to generalize new knowledge – as in most other sciences.
To place the practice of empirical software engineering in a broader context, Figure 13 shows how it relates to the software engineering discipline, and how it forms connections between research, industry, and users of technology:
Figure 13 - Empirical software engineering
This value chain shows the role of the software engineering industry and its relations,
both to their customers and users, and to the research community. The value chain here is two ways; from left to the right it shows how knowledge, based on experience (empiricism), is applied in the construction and application of software systems. Going the other way, it shows how experience from the use of software systems and development processes is captured and externalized. In this way the figure shows a learning loop where empirical software engineering is the vehicle.
Research approach
- 44 -
There exist several strategies and tools to plan, execute, and report empirical studies. These are general approaches applicable to many domains and disciplines, including software engineering. Software engineering is in many respects an immature engineering discipline, and so is empirical software engineering. For example, looking to medicine or social sciences, we find a much more mature and widespread use of empirical methods that we can learn from (Kitchenham, Dybå et al., 2004). However, there are differences between fields, and software engineering has its peculiarities that influence how we can utilize these methods. Further on, we will give a brief overview of a set of empirical approaches with emphasis on the field of software engineering.
Wohlin et al. (2000) refers to two main types of empirical studies: qualitative and
quantitative, a very high-level taxonomy of research approaches. Qualitative studies refer to studies where data is collected from humans (either
directly or through their actions) using techniques such as interviews, surveys, and observations etc. This type of data is to some extent subjective and can be biased, since the subject’s or the observer’s viewpoints can be affected by belief and personal standards. Qualitative research is mainly concerned with phenomena not possible to be expressed or measured quantitatively. Raw data is often represented in the form of words (written or spoken) or as pictures or illustrations (Seaman, 1999), or through instruments like questionnaires using psychometric scales (Cooper and Schindler, 2006). The type of data being collected naturally influences the type of analysis that can be performed. Typically, qualitative data cannot be analyzed mechanistically but need to be interpreted in some way, thus risking a bias. Qualitative research methods have been developed mainly by researchers in the social disciplines focusing on humans, their relations, and their actions (Seaman, 1999).
Quantitative research refers to studies that quantify properties of the phenomenon being studied, thus representing data as numbers which can be of nominal, ordinal, interval, or ratio types (Cooper and Schindler, 2006) (p. 312). This approach comes mainly from the natural sciences where data is available through measurement either by direct empirical observations or by the use of instruments and calculations. Having data in a numerical form makes it possible to do sample-based statistical calculations to find values describing populations and to compare results mathematically (less, greater or equal). Doing quantitative studies also makes it possible to demonstrate the quality of the analysis through measures of significance and validity, which are important factors when findings are to be applied in practice. However, there are many pitfalls that may introduce errors during data collection, analysis, and reporting, for instance poor construct validity.
Research approach
- 45 -
Wohlin et al. define three high-level types of empirical strategies: surveys, case studies, and experiments. Variations of these and more types can be added, as there is no commonly accepted taxonomy of empirical strategies within the field of software engineering as yet.
A survey is a retrospective type of study where data, quantitative or qualitative or both, are collected after an event, thus not being able to control the event, for example a development project. Questionnaires and interviews are commonly used techniques for data collection. Data can be interpreted to reach either descriptive or explanatory conclusions.
A case study is basically a study of a phenomenon over time, which makes it an observational type of study. Both quantitative and qualitative data may be collected, either alone or in combination. A combination, often referred to as triangulation, is often productive as one type of data may support or complement another type (Davison, Martinsons et al., 2004). The term case study is not well defined, and it may take many forms with respect to intervention, data collection techniques, and analysis (Yin and Campbell, 2002), however it can be defined as simple as ‘the study of a restricted case’.
Experimentation implies a high level of control (as opposed to a survey). An experiment can be done in a laboratory-like setting or in a real-life (field) setting. However, the most prominent feature is the ability to control the variables being studied. Control, in this context, means that the values of the variables are known through some kind of manipulation, and can be measured with a certain known level of accuracy. There are three types of variables:
1. Dependent variables, also called response variables, which represent the effect or outcome in an experiment. An example of a dependent variable may be the productivity or quality of a software development process, which again can be operationalized in various ways.
2. Independent variables are controlled or manipulated in the experiment setting. Examples are the type of design technique or development tool being used by different experiment groups. An experiment can be planned to evaluate the co-variance between several variables. The main goal of the experiment is to study how the independent variables influence the dependent variables or how changes in variables correlate.
3. In addition there may be context variables, which are also believed to have an influence on the dependent variables. It is important to know these variables and their (interfering) effect on the dependent variables to be able to better evaluate the direct relationship between the independent and the dependent variables.
Control may ensure accuracy and data of high quality, but it also imposes costs and a
risk of bias. As stated by several authors, e.g. (Fenton, 1994; Kitchenham, Pfleeger et
Research approach
- 46 -
al., 2002; Tichy, 1998), both the level and the quality of experimentation is poor in the field of software engineering.
Wohlin et al. present one interpretation of the term controlled experimentation, but
there exist others. Zelkowitz and Wallace (1998) give a more detailed definition and explanation of the term as they suggest a taxonomy for software engineering experimentation. First, they define four categories of experimentation: the scientific method (a theory explaining a phenomenon is validated through testing of a hypothesis variation), the engineering method (a solution to a hypothesis is developed, tested, and refined until no further improvement is needed), the empirical method (a hypothesis is validated by the use of statistical methods), and the analytical method (a formal theory is developed and results are compared with empirical observations). Through a review of multiple examples of technology validation approaches, Zelkowitz and Wallace identified 12 different experimental approaches grouped into three categories: observational, historical, and controlled. In the observational category they define project monitoring, case study, assertion, and field study. In the historical category they define literature search, legacy, lessons learned, and statistical analysis. In the third category, controlled, they define replicated experiments, synthetic environment experiments, dynamic analysis, and simulation. Some of these approaches resemble the description by Wohlin et al. Some terms are used in a similar ways, others differ.
Some other descriptions give other variations, different focus, and levels of detail, for example in the paper Preliminary guidelines for empirical research in software engineering by Kitchenham et al. (2002). The title is unclear, as the paper focuses mostly on experimental design, using terms slightly different from the other referenced sources. This simply illustrates the diversity of definitions and descriptions for methods of empirical research within software engineering. This diversity is believed to contribute to a certain level of confusion in the various study reports that mix terms as well as in the actual use of methods, and as a consequence, an unsatisfactory level of quality of empirical studies (Dybå and Dingsøyr, 2008).
In addition to the empirical strategies discussed here, some other approaches also deserve to be described. First, action research may be viewed either as an empirical strategy or a research method. However, it is mostly a framework that defines how the researcher(s) should cooperate and intervene with client(s) to plan, carry out, and evaluate a study. Davison et al. (2004) defines the concept of canonical action research through five principles. These are diagnosing (what is the problem?), action planning (what can we do about it?), intervention or action taking (do it), evaluation (assess the effect of the change with respect to the defined problem), and finally reflection (learn from the experience, both specifically and generally).
Research approach
- 47 -
Another important meta-strategy related to empirical research is systematic literature reviews as described by Kitchenham (Kitchenham, Pfleeger et al., 2002). This is an approach to systematically and rigorously search, evaluate, and systematize published research results in a given field or for a given topic or problem. Systematic reviews are not an empirical strategy in the direct sense. However, it is a useful approach to interpret and understand published results from existing empirical studies to build a wider and composite view and understanding, or at least an updated overview of a research field.
The strategies briefly presented and discussed here do not give the whole picture;
there are variants and combinations of these, as well as other approaches that may have been missed out in this overview.
Common to most strategies and guidelines for empirical research, regardless of flavor, is the definition of the overall process describing a set of phases. These do vary, however in general they address the following five phases:
1. Problem definition - that is, identifying and justifying a problem or question, sometimes related to a need for improvement. This may also include planning of a potential action to correct the problem or improve some performance. This phase is crucial to identify the most important problem to be addressed with respect to criticality and significance.
2. Planning of the study may include the definition of one or more hypotheses to be tested or some other type of activities to evaluate a case (producing experience) to gain knowledge about the defined problem. For historical types of studies such planning may be irrelevant, but in any case requiring some kind of intervention or data collection during an event, this must be planned.
3. Operation or execution is the implementation of the plan. This may cover manipulation of a subject, e.g. to carry out a treatment, or simply observations to collect data without any intervention.
4. Analysis is the phase where data is interpreted either to test a defined hypothesis or to build, edit or verify a theory or hypothesis.
5. Presentation and dissemination is the final phase where results are made available externally, thus constituting a potential addition to the existing knowledge base on the topic.
Research approach
- 48 -
3.2 The applied research process of the thesis
As we have shown, the variety in study approaches and techniques is great, and it depends on the type of research question and the context of the study. The particular study being reported in this thesis can be described using the following attributes:
(1) A case study of a single industry organization. (2) An interpretative field study. (3) A longitudinal study. Given these premises for the study of CSoft, we have applied the following principal
guidelines for planning and conducting our studies: (1) Case study: All four studies underlying this thesis have been exploratory and
qualitative, and the focus and the defined research goals were rather widely defined. This made the case study approach the preferred overall research strategy. In contrast, controlled experimentation based on a few isolated and controlled variables, and their hypothesized relationships was not found to be suitable for understanding the selected case organization and their processes, especially as we were interested in following a development over time and its unforeseen outcomes. Further more, the state of research on agile methods, which lies at the basis of this study, is in a rather nascent state. Despite the widespread adoption of agile processes and techniques in the software industry - and the correspondingly large interest in academia - there are relatively few empirical studies of good quality (Dybå and Dingsøyr, 2008). There is nearly no theorizing in the present studies. This calls for more qualitative and explorative studies (ibid., p.852).
According to Yin (2002), case studies are the preferred research strategy “…when a «how» or «why» question is being asked about a contemporary set of events over which the investigator has little or no control.” (p. 9). This is a very adequate description of the context of our studies of CSoft.
Research approach
- 49 -
(2) Interpretive field study: More specifically, our study of CSoft can be classified as an interpretive field study according to a definition given by Klein and Myers (Klein and Myers, 1999). This definition includes seven principles for conducting interpretive field research that suits the aim of our study:
Table 6 - The application of the seven principles of interpretive field studies
Principles (from Klein and Myers (Klein and Myers, 1999), p. 72)
Practiced in the case study
1. The Principle of the Hermeneutic Circle This principle suggests that all human understanding is achieved by iterating between considering the interdependent meaning of parts and the whole that they form. This principle of human understanding is fundamental to all the other principles.
Data are collected through repeated interviews with actors playing various roles. The data collection is supported by observations and collection of relevant documentation. The growing knowledge of the case has guided the data collection.
2. The Principle of Contextualization Requires critical reflection of the social and historical background of the research setting, so that the intended audience can see how the current situation under investigation emerged.
The study of the case is conducted from two viewpoints – the present organization, and how it has emerged over time.
3. The Principle of Interaction Between the Researchers and the Subjects Requires critical reflection on how the research materials (or “data”) were socially constructed through the interaction between the researchers and participants.
A large part of the collected data is based on semi-structured interviews (Seaman, 1999) that followed open interview guidelines to ensure a balance between thematic focus and room for reflection, correction, and discussions. This allows for unplanned but relevant topics to be addressed.
4. The Principle of Abstraction and Generalization Requires relating the idiographic details revealed by the data interpretation through the application of principles one and two to theoretical, general concepts that describe the nature of human understanding and social action.
Findings are related to the concept of organizational ecology (Trist, 1977). Key principles from this theoretical background are applied to the studied case (section 5.3).
5. The Principle of Dialogical Reasoning Requires sensitivity to possible contradictions between the theoretical preconceptions guiding the research design and actual findings (“the story which the data tell”) with subsequent cycles of revision.
The theory applied to the case was not used to plan and guide the data collection. The applicability of the theory became evident through the analysis after the data had been collected.
Research approach
- 50 -
Principles (from Klein and Myers (Klein and Myers, 1999), p. 72)
Practiced in the case study
6. The Principle of Multiple Interpretations Requires sensitivity to possible differences in interpretations among the participants as are typically expressed in multiple narratives or stories of the same sequence of events under study. Similar to multiple witness accounts even if all tell it as they saw it.
This principle was followed by collecting data from both external actors and people with various roles in the product line organization.
7. The Principle of Suspicion Requires sensitivity to possible “biases” and systematic “distortions” in the narratives collected from the participants.
The data were collected and analyzed by the author, who is external to the organization, having no formal responsibilities, interests or agenda, except to create an unbiased view of the organization and its development.
(3) Longitudinal study: An important feature of the study of CSoft is that it was a
longitudinal study with data collected from 2004 to 2009. This means that we can describe how the organization has developed over time, some of the reasons for the changes that took place, and the results of these changes. From the start we have acknowledged that the organizational change is usually not a plan-driven and strictly controlled process. Given turbulent, flexible, and uncertain organizational and environmental conditions in today’s industry, such an approach would be less appropriate (Orlikowski and Hofman, 1997). On the contrary, power, chance, opportunity, and accident are as influential in shaping outcomes as are design, negotiated agreements, and master plans (Pettigrew, 1990).
When the focus of a study is the change of an organization, a longitudinal approach is the only approach to understand the change. Tushman and Romanelli (1985) (p. 174) say:
“The call for longitudinal, historical perspectives stems (1) from a pervasive dissatisfaction with static, cross-sectional views of organizations that illuminate covariant attributes to organizations, but tell little of the impact of history and precedent on current organization behavior; and (2) from simple curiosity for answers to such questions as, “How and why did this firm evolve? Why did certain firms succeed while others did not?”
This view is supported by Pettigrew (1990) who describes a theory of method for longitudinal field research on change. His fundamental view is that theoretically sound and practically useful research on change should explore the contexts, content, and process of change together with their interconnections through time (ibid., p.268). We have applied these guidelines by explaining the context of our study and the change that the case organization has undergone (see section 2.2.4).
Research approach
- 51 -
3.2.1 The selection of the case organization
The selection of CSoft as the sole organization to study was based on a long-term relationship between the researchers and the company. The author and others have studied this company and their product line more or less continuously over ten years (Johansen, 2005; Moe, Dingsøyr et al., 2002). The study reported herein adds to the previous studies. Together, the reports describe how CSoft have moved from a start-up company to an organization with a mature product. The present day status of the organization is not the result of a sudden adoption of a defined SPLE method; rather, it is the result of years of organizational development and evolution. This organization and the course of its development is, therefore, a relevant subject for an exploratory and descriptive case study, such as the one reported here. CSoft is a particularly interesting case, given that the organization’s use of SPLE also incorporates some of the fundamental principles of ASD. This combination of the two approaches is the result of being able to move outside disciplinary boundaries (Hughes, 1989). Other organizations considering making similar choices may well find it productive to turn to the CSoft case as a valuable source of experience.
When the object or phenomenon being studied is a large, diverse, and complex organization like CSoft, gaining access to necessary data may be difficult, for reasons of both practicality and confidentiality. In the case of CSoft, a high level of trust has been developed over several years, making it possible to access people and data that would otherwise not be available. Such sources may reveal information confidential to the organization and include even negative information about the company or its product. To formalize this relationship, we have signed a non-disclosure agreement with CSoft.
The empirical studies underpinning this thesis have all been conducted within this single case organization, CSoft. The first study started in 2004 (reported in Paper 1 (Hanssen and Fægri, 2006)), the last finished in 2009 (reported in Paper 6 – a manuscript submitted to a special issue of the Journal on Systems and Software). Together, four consecutive studies were reported in six publications, covering approximately five years of the development at CSoft.
Research approach
- 52 -
3.2.2 Data sources
Nearly all data that were collected throughout the studies of CSoft were qualitative, and their sources were interviews, workshops, observations, and collections of relevant documents. All interviews were semi-structured, and most of them followed a predefined interview guide but with a clear rule of conduct for allowing discussions outside the guide (meeting the third principle of Klein and Myers (1999) p.72). Table 7 - Data sources, shows which interviews that were based on a guide and appendix A lists all interview guides used in all four studies as well as some background of other types of data collection. In total, we did two group interviews and twenty single person interviews. All interviews and some observations were recorded digitally and transcribed completely. Observations that were not suitable for recording, for example ad hoc conversations were documented through notes. In this way, all data became textual. Table 7 lists all collected data, explaining study number (ref. Figure 1 - Study design), data source and whether appendix A gives more details on interview guides and techniques for data collection. In addition to these data, we have had continual discussions with various members of the case organization throughout the whole study. The outcome of such discussions has not been documented explicitly but has contributed to a better understanding of the case. The table also indicates which papers that report the data. Each paper give more detail on data collection, analysis and results.
Research approach
- 53 -
Table 7 - Data sources
Study Data sources A Interviews 1. One group interview with six developers using the PMA technique
(Birk, Dingsøyr et al., 2002) 2. Three semi-structured interviews with the participating customers
Study 1 (paper 1 & 2)
3. Five semi-structured interviews with the PMT members Interviews 4. One semi-structured interview with the CTO 5. One semi-structured interview with the PMT manager Documents
Study 2 (paper 3)
6. Six business plans for product line modules and main features Reusing data from study 1 and 2. New data: Interviews 7. One group interview with two of three members of the architecture
team. The interview was done after the architecture analysis as an open discussion about software entropy.
8. One semi-structured interview with one of the team leaders 9. One semi-structured interview with a team member/developer Other
Study 3 (paper 4)
10. Architecture analysis using NDepend5. Results are documented in (Hanssen, Yamashita et al., 2010; Hanssen, Yamshita et al., 2009; Smaccia, 2008).
Interviews 11. One interview with the R&D manager 12. One interview with the manager of Professional Services 13. One interview with the PSG manager 14. Three interviews with PSG members 15. One interview with the a Technical Account manager 16. One follow up interview with the PSG manager to clarify issues and
notes after observation of customer review meeting (source #18).
Observations 17. Product conference, London 2008 18. Customer review meeting, 2008 Documentation
Study 4 (paper 5&6)
19. Five module product plans, four presentations at the 2008 product conference (CSO’s, CEO’s, VP Product Marketing’s presentation and customer’s presentation)
5 http://www.ndepend.com/
Research approach
- 54 -
The selection of interview respondents throughout the studies have been based on a growing knowledge of the case in combination with the focus and research aim of each study. For some cases, we have based the selection of interview respondents based on the input or even direct guidance from the previous respondent, an approach similar to snowball sampling (Cooper and Schindler, 2006) (p.204).
Due to the geographical distance between the researchers and the case organization as well as very limited opportunities for on-site visits for observational studies, a large portion of the collected data have been done through telephone interviews. Further on, since a large part of the data in the studies comes from interviews we risk that the collected information can be biased, incomplete or even wrongful due to misunderstandings, lack of insight etc. To reduce this risk we have (when possible/feasible): (1) Done interviews with several individuals sharing the same function in the
organization. For example, in study 1, we interviewed three customer representatives and all five members of the product management team. In study four we interviewed four out of five members if the product strategy group.
(2) We have also sought to cover a variety of organizational roles to get various viewpoints in order to be able to build a balanced overview of the organization, their product line and their development processes.
(3) Besides interviews, we have done observations and collected documentation, which have been used to complement data from the interviews.
3.2.3 Data analysis
When working with qualitative data, such as interview transcripts, documentation, and field notes, several strategies are available for analyzing the material. These depend partly on the form of the data and partly on how the results are going to be presented. Langley (1999) defines seven strategies for what she calls sense making. Of these, the grounded theory strategy was found to suit the CSoft case study best because it is appropriate for the analysis of eclectic and ambiguous data. Other strategies have a varying potential for accuracy, simplicity, and generality.
According to Langley, grounded theory has the potential for high accuracy, but it can be difficult to move from substantive theory to a more general level. The level of generality that can be achieved depends on a number of factors, such as the degree of the scope of replication and the source of ideas. In the study of CSoft, the combination of SPLE and ASD was investigated. These are both well-established strategies for software engineering, each with a considerable legacy of research, models, methods, and nascent theoretical foundations. This allows for a high degree of generality, because the results can be compared to a large body of recent research. The overall goal of the CSoft study has been to contribute to the development of generally applicable theories
Research approach
- 55 -
for agile software product line engineering and to the more recent concept of software ecosystems.
Grounded theory aims towards theory building. This approach to analysis was originally defined by Lewin (1946), later revitalized by Glaser and Strauss (1967), and then developed into a few variants. Regardless of variant, the main principle is to analyze (usually textual) qualitative data step-wise and bottom-up by coding text fragments, adding a meaning to data, thus following an inductive process. Codes, which are short textual descriptors of the text, can then be grouped to reveal higher-level concepts in the data material. These concepts can be further grouped into categories, which can eventually be used to put forward and support a theory explaining the phenomena being studied.
This approach to analyzing data is also referred to as constant comparison, which according to Seaman (Seaman, 1999) is the classical method for theory construction. In the various studies supporting this thesis, textual data were reviewed constantly while they were being collected in order to identify concepts of interest. For example, the concept of cross-organizational participation in planning was formulated and developed in the course of data collection. The increasing knowledge of the case was used to guide the collection of new data, thus making it an iterative process, where understanding of the case is built incrementally. Recent findings were typically used in interviews, for example to elaborate seemingly interesting concepts or to clarify confusions. This continual interplay between the use of information that had already been acquired and the information that was being acquired is an example of the hermeneutic circle as described by Klein and Myers (1999).
Due to the nature of the data that have been collected in the four studies of CSoft, which in sum are a large amount of text, it is not feasible to include all this material in this summary. Thus, we give details on how the analysis in each study have been done and supplement these descriptions with examples.
Study 1 – Adoption of Evo Transcripts from all eight interviews and the postmortem analysis with a team of
developers were analyzed using NVivo™, a software tool for analyzing textual data by tagging data and then grouping analytical fragments into larger constructs. Each tag (called a node in NVivo) explains the meaning of the coded text. For example, one of the interviewed customers explained the reason for spending time as an external stakeholder in the Evo project:
Interviewer [following up on previous statements in the interview]: “..do you
actually define requirements for the product?”
Research approach
- 56 -
Respondent: “Yes, and very detailed to. Only yesterday we did two requests about two things that didn’t work in the reporting module, which is very important to us.”
This and other text fragments was tagged with “Evo enabled participation in
development”. Another customer were telling about how they provided ideas to the development project:
“We provided development ideas and feedback for CSoft’s new Panel Sample
Builder. We were asked to provide input on what we require for pulling panel sample efficiently, and how we could help improve CSoft’s existing sampling tool. “
This and other text fragments were tagged with “Providing concrete input”. These
two tags, and others, were later in the analysis grouped together under a higher-order tag (called tree nodes in NVivo) and called “Motivation to participate”.
Through several iterations of analysis, done by the two authors of paper 1 and 2
(Fægri and Hanssen, 2007; Hanssen and Fægri, 2006), a structure of tags was developed. This can be seen as an iterative and incremental approach to analyzing qualitative (textual) data and a case of constant comparison. The defined aim of the study was “to present a real case of agile customer engagement showing prerequisites, benefits, costs and risks in a software product setting”. This study aim was used as an analytical lens in the study, both for designing the interview guides and for analyzing the data.
Research approach
- 57 -
To illustrate the outcome of this type of analysis we list the structure of the tags developed for the interviews with the customers (45 nodes ordered under 10 tree-nodes):
Table 8 - Example coding nodes
Premises for Evo o Customer participation needs preparations o Knowledge of each other is important
Engagement o Customer was invited to participate
Communication o Customer appreciated dialogue o CSoft listened o Good communication o Frequent communication o Used emails o Used telephone o Wants a close dialogue
Customers perception of the process o Good management of expectations o Spent a small amount of time o Detailed requirements from customer o Was not able to test administration and
performance issues o Satisfied with the participation o Satisfied with CSoft’s response o Informative o Little interaction o Wants to be involved in future projects o Missed focus on some important
requirements o Want explicit feedback from CSoft o Weekly iterations are too short
Customers’ practical tasks o Found errors, which was corrected o Did not use the test server o Requested, and got a local installation o Tested the software and provided feedback o Used the test server o Customers’ tasks perceived as casual
Customers’ background o Have an existing relationship o Customer is an expert user o Have a close relationship with CSoft o Considered to be a large and important
customer
Educating customers o Training (in the Evo process) was simple o Training customers (for participation) is
important
Motivation to participate o Evo enables customer to participate o Expressing concrete requirements o Customer is highly engaged in the product o Need more in return for the time spent
Local organization at the customer o Involved other persons from own
organization o Involved own customers
o Previous practice o Did not see new versions until final release o Support is perceived to be impersonal o Feedback used to go through customer
support only o A low level of communication o Development was perceived as informal
Research approach
- 58 -
(Similar structures were developed for the interviews with the five members of the product management team and from the postmortem analysis with the developers.)
Having developed this structure of tags and an overview of the textual data, we used it in the papers to report on what we found to be the most interesting results with respect to the defined study aim. Due to the nature of the data collected (several pages of text), it is not feasible to present data in its raw form. We used the NVivo tool to extract tagged text for nodes we found interesting or even combination of nodes, for example, text fragments that were coded with more than one node.
These insights were used to develop the discussions in the papers. For example, in paper 1 (Hanssen and Fægri, 2006), we have sections discussing “Stakeholders’ practical roles”, “Specifying quality goals”, and others.
Study 2 – Agile SPLE In study 2 we reused the data and NVivo-analysis from study 1, and added two extra
interviews. The outcome of the analysis was presented in paper 3 (Hanssen and Fægri, 2008), discussing findings related to the defined research aim of the study, which was “to describe and analyze an industrial case to understand how SPLE and ASD can be combined and to clarify associated costs and gains.” Like in paper 1 and 2, we use the analyzed data to discuss relevant topics and findings. This was supported by a set of business plans for a release of the CSoft product line.
Study 3 – Software entropy The three transcribed interviews were analyzed in NVivo, using the same approach
as described under study 1. The research goals of the study guided this analysis: 1) How may system entropy and agile processes mutually negatively affect each other? And, 2) Can code smell analysis and refactoring be a viable solution?
This analysis was supported by the outcome of the architecture analysis, which documented facts about the internal structure of the CSoft product line. The results were presented in paper 3 which reports and discusses the facts from the architecture analysis and the most important findings from the qualitative analysis of the interview data which were 1) analyzability and comprehensibility of the product line, 2) modifiability and deployability, 3) testability and stability, and 4) organization and process.
Study 4 – Software ecosystems Eight new interviews were made as well as two observations and collection of
relevant documentation. Data was analyzed in two steps: Step1 – All data were first examined to produce an intermediate analysis report in
order to produce an updated overview of the CSoft development process in terms of roles, activities, and artifacts, in addition to high-level concepts, necessary to understand
Research approach
- 59 -
how product planning and development is conducted. This analysis created a structure by grouping information coming from the various data sources. Examples of such concepts are teamwork, planning, customer interaction and innovation.
Step 2 – All data, in textual format, was analyzed using NVivo, using the same approach as explained for the other studies.
The outcome of this analysis are presented in paper 5 (Hanssen, 2010) and 6 (Hanssen, 2010). The latter includes several extracts from the data to illustrate the concepts being discussed.
The case description The description of the case organization, the product line and their processes in
section 2.2, is a summary of all information which have collected in all four studies. The overview in the subsections 2.2.1 to 2.2.3 represents the present organization and situation, while section 2.2.4 explains the development towards the present situation. This information is the result of several interviews and observations, done over several years and is supported and complemented with information from the organizations web site, marketing information and company presentations. Some of these data sources are listed in Table 7. These descriptions of facts about the case have been shared with persons from the case organization, which have verified the correctness. For example, we shared the manuscript of paper 4 with the architecture team.
Discussion and implications
- 60 -
4 Results This chapter presents a synopsis of the results from the four studies reported in the
six papers that support this thesis. As the thesis is solely based on the supporting publications (listed in section 1.5), no additional results or data are added in this chapter.
Each section gives an introduction explaining the aim of each of the studies and papers, and the main results. Further details and information about study approach, related research, analysis, discussions, and conclusions can be found in the original papers in part II of this thesis.
4.1 Study 1: Adoption of Evo
The first study of CSoft did coincide with the first attempt to try the agile method Evo in 2004, as an alternative to the plan-driven approach that CSoft had used for several years. Due to declining process performance and problems related to the late user feedback on new functionality, the development managers decided to try Evo for one of the modules in the product line. One of the most obvious changes was the change from long-lasting sequential release projects to extremely short development iterations with frequent interactions with external actors. This case represented an unexpected and unique opportunity for the researchers, who followed this process trial. Evo was used for two main releases and two intermediate releases, covering a year and a half of activity. This study resulted in two publications, focusing on important aspects of the adoption of Evo, respectively customer engagement (Hanssen and Fægri, 2006) and process fragility (Fægri and Hanssen, 2007).
In sum, our focus in study 1 was the initial adoption of the agile method Evo in a waterfall-oriented product line organization, with an emphasis on how developers relate to customers and the fragility of the agile approach.
4.1.1 Paper 1 - Agile Customer Engagement - a Longitudinal Qualitative Case Study (Hanssen and Fægri, 2006)
Due to the sudden start of this software process improvement initiative and our study of it, the study aim was simply to investigate and present an industrial case of agile customer engagement showing prerequisites, benefits, costs, and risks in a software product setting.
Our analysis of the interviews identifies a number of prerequisites for succeeding with an agile development approach in a software product line setting. Proactive
Discussion and implications
- 61 -
stakeholder management is the foundation. CSoft operates in a dynamic market in which long-term planning in advance of development is unrealistic. This applies equally to the customers, which are inherently unpredictable in their role as collaborative partners. CSoft depends upon mutual benefits in the collaboration to maintain motivation and contribution. Lack of continuity has significant bad effects on the performance. In turn, achieving these mutual benefits depends upon selecting relevant customers with sufficient expertise. These are all volatile and difficult factors. Thus, the capability to constantly review and manage the selection of stakeholders is critical. Furthermore, our analysis shows that Evo is a highly demanding process with respect to personal discipline and professional behavior. Due to the frequent iterations there is little room for unrestrained activity. All roles must be meticulously filled. Additionally, some sophistication in technical infrastructure, such as continuous integration of the solution under development, simple backlogs of requirements (called impact estimation tables (IET) in this case) and ASP-based software delivery, has been essential in CSoft’s use of Evo.
Further on, we have seen that CSoft has achieved a number of benefits as a result of Evo and the introduction of the product management team (PMT). First, close customer cooperation has a highly motivating effect on the developers. Second, developers’ confidence has increased as a result of continuous settlement of expectations in that stakeholders assist in the prioritization of goals. The direct cooperation with users is a positive experience for the developers as it increases the quality of the communication and leads to an improved understanding of customers’ business problems. Third, Evo has increased the visibility of the process internally in the organization and externally among the stakeholders.
Although these benefits, we have seen that the adoption of agile customer engagement practices has incurred additional costs. Our analysis emphasizes the extra overhead in actually running the process and the human resources required. Essentially, each Evo week/iteration is a complete development process, spanning a number of phases that demands frequent changes of context and thus occupy significant resources. Also, the technical infrastructure, being a prerequisite, is costly. Furthermore, the analysis shows a significant cost incurred by the continuous need to maintain it.
Through our analysis we have come to consider increased exposure to risk as a cost. First, short iterations with insufficient attention to management and process compliance increase the fragility of the software process and create a risk of leaving developers with a high workload. Secondly, engaging in this kind of strong cooperation with a small selection of customers also means a reduced capability to capture the needs of other, non-appointed customers.
Discussion and implications
- 62 -
4.1.2 Paper 2 - Collaboration and process fragility in evolutionarily product development
Following the results from paper 1, using the same data material, we investigated and discussed in more detail the transition from a waterfall-oriented approach to an agile approach and in particular the collaboration with external stakeholders and the related fragility in the new process.
Our study of the introduction of Evo at CSoft showed that the agile process made the relationships with the customers more collaboratively and that a constant focus on delivering customer value increases the synergetic value of these relationships. This becomes a significant competitive advantage, enabling more efficient innovation transfer between CSoft and the collaborating customers. However, we also clearly saw that maintaining a watchful, vigorous stakeholder management capability, as implemented by the product management team, is paramount for Evo’s success. Further more, Evo increased the product’s exposure and resource usage compared to the original process. More people can observe or criticize prioritizations. This can help maintain focus on generating value, but it also influences the nature of the demands on the product. This follows implicitly from the increase in direct stakeholder feedback. However, the R&D department also saw a lack of attention to software engineering principles, therefore, companies should also select stakeholders with a particular interest in the product as an engineering artifact (called “internal stakeholders” in Evo) at intervals to counter architectural erosion and subsequent excessive costs of implementing improved quality. We also learned that short cycles of specification, design, development, and testing reduce tolerance toward inefficient tools, for example tools for automated testing and continuous integration. On the other hand, we believe that these rapid, direct feedback loops explain most of the increase in employee motivation and enthusiasm that we saw, particularly in R&D. Rapid feedback on developed code, appropriate metrics, and effective decision making help reduce uncertainty and thus increase developer motivation. The positive experiences reported by stakeholders stemmed mainly from a feeling of being listened to and able to affect the development.
Evo is an agile process but also a sophisticated one, highly dependent on diligent fulfillment of the specified roles and activities. This sophistication is also Evo’s Achilles’ heel. We have found that Evo, as practiced in the case we have studied, is very vulnerable to irregularities. This process fragility increases with the frequency of iterations, which put higher demands on the timeliness of role fulfillment and activities. Unresponsive stakeholders expose this fragility.
Discussion and implications
- 63 -
4.2 Study 2 – Combining agile software development and software product line engineering
Based on the results from study 1 and an initial understanding of some of the effects and challenges we observed, we continued our study of CSoft and their continuous adoption of Evo. Now in a larger scale, covering more parts of the.
The focus of study 2 was the combination of ASD and SPLE and the coordination between the strategic, tactical, and operational processes.
4.2.1 Paper 3 - Process fusion - an industrial case study on agile software product line engineering
Building on the results reported in paper 1 and 2 we collected more data to investigate the broader consequences of combining an ASD process with a SPLE practice. We were interested in learning how this combination affected various levels of the organization, not only the development department. The aim of the study reported in this paper was to describe and analyze an industrial case to understand how SPLE and ASD can be combined and to clarify associated costs and gains.
At a high level, we have found that the integrated software processes at CSoft support three key virtues of product development. (1) Technical excellence: an open and modular platform architecture implemented using industry-standard technology enables simple development and maintenance of the product line. (2) Market knowledge and relevance: the well organized, yet nimble strategic process provides adequate decision support for company management and guidance for the development projects. (3) Agility: the adoption of Evo, and agile principles such as short and frequent iterations with active participation by external stakeholders, enables fast response to changes in stakeholder requirements and accurate delivery of desired features and qualities to the users.
An important observation from our findings is that CSoft’s integrated development process is not just an accidental collection of strategic, tactical and operational processes. Rather, these three processes play distinct, supplemental and important roles at three different levels in CSoft’s overall software development business, all three being lightweight to reduce unnecessary work. We also saw that these three processes together have the same function and effects as the more general Shewhart/Deming improvement cycle (Deming, 2000) (See Figure 7). This is also known as the PDCA cycle, from its four main activities: Plan (plan how to satisfy improvement requirements); Do (accomplish planned actions); Check (monitor the actions and verify the outcome with respect to the planned effects); and Act (implement actions for improvement based on the acquired information).
Discussion and implications
- 64 -
Although a feasible combination, this approach has its costs; running the strategic process and working closely with stakeholders entails a considerable extra overhead, resources that could have been invested in development (as was the previous practice). Arguably, the most important benefit of this process configuration is that it helps CSoft to exploit both strategic long-term ambitions for innovation and smaller-scale tactical innovations, such as refinement of the software to meet more detailed end-user requirements.
Further, the process configuration is a potent foundation for further process innovations. As long as the PDCA cycle is maintained, any component in the configuration can be further refined to improve performance. If the company experiences the expected growth, the modularization gives extra organizational flexibility. For example, introduction of separate roles for platform and product development within R&D can readily be supported because the objectives of strategic and tactical development are already defined and well established. Additionally, the explicit engagement of primarily external stakeholders in the tactical development process ensures that the company is able to supplement strategic planning by selecting stakeholders based on long-term as well as short-term interest.
Engaged stakeholders provide requirements and feedback on a detailed level but it is the responsibility of the development projects to suggest practical solutions that will meet the quality goals stated by the stakeholders. This close cooperation with a few selected stakeholders promotes valuable creativity. If process innovation at CSoft had a more revolutionary style, replacing all existing practices with a pure and very formal SPLE approach, this potential could have been lost.
4.3 Study 3 – Process agility and software entropy
Through the initial studies of CSoft and their adoption of Evo we noticed a growing concern regarding problems related to the increasing entropy in the code base. Actors being close to the code, like developers and in particular the dedicated software architecture team, were worried about the increasing level of entropy in the system as an invidious result of the agile software product line engineering practice which by now had been adopted in the whole organization. Based on this, we initiated a study to understand better the relationship between the agile development process and the entropy of the system.
The focus of study 3 was the emerging problem of software entropy, the potential reinforcing effect from the agile development process, and potential solutions to improve the situation.
Discussion and implications
- 65 -
4.3.1 Paper 4 - Software entropy in agile product evolution
Based on the increased understanding of the agile software product line engineering at CSoft we sought to investigate a concern of software entropy, which have emerged from the previous studies. We set out to answer to research questions: First, do system entropy and agile processes negatively affect each other? And secondly, can “code-smell” analysis and refactoring be a viable solution?
First of all, we constructed an overview of the technical structure of the CSoft product line, needed to understand problems of system entropy. The system has been under constant development since 1996 and is based on several technologies that have emerged over those years. Aging solutions from years ago are still part of the system, such as older ASP solutions, COM+ components, VB6 code and other legacy technologies. Today, most new code is developed in C#, and is spread over approximately 160 .Net assemblies. The complete product is best described as a traditional three-tier system with an MS SQL Server driving the data layer, a business layer and a presentation layer based on a dozen ASP.Net applications. There is a clean separation between the presentation- and the business layer. However the most obvious problem in the software is what the architects refer to as “the Blob”: a very large assembly (aptly named Core) consisting of approximately 150K lines of code in 144 namespaces. Section 1.1.1 contains more details.
Through interviews with the architecture team and actors from one of the development teams we uncovered four types of problems related to the level of entropy in the system.
(a) Analyzability and comprehensibility. Due to the high complexity of the system, it is very hard for developers to get an overview of the code and its structure. Especially the central component has grown extremely large and has many internal references (each namespace depends directly or indirectly on another namespace), making it difficult to understand how it really works. This was clearly not by design, but the result of years of intense development. The system is structured as vertical modules, but as it is now there are too many relationships between the verticals – changing one will inevitable affect many others. New developers joining R&D have a steep learning curve and require close follow-up over a long period of time by more experienced developers. There exists no documentation or models that explain the structure of the system, even though this clearly would be highly useful both to existing and new developers. Even worse, having problems understanding how the code is structured leads to a fear of changing the code, both for adding new features and for improving existing code. The unclear internal structure creates a cognitive overload and a common (unfortunate) way to deal with this is code duplication: instead of modifying existing code, developers create their own copy over which they have full control. This leads to a larger cognitive
Discussion and implications
- 66 -
overload for other developers; only making the problem worse – it has become a self-reinforcing effect.
(b) Modifiability and deployability. As a result of the duplication and entanglement of code, developers frequently need to perform so-called ‘shotgun surgery’, meaning that even the modification of a small detail forces them to identify and change code in many places. These problems slow down the development process and the potential for errors increases due to the high chance of overlooking one or more locations. Having to deal with bad code is frustrating to the developers as they in some ways in practical terms are enforced to build bad code on bad code as there is no room to actually resolve the problem. Besides development and maintenance, also deployment of the product suffers from its structure: The current core component aggregates features and functionality for every possible configuration of the product and it has to be released as a whole, even though only a fraction of the functionality may actually be needed for a particular configuration.
(c) Testability and stability. Due to the size of the code and the many cross-references, there are too many paths through the code to test them all systematically. The test coverage is not high enough and existing tests have shown to be unstable and inconsistent. For example, the same tests run on similar systems may produce different outcomes that are hard to explain. Also, a lot of the existing tests are extremely large, meaning that they too are hard to maintain and use. When a test fail, it often takes a lot of time to locate and fix the actual problem that triggered the failure. Although such tests are supposed to act as a safety net and give developers the courage to make changes they are not trusted. This increases the fear or at least reluctance to change existing code – since the effects of a change are hard to foresee and errors can have considerable negative effects. Nevertheless, regression testing is done, albeit with a lower than desired quality.
(d) Organization and process. As both the business domain and the system are highly complex, each of the development teams (4-6 developers in each) has an expert (the so-called guru). This guru has high technical skills and extensive experience with the code, which is vital for the team to solve its tasks. Consequently, this organization represents a considerable vulnerability; losing just a few of these gurus would have devastating effects on the development. The development process is based on two-week iterations and it is a strong focus on delivering working software by the end of each iteration. A negative effect of this focus is that delivering quality software is at times traded in for creating a working version. Each iteration ends with a review, but the high velocity typically does not give enough time to catch all issues. This causes extra work close to a release when the system is thoroughly tested as a whole, yet entropy is allowed to grow from release to release. The development teams are set up to have separate areas of concern, each team being responsible for a part of the total product,
Discussion and implications
- 67 -
e.g., the reporting solution or the data storage. The idea is to build competence around a well-defined part. Unfortunately, the structure of the system does not reflect this organization in practice, because functionality is spread throughout the code. This forces the teams to operate outside their area of concern, which has shown to negatively affect their ability to produce enough new and improved features of the product in their releases. The total request for improvements from the market is constantly higher than what actually is delivered, thus indicating a need to improve development efficiency.
As part of the discussions with the architects, we also collected several of their high-
level ideas to further improve the product and development process: (a) Process automation. Currently too much testing is done manually and more
automation is desired. In addition, to establish an efficient and trustworthy safety net for the developers, tests need to become more stable and trustworthy. With this in place, the architects can introduce what they call “pain-driven development”. That is, when a developer introduces or changes code that breaks the tests, he or she will get notified immediately to correct it.
(b) Restructuring and refactoring goals. The architects feel that components of the software need to be de-coupled from the core and the overlapping and duplicated code has to be removed. They also agreed that the system should have a clearer separation of concerns were vertical modularization should reflect business segments and horizontally, the system should better separate business and platform related code.
(c) Continuous monitoring of quality. The architects proposed a principle that they refer to as “quality-from-now”, meaning that any change to the code should be analyzed at development time, to check that it does not conflict with defined rules of good design. This can, for example, be achieved using tools to detect code smells and monitor potential problems nearly constantly during development. The architects believe that this approach would considerably reduce the fear of changing the code.
4.4 Study 4 – Software ecosystem emergence
By now, having followed CSoft over five years - observing how they have changed their development process - it became evident that this change also affected how they relate to external actors, beyond single customers acting as stakeholders. This initiated a new iteration of data collection and analysis with the primary objective of understanding how the organization relates to its external organizational environment. These new data revealed an image of an organization that was opening up their strategy and development processes, moving towards a situation that can be understood as a software ecosystem in development.
Discussion and implications
- 68 -
The focus of this final study of CSoft was the increasing openness of information sharing and cross-organizational collaboration.
4.4.1 Paper 5 – Opening up product line engineering
Based on the lessons learned from the study of CSoft so far, this paper looks into the change that the product line organization is experiencing. The aim of the paper was (i) to explain how the SPLE process can be opened up and why this may be a necessary development, (ii) to illustrate some practical issues, and (iii) to provide some preliminary guidelines to industry and indicators for future research. Actions taken: The initial waterfall-like approach with extensive up-front planning required a considerable amount of resources and blocked input from customers and other important stakeholders during the development phase. Through the adoption of core agile development principles, fewer resources are spent on making detailed up-front plans and controlling plan conformance during development. The iterative process opens for continuous corrections during development based on a close dialogue with stakeholders, including invited customers. This enables a reactive approach to SPLE where no assets are changed or added to the line unless they are explicitly founded on concrete needs. Observed effects: The resulting SPLE process at CSoft helps addressing some of the inherent risks that comes with this approach. McGregor (2008) describes that, for a pro-active approach, there is a risk of developing assets that will become obsolete and not used in later applications. For a reactive approach, there is a risk of missing short-termed business opportunities due to increased time-to-market and that it might require considerable effort to, reactively, prepare assets for later reuse. CSoft’s incremental and iterative approach meets these risks as a compromise between the proactive and reactive approaches. Continuous corrections by involved stakeholders ensure that changes and additions to the product line actually will be used. The process also opens for corrections close to the release, if found necessary. In addition, the proximity of representatives from the business domain and the extensive use of direct and conversational exchange of information improve the ability to catch both explicit and implicit (tacit) requirements (Grunbacher and Briggs, 2001). One potential effect on the negative side is a reduced maintainability of the product line. The agile development process emphasizing a continuous focus on short termed goals seems to reduce the focus on maintaining the code and architecture properly, thus causing escalating system entropy which seriously hampers the organizations ability to improve and develop the existing code base. Enabling factors: The opening of the SPLE process at CSoft has been supported by some additional measures. One of the most noteworthy is that a community of third
Discussion and implications
- 69 -
party organizations has emerged, partly supported by CSoft and partly as an effect of the accessibility to the product line via the API’s. Approximately 60 organizations are now offering software products and/or services partly or completely based on the CSoft product line. This means that CSoft may keep full focus on developing the core product line and that customers are offered a large number of specializations and applications of the product line. The emergence and growth of this community come as a result of not only the technical accessibility via the API’s but also the sharing and external visibility of product plans and roadmaps. Key clients are given insight into these plans; some are even visiting the R&D department to elaborate business needs and ideas, and to discuss potential solutions directly with the development teams. CSoft has also (once so far) organized a large product conference where strategies were announced, new features introduced and third parties got to expose their solutions. Contextual factors: This change in CSoft’s approach to developing their product line is moderated by a set of contextual factors. First of all, the changes done to the development process is a reaction to the volatility of the business domain being addressed. This particular domain is still shaping, its boundaries still being determined. This is an alternating process; the technology being developed and offered through the product line is creating new business opportunities, which in return leads to new requirements. In this context, it is more important to be able to respond quickly to the market than to produce reusable components that might be useful at a later moment. Enabling an agile and responsive development organization gives it control over how the domain and its supportive technology develops – a clear advantage in a competitive business. Another factor affecting the shaping of the SPLE process is the strongly emerging software-as-a-service (SAAS) delivery model. Software is not procured as a commodity, installed and managed locally, but rather as a service accessed over the Internet. This simplifies the delivery of software and consequently also the development process. For example, this model makes it practical to engage selected stakeholders, which frequently and with a minimum of effort can access the latest increments to assess and provide corrective feedback.
4.4.2 Paper 6 - A Longitudinal Case Study of an Emerging Software Ecosystem: Implications for Practice and Theory
Based on the previous studies of CSoft we collected more data, now from a more mature organization than for the first studies, to develop a more thorough understanding of the development towards a situation which can be named as a software ecosystem. Our research question was: Why and how is software product line engineering developing towards a software ecosystem?
Discussion and implications
- 70 -
The results from this study relates to CSoft’s engagement of customers and the emerging third party community constituting a software ecosystem.
The main motivation for customers to spend time participating in the development projects is the ability to affect development: no payment or any other compensation is provided. One of the product managers explained:
“…they see their wishes or their requirements or whatever in the product at the end. And then you get very nice feedback like ‘I can see that I said this and that, and in the next release you did it’”.
This collaboration forms a self-regulating system where the supplier and the stakeholders mutually adapt to each other through their shared interest in developing the software product line. This usually works well, but there is always a risk of having external stakeholders, which do not provide the necessary input, as explained by the manager of the PSG team:
“Everybody has busy jobs and projects that need to be on time etc. It happens quite often that we have to cancel these meetings or that they haven’t done anything since the last time. Then we can only show them what we’ve done and get some ad-hoc feedback...”
The PSG manager also explained that it is relatively easy to discuss ideas, but that it is more of a challenge when they are included in the development process:
“There’s no problem to get them to discuss high-level plans, but it varies when it comes to the development process”
Maintaining the motivation for participating is an important task for the PSG. Interestingly, large and leading customers tend to expect and demand to be more and
more involved at both planning and development stages. One example is a product conference keynote given by the VP from one of the large customers. He stated several ‘requirements’ (this was the word he used) for being involved, for example:
“Regular meetings with product development teams”, “To work as a stakeholder on new software developments that are key to us.”, “Help to guide product strategy.”, and others.
Finding the “right” stakeholders for participation in the development projects is not done through a formal and structured process, but is mostly based on the collective knowledge about the customers. The PSG manager says,
“We don’t have a formal process for selecting stakeholders. We have internal discussions, listening to sales people etc. We know which customers have asked for certain features or improvements or those that are heavy users of a particular type of functionality.”
In addition, experience from previous participation is also useful as explained by the R&D manager:
“We have become better at selecting [external stakeholders]. Those that have disappointed us are never asked again. You end up with a pool of persons that you know you can trust.”
Discussion and implications
- 71 -
An important part of recruiting customers as external stakeholders is to communicate to them clearly the opportunity, which is given to them. The PSG manager explains:
“..quite often we talk about our development process. We do it in sales situations because it tells that our goal is to solve business problems for our customers. This level of interaction and the way we try to listen has been well received, and now some customers insist on being involved in development.”
In the very start when Evo and collaboration with external stakeholders were at an experimental stage it took quite some effort to recruit stakeholders to the development projects and to keep them active (Fægri and Hanssen, 2007). After some releases where collaboration with external stakeholders have become an integrated part of the development process the situation is turned upside-down. When asked to explain this relationship, one of the product managers told us:
“It's almost a problem because as soon as you offer the capability of being a stakeholder, the hardest part is rejecting people, turning them away from actively participate. So people are very keen on participating.”
Co-creating the product line is one of the most significant effects of engaging and communicating with external stakeholders. The rationale is simple, CSoft have the most up to date knowledge of the technology, and the ability to make use of it in the development. Likewise, customers hold the most up to date knowledge of their own business domain, and how it seems to develop. These two pools of knowledge and competence are joined in several ways. One important arena for sharing and gaining knowledge is the product conference where management, strategists, developers and other internal actors get to meet externals from various customers and third parties. Equally important – customers can meet other customers, third parties can meet customers or internals etc. This shows the networked character of the ecosystem that is shaped around the product line. Some examples from the product conference in 2008:
A former customer of a competing solution was seeking experienced customers to discuss the product line and share experience. Several providers of third party products and services were having stands at the conference, communicating with both existing and potential customers and developers from CSoft.
Another major event, which is more directly focused on the development of the product line, is the annual Advisory Board meeting. Top management from some of the largest and most demanding clients meet with the PSG and other actors who are involved in the shaping of the product strategy. A PSG member explains that they meet to:
“…discuss high level product strategy and how the demands of their companies and the market are developing.” Bringing together major competitors like this was a daring thing to do according to the PSG manager: “The first time we did this it was a bit exciting – would they discuss issues openly, and would they open up? It turned out that they did very fast. They have many concurrent needs, and even though they are competitors they see the value of doing this.”
Discussion and implications
- 72 -
From a practical viewpoint, we see that tools and infrastructure for collaboration are important enablers for co-creating the product line. Especially the Webex™6 online meeting solution lowers the threshold for having frequent and detailed meetings with stakeholders:
From our observation of one of the customer review meetings we saw a lot of very detailed discussions that were made possible by on-the-fly demonstration of the software through the screen sharing solution. This sparked detailed discussions both on the customer side and among the development team. The meeting resulted in a list of clear actions points to be addressed in the next development iteration.
Close corrective feedback in the Evo development projects is another approach to co-creating the product line, but on a tactical level. One of the developers describes the meeting with the external stakeholder at the end of the two-week Evo iteration:
“What you get during a meeting is often very valuable. Especially when you are about to move in the wrong direction, which you can adjust. We get feedback saying that our solution is not quite what they had in mind or what they need.”
This demonstrates one important function of the agile process; the development teams get nearly immediate (within two weeks) and detailed (face-to-face) feedback. This closeness to a few selected customers means that CSoft must also consider the needs of other customers, as they are the referent organization, which always has the last word in the development of the product line. This is partly achieved through Evo’s focus on product qualities instead of product features, which are typically emphasized in plan-driven development methods. This is a useful abstraction, and it turns the focus from predefined design (features) to effect and impact (qualities). Both the product roadmaps and the evaluation meetings at the end of the Evo iterations evaluate the product qualities. This means that both the development teams and the external stakeholders have to consider why something is needed, leaving the how to the developers. The PSG manager explains:
“..we take one step back, and try to think about why our stakeholder needs this, and then rethink other ways of solving their problem. It is in this type of process that the smart things can turn up – that your thinking is totally new and that you come up with a solution which may be a totally different way of doing it, maybe faster...”.
Catching and following up on customer ideas on an ad-hoc basis is equally important as involving customers in regular processes such as roadmapping and the Evo development projects. At the product conference:
A customer representative told about a case where his company gave input to CSoft on some changes they would have liked to see. This led CSoft to invite a delegation from the (abroad) customer to the R&D department in Oslo. Ten CSoft people spent the whole day discussing the solution with four representatives of the customer. This was perceived very positively, and in the end actually affected the software.
6 www.webex.com
Discussion and implications
- 73 -
Some of the largest customers may also request dedicated workshops to discuss needs and ideas. The PSG manager talks about this:
“…for some of our largest customers, mostly by their initiative, we organize workshops once a year, usually on a strategic level. They want to know what the roadmaps may bring for the next couple of years, and talk a lot about their needs etc.”.
The close contact with customers is also a valuable source of learning about competing solutions. One of the team leaders talks about customers visiting the R&D department:
“In these meetings they demonstrated the solution they used today, and actually demonstrated how they used the competing solutions – what worked well and what needed improvements, as well as ideas they might have. These meetings gave the team a wealth of details, and it was quite clear what to deliver to the stakeholder.”
The PSG manager also explains the value of learning of the use of competing solutions:
“…alternatively they do it using other tools today when not using our solution. The option to work more closely with them and to get that knowledge made us more capable to meet their needs better than before when the development was more of the black-box type”.
A phrase from one of the roadmaps illustrates the business impact this may have: “Through a client we have been given a thorough demonstration of the competing
solution NN, and by implementing support for [some advanced functionality] and a couple of small features, the X-module will by far exceed their corresponding functionality. These improvements alone will ensure we win one [sales] deal, and have also been brought up by several other clients/prospects.”
Learning the business processes and domain is another valuable outcome from the direct contact with selected stakeholders. One of the developers talks about one of the stakeholder meetings in an Evo project:
“…we have tried to solve a task in a way we believed would be reasonable, but to people who actually use this it is obvious that we have misunderstood the process. This gives us guidance as early as possible.”
A PSG member tells about another case: “They [the customer] were here in a workshop for two days. We presented the
roadmap [for module X] and they presented their wishes and their business, [related to] what they are doing.”
This illustrates the shared interest that the customers and the supplier have – customers want to learn about the product line and its development. Correspondingly, CSoft learns about the business that their product line is supporting.
At present around 60 external organizations base their business completely or partly on CSoft as a platform. This can be value-adding solutions or products, related services, and consulting. Examples are solutions for data visualization or voice data capture technology, assistance in using various components in the product line, and training.
Discussion and implications
- 74 -
This networked community (Fricker, 2009) has not been planned and deliberately established by CSoft, it has emerged spontaneously over the past years. This emergence is mostly driven by customers’ need for additional features and services on one hand, and the opportunity to extend and use the product line as a platform on the other hand. Also, building solutions and providing services based on the product line means that external organizations get immediate access to a large group of established users of the product line.
Providers of third-party solutions are considered to be important external stakeholders, and are included in the development of the product line in very much the same way as customers.
During a product conference, a representative from a third party company, delivering an integrated product, explained that when they needed to improve the integration with the CSoft platform they took on the role of an Evo stakeholder. Communication was mostly done by phone, supported by web meetings with screen sharing.
Offering an efficient integration technology enables a third party community. Over the past few years a set of simple APIs have been offered to enable external actors to make extensions to the product line. The development of these APIs have followed the development of the product line, where each new release has improved existing and offered new APIs due to requests from external actors. This means that there is a long (a year) connection time between a request for an interface and its actual release. As more and more externals have made use of this connection point to the software it has been given increasingly higher priority in the development of the product line. An excerpt from one of the roadmaps exemplifies this:
“We are in dialogue with some clients/prospects who are building their portal in a Content Management System, and need to integrate content from [module X] into it. Some competitors seem to have APIs that are easier to use than our SOAP7 based APIs, making it easier to integrate with other portals/communities. It is therefore an ambition to provide an easier API for including [module X] content into an external portal.”
Due to the extensive use of the API’s by externals and their increasing demand for integration with the product line it became clear that the simple web-service based interface had become obsolete. This has led CSoft to develop and offer a new API called FlexibilityFramework8 (FF), which enables a closer integration to core services in the product line than the previous (and still existing) simple messaging-based APIs offer. A recent webcast, where the CSO presents FF explains further the motivation for this improved interface:
“CSoft is like a supertanker. It is large, can take huge loads, travel far, and take heavy weather. These are all very positive things; on the other hand, the consequence
7 Simple Object Access Protocol, http://www.w3.org/TR/soap12-part1/ 8 A fictitious name to protect the anonymity of the case.
Discussion and implications
- 75 -
of that approach is that we are quite careful at looking after the supertanker. That means various procedures, on policy, on quality assurance and so forth. And that means that we get less nimble than we would like. The question we posed ourselves is how can we behave like a speedboat while having all the benefits of the supertanker? I’d like you to think of FF as the speedboat. The tanker is still there. It will still take heavy loads and perform extremely well, but in order to be nimble we can build a few speedboats. And they have independent lives from the supertanker and can run on different development schedules.”
The last argument is worth a comment; with this new interface to the product line external actors are disconnected from the long release cycles of the product line, and can develop value-adding solutions independently. This is likely to further drive the growth of the third party community.
Actively supporting the community has become a regular activity in addition to the continuous development of the product line. As this community has emerged and grown, CSoft have seen its value, and started to actively support it. In 2007 a dedicated web-portal was launched to make this community visible and each partner is listed and presented. There are five types of partners, those offering technology that is integrated with the product line, those offering value adding services, some can prepare the use of the product line, some can use it on behalf of clients and some offer consultancy services.
Discussion and implications
- 76 -
5 Discussion and implications In the discussion of the results of the studies of CSoft we will emphasize 1) how the
combination of SPLE and ASD created a more open development organization and 2) how this change leads towards the formation of a software ecosystem. Based on this discussion we state implications for theory and practice.
5.1 An agile software product line engineering process
5.1.1 Actions taken to combine the processes
By comparing the processes and organization at CSoft with what can be referred to as a traditional SPLE approach we see that CSoft has taken two major steps to shape their development process: (i) the overall simplification of processes and organization, and (ii) the close engagement of customers, both in planning and in development.
Simplification of process and organization: CSoft has simplified their
development process in the following three ways. 1) There is no clear separation between core asset development and product
development, which is an important distinction in traditional SPLE. The idea of having two distinct processes is to enable the separation of the different concerns of building a robust platform on the one hand, and of effectively building products based on that platform on the other. In the CSoft case, there is no dedicated sub-process for core asset development, yet the total organization manages to handle both the development of core assets and applications simultaneously (Sanchez and Mahoney, 1996). Variability is implemented through a simple system where features are activated or deactivated according to the needs of customers. This simplification with respect to product development and variability management means that there are fewer roles and fewer dedicated sub-activities. Although simpler, it also means that the product line has less variability. This drawback is balanced because it is easier to manage the CSoft product line with the development resources focused on the next release, not on managing an extensive set of variations, which can be costly (Bosch, Florijn et al., 2001).
2) The second simplification is how requirements are managed. Related to the first type of simplification discussed above, CSoft does not implement a dual requirements engineering processes, one for core assets and one for application development as explained by traditional SPLE approaches such as the SEI guidelines (Clements and Northrop, 2002) (p.109). The equivalent is the strategic and tactical processes as described in section 2.2.3, but with the difference that the roadmaps represent high-level guidelines for the long-term development of the product line and not requirements for
Discussion and implications
- 77 -
developing core assets in the product line. At CSoft, requirements engineering is done as part of every iteration where external stakeholders define and re-define requirements based on experience from the previous iterations – in line with principles of ASD (Takeuchi and Nonaka, 1986). Furthermore, CSoft’s agile SPLE approach means that the product managers in the PSG and the development teams does make use of practices such as domain analysis techniques, stakeholder-view modeling, feature modeling etc. (described e.g. by the SEI guidelines, (Clements and Northrop, 2002) (p. 114)).
3) The third type of simplification concerns internal communication and coordination. The general principle of openness, visibility, and direct communication is adopted from ASD (Cockburn, 2002). The status of development of roadmaps and the status of the ongoing software development is made visible to everyone internally, and to some extent also to external stakeholders. The primary artifact that communication is based on is the most recent version of the working software, rather than a set of models or documentation of the product line. In addition, direct face-to-face communication is emphasized. Written communication is kept to a minimum. This strategy improves the efficiency of communication (Daft, Lengel et al., 1987). Teams usually work in open offices to enhance communication. The product managers from the PSG are present at most meetings. When someone cannot be physically present at a meeting (typically external stakeholders abroad), the development teams use an online conferencing system with voice communication and screen sharing for live demonstrations of the software. Feedback from involved stakeholders is provided primarily as direct oral feedback, not as written communiqués, thereby improving efficiency (Bosch, 2001).
Engagement of customers: Another major step taken to enable the agile software
product line engineering process is the engagement of customers in both planning and development of the product line. Interacting with customers is an obvious but nevertheless critical factor for success in any type of software engineering (Reel, 1999), software product line engineering included. From a purely business perspective, collaboration with actors outside the organization is considered to be important for creating and sustaining competitive advantage (Sawhney, Verona et al., 2005). The objective of interacting with the customer is to define short and long term requirements, to gain access to domain and business knowledge, to verify and to validate results, and even in some cases to involve the customers in the innovation process (von Hippel, 1996). According to Damodarran (1996), collaborating wisely may yield several benefits for software development, such as improved system quality due to more accurate user requirements, the avoidance of costly system features that the user does not want or cannot use, improved levels of end-user acceptance of the system, and better understanding of the system by the user resulting in more effective use.
Discussion and implications
- 78 -
In contract-regulated development, it is obvious who the customer is and what the conditions for collaboration are. In SPLE, this is not evident at all due to the large number of customers. At CSoft, the multiple links between internal and external stakeholders play an important role. Members of the PSG interact frequently with existing customers, prospective customers, and third parties such as producers of related products and services. Others work in technical account management, support services, and consulting services, and assist customers in using the products. In addition, there is the annual Advisory Board meeting, several dedicated meetings with specific customers, and product conferences – events enabling customer communication. All in all, this means that CSoft has wide and rich communication with direct and indirect users of their product line. In some cases, the relationship with given customers is very informal.
Acquiring information from the customer base and other important actors, and maintaining relationships, takes a lot of resources. This however pays off because it has become a valuable source of knowledge for the company. Communication is mainly direct and verbal (usually over the telephone), not on formal documentation (indirect). This is what Keil and Carmel (1995) define as direct customer-developer links. They show that direct communication between the supplier and a participating customer reduces problems caused by filtering and distortion of information. The more links that are used, the better the communication becomes, up to a certain point. In short, the value of this direct communication is twofold: (i) it is simpler, and (ii) it improves the quality of the communication. The claimed value of direct communication is supported by the media richness theory (MRT) (Daft, Lengel et al., 1987), which ranks several media according to “richness” in communication (face-to-face being the richest, written documents the poorest).
Although valuable, we have also seen that customer relationships might be fragile, and there is a clear need to manage the multiple inputs received from, and the expectations of, numerous external actors. This concern, that the customer role can be very demanding have been pointed out in recent research on the customer role in ASD. For example, Martin (2009) did a series of case studies of XP-projects and found that participating customers were experiencing fatigue over time, threatening sustainability of the customer role.
One important aspect of a customer’s motivation to invest time and resources is the ability to affect the course of development. However, input, requirements, and ideas may sometimes conflict with other concerns of the product line, which means that the PSG need to consider the input carefully, and that the final decision on which changes should be implemented in the product line must be an internal matter. In some cases, this may be difficult and there is an evident risk of losing the engagement of a customer if the response to his input is considered too weak or insignificant. As an example, when
Discussion and implications
- 79 -
we studied the initial adoption of Evo at CSoft, we observed that one customer (stakeholder) withdrew from its allocated project because the goals had been reached successfully (Fægri and Hanssen, 2007) (p. 102). However, this is an inevitable price that the development organization has to pay. Users’ involvement in the development process may vary from informative, through consultative, to participative (Damodarran (Damodaran, 1996)). CSoft users may be considered to be informative for planning and consultative for development, but never truly participative with respect to decision-making. One risk of not giving customers full participative powers is that the development may “…fail to reflect real human and organizational needs” (ibid.).
The development iterations are short and the increments are small and focused, thus driving what may be called incremental innovations as described by e.g. Lettl (2007). In the long term, focusing on incremental innovations at the cost of radical innovations can be a serious threat to the product line and consequently the business. Böckle (2005) refers to this problem as innovation lock-in, and points out that this can be a barrier in software product line engineering, because components and variability are predefined. This problem is recognized within CSoft, but no changes are planned as yet to address it. It is likely that if they were given the opportunity, CSoft customers could play an active role in radical innovations. Lettl’s study (2007) gives examples of capable and knowledgeable end-users that act as technology inventors. However, doing so would probably require an extension (and complication) of the present development process.
When discussing ASD and customer engagement we also have to address one of the most fundamental practices in ASD, which is the ‘customer on-site’ practice, an important part of XP (Beck and Andres, 2004). According to a systematic literature review by Dybå and Dingsøyr (2008), this is also one of the most researched practices within ASD. Examples are (Hanssen and Fægri, 2006; Hansson, Dittrich et al., 2004; Korkala, Abrahamsson et al., 2006; Korkala, Pikkarainen et al., 2009; Koskela and Abrahamsson, 2004; Martin, Biddle et al., 2004; Martin, 2009). With respect to the study of CSoft it is important to notice that the principle of customer on-site is relevant to smaller and simpler development projects with a single customer or a small group of customers, which is the initial home ground of most agile methods, at least XP. However, in cases where a software organization serves a large market with a high number of customers (like CSoft) this approach to customer interaction is not feasible due to the high number of customers, their diversity and their geographically distribution. Still, applying ASD in such a context needs to find ways to collaborate closely with these customers. Scaling up ASD has emerged as a sub-field of its own (Eckstein, 2004; Larman and Vodde, 2008; Leffingwell, 2007), but the understanding on how to establish efficient customer interaction in large-scale development is still only nascent. We believe that the study of CSoft may contribute to some deeper understanding of customer engagement in large-scale product line development.
Discussion and implications
- 80 -
5.1.2 Effects of the combined process
Risk reduction: Developing and improving a software product line may carry with it a set of risks that need to be managed. McGregor (2008) describes a set of risks that are inherent in SPLE, depending on whether a proactive or reactive strategy is applied. With a proactive strategy, where assets are predeveloped with the assumption that they will be used in future applications, there is a risk that these assets may become obsolete, thereby becoming a lost investment of resources. With a reactive strategy where assets are harvested from applications for later reuse, there is a risk that short-term business opportunities will be missed because the production of applications is not as fast as it could have been. There is also a risk that a lot of rework will be needed to prepare such assets for future and more generalized reuse. CSoft’s approach for managing their product line is best defined as incremental. Further it constitutes a compromise between the proactive and reactive approaches, which reduces the risks mentioned above. Given that all development is aimed towards the next release a year ahead with external stakeholders involved in each increment, there is a low risk of developing features that will not be used. This is actually a central motivation for using the Evo process (Gilb, 2005), which is driven by short-term goals. Another observed effect of the agile approach to requirements management is that frequency of interaction and closeness to customers increase the ability to capture both explicit and implicit (tacit) requirements (Grunbacher and Briggs, 2001). An example of this, from an iteration review meeting with one of the development teams and a customer team, is when the product manager demonstrates a new feature and says “we’re al anxious to see what happens”. The customer breaks in “this is nice, because in the current version you have to do this manually”. This initiated more detailed discussions between the team leader and the customer.
Organizational development: Over a period of 13 years, CSoft has gone through
roughly three phases of organizational development, as described in more detail in section 2.2.4. The introduction of ASD (Evo) has been a vital factor for re-establishing efficiency and the ability to respond to the market (Fægri and Hanssen, 2007). This adds to the findings of a study of key business factors in SPLE by Ahmed and Capretz (2006), who conclude that an SPLE organization needs to understand that the customers’ business process supports the product line, which in turn supports the business process. This mutual support is one of the important benefits of interacting closely with customers. The third-party community of related companies that base their business on the CSoft product line contributes greatly to the development of the organization because this symbiotic relationship helps CSoft to maintain a strict focus on the continuous development of the product line, leaving specialized ways of use and extensions to others.
Discussion and implications
- 81 -
Reduced maintainability: The present situation at CSoft has emerged over several
releases as the organization made it a priority to serve the market rapidly with new and improved features and functional qualities at the expense of internal “tidiness”. This may have been a wise strategy to establish a position as the present market leader, however, this more or less deliberate strategy for the shaping of the organization and the product line has come at a cost of escalating system entropy, which makes it more and more difficult to improve the product line. Through a recent study of the maintainability of CSoft product line architecture (Hanssen, Yamashita et al., 2010; Hanssen, Yamshita et al., 2009) we found that the R&D department experiences severe problems with respect to maintainability of the product line related to the agile development process. The two-week Evo iterations means that the development teams needs to focus on short-term goals, at the expense of the overall structuring of the product line. It is very difficult for even experienced developers to understand the inner structure and workings of the core parts of the product line. We found that the CSoft product line has a high complexity, which negatively affects the developer’s analyzability and comprehensibility, which again leads to, reduced modifiability and deployability of the software. We also found that the system is hard to test and that developers actually may display a fear of changing the code because it is hard to see effects of changes. To deal with the complexity, each development team needs a team leader that is highly skilled in one of the modules in the product line. Loosing one of these experts represents a great vulnerability. All in all we see that the agile approach to evolving the product line makes these problems permanent.
Community building: The products and services from the third-party community,
which is supported by CSoft through the API and by exposing their partners, represents in total a large variability and flexibility in the CSoft product line, far larger than CSoft would have been able to develop and manage by themselves. As a product line grows in size and complexity, it becomes a platform that opens up opportunities for the development of related products and services and, which represents a business potential but also present a large challenge for the organization (Cusumano, Kahl et al., 2006). Opening up the technical interface has earlier shown to support the growth of an external community, for example IBM’s opening of the PC architecture. This enabled the establishment and growth of a community (Moore, 1993), but also the growth of competitors.
The existence of the third-party community supports CSoft establishment of their agile product line organization in two respects. Firstly, it allows CSoft to maintain the maximum focus on the development and progression of the core product line (Zook, 2010). This is important - if the level of complexity were to become too high, their
Discussion and implications
- 82 -
ability to improve the product line would be seriously impeded. Secondly, the third-party community increases the actual value of the product line to the customers, growing sales, and thus increasing the probability of a self-funding and economically sustainable SPLE process. In order to enable the third-party community to direct their efforts and development appropriately, CSoft share their high-level plans openly and, in some cases, invite third parties to planning and development meetings, just as they do with their customers.
This nearly symbiotic relationship between the third-party community and CSoft, as a SPLE organization and a provider of a partially open platform, can be described as a software ecosystem (Bosch, 2009), which is an emerging concept within software engineering and a potentially important shift in how the industry relates to other actors in the market. The term “ecosystem” is borrowed from biology (just like the term “evolution” is), and can be defined as a unit of mutually dependent organisms co-existing in the same habitat. Here, it is used as an analogy to the software industry where organizations are organisms and the marketplace is the habitat. Bosch (ibid.) discusses this concept in relation to SPLE and comments that “…enabling a software ecosystem causes processes optimized for intra-organizational purposes to no longer work” (p. 177). He calls for a change from a centralized approach to a decentralized one, in which individual and partially self-managed teams are able to develop their solutions without having to coordinate with a central organization. This is what happened at CSoft, as part of the adoption of agile development practices. It is reasonable to believe that Evo, with participation from actors outside the organization and the self-managed component teams, has been an enabling factor in the establishment of a viable software ecosystem.
Openness and visibility: CSoft has developed into a partially open organization,
making their plans and processes available, both internally and in part externally. They also promote frequent communication across various levels and roles in the organization, not necessarily following a regular pattern from case to case. The work is driven by the PSG, but is best seen as a collective effort with contributions being welcomed from practically all parts of the organization. Besides creating a common responsibility for and ownership of the process, this modus operandi also opens up the planning process to the influx of ideas from external sources, and helps to balance the concerns of various stakeholders. This openness is also very much in evidence outside the company, for example, at seminars and product conferences. Long-term visions and ideas, and more detailed and short-term plans, are communicated to a mix of new and potential customers as well as third parties. These events open up a dialogue with external actors that may have complementary needs or ideas and even corrective feedback, which may become valuable input to the planning process.
Discussion and implications
- 83 -
This way of interacting with customers and third parties resembles what Chesbrough and others have termed open innovation (Chesbrough, 2003; Enkel, Gassman et al., 2009). Open innovation is, as the name implies, an approach in which innovation takes place partly across organizational boundaries, rather than within them. Such boundary-crossing processes of innovation may take many forms. Open source software development is an example of open innovation. As Chesbrough (2003) says, “…open innovation is based on a landscape of abundant knowledge, which must be used readily if it is to provide value for the company that created it.”. In the case of CSoft, the rapid development cycles and frequent corrections from invited stakeholders enable the development organization to achieve this readiness.
Enkel et al. (2009) presents open innovation as consisting of one of three core processes. The first, which is called the outside-in process, enriches the company’s own knowledge base through the integration of supplier, customers, and external knowledge sourcing. The second is called the inside-out process, in which ideas are brought to the market, which may generate new streams of income through spin-offs or joint ventures. The third, which is called the coupled process, combines the first two and best describes CSoft’s approach. This openness across organizational borders and the ability to manage both internal and external innovation processes may constitute a clear competitive advantage (von Hippel, 2005). In the case of CSoft, it has enabled the organization to improve the processes by which their software product line continues to develop.
Company culture: Since its incorporation about 13 years ago, CSoft has grown
considerably. Despite its growth in size and complexity, CSoft has maintained an interest in taking opportunities to improve, and has retained the willingness to change. For example, prior to the introduction of Evo, the situation became untenable because of the development process rigidity. The decision to dramatically change the development process was made very swiftly and took effect within weeks (Fægri and Hanssen, 2007). This is an example of the company culture, which is, to a large degree, aware of the company’s weaknesses and open to new approaches. Although inspired by the latest trends in the software industry, the organization has managed to maintain a rather sober attitude, trying to consider what may and may not be relevant to it in practice. Hughes refers to this as “enthusiastic problem solving and dedicated system building” without being constrained by disciplinary and knowledge boundaries (Hughes, 1989). At CSoft, this culture has enabled the organization to adopt their agile software product line approach. Studying standard descriptions of ASD or SPLE may give the impression that these two approaches are irreconcilable. However, by selecting process components from each, without being constrained by what may be called the disciplinary rules of
Discussion and implications
- 84 -
each approach, CSoft have managed to apply a simple, yet seemingly powerful combination.
5.1.3 Contextual factors
Domain volatility: One assumption about SPLE organizations and their target domains is that the detailed practice of product line engineering as described by e.g. the Software Engineering Institute (SEI) is suitable where the domain is fairly stable (McGregor, 2008). Serving a domain that does not change too much, means that there is a lower risk in pre-developing assets for later (re)use in application engineering. In contrast, the domain that CSoft serves is unstable, because its boundaries are still being determined. New technologies and business ideas emerge frequently; hence, it is more important to be able to respond quickly to the market than to produce reusable components that might be useful at some later time, but are not sure to be.
The software-as-a-service delivery model: Early in the development of the CSoft
product line, the software-as-a-service (SAAS) deployment model (Dubey and Wagle, 2007) was adopted. When a new release is ready, all parts of the product line are released simultaneously and are deployed on a server farm. This allows customers to use the new release instantly through an ordinary web browser. This model ensures simple release of new versions of the product line, offers an easy upgrade path for the customers, and provides opportunities for reducing the costs since no local operational infrastructure and services are needed. In addition, the model is advantageous when external stakeholders are providing feedback on the recent increment – which is released on a test server, as SAAS.
Discussion and implications
- 85 -
5.2 An emerging software ecosystem
Building on the understanding of how the combined processes work in the product line organization and the changes this has lead to, we continue the discussion by looking into how this has enabled a change from software product line engineering towards a situation which eventually can be described as a software ecosystem. We summarize this part of the discussion by proposing a model of a software ecosystem.
5.2.1 Changing from agile software product line engineering to a software ecosystem
The initial motivation for changing the waterfall-like development process by adopting Evo in 2004/2005 was that CSoft struggled with unstable requirements, incurring high costs due to little flexibility in the process. Much emphasis was given to extensive and thorough requirements engineering upfront, but with little effect (Hanssen and Fægri, 2006). The immediate experience from involving stakeholders in the short Evo development iterations was that developers felt more comfortable and secure by having this close and continuous dialogue on requirements and results (ibid.). However, in the first release projects using Evo, it became a considerable challenge to maintain the motivation of the external stakeholders throughout the project. As we have seen, the new process was fragile (Fægri and Hanssen, 2007).
(Change 1 – engaging customers) From the more recent study (Hanssen and Fægri, 2008) we see that this has clearly changed; now external stakeholders are keen to participate – CSoft actually have to turn down stakeholder candidates. This change is the result of a learning process that took place during the first years of using Evo – customers have gotten to know of this practice and some gained experience as stakeholders. The engagement of customers and users is generally considered to be an important success factor in any kind of software development (Chiasson and Green, 2007; Keil and Carmel, 1995).
(Change 2 – learning by doing) We can also observe another change that took place internally at CSoft. The first experiment with Evo was done as an R&D-internal matter, like a kitchen experiment. However, as this turned out to be an improvement of the development practice, this way of working eventually became adopted by the rest of the organization. Now, all parts of the organization, from operational support to the top management, are supporting this practice. An example is the CEO explaining the software development process Evo and its strategic importance in his keynote at a large product conference. Another example is the strengthening of the PSG, which has a liaison function between customers and development teams. This tells us that changing a product line organization takes effort and time, and that both internal and external
Discussion and implications
- 86 -
actors need to learn from practice to accept this opening of the organization and its work processes.
(Change 3 – increasing visibility) Another change we can see from the results is an increasingly higher external visibility of plans and strategies. Initially this kind of information was kept confidential, but it is now more and more openly communicated through various channels such as product conferences, at ad-hoc meetings with customers and in development projects. It has turned out that doing so does not introduce the presumed risk of leaking vital information to competitors, but that it is rather an advantage. As external actors see what might be coming, they can relate it to their own business, and potentially respond to it.
(Change 4 – increasing extensibility) Another related change is the opening of the product line at the technical level with the APIs. Initially this represented a minimal and very limited opportunity for extending the product line, but it quickly grew to a considerable extent as it represented a tangible business value. This aspect has eventually been given more attention, and has been designated as strategically important in some of the roadmaps. We see several benefits from allowing externals to use the product line as a platform. Firstly, it increases the variability of the product line – it can be used in more specialized ways, serving more needs. Secondly, existing users represent a great opportunity to the third parties (being the second component of a symbiosis-like relationship). Thirdly, letting externals deal with specialization and minor extensions enables the product line organization itself to maintain focus on developing the core product line (Zook, 2010). This may be the most important effect.
5.2.2 A case of improvisational change
As the four studies of CSoft show, the organization has undergone an extensive change over the time under study. Using the theoretical background on organizational change from section 2.1.5 as a lens, we see that CSoft is not a case of episodic change as described in Lewin’s model (1951). The organization was not “unfrozen” in terms of removing inertia and anxiety of change. Rather, the state of the organization had stretched to a limit where the “pain” (poor process performance, low motivation, and code errors) was so high that nearly any change was perceived as better than none. One of the product managers even stated laconically: “Whatever changes we implemented, it would probably be a good thing.” Thus, the motivation to change had grown in parallel with, and as a consequence of, the increasingly bad performance. The decision to try Evo can be seen as a case of opportunity-based change (Orlikowski and Hofman, 1997) as the decision was made quite abruptly, based on the newly acquired knowledge about Evo and iterative and incremental development.
Following CSoft through this period of time (Fægri and Hanssen, 2007) we observed several iterations of the Deming/Shewhart improvement cycle. At first only a few
Discussion and implications
- 87 -
aspects of the new process were applied and only in one project. Then, having verified that iterative work was applicable, external stakeholders were invited in the next cycle.
The introduction and adoption of Evo can be seen as a large opportunity-based change, taking several cycles to implement. From our continued study of CSoft we have also seen examples of other types of change. For example, we saw that the iteration length was increased from one to two weeks (in release 9.1 - the third release where Evo was used (ibid., p.101)), and we later observed the introduction of the green-week concept (an week dedicated to error correction and stabilization of the system). These may be seen as cases of emergent change (Orlikowski and Hofman, 1997).
All in all – we see CSoft and the development they have undergone over the past years as an example of improvisational change. Further, we believe that this case also exemplifies that a low level of formalization of processes (self-managed teams, little process documentation, no domain models) and the flat organizational structure of this company makes improvisational change easier to implement. As Dybå (2000) says: “Too much reliance on previously learned patterns tends to limit the explorative behavior necessary for improvisation.” (p. 83).
5.2.3 A conceptual model of a software ecosystem
The study of CSoft, as a supplier of a product line, and their collaboration with external actors leads us to develop a conceptual model of a software ecosystem of the web/application type (Bosch, 2009).
CSoft has changed their role to become a central actor in a network of organizations having an interest in the product line. Customers and third parties form the external environment and can participate in both strategy making and actual development projects. Likewise, third parties can also collaborate with the supplier, as well as its customers through offering value-adding products and/or services. The product line is the central asset that collaborations and interactions are based upon. This networked organization is not deliberately created – it has developed through symbiotic relationships, meaning that each actor in some way benefits from the relationship with the others. These benefits must be in place in order to make the ecosystem work as a whole.
Discussion and implications
- 88 -
Table 9 - Values from being a network actor
The supplier has the central role and controls the development of the ecosystem to a
large extent by controlling the flow of information and involvement of other participants. This controlling role is called the keystone advantage by Iansiti & Levien (2004). In the type of software ecosystem being described here, this lack of “democratic” influence is compensated by the deliberate support from the product line supplier. External actors are supported through information about strategies and plans and – for a few selected external actors – through direct involvement in both planning and development.
A fundamental aspect of an ecosystem is the opportunity the various actors have to learn about the product line, its development, and its use. Also, talking directly to customers, allows for learning about the domain. On the other hand, customers and third parties learn about the ongoing development. Such proximity are found to be important to balance long-term strategic objectives of SPLE and the short-term tactical objectives of ASD, and to make the development organization adaptive with respect to its external environment (Mohan, Ramesh et al., 2010).
However, there is also a reason to be concerned with the effects of increased customer proximity and iterative development. As we have seen from CSoft (Hanssen, Yamashita et al., 2010) this can aggravate software entropy due to the strong focus on low-level product qualities, at the cost of high-level qualities such as for example system architecture.
Developing a product line within such an ecosystem has the potential of producing some beneficial effects. By having external organizations dealing with minor extensions or specialized variants, the supplier can maintain focus on developing the core (Zook, 2010). Also, having externals to build on the product line, using it as a platform, increases the diversity and potential value of the product line and increases its variability. Collaborating with deliberately selected external actors enables open innovation (Chesbrough, 2003), both in the long-term in strategic planning and in the
Discussion and implications
- 89 -
short-term. We interpret CSoft’s ecosystem and its evolutionary development process as a variant of Chesbrough’s open innovation model (ibid., p.xxv):
Figure 14 - Open innovation
The borders of the organization are permeable in the sense that external organizations
influence and even contribute to research (strategic planning and road-mapping) and development (agile development projects). New releases enter the market, which is extended by external actors who run their own research and development processes, building on and extending the product line.
Another aspect of the ecosystem is how it relates to knowledge creation and dissemination. Compared with a closed approach (CSoft prior to the adoption of Evo and the engagement of external stakeholders) an ecosystem enables new spaces for creation of knowledge. We have described some of these, for example the advisory board, ad-hoc meetings with customers, and the participation of stakeholders in the development projects. Nonaka and Konno (2008) refer to such spaces as “ba” - Japanese for space. “Ba” can be physical (e.g. office space), virtual (e.g. e-mail conversations), mental (e.g. ideas), or any combination of the above. Knowledge creation is a spiraling process of interactions between explicit and tacit knowledge. Nonaka’s SECI9-model describes four types of knowledge transition (ibid., p.43). Socialization is the transfer of tacit knowledge between individuals. Externalization is the transition of tacit knowledge to explicit knowledge, which can be understood by others indirectly. Combination is the construction of more complex sets of explicit knowledge. Internalization is the transition of explicit knowledge into the organization’s collective tacit knowledge. Applying these principles to the study of CSoft, we see that the establishment of the ecosystem, with its spaces/ba of contact
9 Socialization, Externalization, Combination, Internalization
Discussion and implications
- 90 -
between internals and externals enables knowledge creation through such transitions. For example, frequent meetings between teams of developers and external stakeholders (as close to face-to-face as practically and technically possible) enable transition of tacit knowledge (socialization). During the iteration evaluation meetings stakeholders can explain directly to the development team how they work and how they use the software product under development. This demonstrates the effect of the agile components in the SPLE process where direct and frequent communication enables knowledge creation.
To wrap up this discussion, we propose a simple conceptual model of a software ecosystem. This model represents the case we have studied, and could serve as a basis to reflect on and guide other cases.
Figure 15 - A conceptual model of a software ecosystem
The model illustrates the main actors in a software ecosystem. The supplier
(keystone organization) develops the product line. This development is guided by a strategy, which points out needs and opportunities and main paths of development. Both the strategy and the development of the product line are to some extent visible to external actors. These consist of (at least) customers and third parties, but can also involve others. Third parties use the product line as a platform to serve customers with additional solutions and services. Being a part of an ecosystem means that these actors learn about each other. The supplier learns about requirements, needs, ideas, opportunities etc. In return, external actors learn about the development of the
Discussion and implications
- 91 -
technology of common interest (the product line), and may even participate actively in the development.
5.3 Implications for theory
We believe that software ecosystems have the potential of becoming an important field of practice and research in the years to come. As shown in section 2.1.4, the interest is growing and some work has been done, but the overall impression is that this concept is yet poorly developed and understood. As a contribution to future development of this concept, we propose to form a theoretical platform. Just like the taxonomy suggested by Bosch (2009), a theory of software ecosystems, as suggested in this section, can become valuable and useful to generalize the concept and bring together results from more empirical studies. It may over time develop towards a unified and empirically justified understanding of the concept (Sjøberg, Dybå et al., 2007), being useful to both research and practice.
The theory of organizational ecology (Trist, 1977) which was briefly presented in the
background section (2.3), are used a framework for the following reasons: -‐ It builds on the concept of socio-technical systems, which covers the interaction
between humans and technology. ‘Technology’ also covers organizational structures, processes, techniques etc. This is a useful concept in order to understand the CSoft case, which is a social system, a technical system and the interaction between these. For example, the people in this organization collaborate internally to continuously develop a software product line, the technology. The other way around, this technology clearly affects how these people develop the technology, for example illustrated by the problems related to the system entropy.
-‐ It defines the concept of the external environment. This has become a very important aspect for CSoft as they have moved from a closed water-fall like approach with little emphasis on, and contact with, external actors to the present practice where customers and third parties are closely involved in planning and development, supported by ASD principles such as short iterations, incremental development and frequent feedback through interaction with external actors.
-‐ It discusses types of organizational environments, ordered according to complexity whereof the most complex type and most recent in development, called turbulent fields describes CSoft’s environment. CSoft operates in a turbulent domain where their product line is constantly changing, the development organization is changing, as well as the technology they use and the business domain they serve. Trist says “..with the increasing salience of turbulent conditions, systems of aggregate control
Discussion and implications
- 92 -
are becoming increasingly insufficient and inefficient, no matter how large the outgrowths.” (ibid., p.169).
Related to the study of CSoft reported in this thesis, we use Trist’s theory as a framework and derive a set of theoretical propositions suitable for describing software ecosystems. This means that we have not done a 1:1 adoption of concepts from Trist’s description, but used it as a basis for our own. Sjøberg et al. (2007) explains three ways in which theories are built, whereof the second type is applied here: “Theories from other disciplines may be adapted to software engineering before use.” (p. 5). This means here that the theory of organizational ecology identifies concepts, which we have found to be relevant to the field of software engineering. As a mean to describe this theory we have looked to the guidelines and a five-step approach provided by Sjøberg et al. (ibid., p.13) and thus describe it as a set of propositions, which explains how basic constructs relate to each other (step 1 and 2). Five propositions have been derived from Trist’s work and adapted to software engineering. In addition, we also propose two new ones, based on the study of CSoft. Each of the seven propositions is supported by explanations from our case study to justify the theory (step 3). Finally we discuss the scope of the theory (step 4). Step 5 is to test the theory through empirical research and will be left to later work. Later empirical studies may extend, change, confirm or criticize the theory, as presented here.
1. Member organizations in a software ecosystem are linked to a key organization
among them, which acts as a central referent organization, doing so even though many of the members are only partially under its control or linked to it only through interface relations. (This is a Class 1 system according to Trist’s classification (1977) ( p. 165))
CSoft is an example of a referent organization in a software ecosystem, where other members are their customers, and third party organizations that use CSoft’s technology as a platform for providing related products and/or services. None of these external organizations are formally controlled by CSoft. However, all activity in the ecosystem is related to the product line, which is owned, developed and thus controlled by CSoft. These relationships resemble those in open source initiatives, however with the distinction that the software is closed and owned by one actor10. The interface relations between the referent organization and other members are nurtured by ASD practices such as customer engagement in the planning of iterations and the evaluation of the outcome and by the open communication of product strategies, for example on product conferences or on an ad-hoc basis.
10 Future research on open source ecosystems may consider Trist’s second class of socio-
ecological organization as an explanatory model.
Discussion and implications
- 93 -
2. Software ecosystems promote self-regulation.
Our study of CSoft shows that the collaborative approach can be seen as a self-regulating system in that the referent organization to a large degree adapts to its external environment, and that the external environment adapts to the referent organization. The main enabler for this is the close and frequent co-operation between CSoft and externals, which takes place in the development iterations. Demonstration of results, feedback and correction takes place in each iteration. This stands in contrast to the previously centralized control that was applied in the development of the product line where feedback was rare.
3. Software ecosystems have a networked character.
CSoft and its external environment constitute an open-ended network of customers and third party organizations, where all relate to the referent organization but with the additional opportunity of relationships between member organizations outside the direct control of the referent organization. For example, some of CSoft’s customers are also customers of third part organizations, using related services or extensions or specializations of the product line. The referent organization benefits from this network and in the case of CSoft also support it actively, for example by marketing third parties and by offering technological interfaces enabling these third-party operations.
4. Software ecosystems exist through the use of information and communication technology (ICT).
The ecosystem, which CSoft is a part of, relies considerably on the use of ICT to enable the collaboration with its external environment. Examples are the web-meetings with external stakeholders in the development iterations. This enables collaboration with actors at different locations than the development teams. We have also seen that the software-as-a-service deployment model makes distribution of increments easy and efficient, which enables external stakeholders to test increments and provide feedback.
5. Software ecosystems exhibit shared values.
In the CSoft ecosystem the software (product line) is a shared value. For CSoft, the value is revenue from licenses and services, for the customers the value is improved business operations, and for the third parties the value is revenue from sales of value-adding services and solutions (see Table 9). This common interest creates a motivation to care collaboratively for the shared value. For example, since the
Discussion and implications
- 94 -
product line has become an inevitable part of the business of the customers, they are willing to invest time to participate in the development of this value.
These five propositions constitute a foundation of a theory for software ecosystems. In addition to these principles adopted from Trist’s work (ibid.) we also propose two extensions:
6. The shared value of a software ecosystem is both the software product and the
business domain.
The focus of the referent organization is not restricted to the product line only. It also covers the business domain that is supported/enabled by the product line. The better the product line is at supporting the business of customers and third parties, the stronger the incentive to use the product line becomes as well as the incentive to participate in the continuous development of it. The continuous dialogue between the referent organization and external stakeholders, in each iteration in the development projects, addresses both how to develop or improve the product line as well as how the product line supports the business of the stakeholders. We see this as a case of open innovation.
7. As a software ecosystem emerges, control moves from the supplier of the software to its users.
Prior to the adoption of Evo, the only guiding input from the customers was through formal requirements documents. Feedback came late, very close to release. CSoft controlled all aspects of the development. However, this was not a beneficial situation due to low input from customers. Now, some of this control has moved to (participating) external stakeholders that to a larger extent control the development of the product line. Yet, CSoft still owns the product line and controls the main lines of development, but now in combination with vital input from external actors. Here lies a clear distinction from open source initiatives.
The fourth step of building a software engineering theory (Sjøberg, Dybå et al., 2007) is to determine the scope. The propositions given here are based on the study of a single software product line organization (and inspired by a general theory on organizational ecology). With this in mind, we identify the following limits of the scope:
-‐ The proposed theory concerns software product line organizations similar to CSoft,
which emphasize iterative and incremental development and participation by externals in strategy making and in development.
Discussion and implications
- 95 -
-‐ We believe that this theory is relevant to software organizations serving business domains that is still shaping and not mature domains.
-‐ We believe that this theory is relevant to software organizations that develop products that are vital to the business performance of the customers and where the use of the product is long-term.
5.4 Implications for practice
Based on our combined study of CSoft, our overview of related research and literature, and our discussions we propose a set of implications for practice. We shape these implications as advice or guidelines for software organizations similar to CSoft – or – software organizations that consider a change towards agile software product line engineering and software ecosystems.
-‐ Support the external environment by being open. Sharing information on plans,
strategies, and development may create a fundament for collaboration and new patterns of innovation. If appropriate, encouraging and supporting a third party community can be a valuable extension to the normal development of the product line.
-‐ Reducing the variability in the product line consequently reduces the costs of maintaining and managing variability. The potential drawback of reduced variability can be compensated through offering interfaces for extending the product line by externals, and thus increasing the overall variability.
-‐ Establishing and benefiting from a software ecosystem takes time. A successful development relies on repeated cycles of experimentation and learning. This learning process needs to involve all types of actors.
5.5 Limitations
The case study of CSoft is subject to four limitations. (1) This is a single case study, which naturally affects the generalizability of the
conclusions. Yet there are good reasons for choosing such an approach. First of all, the number of relevant cases is still low. Nevertheless, we have had the opportunity to study a case, which we believe is both advanced and relevant to other parts of the software industry. In addition, focusing on a single case means that the study can be more thorough than a study of multiple cases, with respect to available resources. Yin (2002) discusses the single case study design (p. 38-41) and presents several arguments in favor of choosing such a design. One of these is particularly applicable to CSoft, namely
Discussion and implications
- 96 -
that it is a unique case. According to Yin, such a study may act as a prelude to further studies of a relatively new topic, such as software ecosystems in this case.
(2) A large part of the data that has been collected comes from group and single-respondent interviews with internals and externals. This type of data can potentially be biased, incomplete or even wrongful due to misunderstandings, lack of insight etc. We have sought to address this threat to validity by collecting data from various respondents and supplementing these data with documents and observations.
(3) The third limitation concerns the completeness of the study. Only a subset of the employees was contacted. Likewise, relatively few samples of all available documentation were collected and analyzed. This is due to natural limitations such as limited time and resources.
(4) The fourth limitation concerns the applicability of the findings and conclusions of this study. The organization investigated is a medium-size product line organization and a web/application type of ecosystem (according to the taxonomy proposed by Bosch (2009)). Thus, results do not necessarily apply to all other types of software ecosystems.
Conclusions
- 97 -
6 Conclusions and further work The overall goal of this study has been to understand the need for and implications of
a more open approach to software product line engineering. To establish this understanding, we have followed a growing software product line organization and the changes they have undergone over a period of approximately six years.
This longitudinal study of CSoft and their development began by investigating the abrupt change of the development process through the adoption of some key agile principles in the inflexible development process. The main focus of study 1 was how the development organization changed the way they related to their customers. The insight into this change and the results of the study led to the collection of new data used in studies 2 and 3, focusing on agile software product line engineering and software entropy, respectively. Through these studies, we became aware of how the relationships with external actors were changing, and we initiated the collection of more data to investigate in more detail how this organization developed their relationship to external actors, leading to study 4.
This brief retrospect shows that the focus and the motivation of the studies have shaped and developed during the course of the study of CSoft. This means that the research questions have also developed in this way.
6.1 Answering the research questions
RQ1: How can software product line engineering and agile software development be combined?
Firstly, we have shown that combining the plan-driven product line engineering
approach with an incremental and iterative agile development process is practically feasible. Developing and evolving a product line requires long term planning, which results in high-level plans. These are used to initiate shorter-term agile development projects. We have described the process combination as a sum of three processes with varying time-horizons. Long-term planning and activities related to the evolution of the product line are organized within a strategic process. Development is organized as a tactical process. These two interact with the operational process, encompassing day-to-day functions. Active engagement of and collaboration with external stakeholders is the driving force in this process combination. We have found several desirable effects of collaborating with externals such as increased motivation among developers as well as faster and more frequent feedback. Continuous collaboration with externals requires
Conclusions
- 98 -
continuous management of the relationship to maintain motivation and to replace stakeholders if needed. Frequent feedback and re-planning from short iterations come with the risk of architectural erosion and increased system entropy. We have seen the need to compensate potential erosion by a dedicated function (the architecture team), which ensures the maintenance and development of the overall architecture of the product line. We have also observed process fragility caused by the increased frequency of iterative development. An agile development process within a product line organization requires a high level of process discipline, and small deviations may cause process instability. We have observed such fragility when external stakeholders failed to fulfill their role. Also, we have demonstrated that frequent collaboration with external stakeholders in strategic, tactical, and operational processes creates additional overhead as compared to a traditional plan-driven product line approach.
RQ2: How does a software ecosystem shape?
Based on the answer to RQ 1, we have further investigated how the process
combination led towards a situation that we describe as a software ecosystem. Fundamental to this change is the process of opening up. We have shown how the case organization moved towards a software ecosystem by 1) opening the information flow internally and externally, 2) opening the innovation processes by collaborating with external actors at various levels, and 3) opening the technical interfaces, enabling external actors to use the product line as a platform for additional services and solutions. The transition to an ecosystem has come along with a change from a traditional closed systems mindset to an open systems mindset. We have shown how actors in the external environment contribute to both long-term strategic planning and to short-term tactical development. Externals can be both customers/end-users and third parties. We have also observed how the product line organization actively supports a community of external actors and the benefits that come from such collaboration. Examples of such benefits are: increased focus on the development of the core product line, increased variability of the product line, and business opportunities for third parties.
The change from a closed product line approach to a partly open ecosystem happened as an improvisational change process. It did not happen through the implementation of a plan of change but began with careful testing of some core principles in practice. Based on experience, new changes where introduced, forming a continuous process improvement initiative.
To foster further studies of the emergence and functions of software ecosystems we proposed a theory of software ecosystems, derived from socio-technical theory and organizational ecology.
Conclusions
- 99 -
Finally, we revisit our overall research objective:
Understanding the need for and implications of a more open approach to software product line engineering.
The software product industry recognizes an increasing need to become more open
and responsive to its market. Software technology becomes an increasingly important part of most businesses, and the users of this technology expect a more influential role for themselves in its development (Messerschmitt and Szyperski, 2003; von Hippel, 2005). This attitude is reinforced by an increased opportunity to collaborate in terms of technology, tools, and agile processes.
The implications of this change are an increased proximity to the customers and a potential to better capture needs and opportunities in the market. The software product supplier gets a larger role in the shaping of both software products and services, and the business domain they support. Understanding, supporting, and exploiting software ecosystems become a competitive advantage.
6.2 Directions for future work
(1) As Jansen et al. also point out (2009), we need to see more empirical studies of various types of software ecosystems, how they develop, and the effects they produce. Such studies should naturally be focused towards the industry, and be longitudinal as well as exploratory.
(2) To build a common understanding of software ecosystems: how they shape, how they work, and what their effects are, we advise a further refining of a theory of software ecosystems, as the one proposed in section 5.3.
(3) The emergence of software ecosystems comes with new business models affecting intellectual property rights, economic models, competition etc. We need to see more dedicated studies of these issues to realize the potential of ecosystems.
(4) Software ecosystems are closely related to the more mature concept of open source software development. We need to better understand the similarities and the differences in order to transfer knowledge between these two related domains (Fitzgerald, 2006).
(5) The engine of a software ecosystem is the collaboration with external actors. We have showed some examples through our studies, but this is a broad topic that needs further investigation.
Conclusions
- 100 -
(6) The study of software ecosystems potentially relates to several disciplines such as business strategy, sociology, technology and innovation management, economy, and others. We have briefly touched a few of these and we see a need to investigate these links further.
(7) Software ecosystems affect the shape of control structures. We believe that control shifts from the supplier towards the users, a transition that needs to be better understood.
References
- 101 -
References Abrahamsson, P., Salo, O., Ronkainen, J. and Warsta, J. (2002). Agile software development methods - Review and analysis, VTT Electronics: 112. Adler, P. S. (2005). The Evolving Object of Software Development. Organization 12(3): 401-435. Ahmed, F. and Capretz, L. F. (2006). Managing the business of software product line: An empirical investigation of key business factors. Journal on Information and Software Technology 49(2): 194-208. Alspaugh, T. A., Hazeline, U. A. and Scacchi, W. (2009). The Role of Software Licenses in Open Architecture Ecosystems. In proceedings of First International Workshop on Software Ecosystems (IWSECO´09), Milan, Italy, September 27. 4-18. Arisholm, E., Gallis, H. G., Dybå, T. and Sjøberg, D. I. K. (2007). Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise. IEEE Transactions on Software Engineering 33(2): 65-86. Babar, M. A., Ihme, T. and Pikkarainen, M. (2009). An industrial case of exploiting product line architectures in agile software development. In proceedings of 13th International Software Product Line Conference, San Fransisco, California, 24-28 August. 171-179. Basili, V. R. (1996). The Role of Experimentation in Software Engineering: Past, Current, and Future. In proceedings of 18th International Conference on Software Engineering (ICSE´96), Berlin, Germany, March 25-29. 442-449. Baskerville, R., Ramesh, B., Levine, L., Pries-Heje, J. and Slaughter, S. (2003). Is "Internet-speed" software development different? IEEE Software 20(6): 70-77. Bayer, J., Gacek, C., Muthig, D. and Widen, T. (2000). PuLSE-I: Deriving Instances from a Product Line Infrastructure. In proceedings of Seventh IEEE International Conference and Workshop on the Engineering of Computer-Based Systems (ECBS 2000), Edinburgh, Scotland, 3-7 April. 237-245. Beck, K. (1999). Extreme programming explained: embrace change. Boston: Addison-Wesley Professional. Beck, K. and Andres, C. (2004). Extreme programming explained: embrace change (2nd Edition). Boston: Addison-Wesley Professional. Birk, A., Dingsøyr, T. and Stålhane, T. (2002). Postmortem: Never Leave a Project without It. IEEE Software 19(3): 43 - 45. Böckle, G. (2005). Innovation management for product line engineering organizations. In proceedings of 9th International Software Product Line Engineering Conference (SPLE'05), Rennes, France, 26-29 September. 124-134. Böckle, G., Clements, P., McGregor, J. D. and Muthig, D. (2004). Calculating ROI for software product lines. IEEE Software 21(3): 23-31. Boehm, B. (2006). A view of 20th and 21st century software engineering International conference on Software engineering (ICSE). Boehm, B. and Turner, R. (2004). Balancing Agility and Discipline - A Guide for the Perplexed. Boston: Addison-Wesley.
References
- 102 -
Bosch, J. (2001). Software product lines: organizational alternatives. In proceedings of 23d International Conference on Software Engineering (ICSE'01), Toronto, Canada 12-19 May. 91-100. Bosch, J. (2009). From Software Product Lines to Software Ecosystems. In proceedings of 13th International Software Product Line Conference (SPLC'09), San Fransisco, USA, 24-28 August. 111-119. Bosch, J. and Bosch-Sijtsema, P. (2009). From integration to composition: On the impact of software product lines, global development and ecosystems. Journal on Systems and Software 83(1): 67-76. Bosch, J., Florijn, G., Greefhorst, D., Kuusela, J., Obbink, H. and K., P. (2001). Variability Issues in Software Product Lines. In proceedings of Fourth International Workshop on Product Family Engineering, Bilbao, Spain, 3-5 October. 1-9. Brown, A. W. and Booch, G. (2002). Reusing Open-Source Software and Practices: The Impact of Open-Source on Commercial Vendors. In proceedings of 7th International Conference on Software Reuse: Methods, Techniques, and Tools, Austin, USA, April 15-19. 123-136. Buhrdorf, R., Churchett, D. and Krueger, C. W. (2004). Salion’s Experience with a Reactive Software Product Line Approach. In proceedings of Software Product-Family Engineering, Siena, Italy. 317-322. Carbon, R., Lindvall, M., Muthig, D. and Costa, P. (2006). Integrating Product Line Engineering and Agile Methods: Flexible Design Up-front vs. Incremental Design. In proceedings of 1st International Workshop on Agile Product Line Engineering (APLE'06), Baltimore, USA, 21 August. 1-8. Chesbrough, H. (2003). Open Innovation: The New Imperative for Creating And Profiting from Technology. Boston: Harvard Business School Publishing Corporation. Chesbrough, H. (2006). Open Innovation: A New Paradigm for Understanding Industrial Innovation. In Open Innovation: Researching a New Paradigm. Chesbrough, H., Vanhaverbeke, W. and West, J. (eds.). Oxford: Oxford University Press: 1-12. Chesbrough, H. W. (2003). The Era of Open Innovation. MIT Sloan Management Review 44(3): 9. Chiasson, M. W. and Green, L. W. (2007). Questioning the IT artefact: user practices that can, could, and cannot be supported in packaged-software designs. European Journal of Information Systems(16): 542-554. Clements, P. C. and Northrop, L. (2002). Software Product Lines: Practices and Patterns. New Jersey: Addison-Wesley. Cockburn, A. (2002). Agile Software Development. Boston: Addison-Wesley. Cooper, D. R. and Schindler, P. S. (2006). Business Research Methods. New York: McGraw-Hill. Cusumano, M., Kahl, S. and Suarez, F. F. (2006). Product, Process, and Service: A New Industry Lifecycle Model (Working Paper), Harvard Business School. Daft, R. L., Lengel, R. H. and Trevino, L. K. (1987). Message Equivocality, Media Selection, and Manager Performance: Implications for Information Systems. MIS Quarterly 11(3): 355-366. Damodaran, L. (1996). User involvement in the systems design process - a practical guide for users. Behavior & Information Technology 15(6): 363-377. Davison, R. M., Martinsons, M. G. and Kock, N. (2004). Principles of canonical action research. Information Systems Journal 14(1): 65-86. Deming, W. E. (2000). Out of the Crisis. Cambridge: The MIT Press.
References
- 103 -
Dubey, A. and Wagle, D. (2007). Delivering software as a service. The McKinsey Quarterly May: 1-12. Dybå, T. (2000). Improvisation in small software organizations. IEEE Software 17(5): 82 - 87. Dybå, T. and Dingsøyr, T. (2008). Empirical Studies of Agile Software Development: A Systematic Review. Information and Software Technology 50(9-10): 833-859. Dybå, T., Dingsøyr, T. and Moe, N. B. (2004). Process Improvement in Practice - A Handbook for IT Companies. Dordrecht: Kluwer Academic Publishers. Eckstein, J. (2004). Agile Software Development in the Large - Diving Into the Deep. New York: Dorset House Publishing. Emery, F. E. and Trist, E. L. (1965). The Causal Texture of Organizational Environments. Human Relations 18: 21-32. Enkel, E., Gassman, O. and Chesbrough, H. (2009). Open R&D and open innovation: exploring the phenomenon. R&D Management 39(4): 311-316. Erdogmus, H. and Morisio, M. (2005). On the Effectiveness of the Test-First Approach to Programming. IEEE Transactions on Software Engineering 31(3): 226-237. Fægri, T. E. and Hanssen, G. K. (2007). Collaboration and Process Fragility in Evolutionarily Product Development. IEEE Software 24(3): 96-104. Fenton, N. (1994). Software Measurement: A Necessary Scientific Basis. IEEE Transactions on Software Engineering 20(3): 199-205. Fitzgerald, B. (2006). The Transformation of Open Source Software. MIS Quarterly 30(3): 587-598. Fricker, S. (2009). Specification and Analysis of Requirements Negotiation Strategy in Software Ecosystems. In proceedings of First International Workshop on Software Ecosystems, Milan, Italy, 28 September. 19-33. Ghanam, Y. and Maurer, F. (2009). Extreme Product Line Engineering: Managing Variability & Traceability via Executable Specifications. In proceedings of Agile Conference 2009, Chicago, USA, 24-28 August. 41-48. Ghanam, Y. and Maurer, F. (2010). Extreme Product Line Engineering – Refactoring for Variability: A Test-Driven Approach. In proceedings of 11th International Conference on Extreme Programming (XP2010), Trondheim, Norway, 1-4 June. 43-57. Gilb, T. (2005). Competitive Engineering: A handbook for systems engineering, requirements engineering, and software engineering using Planguage. Burlington: Elsevier Butterworth-Heinemann. Glaser, B. G. and Strauss, A. L. (1967). The Discovery of Grounded Theory: Strategies for Qualitative Research. New York: Aldine Transaction. Grunbacher, P. and Briggs, R. O. (2001). Surfacing tacit knowledge in requirements negotiation: experiences using EasyWinWin. In proceedings of 34th Hawaii International Conference on System Sciences (HICSS'01), Hawaii, USA, 3-6 January. 1-8. Hanssen, G. K. (2010). A Longitudinal Case Study of an Emerging Software Ecosystem: Implications for Practice and Theory Journal on Systems and Software Under review. Hanssen, G. K. (2010). Opening up Software Product Line Engineering. In proceedings of 1st
References
- 104 -
International Workshop on Product Line Approaches in Software Engineering, in conjunction with the 32'nd International Conference on Software Engineering (ICSE), Cape Town, 2 May. 1-7. Hanssen, G. K. and Fægri, T. E. (2006). Agile Customer Engagement: a Longitudinal Qualitative Case Study. In proceedings of 5th International Symposium on Empirical Software Engineering (ISESE'06), Rio de Janeiro, Brazil, 21-22 September. 164-173 Hanssen, G. K. and Fægri, T. E. (2008). Process Fusion - Agile Product Line Engineering: an Industrial Case Study. Journal of Systems and Software 81: 843-854. Hanssen, G. K., Yamashita, A. F., Conradi, R. and Moonen, L. (2010). Software Entropy in Agile Product Evolution. In proceedings of 43d Hawaiian International Conference on System Sciences (HICSS'10), Hawaii, USA, 4-7 January. 1-10. Hanssen, G. K., Yamshita, A. F., Conradi, R. and Moonen, L. (2009). Maintenance and agile development: challenges, opportunities and future directions. In proceedings of 25th International Conference on Software Maintenance (ICSM'09), Edmonton, Canada, 20-24 September. 487-490. Hansson, C., Dittrich, Y. and Randall, D. (2004). Agile processes enhancing user participation for small providers of off-the-shelf software. In proceedings of Extreme Programming and Agile Processes in Software Engineering (XP 2004), Garmisch-Partenkirchen, Germany. 175-183. Hughes, T. P. (1989). The Evolution of Large Technological Systems. In The Social Construction of Technological Systems. Bijker, W., Hughes, T. P. and Pinch, T. (eds.). Camebridge: The MIT Press: 51-82. Iansiti, M. and Levien, R. (2004). Strategy as Ecology. Harward Business Review(March): 1-12. Jansen, S., Brinkkemper, S. and Finkelstein, A. (2009). Business Network Management as a Survival Strategy: A Tale of Two Software Ecosystems. In proceedings of First International Workshop on Software Ecosystems, Milan, Italy, 28 September. 34-48. Jansen, S., Finkelstein, A. and Brinkkemper, S. (2009). A sense of community: A research agenda for software ecosystems. In proceedings of 31st International Conference on Software Engineering (ICSE'09), Vancouver, Canada, 16-24 May. 187-190. Johansen, T. (2005). Using evolutionary project management (Evo) to create faster, more userfriendly and more productive software. Experience report from FIRM AS. In proceedings of 6th International Conference on Product Focused Software Process Improvement (PROFES'05), Oulu, Finland, 13-15 June. 216-223. Keil, M. and Carmel, E. (1995). Customer-Developer Links in Software Development. Communications of the ACM 38(5): 33-44. Kiesgen, T. and Verlage, M. (2005). Five years of product line engineering in a small company. Kitchenham, B. A., Dybå, T. and Jørgensen, M. (2004). Evidence-based Software Engineering. In proceedings of International Conference on Software Engineering (ICSE´04), Edinburgh, Scotland, 23-28 May. 273–281. Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones, P. W., Hoaglin, D. C., El Emam, K. and Rosenberg, J. (2002). Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28(8): 721 -734. Klein, H. K. and Myers, M. D. (1999). A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Quarterly 23(1): 67 - 93.
References
- 105 -
Kniberg, H. (2007). Scrum and XP from the Trenches - How we do Scrum: InfoQ. Korkala, M., Abrahamsson, P. and Kyllönen, P. (2006). A Case Study on the Impact of Customer Communication on Defects in Agile Software Development. In proceedings of Agile Conference, 23-28 July. 76-88. Korkala, M., Pikkarainen, M. and Conboy, K. (2009). Distributed Agile Development: A Case Study of Customer Communication Challenges. In proceedings of Agile Processes in Software Engineering and Extreme Programming (XP´09), Sardinia, Italy. 161-167. Koskela, J. and Abrahamsson, P. (2004). On-site customer in an XP project: Empirical results from a case study. In proceedings of European Conference on Software Process Improvement (EuroSPI 2004), Trondheim, Norway, 10-12 November. 1-11. Langley, A. (1999). Strategies for Theorizing from Process Data. Academy of Management Journal 24(4): 691-710. Larman, C. and Vodde, B. (2008). Scaling Lean & Agile Development - Thinking and Organizing Tools for Large-Scale Scrum. Boston: Pearson Eduction Inc. Leffingwell, D. (2007). Scaling Software Agility - Best Practices for Large Enterprises. Boston: Pearson Education Inc. Lettl, C. (2007). User involvement competence for radical innovation. Journal of Engineering and technology management 24(1-2): 53-75. Lewin, K. (1946). Action Research and Minority Problems. Journal of Social Issues 2(4): 34-46. Lewin, K. (1951). Field Theory in Social Sciences. New York: Harper & Row. Linden, F. v. d. (2002). Software product families in Europe: The ESAPS and CAFÉ projects. IEEE Software. 19: 41-49. Linden, F. v. d., Schmid, K. and Rommes, E. (2007). Software Product Lines in Action. Berlin Heidelberg: Springer Verlag. Martin, A., Biddle, R. and Noble, J. (2004). The XP customer role in practice: three studies. In proceedings of Agile Development Conference, 2004, Salt Lake City, USA, 22-26 June. 42-54. Martin, A. M. (2009). The role of customers in extreme programming projects (thesis). Victoria University of Wellington. McGregor, J. D. (2008). Agile Software Product Lines, Deconstructed. Journal of Object Technology 7(8): 7-19. Merisalo-Rantanen H., T. T., Rossi M. (2005). Is Extreme Programming Just Old Wine in New Bottles: A Comparison of Two Cases. Journal of Database Management 16(4): 41-61. Messerschmitt, D. G. and Szyperski, C. (2003). Software Ecosystems, Understanding an Indespensable Technology and Industry. Cambridge: The MIT Press. Moe, N. B., Dingsøyr, T., Dybå, T. and Johansen, T. (2002). Process guides as software process improvement in a small company. In proceedings of EuroSPI, Nuremberg, Germany, 18-20 September. 177-188. Mohan, K., Ramesh, B. and Sugumaran, V. (2010). Integrating Software Product Line Engineering and Agile Development. IEEE Software 27(3): 48-55.
References
- 106 -
Moore, J. F. (1993). Predators and prey: A new ecology of competition. Harvard Business Review 71(3): 75-86. Müller, M. and Hagner, O. (2002). Experiment about test-first programming. Software, IEE Proceedings 149(5): 131-136. Nonaka, I. and Konno, N. (2008). The concept of "ba": Building a foundation for knowledge creation. California Management Review 40(3): 40-54. Noor, M., Rabiser, R. and Grünbacher, P. (2008). Agile product line planning: A collaborative approach and a case study. Journal of Systems and Software 81: 868-882. Orlikowski, W. J. and Hofman, J. D. (1997). An Improvisational Model for Change Management: The Case of Groupware Technologies. Sloan Management Review Winter. Pettigrew, A. M. (1990). Longitudinal Field Research on Change: Theory and Practice Organization Science 1(3): 267-292. Pohl, K., Böckle, G. and F., v. d. L. (2005). Software Product Line Engineering: Foundations, Principles, and Techniques. Heidelberg: Springer Verlag. Poppendieck, M. and Poppendieck, T. (2003). Lean Software Development: An Agile Toolkit for Software Development Managers. New Jersey: Addison Wesley. Reel, J. S. (1999). Critical Success Factors In Software Projects. IEEE Software 16(3): 18-23. Royce, W. W. (1970). Managing the development of large software systems. In proceedings of IEEE WESCON, Loas Angeles, USA, 25-28 August. 1-9. Sanchez, R. and Mahoney, J. T. (1996). Modularity, Flexibility, and Knowledge Management in Product and Organization Design. Strategic Management Journal 17(Special Issue: Knowledge and the Firm): 63-76. Sawhney, M., Verona, G. and Prandelli, E. (2005). Collaborating to Create: the Internet as a Platform for Customer Engagement in Product Evolution. Journal on Interactive Marketing 19(4): 4-28. Schwaber, K., Beedle, M. (2001). Agile Software Development with Scrum. New Jersey: Prentice Hall. Seaman, C. B. (1999). Qualitative methods in empirical studies in software engineering. IEEE Transactions on Software Engineering 25(4): 557-572. Sjøberg, D. I. K., Dybå, T., Anda, B. C. D. and Hannay, J. E. (2007). Building Theories in Software Engineering. In Advanced Topics in Empirical Software Engineering. Shull, F., Singer, J. and Sjøberg, D. I. K. (eds.). Heidelberg: Springer Verlag. Smaccia, P. (2008). Getting rid of spaghetti code in the real-world: a Case Study. Retrieved March, 2009, from http://codebetter.com/blogs/patricksmacchia/archive/2008/09/23/getting-rid-of-spaghetti-code-in-the-real-world.aspx. Takeuchi, H. and Nonaka, I. (1986). The New New Product Development Game. Harward Buisiness Review(Jan/Feb). Tian, K. and Cooper, K. (2006). Agile and Software Product Line Methods: Are They So Different? In proceedings of 1st International Workshop on Agile Product Line Engineering (APLE'06), Baltimore, USA, 21 August.
References
- 107 -
Tichy, W. (1998). Should Computer Scientists Experiment More? IEEE Computer 31(5): 32-40. Trist, E. L. (1977). A Concept of Organizational Ecology. Australian Journal of Management 2(2): 161-175. Trist, E. L. and Bamforth, K. W. (1951). Some social and psychological consequences of the longwall method of coal-getting. Human Relations 4(1): 3-38. Tushman, M. L. and Romanelli, E. (1985). Organizational Evolution: A Metamorphosis Model of Convergence and Reorientation. In Organization Change: A Comprehensive Reader. Burke, W. W., Lake, D. G. and Paine, J. W. (eds.). San Fransisco: Jossey-Bass. Vliet, H. v. (2002). Software Engineering - Principles and Practice. New York: John Wiley & Sons. von Hippel, E. (1996). Lead Users: A Source of Novel Product Concepts. Management Science 32(7): 791-805. von Hippel, E. (2005). Democratizing innovation. Cambrige: MIT Press. Weick, K. E. and Quinn, R. E. (1999). Organizational Change and Development. Annual review of psychology 50(Feb.): 361-386. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B. and Wesslén, A. (2000). Experimentation in software engineering: an introduction. Boston: Kluwer Academic Publishers. Yin, R. and Campbell, D. T. (2002). Case Study Research. Thousand Oaks: Sage Publications Inc. Zelkowitz, M. V. and Wallace, D. R. (1998). Experimental Models for Validating Technology. IEEE Computer 31(5): 23-31. Zook, C. (2010). Profit from the core: a return to growth in turbulent times. Boston: Bain & Company, Inc.
Appendix A
- 108 -
Appendix A
Study 1 – Adoption of Evo
Data #1: Group interview (postmortem analysis) with 6 developers A PMA is a group exercise where a homogenous group, in this case all developers from one of the development teams, is given the task of documenting positive and negative experiences related to a defined topic. In this case the topic was the software development process and the recent adoption of the agile software development method Evo. In short, the participants are asked to write short statements on post-‐it notes to explain positive experiences. Then each participant places each note on a whiteboard explaining the meaning to the rest of the group. After this, the group together rearranges the notes to group similar or related topics together. Finally the group classifies each group of notes by giving it a name. Relationships between groups can also be indicated. The same procedure is repeated for negative experiences, which usually gives input to improvement actions. Having this map of experience, the participants can prioritize the groups of negative experience according to importance or severity etc. The highest prioritized topic(s) can then undergo a root-‐cause analysis (RCA). This is simply done by having the same group drawing an Ishikawa diagram (fishbone), which shows causal relationships leading to the problem or issue. A PMA is a lightweight technique that establishes a first overview of the topic under investigation. This information is high-‐level and based on subjective experience/opinions. The strength of this method is that the results represent the consensus of the group. See (Dybå, Dingsøyr et al., 2004) for more details on these techniques.
Data #2: Guide for three semi structured interviews with participating customers
-‐ Customer background information (briefly) o Personal
Name (person and company) Briefly about your background/experience Role(s) and responsibilities CSoft experience level
o Company Type of business Years as CSoft customer (Company) Type of customer (ASP or server) Type of CSoft-use
-‐ Have you participated in previous projects and, if so, how? -‐ Describe the CSoft 9.0 project
Appendix A
- 109 -
o How did you, as a representative from the company, get appointed? o Describe the project start o Describe your every-day routines in the project.
Activities/contact with CSoft Responsibilities/roles Other…
o What happened if you, personally, couldn’t respond? -‐ If not mentioned before: how did you express your requirements and needs, and what do
you think of it? -‐ Did CSoft respond to your feedback?
o Can you see that your contribution has affected the product? o Was your contribution to the project worth the effort? o In general: was this a good process?
-‐ Three most negative aspects of the project? -‐ Three most positive aspects of the project? -‐ How do you, internally in your company, determine requirements and response to
CSoft? o How do you function as a representative for your company?
-‐ How do you compare your experiences with this project to previous projects (if you have any such relevant experience)?
-‐ Did you get any knowledge about the practical process prior to, or as part of, the project? -‐ How and why did you get involved in the project? -‐ Would you participate in a new project if asked by CSoft and do you have any
improvement suggestions to the process? -‐ Are there other aspects that you feel are important to the study?
Data #3: Guide for five semi structured interviews with PMT members -‐ Personal
o Name o Role o Role in PMT o Years in CSoft o Years in PMT o EVO knowledge/training? o How well do you know EVO? o How did you get this knowledge?
-‐ About PMT o Could you describe PMT briefly, with respect to function, tasks,
responsibilities and interaction with other parts of CSoft and interaction with customers and users?
o How do you get customers’ requirements and requests for features? o How do you document requirements and requests for features? o Do you interact with customers and if so, how? o How can you assure that you get the right picture of the customer’s
requirements and requests for features, both the explicit and the tacit/unexpressed?
o How do you interact with the development project? -‐ About EVO and the role of PMT, seen from a PMT perspective
o How do you interact with customer?
Appendix A
- 110 -
o How do you interact with the development projects? o What are the most negative aspects of this? o What are the most positive aspects of this?
-‐ Expectations to EVO? (before project start) o From a PMT perspective? o Generally?
-‐ Positive effects of EVO, seen from a PMT perspective o What are the most positive effects of EVO, so far.
-‐ Negative effects of EVO, seen from a PMT perspective o What are the most negative effects of EVO, so far?
-‐ Product quality; how does EVO affect the product quality? -‐ About the realism in being a surrogate user
o How realistic is it to represent the users/customers? o Have you any suggestions to improve this function?
-‐ Other issues o In this context, have you any other information of importance.
Study 2 – Agile SPL
Data #4: Guide for one semi structured interview with the CTO -‐ The architecture proper
o Can you name key characteristics prescribed by the CSoft architecture? General principles (e.g. components, Service Oriented
Architecture…) Patterns (e.g. layering, client-server, facade etc.) Are the server-version and the ASP-version of CSoft
architecturally identical? o How is the architecture proper represented/maintained? (e.g. documents,
models supported by tools…) Are developers routinely trained, briefed or simply expected to
“teach themselves” the architecture on a need-to-know basis? o What are the oldest “explicit” architecture elements in the current
architecture? (and how old are they?) o Can you roughly estimate the size of the CSoft code today? What is the
expected growth rate? -‐ Designing and maintaining the architecture
o How was the architecture of CSoft first designed? o What are the main architectural drivers in the current generation of CSoft
products? Name 2-3 of them (for example: modularization, supporting multiple product variants, performance, maintainability, testability…)
o Is the architecture of CSoft adapted/designed/constructed to suit the agile development approach Evo?
In particular: Is it your opinion that the introduction of Evo increased or decreased the need to pay attention to architectural issues?
o Name some examples of events/situations/business opportunities that have caused major new revisions to the CSoft architecture
Appendix A
- 111 -
Do you see different stimuli causing architecture revisions now, using an agile software process, compared to before, when using a sequential process?
o Some years ago, the CSoft architecture was “modernized” as it was ported to the .Net platform. Which changes did you do to CSoft? Did you add “something” to support customization and potential future extensions?
o How is the architecture of CSoft maintained? Which stakeholders are involved and what are their
responsibilities? CTO: You interact with the Evo-process once per iteration to
check decisions that affects the overall structure/architecture of the product; what guidelines do you use when doing so? (For example: roadmaps, overall plans, design guidelines etc.)
Do you have any examples of conflicts and how they were resolved?
A formal or informal process? How do you deal with conflicts between short-term
project/solution objectives, CSoft architecture, and product roadmaps?
We have heard so-called “interim releases” of CSoft being mentioned – are they mainly motivated by the need to target platform/architecture-oriented work?
Please describe the influence of the product roadmaps with respect to the maintenance of the CSoft architecture
o Have you ever been forced to say no to a business opportunity because of too large divergence from CSoft architecture?
Data #5: Guide for one semi structured interview with PMT manager -‐ Product roadmaps
o Can you describe a product roadmap? o What is the main objective(s) of a product roadmap? o Who ”owns” the product roadmap? o How does guidelines in a roadmap (long term) relate to project-level
requirements (short term) o Are the various solutions included in the roadmap? o Do you have an “architecture roadmap”?
-‐ Product planning in general o How is product planning done?
Formal/informal process? Which stakeholders are involved? How do you capture future needs and how do these get into
development projects? • What are the main sources of ideas for future development
plans? In your experience, has this changed as a result of using Evo?
• Do you see a difference in the kind of ideas stemming from different sources?
• If yes, do you use this consciously to direct a “balance” in requirements towards the product?
Appendix A
- 112 -
Do you consciously deal with the balance between short-term and long-term (architectural) issues during product planning?
How are product roadmaps used in projects? What is the typical lifecycle of a road map? Are there different
phases of product planning? o Which events/situations/occurrences may trigger major rewriting of road
maps? o Apart from the problem of predicting the future, what is the most difficult
aspect of product planning? o In your opinion; is the “nature” of product planning different in agile
software development compared to traditional, plan-based/sequential development? If so, how?
o Do you see particular challenges/critical factors stemming from agile development that affects product planning?
-‐ Product planning with multiple solutions o What are the most important effects on product planning caused by
offering multiple solutions? o How do you define the scope of CSoft? When/how do you extend the
scope? -‐ Working with product plans
o Who are the main users of product plans and/or product roadmaps? o How do the development projects use the roadmaps? o Are roadmaps affecting how you select and communicate with customers
being engaged in development projects? -‐ Product planning and software architecture
o Does (expected) future needs affect product architecture? Are parts of the solution (components) pre-developed?
o What is your opinion of the architecture of CSoft with respect to being an enabler or inhibiter of innovation? Please elaborate.
-‐ Misc o How many versions of CSoft exist?
Only one version alive (except several older ones in use in the market)?
o A bit about the five standard configurations: We assume that the close engagement of selected customers
provides a rich source for detailed feedback and requirements to CSoft functionality. Is it fair to say that this primarily address user-near parts of the total system (e.g. the GUI-layer?)
What type of directions do you use to develop and improve other (non-visual) parts of CSoft?
o When extending CSoft (new features/modules/etc.); what types of input do you use? Custom development, internal decisions (based on market analysis?), others?
o What role does feedback to support play in developing roadmaps and in development projects?
Appendix A
- 113 -
Study 3 – Software entropy
Data #8: Guide for one semi-structured interview with one of the team leaders
-‐ How does the team know what do develop? o How does the developers get to know requirements and specifications?
-‐ How do you test and verify new and improved code? -‐ How do you and your team contribute in the planning of the next release and ahead? -‐ How often do you speak with users and customers? -‐ How do you do this? -‐ Who else are you in touch with? -‐ How much of your work can be described as refactoring or improvement of architecture
and structure? -‐ Which challenges do you have with respect to improving the systems architecture and
structure? -‐ How and by whom are decisions of such work made? -‐ We have learned that the structure in CSoft has some weaknesses – how does this affect
your job? -‐ What is the largest change with respect to Evo as compared with the old process?
Data #9: Guide for one semi structured interview with one of the developers
-‐ Can you please describe what your job is about? -‐ Do you and the team contribute in the planning of new releases (roadmaps and plans)? -‐ In which ways do you have contact with external stakeholders during development?
How does this affect the job you do? -‐ What is the best about the way you work today and which are the biggest challenges? -‐ We have learned that the structure in CSoft has some weaknesses – how does this affect
your job? -‐ What are your ideas to improve the situation?
Study 4 – Software ecosystems
Data #11: Guide for one interview with the R&D manager -‐ [In this interview I would like to focus on how you communicate with the market and
with your customers] -‐ Would you say that the roadmap is the start of requirements specification? -‐ What is new about your process as compared to before? -‐ When developing roadmaps, who do you talk with, which type of information do you get
and how are decisions made? -‐ Could you tell a bit about the annual customer panel meeting? -‐ Who are your major competitors in the market? -‐ Can you tell a bit about your pricing policy? -‐ Are the development teams co-located or distributed?
Appendix A
- 114 -
-‐ Do you still practice the green-week concept? How do you go about verifying the deliveries from the iterations?
-‐ How does a stakeholder provide feedback? -‐ Do you have an active software process initiative at the time?
Data #12: Guide for one interview with the manager of professional services
-‐ I would like to hear a bit about your role at CSoft, your tasks and the area of responsibility you have.
-‐ Do you (Professional Services) participate in the development of roadmaps, which is managed by the PSG?
-‐ Could you tell a bit about the department you run? -‐ I have recently observed one of your customer review meetings; do you communicate
with the development teams in a similar fashion? -‐ Could you explain what the ‘quality center’ is? -‐ Do you/how do you participate in the development of roadmaps? -‐ Who operates your solution? -‐ How do you respond when customers report bugs or errors?
Data #13: Guide for one interview with the PSG manager -‐ About you
o Roles, responsibility and tasks? -‐ About PSG
o Who is part of PSG? o What is the role of PSG? o How does the PSG differ from PMT? o What was the rationale behind establishing the PSG?
-‐ About customer engagement o What is the rationale of involving customers? o How does customers contribute in strategic planning? o How does customers contribute in product development (Evo)? o How do you motivate customers to participate? o How do you identify/select customers to participate? o How does the ideal customer (for participation) look like? o Engagement of third parties?
-‐ Modes of involvement o What channels/practices do you use for engaging customers?
Operational and strategic user groups Sales/market Forums Others?
-‐ Third parties o How do you work with third parties?
-‐ Innovation o How does new ideas/solutions come up? o Do you pilot ideas, make mock-ups, demos etc.? How do you evaluate
these? -‐ Connection with R&D
o In strategic planning
Appendix A
- 115 -
o During development -‐ Roadmap
o What is the role of product roadmaps in strategic planning and in R&D? o How is the process of developing roadmaps?
Who participates? How do PSG contribute in developing roadmaps?
Data #14: Guide for three interviews with PSG members We used the same guide as for the interview with the PSG manager (Data #13, above).
Data #15: Guide for one interview with a Technical Account manager -‐ About you
o Roles, responsibility and tasks? -‐ About TAM -‐ Are you mostly involved into sales and dealing with prospects or do you do any kind of
support? -‐ Do you participate in any way in making roadmaps? -‐ Do you meet external stakeholders in web conferences like the development teams do? -‐ Having gained some experience with the Evo process; what do you think of it?
Data #17: Observation of the 2008 product conference The author participated at the whole event (three days). Selected sessions were observed and notes were made. Relevant presentations were collected. Relevant discussions in breaks and at social events were also documented.
Data #18: Observation of review meeting A customer review meeting was observed via a web-‐meeting tool. The conversation in the meeting was recorded and later transcribed. Screenshots were taken throughout the presentation and demonstration. One example:
PART II – selected publications
Paper 1
Hanssen, G. K. and Fægri, T. E. (2006). Agile Customer Engagement: a Longitudinal Qualitative Case Study. In proceedings of 5th International Symposium on Empirical Software Engineering (ISESE'06), Rio de Janeiro, Brazil, 21-22 September. 164-173
Agile Customer Engagement: a Longitudinal Qualitative Case Study
Geir Kjetil Hanssen SINTEF ICT
S.P. Andersens vei 15 NO-7465 Trondheim, Norway
+47 73597081
Tor Erlend Fægri SINTEF ICT
S.P. Andersens vei 15 NO-7465 Trondheim, Norway
+47 73593007
ABSTRACT In this longitudinal case study we have followed a small software product company that has turned from a waterfall-like process to evolutionary project management (Evo). The most prominent feature of the new process is the close engagement of customers. We have interviewed both internals and customers to investigate the practicalities, costs, gains and prerequisites of such a transition. We have gathered data from a period of two years covering four consecutive release projects using the new process and analyzed the material in detail. Our findings implicate that close customer engagement does give certain benefits but that it comes with a cost and needs careful attention to management.
Categories and Subject Descriptors D.2.9 [Software Engineering]: Management – productivity, programming teams, software process models, software quality assurance.
General Terms Design, Human Factors, Management, Measurement, Performance.
Keywords Customer Engagement, Agile Process, Stakeholder Management, Process Transition.
1. INTRODUCTION Software engineering is a highly versatile practice and can take many forms from small tasks to huge projects, lasting for years. Technologies, complexity, criticality and business domains may vary greatly – however, one piece is always a part of the game - the customers. It may be a single person, an organization or a larger diverse group. As there are many theories, strategies and methodologies for software engineering there are many ways of
engaging the customers. Nevertheless, doing it right is one of the main success factors for software engineering [1]. The fundamental challenge is to understand and capture expressed and/or un-expressed requirements and needs from the customers and to answer to them as completely and correctly as possible. Traditional plan-based approaches tend to favor a large up-front investment in requirements analysis and design having low flexibility in customer dialogue in later phases [2]. As a reaction to this, and other characteristics of traditional approaches, agile methodologies suggest a different approach, emphasizing close and continuous cooperation with the customers throughout a larger part of the development cycle. There is only scant evidence holding sufficient scientific quality on agile practices [3-5]. Furthermore, we were particularly interested to determine the effects of agile development in a product organization, i.e. a supplier of packaged software. Most published work on agile processes focus on single-customer development, mainly in IS focused forums. We believe this is due to the stronger tradition of empirical research within the IS domain. To investigate practical agile customer engagement we have followed a small sized Norwegian software product company during two years and four product releases in their transition from a plan-based development process to an evolutionary software development process. The agile method being used in this case is Evo [6]. The basic feature of Evo is that it is evolutionary, however it can also be classified as an agile process as it shares many of the concepts defined in the Agile Manifesto (www.agilemanifesto.com). The aim of this paper is to present a real case of agile customer engagement showing prerequisites, benefits, costs and risks in a software product setting. We focus here on the effects on the people participating in the software process; we do not discuss potential effects on the resulting product as such or how the process affects other stakeholders than the customers, developers and product management. We find the focus on customer engagement to be particular interesting within the context of agile processes as this is one of the four foundational values of the agile manifesto. Our findings are based upon a qualitative analysis of data collected using interviews with customers, product management and developers. Our findings indicate that implementing some of the basic principles of agile methodologies do work, however they come with a cost and an absolute premise of disciplined management. The continuation of the paper first gives a brief overview of the state of the practice with respect to customer
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ISESE'06, September 21–22, 2006, Rio de Janeiro, Brazil. Copyright 2006 ACM 1-59593-218-6/06/0009...$5.00. .
engagement to identify gaps in current knowledge (section 2). Then the study context is explained (section 3) followed by a description of the study method (section 4). Further on the results and an analysis is presented (section 5). Finally we give some conclusions (section 6).
2. STATE OF THE PRACTICE Professional software product development deals with the effective and efficient construction of software artifacts that bring value to the customer organization through benefits to the users, efficiency gains in work processes and implementation of miscellaneous company goals. Most people will agree that this is a non-trivial endeavor. Two parallel knowledge transfer processes constitute the platform of software engineering: Firstly, the knowledge of the particulars about the problem domain, the business functions, and the inherent qualities of these functions must be transferred from the customer organization to the supplier organization. Secondly, pragmatic knowledge related to the realistic fulfillment of these criteria must be transferred from the supplier back to the customer (and other stakeholders). This is basis for setting sensible expectations to the collaboration. In practice, the schedules of these processes are typically firmly embedded within the software development process. Although there might be a vast number of stakeholders in both organizations that have significant interests in these processes (politically or otherwise) and the resulting product we focus here on the issues related to the practical development and use of the product. Thus, developing a software system that will ultimately satisfy its users is critically dependant upon effective knowledge transfer between developers and users. As a profession, software engineering is in a maturing state. This is evident from the high failure rate for IT projects. As referred to in [7], average cost overruns lies close to 34% reminding us about the need for further improvement. It is claimed that problems related to the customer relationship is the most prominent cause [8]. Researchers and practitioners seem to agree that increased user engagement is the answer to the problem [9, 10], exemplified by agile development that embraces user engagement. Value number one of the agile manifesto advocates “individuals and interactions over processes and tools”. The positive effects that are credited to user engagement include: improved product quality through better understanding of the users needs, improved knowledge of customers’ organization, reduced risk of producing unnecessary or unacceptable functionality, improved ability to negotiate expectations among users, improved ability to resolve conflicts regarding the design of the system, increased feeling of ownership among users, reduction in the natural resistance towards change in work practices, remedies lack of decision capability in management, improved project performance and an increased willingness to experiment and improvise in search for solutions [11-14]. Furthermore, and of strategically high importance, user engagement might help the supplier organization to embrace innovation by improving the ability to capture users’ ideas and needs [9, 15]. There is only a small amount of evidential knowledge that provides support for these claims. Most studies report both varied [16, 13, 17, 18] and somewhat contradictory results [19]. We believe that this is due to a lack of knowledge of the
inherent complexities that must be addressed by the software organization embracing user engagement [9]. This helps to explain why user engagement is still not widely used in practice [11]. So, what does the evidence tell us? Arguably, personal, team-oriented skills are more important for developers working within an agile software process compared to plan-based processes [4]. Some factors that are consistently associated with benefits are of the psychological kind, relating to the personal qualities of the people and the environment in which the communication occurs. Examples of factors include the ability to accept a variety of responsibilities, discipline in communication, appropriate office designs in addition to incentives for both personal and group related achievements [11, 16, 20, 12]. Trust, of key importance to good communication, must be built across all tiers of the cooperation between the supplier and the customer organization [14, 1]. This includes the constituting, contractual and development level. Difficulties in the two former tiers are likely to affect the cooperative nature of the relationship between the developer and the end users. Although communication is a vital process it will only benefit the knowledge transfer between supplier and customer if it is employed mindfully. Too much, and it will most likely degrade the performance of the developers [19]. Naturally, if there is too little, developers face the risk of developing the wrong software. The risk associated with the development effort should be used to tailor the collaboration between developers and end users [21, 22]. This is particularly important during development of software with high complexity [18]. Sound tradeoffs must be made here – it is important that developers are given time to focus sufficiently on the engineering problems. At the same time, an important aspect is to maintain the trust and confidence among the developers that the relevant problems are solved, and solved in the right way. This requires attention to the continuity of the collaboration. Tasks related to user engagement consume a significant effort from the developers [23], so maintaining this continuity should not be left to the developers, as it will only disturb their work routines further. As we will discuss later in this paper, discontinuity may have devastating effects on developers’ confidence and performance. Engaging users takes on many forms. Users might be involved, meaning that they are mentally part of the development process but might not have any specific tasks or responsibilities. Examples of user engagement are customer surveys or user forums. On the other extreme, users might be participating in the process – implying that they are expected to play certain roles and perform certain tasks. Examples include participatory design [24], Soft Systems [25] and ETHICS [26]. Users are occasionally promoted to decision makers – essentially directing the course of development [27]. Although this might be good for projects lacking sufficient knowledge for good decision making within project management [11] this would pose problems for a product company that depends on the adherence to product development plans, technology strategies etc., established within the supplier organization. The underlying reason is the fundamental problem of management without the appropriate mandate. However, the majority of research on user engagement has focused on custom development projects, very few have investigated user engagement in packaged product development [28, 9]. We regard the lack of empirical evidence in this field is a sound motivation for this work.
3. STUDY CONTEXT 3.1 Case company profile The case company, CompNN, is a medium-sized Norwegian software company that provides a packaged software product for marketing and customer surveys called ProdNN. (All names are made anonymous). The product comes in two variants; an ASP-solution and a stand-alone server installation. They have a wide customer base including some of the world’s largest market research agencies. The company was established by three friends in 1996 and has since grown steadily to the present situation with about 80 employees with offices in Norway, UK, Vietnam, and USA. There has been a gradual shift from building custom-made applications to a packaged product. Today, ProdNN licenses constitute most of the revenue.
3.2 The situation prior to Evo The development process had from the start matured from ad-hoc to a well defined non-iterative waterfall-like process. This was documented in a web-based process guide and included the ordinary phases from requirements engineering through design, implementation and testing. Initially, it seemed to address their current problems with lack of control and quality management [29]. But it also introduced some new problems. CompNN struggled with changing requirements that incurred high costs due to little flexibility in the process. Much emphasis was given to extensive and thorough requirements engineering upfront but with little effect. Product planning was at this stage a collaborative effort between R&D and management. A CompNN representative explained: “Development was mainly driven by casual discussions between R&D and management. Gradually, the need to structure the work better and allocate responsibilities for the research ahead of product planning materialized.” Customers also raised the question of closer collaboration with CompNN. One customer said: “Input to the development plans was very informal. We only saw the products released by CompNN.” The somewhat ad-hoc product planning resulted in a failure to address many existing business requirements from end-users. A CompNN representative said: “We were really only focusing on features. Non-functional qualities were added last, and only in very vague terms.” Another representative commented “By focusing primarily on features, we somehow forgot to think about how these features should be used, and the user-perceived quality of these features.” The first point of customer feedback came late – at the phase of acceptance testing – leaving little time to improve the software before delivery. Also, effort estimates had been too optimistic, forcing developers to cope with intense amounts of overtime near the release date. As a result the product was delivered with many errors. Coincidentally, the Quality Assurance (QA) manager and the Chief Technical Officer (CTO) of CompNN met Tom Gilb at a software engineering research forum in November 2003 where he gave a presentation of Evo. CompNN’s representative found the ideas appealing and within a month CompNN decided to apply some key Evo practices to the next product version, already in the initial phase and due for release May 2004.
3.3 Evo described Evo [6] – short for ‘Evolutionary project management’ – is an agile process in which the product evolves iteratively as new requirements are discovered, constructed into the code, and subsequently tested. Stakeholders are expected to give feedback on each of the iterations of the product in order to guide the development of the next iteration. Finally, suites of integration tests are performed before the final system test. Evo is a rigorous process that shares some aspects of the risk-driven Spiral model [30] but is ultimately driven by the highest possible stakeholder value in each iteration together with risk validation and reduction. The core process of Evo is the requirements management process that serves two purposes per iteration. Firstly, selected stakeholders evaluate the achievements in the previous iteration in terms of a hands-on test. This helps to assure that progress is in line with stakeholders’ expectations. Secondly, based on stakeholders’ feedback the next iteration is planned – what requirements needs further improvement and what new requirements should be targeted? Requirements are not specified as functions but as product goals – addressing real customer business concerns. Great emphasis is given to identify and associate metrics to each goal. CompNN uses Impact Estimation Tables (IET) as the key project management tool. Each of the projects within a release has an IET. The IET is essentially a spreadsheet that lists goals vertically and project iterations horizontally. Each of the iterations is broken down into one or two solutions (think of them as design proposals) that are believed to address the goals. The structure helps to separate requirements from solutions. By documenting both requirements and project progress (measured results) per iteration, the IET becomes a combined plan and historical record of the project. Before release initiation, selected customers are invited to give input to product plans with initial goals for the release. During project initiation, spanning from one to two weeks, the most promising solutions to address the initial goals are captured and laid out for a number of future iterations in the IET. CompNN’s CTO explained: “This plan helps us ensure adherence to an overall technical strategy for product development.” Evo at CompNN is a highly scheduled process where weekly iterations follow a fixed schedule (called the “Evo-week”) that defines each role’s responsibilities per day. For example, on Fridays there is a management design review meeting for iteration N, on Mondays stakeholders uses (tests) the product of iteration N-1. As an extension to Evo, CompNN has included an additional role; the product management team (PMT). This is a group of five persons with responsibilities within marketing, sales, dedicated support and key account management, thus being the role having best access to knowledge of the market and the customers [9]. In the release projects they have the responsibility of appointing customers and maintain the relationship throughout the release. During the second Evo-release, CompNN extended the adoption of Evo by introducing the green week. This is a week dedicated to error correction and improvements responding to the requirements/needs of other stakeholders, thus not directly engaging customers. This is according to the Evo methodology
which recommends using about 20% of the time to improve qualities not directly related to end-users. Not a part of Evo but nevertheless a great aid is the Continuous Integration framework (CI). This is a set of well integrated tools for code management, automated tests and builds that enable swift production of product versions ready for stakeholder testing.
3.4 The four Evo-releases studied Our case study of the introduction, use and maturing of Evo covers four ProdNN releases over two years. This section briefly describes each release to explain main process innovations and experiences. See figure 1 for a brief timeline.
Release 8.5: The first Evo-release started soon after a one-day workshop covering the main Evo practices, held at CompNN’s main office, given by Tom and Kai Gilb. As this was the first Evo-project CompNN wanted a gentle start. It was decided not to involve any customers directly. The product management team would act as surrogate customers. In cooperation with customers PMT put up a list of improvements and innovations for the upcoming release. A release plan was defined dividing the release into four projects, each addressing a specific product module. Each project had an R&D-team of 2-4 developers and each team was appointed one PMT-representative to act as a surrogate stakeholder. All projects followed the schedule of the Evo-week, used the CI-framework and an IET. CompNN, especially R&D, soon began to see some first effects of this new practice. The amount of overtime at the end of the release period was almost eliminated and interviewees from R&D reported an increased level of comfort. Version 8.5 of ProdNN was released in April 2004. Release 9.0: The second release started in May 2004. The main experience was positive, and CompNN decided to include real customers as stakeholders. PMT made a thorough judgment on whom to invite to ensure stakeholder competence and
motivation to contribute. Three stakeholders, all with an existing and close relationship with CompNN, were asked to participate in the development process and after a brief introduction of Evo they accepted. The stakeholders were allocated to individual projects within the release. Stakeholders and developers soon became accustomed with the feedback loop which for the most part worked according to the description given in the section above. However, the development teams experienced that the fast pace due to the weekly iterations reduced the time for testing, thus resulting in an increasing number of errors. As a corrective action midways in the release CompNN introduced the green week to get time to catch up. Release 9.0 ended in November 2004.
Release 9.1: Release 9.0 was considered to be a success with respect to the feedback from customers and its effect on the perceived product quality; however CompNN realized that too many errors were introduced – even if the green weeks had helped to some point. CompNN decided to run an interim release to correct and improve ProdNN. Release 9.1 was started in January 2005. Only one external stakeholder was involved, the other projects focused solely on internal improvements. The release ended in April 2005.
Release 10.0: The fourth release being studied here was started in May 2005. After the interim release CompNN now set a strong focus on Evo – including stakeholders – and soon recovered the spirit from the 9.0 release. However, it shortly became a problem with customers dropping out of the projects for several reasons, the summer holidays being one of them. This partly brought the developers back to old practice as they were left alone. Lacking corrective directions from customers, developers included too many requirements and the release ended as an exhaustive all-out effort. The 10.0 release was completed in October 2005.
8.5 9.0release timeline
main process objectives
data collection(interviews)
• experiment with Evo
• maturing Evo
• external stakeholders
9.1 10.0
• testing and corrections
• revitalize Evo
jan‘04 may‘04 jan‘05apr‘04 nov‘04 apr‘05 may‘05 oct‘05
customer 1 & 2 customer 3
PMT 1 developers (group)
main process learnings
• reduced all-out effort
• confidence with Evo
• increasing number of errors
• need of green-week
• motivation suffered
• stakeholder management failed
PMT 2 & 3 PMT 4 PMT 5
Figure 1. Case timeline and points of data collection
4. STUDY METHOD 4.1 Rationale At the time this study was planned there was a great amount of uncertainty at CompNN with respect to the new process. The decision to try Evo as a development process was made promptly, based on a brief introduction of Evo. The motive to perform such a radical change in process was funded on a feeling of the need of moving on, in this case from a cumbersome and inflexible waterfall like process. No clear improvement goals or overall objectives were defined – CompNN just wanted to try a new and promising process. As seen from the researcher’s point of view we had no specific process performance goals to examine nor any baseline to contrast any qualitative data and therefore decided to follow the introduction broadly to document the details of the process innovations, the process values, costs and prerequisites. With this background we decided to perform this as an open case study [31]. To collect a widest possible foundation of data we prepared for a longitudinal study, covering several releases. This has given us the opportunity to study how CompNN has used gained knowledge from one release to the next to optimize the customer engagement practice [9]. From the start of the study in January 2004 to the end in December 2005 the study covers four consecutive releases of ProdNN described in section 3.4.
4.2 Data collection As the main source of data we have interviewed several persons filling the three most central roles with respect to Evo; the developers (6 persons), participating customers (1 person from each of the 3 companies) and all members of the product management team (5 persons). Additionally we have had numerous discussions with the QA-manager and the CTO. The timing of the interviews is depicted in figure 1. All three customers involved in the second Evo release were interviewed using an interview guide after the release had completed. All five members of the product management team were interviewed also using a guide. The first was interviewed late in the first release, the two next early in the second release, the fourth in the start of the third release and finally the fifth near the end of the fourth and latest release. Six developers from R&D were interviewed in a group interview after the completion of the fourth Evo-release. During this interview the group first drew a timeline for all the four releases on a whiteboard. Then we organized a postmortem analysis [32] asking the developers describe their positive and negative experiences with Evo. All interviews, nine in total, were taped. We sought to support the interview data with quantitative data on costs and effects of the process being studied. It turned out to be hard to obtain this type of data during or after the four releases, thus making this study a solely qualitative study.
4.3 Data analysis All taped interviews were transcribed to form a basis for a thorough and detailed textual analysis. Interview results per interviewee-group were analyzed using constant comparison [33] to find lessons learned from the introduction of Evo at CompNN with respect to practical issues (how to), effects, costs and prerequisites. The analysis was conducted with the help of NVivo™, a software tool for analyzing textual data by tagging data with meaning and grouping analytical fragments into larger
constructs. The results from the analysis of each group of interview results were used as the basis for the analysis presented in section 5.
4.4 Bias and limitations There are several factors restricting the credibility and generalizability of this study. First of all we have collected and analyzed data from one single company. However, this opens up for an in-depth study having been harder to perform involving more case companies (with the same amount of resources). Besides that, we only know – and have access to – this one company taking into use this particular software development method. Further on, restricting a study to a single organization puts limitations on the generalizability of the results to other organizations as there are no contrast cases. However, by providing contextual information a reader may be able to see which experiences that may be transferred to another software development organization. The most prominent contextual features of our case organization is the size and the fact that they are a provider of a standard software product with an established user group – as opposed to project or consultancy organizations. Finally, there is a threat with respect to the completeness of our study. We have not interviewed all parties affected by the process transition but we believe we have selected the most central ones that could give us the most reflective and nuanced view. To summarize; the reader should have these limitations in mind when reading the results of this study, however, our conclusions should be useful to those that find a similar context and at least they could serve as a basis for further research.
5. ANALYSIS CompNN’s motivation for increasing customer collaboration throughout the development cycle was described in section 3.2. In short, CompNN saw a need to better respond to changing requirements and to improve the attention to customers’ perceived quality of use of the product. Through the introduction of Evo, CompNN has performed an organizational change that enables the engagement of a selected set of customers more tightly into their software development process. Subsequently, CompNN has experienced effects on both of the two key knowledge transfer processes mentioned in section 2. Generally speaking, CompNN’s ability to transfer knowledge from the users to the developers has improved significantly. As our following analysis will explain in greater detail, CompNN has obtained a rich set of experiences related to this process. Likewise, CompNN’s ability to clarify expectations towards the involved customers has improved. We also provide detailed discussions on this issue.
5.1 Engaging customers The way CompNN has chosen to engage customers can be characterized as advisory. Customers are not project leaders although each customer selected as stakeholder in a project has a strong voice in approving the quality of the developed software. The project leader is in control of the priorities among the goals in the IET. Another advantage in CompNN’s approach is scalability. Each release involves 4-5 rather autonomous development projects with 2-5 developers. Therefore, maintaining an agile environment is relatively straightforward
and associated with modest overheads within the project group. In a bigger group, this might easily become a problem [34, 20]. Furthermore, CompNN restricts a project to only one external stakeholder (multiple stakeholders may be associated to a project during its lifetime however). The process does not restrict the number of stakeholders associated with a project however, so if it was deemed necessary, it would not be overly difficult to include more than one. A very important task is to establish collaboration with the most appropriate customer within each project, i.e. selecting stakeholders [9]. What criteria should be used in the selection process? CompNN has allocated this task to the PMT role which must identify 4-5 customers that shall act as stakeholders. Priority is given to customers that already have a very close and long-lasting relationship with CompNN. A second criterion is customers classified as expert users. Expert users are likely to have a great direct interest in the product and the direction of its development. Customers of the packaged product were not participating in the product development process previous to Evo. Thus, education of the relevant people is important [11]. In this case this was done quite easily and lightly. Customers were given a brief overview of the basic principles with emphasis on their tasks covering specification of desirable product qualities, evaluation and feedback. Based on our interviews it seems that this nimble approach has been sufficient. We think this is due to the customers in this case already being accustomed to software development. Additionally, an appointed customer must have a motive to participate; CompNN provides no form of payment or compensation to stakeholders. Stakeholders are not given any formal power either, the sole reason to get involved as seen from the customers’ point of view is the opportunity to affect the course of development. All three customers interviewed agreed to this as their main motivation. Without any guarantees for control the value of the collaboration is moderate, compared to collaborations in traditional custom development projects [11]. One did even indicate that if he was to contribute more extensively, the contribution would have to be reimbursed. This is not unreasonable as taking on the role as an active stakeholder in the development project may imply an extensive resource usage. This particular customer invested only a few days during the project period. Another customer had a greater level of involvement, going so far as to set up a designated group within the organization and also consulting their own customers. In practice, we found that as long as CompNN works directly with issues of special interest that hold a promise of business value customers will continue to evaluate consecutive releases and provide detailed and valuable feedback. We see this clearly from the developer’s experiences; as long as the ongoing work affects the customer, the customer stay engaged. As soon as there is no direct interest the customer subsides. A fundamental effect is that customers have a strongly varying motivation to contribute. Thus, monitoring and managing the collaboration emerges as a vital task.
5.2 Stakeholders’ practical roles The stakeholder has two primary roles in a project. The first is to collaborate on the specification of detailed requirements in the form of measurable goals. These goals are subsequently added to the IET by CompNN. The second role is to test a series
of product versions and provide timely feedback to CompNN. The use of ASP-based software delivery made this possible within short iterations and at low cost – no installation and setup is needed (ASP – Application Service Provider). Two customers used this option; the third did not use the product per se, but only got a list of features to comment on. In some cases other ways of exposing new features were used, for example video demonstrations. This worked well for testing end user functionality, but with the negative consequence that it became impossible to test certain system administration qualities of the product (that requires a local server installation). CompNN improvised when necessary however. One of the customers, being a server customer, requested and got a server installation, thus being able to provide this kind of feedback. It is important to note that there is a highly variable cost associated with testing and responding to the different qualities. Another premise to achieve this level of agility was the use of the CI framework for rapid building and partial, but automated testing of new versions of the product.
5.3 R&D experience As mentioned above, customers loose interest quickly if the perceived value is reduced. Subsequently, it does not take long before the customer withdraws from the collaboration. This creates a void for the developers. Having various experiences through four release projects with strongly varying Evo compliance we see this effect clearly. Developers soon got to value the direct and informative dialogue with customers and felt an equivalent strong loss when a customer dropped out. Evidently, it just did not make any sense to provide a release when no one was there to receive it. This close customer dialogue at a personal level had a value beyond detailed functional testing and feedback; it consequently also made the developers feel comfortable and more personally responsible for the customers. The fact that someone took a direct interest in their work was reported to be a huge motivational factor creating a new spirit that was missed before the process change. A PMT-member said: “Well, I felt that... hey - now some of the initial joy is back that we had during the first years where things was very different and where we had to work in an Evo-spirit without really knowing it” One developer even referred to it as being fun.
5.4 Specifying quality goals One of the most distinguished features of Evo is the clear focus on quality requirements as opposed to functional requirements. A user is expected to express desired qualities of the system being developed such as “time to use” or “ease of use” and others. At the same time the customer also defines a measurable goal that can be verified through testing. In this study we see that both customers and developers report this to work as intended in some cases. On one side it gave the user the opportunity to talk “user language” not having to consider any design issues. As seen from the developer’s point of view they got precise requirements with respect to performance giving them freedom do define how to realize the solution. Further on – after testing by customers – developers got clear and consistent feedback on the performance that they could use to either declare the goal as completed or needing further improvement in the next iteration. Developers appreciated this feedback as it became clear when to stop the improvement on a given quality
or feature, thus being able to move on. One of the developers commented: “...feedback from stakeholder minimizes the risk of developing something no one really needs”. One concrete example may be found in Johansen’s description of Evo at CompNN [29] where the goal “Usability.Productivity: Time for the system to generate a survey” used to have a performance of 7200 seconds. The goal was set to be 94 seconds and the result after one of the releases was measured to be 15 seconds. Having obtained this performance the goal was defined as complete. However this practice only worked when the premises were in place, that is, when the user was available and when the goals were clear and measurable. When this practice worked as intended, stakeholders described CompNN a “sensitive to their requirements” and greatly appreciated the close and direct communication during, some times, painstakingly detailed discussions. As this cooperative model involves just a few customers out of many it makes it even more important to select/invite stakeholders that may have requirements that other users potentially share. Including customers this closely into the development process may be a challenge as customers will see the process from their viewpoint. It is therefore extremely important that CompNN makes the final decisions on what requirements to focus on and how to resolve them as they need to have the total market in mind – both existing and future customers. We see from one of the customer interviews that the customer had a close and detailed dialogue with respect to a specific feature that was not included in the final release. In this case CompNN made a strategic decision that deviated from the customer’s desires. For this particular example the customer understood CompNN’s decision, however this type of trade-offs puts an extra dimension to the customer relationship management as development is rather by advice by customers – not by direction. To handle the practicalities and to be able to see the total development process in relation CompNN used IETs. Developers reported this tool only to be a visualization of progress to them. The project leaders used it more extensively as a tool to prioritize requirements. The IET is simply an excel spreadsheet, however some of the project leaders indicate a future need of having a more elaborate tool to handle this new amount of information better and more efficiently.
5.5 Testing Evo, by principle, is a process that incorporates testing during the entire course of a development project; the developers performing unit-testing at each step and the stakeholder perform user acceptance tests. Thus, the system should already be tested completely at the end, just prior to the final release. However, the developers underline the need of a more comprehensive system test before market release as the system has grown in complexity and the fact that testing is never the same as real use. Further on, removing the need of a final system test requires that all aspects of the system have been tested under ways which actually is not happening due to time and resource limitations.
5.6 Experience with the weekly schedule Both PMT members and especially developers commented that weekly iterations may be too short to efficiently cope with the duties prescribed by each step (develop, unit-test, build, deployment, user-test, feedback, feedback evaluation and iteration planning). A certain level of reluctance should come as no surprise, software development is mentally demanding work
that depends upon deep concentration [1]. Additionally, it correlates strongly with the size of a projects development team – two developers may not be able to produce enough value for one iteration – biweekly iterations may be a better option for the smaller teams, as one of the developers directly suggests. Further on, developers refer to the internal testing to be time consuming – eating up the time available for developing new features. As seen from the developers’ point of view it was time consuming to maintain the CI-framework (primarily, this involved maintaining the tests). A developer commented that this problem probably will grow with the code size. The adoption of the green week is a clear example of extra overhead as all time are spent on finding and correcting errors and improving performance. However, this may be equally valuable as just adding new features. We note the fragility of the process. All roles – developers, project leaders, PMT and customers need to stick to the week schedule and diligently tend their responsibilities. If any person or group fails to do so it will strongly affect others directly and thus hamper the whole process. One member of the PMT also underlines the need of process compliance and rigor. Another effect of Evo, only reported by the developers, is that the continuous focus on direct customer value weakens the focus on engineering handcraft such as thorough general design. This may lead to development shortcuts to reach short-termed (next week) goals potentially degrading the overall architecture of the total system. The prevention of architecture erosion is a fundamental challenge in agile development [35]. AT CompNN the CTO has an especially important role to monitor the software architecture and recommend initiatives to maintain a sufficient level of quality – thus being the counter pressure for architectural erosion.
5.7 Stakeholder engagement monitoring and management Several of the interviewed developers emphasized the importance of PMT as monitors and managers of the stakeholder-developer relationship. As we noted in section 5.3 a project experiences the sudden absence of a stakeholder as highly frustrating and devastating. Ideally, when a stakeholder drops out, a PMT resource must either promote a new customer to the role or step into the customer role to keep the vital deploy-test-evaluate loop running. Otherwise, the continuity of the development process may suffer [1]. We see that PMT has been an overloaded resource as the five members also have a wider responsibility including sales, marketing, key-account management and overall product strategy planning. All of the five PMT-resources interviewed blamed a high level of stress and a lack of resources for this. For PMT the direct cooperation with the project leaders and the management of customers came on top of other tasks, this was described as stressful and as we see especially from the 10.0 release this caused PMT not to fulfill their role, bringing some of the projects into trouble. The developers stated directly and clearly that PMT should be extended with more resources. Although it was already identified as a potential problem in the first and the second Evo releases, they still failed to correct this in the last release. We cannot say for sure if this was directly communicated to management, but in interpretation of our data we think this is
explained in R&D’s reluctance of exposing higher costs of the process to management. Besides selecting customers initially and potentially under way, PMT also have a responsibility of making strategic choices of where to direct the development. In cooperation with the project leaders and the CTO they make this kind of strategic decisions, relieving developers from this. However we saw that when a stakeholder left a project and the dedicated PMT resource was unavailable (due to other tasks) developers were forced to make decisions by themselves. This happened in the 10.0 project during the summer holidays and resulted in the inclusion of too many requirements, more than there were resources to handle. This became apparent only close to the release date and led to an intense period with a lot of overtime work to carry the project off.
5.8 Development process visibility In our interviews with the five PMT members they made a strong note of the improved visibility of the development process. They had previously been “outsiders” in this respect, playing a role in the initial rigorous requirements specification but left out after that. One PMT resource noted: “We did not see any visible results until the release was delivered to the market”. Now, as the PMT has an active role in development projects they have both the opportunity to comment on results underway and to discuss and prepare customers on coming features and improvements in front of release. Developers refer to plans of utilizing this process visibility further by including the documentation writers underway, thus being able to complete correct documentation at the time of release.
5.9 Scoping and estimation The fact that CompNN now makes scoping and estimation in each iteration of the Evo process, also means that product marketing can give less precise promises of the expected functionality in the coming release. A member of the PMT explains: “It is hard to promise what to deliver at a given date. We are able to give committing promises concerning coming features only a month before the release date. This is off course frustrating as we think it is important that the customers and market are updated”. This is a fundamental aspect of evolutionary development as estimation is done per iteration and not up-front. However, PMT members commented that the benefit of a tangible product outweighed this disadvantage.
6. CONCLUSIONS Through their process innovations CompNN has enabled us to reveal new knowledge concerning customer engagement in an agile software product development context. Our qualitative study helps to shed light upon the complex problem area of balancing the attention to customer demands while constructing software – arguably one of the world’s most challenging endeavors. This study and our conclusions rely on one single company, however, we believe that the key lessons learned are valid and useful to others software companies, especially those with a product focus and a continuous development process to improve and develop the product. It should be made clear that the process innovations in CompNN have not been a managed process as such. The introduction of Evo occurred due to a need for a change and the incidental encounter with Tom Gilb. However, through their incremental adoption of Evo and the introduction of the PMT role, CompNN
is probably a typical example of process innovation in industry. Some things happen by intent and some things by chance. However, CompNN has both exploited existing competence but also actively embraced the exploration of new possibilities [36]. Mindful engagement of customers is a prerequisite for innovation in product development, it is not an option [37]. Our analysis identifies a number of prerequisites for succeeding with this approach. Proactive stakeholder management is the foundation. CompNN operates in a dynamic market in which long-term planning is unrealistic. This applies equally to the customers which are inherently unpredictable in their role as partner in the collaboration. CompNN depends upon mutual benefits in the collaboration to maintain motivation and contribution. As our analysis shows, lack of continuity has significant bad effects on the performance. In turn, achieving these mutual benefits depends upon selecting relevant customers with sufficient expertise. These are all volatile and difficult factors. Thus, the capability to constantly review and manage the selection of stakeholders is critical. Furthermore, our analysis shows that Evo is a highly demanding process with respect to personal discipline and professional behavior. Due to the frequent iterations there is little room for unrestrained activity. All roles must be meticulously filled. Additionally, some sophistication in technical infrastructure, such as CI, IETs and ASP-based software delivery, has been extremely valuable, and probably essential in CompNN’s use of Evo. Our analysis shows that CompNN has achieved a number of benefits as a result of Evo and the introduction of the PMT. First, close customer cooperation has a highly motivating effect on the developers. Second, developers’ confidence has increased as a result of continuous settlement of expectations in that stakeholders assist in the prioritization of goals. The direct cooperation with users is a positive experience for the developers as it increases the quality of the communication and leads to improved understanding of the real business problems. Third, Evo has increased the visibility of the process internally in the organization and externally among the stakeholders. Adopting agile customer engagement practices has incurred additional costs. Our analysis emphasizes the extra overhead in actually running the process and the human resources required. Essentially, each Evo week is a complete development process, spanning a number of phases that demands frequent changes of context and thus occupy significant resources. Also, the technical infrastructure, being a prerequisite, is costly. Furthermore, the analysis shows a significant cost incurred by the continuous need to maintain it. It is natural to consider increased exposure to risk as a cost. First, short iterations with insufficient attention to management and process compliance increase the fragility of the software process. Our analysis shows that the 10.0 release in particular exposed CompNN to considerable risk in that the developers were left with much higher workload than anticipated – thus being a good example of lack of process compliance and the consequent effects. Secondly, engaging in this kind of strong cooperation with a small selection of customers also means a reduced capability to capture the needs of other, non-appointed customers. For example, it is critical that internal stakeholders are appointed in a conscious manner to take care of the full set
of quality drivers pertaining to the software product (specified in product strategies etc.) Our study uncovers possibilities for further research in this area. It would be valuable to compare these findings with similar investigations in other companies. Combined, these findings might be used as a motivation for more specialized and rigorous studies of effects on customer satisfaction for example. Our study does not reveal information on any effects the process innovation might have on the non-included customers observed quality of the product. Ultimately, studies should be conducted to quantify the benefits and costs of agile customer engagement.
7. ACKNOWLEDGEMENTS We are greatly thankful to the people at CompNN that has contributed to our study and even let us talk to their customers. We would also like to thank the Research Council of Norway for the funding of this work under Grant 156701/220.
8. REFERENCES 1. Reel, J. S., Critical Success Factors In Software Projects,
in IEEE Software. 1999. p. 18-23. 2. Ceschi, M., Sillitti, A., Succi, G. and De Panfilis, S. Project
management in plan-based and agile companies. IEEE Software, 22, 3 (2005), 21-27.
3. Abrahamsson, P., Salo, O., Ronkainen, J. and Warsta, J. Agile software development methods - Review and analysis. VTT Publications 478, VTT Electronics, 2002.
4. Cohen, D., Lindvall, M. and Costa, P. Agile Software Development A DACS State-of-the-Art Report. Fraunhofer Center for Experimental Software Engineering Maryland and The University of Maryland, 2003.
5. Lindvall, M., Basili, V., Boehm, B., P., C., Dangle, K., Shull, F., Tesoriero, R., Williams, L. and Zelkowitz, M. Empirical Findings in Agile Methods. In Proceedings of XP/Agile Universe (Chicago, IL, USA, 2002). Springer-Verlag GmbH.
6. Gilb, T. Competitive Engineering: A handbook for systems engineering, requirements engineering, and software engineering using Planguage. Elsevier Butterworth-Heinemann, 2005.
7. Jørgensen, M. and Moløkken-Østvold, K. How large are software cost overruns? A review of the 1994 CHAOS report. Information and Software Technology, 48, 4 (2006), 297-301.
8. Johnson, J. Turning chaos into success. Software Magazine, (1999), 30-34,39.
9. Keil, M. and Carmel, E. Customer-Developer Links in Software Development. Communications of the ACM, 38, 5 (1995), 33-44.
10. Tait, P. and Vessey, I. The Effect of User Involvement on System Success: A Contingency Approach. MIS Quarterly, 12, 1 (1988), 91-108.
11. Axtell, C. M., Waterson, P. E. and Clegg, C. W. Problems integrating user participation into software development. International Journal of Human-Computer Studies, 47, 2 (1997), 323-345.
12. Boehm, B. and Turner, R. Management challenges to implementing Agile Processes in traditional development organizations. IEEE Software, 22, 5 (2005), 30-39.
13. Lin, W. T. and Shao, B. B. M. The relationship between user participation and system success: a simultaneous contingency approach. Information and Management, 37, 6 (2000), 283-295.
14. Mathiassen, L., Ories-Heje, J. and Ngwenyama, O., Improving software organizations: From principles to practice, in The agile software developement series. 2001, Addison-Wesley Professional. p. 368.
15. Rogers, E. M. Diffusion of innovation. Free Press, 2003. 16. Gallivan, M. J. and Keil, M. The user-developer
communication process: a critical case study. Information Systems Journal, 13, 1 (2003), 37-68.
17. Vredenburg, K., Mao, J. Y., Smith, P. W. and Carey, T. A survey of user-centered design practice. In Proceedings of Conference on Human Factors in Computing Systems (Minneapolis, Minnesota, USA, 2002). ACM Press New York, NY, USA.
18. Wong, E. Y. W. and Tate, G. A Study of User Participation in Information-Systems Development. Journal of Information Technology, 9, 1 (1994), 51-60.
19. Heinbokel, T., Sonnentag, S., Frese, M., Stolte, W. and Brodbeck, F. C. Don't underestimate the problems of user centredness in software development projects - There are many! Behaviour & Information Technology, 15, 4 (1996), 226-236.
20. Hansson, C., Dittrich, Y. and Randall, D. Agile processes enhancing user participation for small providers of off-the-shelf software. In Proceedings of Extreme Programming and Agile Processes in Software Engineering, Proceedings 2004).
21. Boehm, B., Get Ready for Agile Methods, with Care, in IEEE Computer. 2002. p. 64-69.
22. Glass, R. Matching Methodology to Problem Domain. Communications of the ACM, 47, 5 (2004), 19-21.
23. Kujala, S. User involvement: a review of the benefits and challenges. Behaviour & Information Technology, 22, 1 (2003), 1-16.
24. Bødker, K., Kensing, F. and Simonsen, J. Participatory IT Design: Designing for Business and Workplace Realities. MIT Press, 2004.
25. Platt, A. and Warwick, S. Review of soft systems methodology. Industrial Management + Data Systems., 95, 4 (1995), 19-22.
26. Mumford, E. The ETHICS Approach. Communications of the ACM, 36, 4 (1993), 82-83.
27. Hartwick, J. and Barki, H. Communication as a dimension of user participation. Ieee Transactions on Professional Communication, 44, 1 (2001), 21-36.
28. Dagnino, A., Smiley, K., Srikanth, H., Anton, A. I. and Williams, L. Experiences in applying agile software development practices in new product development. In Proceedings of Proceedings of the Eighth IASTED International Conference on Software Engineering and
Applications, Nov 9-11 2004 (Cambridge, MA, United States, 2004). Acta Press, Anaheim, CA, United States.
29. Johansen, T. Using evolutionary project management (Evo) to create faster, more userfriendly and more productive software. Experience report from FIRM AS. In Proceedings of International Conference on Product Focused Software Process Improvment (Oulu, Finland, 2005). Springer Verlag.
30. Boehm, B. A Spiral Model of Software Development and Enhancement. IEEE Compuer, 21, 5 (1988), 61-72.
31. Yin, R. and Campbell, D. T. Case Study Research. Sage Publications Inc, 2002.
32. Birk, A., Dingsøyr, T. and Stålhane, T. Postmortem: Never Leave a Project without It. IEEE Software, 19, 3 (2002), 43 - 45.
33. Seaman, C. B. Qualitative methods in empirical studies in software engineering. IEEE Transactions on Software Engineering, 25, 4 (1999), 557-572.
34. Boehm, B. and Turner, R. Balancing Agility and Dicipline - A Guide for the Perplexed. Addison-Wesley, 2004.
35. MacCormack, A., Verganti, R. and Iansiti, M. Developing products on "internet time": The anatomy of a flexible development process. Management Science, 47, 1 (2001), 133-150.
36. Dybå, T. Improvisation in small software organizations. IEEE Software, 17, 5 (2000), 82 - 87.
37. Carmel, E. and Becker, S. A Process Model for Packaged Software Development. IEEE Transactions on Engineering Management, 42, 1 (1995), 50-61.
Paper 2
Fægri, T. E. and Hanssen, G. K. (2007). Collaboration and Process Fragility in Evolutionarily Product Development. IEEE Software 24(3): 96-104.
9 6 I E E E S O F T W A R E P u b l i s h e d b y t h e I E E E C o m p u t e r S o c i e t y 0 7 4 0 - 7 4 5 9 / 0 7 / $ 2 5 . 0 0 © 2 0 0 7 I E E E
feature
of discussions with Tom Gilb, the company se-lected Evo,1 an evolutionary project manage-ment method that Gilb first detailed in 1981.2
Evo develops the final product iteratively asnew requirements are discovered, constructedinto the code, and subsequently tested.3 Stake-holders give feedback to each product itera-tion to guide the development of the next iter-ation. Integration test suites are performedbefore the final system test.
Evo is a rigorous process encompassingconscious risk management, similar to thelater-developed Spiral model.4 However, eachiteration is ultimately driven by the highestpossible stakeholder value. Despite rigidity inprocess control, Evo builds on commonlyknown agile principles: continuous customer
collaboration, direct feedback to frequent re-leases of working software, and change re-sponsiveness. Here, we detail the company’sexperiences with adopting, refining, and ma-turing Evo. Our aims are to share our insightsregarding the organizational impacts of evolu-tionary development and to fill in some gapsin current knowledge about agile methods andeffects.3,5 The “Study Method” sidebar de-scribes how we collected and analyzed data.
BackgroundCompNN (a pseudonym) is a Norwegian
software company that provides a packagedsoftware product, ProdNN, for marketing andcustomer surveys. The company was foundedin 1996 and has since grown steadily to about
Collaboration, ProcessControl, and Fragility in Evolutionary ProductDevelopment
Evolutionary development is an agile software engineering approachthat embraces continuous customer collaboration to assist the con-struction of a gradually evolving product. Its benefits and chal-lenges are revealed in a longitudinal study we performed of a
medium-sized company’s transition from a traditional plan-based softwareprocess to an evolutionary process. The transition occurred in four consecu-tive product releases during a two-year period in 2004–2005. On the basis
software development
Tor Erlend Fægri and Geir Kjetil Hanssen, SINTEF Information and Communication Technology
A softwarecompany’stransition from a plan-baseddevelopmentprocess to an agile,evolutionaryprocess reveals boththe benefits anddemands it placeson an organization.
80 employees with offices in Norway, Vietnam,the UK, and the US. The company shifted grad-ually from building custom applications to apackaged product. Today, ProdNN licensesconstitute most of the company’s revenue.
CompNN used a noniterative variant of theplan-based waterfall process to manage prod-uct development.6 When the company intro-duced the process, it provided the structurenecessary to maintain a core product, whichthe shift from custom development to pack-aged product required. The process empha-sized thorough requirements engineering upfront. Analysis, specification, design, code,module test, system integration, acceptancetesting, and delivery were performed sequen-tially and diligently, although with only mod-est reflection on intermediate results. Initially,the process seemed to address the company’sproblems by establishing control and qualitymanagement, but it eventually exposed newchallenges. For example, its inflexibility in-curred high costs in the face of unclear orchanging requirements.
CompNN also saw a potential for improv-ing its dialogue with customers. Packaged-product customers asked for closer collabora-tion with CompNN. As one customer said,“Input to the development plans was very in-formal. We only saw the products released byCompNN.” Product planning involved mainlya collaboration between R&D and manage-ment. A CompNN representative explained:“Development was mainly driven by casual dis-cussions between R&D and management.Gradually, the need to structure the work betterand allocate responsibilities for the researchahead of product planning materialized.” Thesomewhat ad hoc product planning failed toaddress many existing business requirementsfrom end users. Another CompNN representa-tive said, “We were really only focusing on fea-tures. Other qualities were added last, and onlyin very vague terms.” And another representa-tive commented, “By focusing primarily on fea-tures, we somehow forgot to think about howthese features should be used and [about] theuser-perceived quality of these features.”
Because the first feedback to the productcame late—during acceptance testing—littletime was available to improve the software be-fore delivery. Also, project effort estimates werefrequently too optimistic, forcing developers tocope with intense amounts of overtime near the
release date. In the pressure to deliver function-ality, testing would often give way to develop-ment. As a result, the delivered product hadmany errors. Furthermore, company manage-ment saw a trend of declining R&D motiva-tion. They feared that these problems wouldlimit the product’s competitiveness.
In November 2003, CompNN’s quality as-surance (QA) manager and the chief technol-ogy officer (CTO) met Tom Gilb at a softwareengineering research forum, where he gave apresentation on Evo. The company represen-tatives found the ideas appealing, and within amonth CompNN decided to apply some keyEvo practices to the next product version, duefor release in May 2004.
Evo practices at CompNNSeveral engineering domains have used the
Evo project management method, and Gilbdiscusses its applicability to systems engineer-ing in his latest book.1 Evo includes manypractices, but here we discuss only thoseadopted and shaped by CompNN.
M a y / J u n e 2 0 0 7 I E E E S O F T W A R E 9 7
CompNN had no defined software process improvement goals prior tointroducing Evo. Therefore, we used an open, qualitative, longitudinal casestudy to investigate potential costs, gains, and risk factors. We covered fourproduct releases produced during 2004–2005.
Data collectionThe main data source was interviews with several persons filling Evo’s
three most central roles: developers (six persons randomly selected), collab-orating customers (one person from each of the three customers), and allproject management team members (five persons). We interviewed the firstgroup after the ProdNN version 10.0 release, using a postmortem tech-nique. We followed an interview guide to conduct semistructured interviewswith the two latter groups at various times during the study. Altogether, weconducted nine interviews, taping them all.
Additionally, we’ve had numerous discussions with the company’s qualityassurance manager and chief technology officer.
Data analysisWe transcribed the interviews for detailed textual analysis. We analyzed
the results by interview group, using constant comparison.1 The softwaretool NVivo supported our analysis by carefully tagging the data and group-ing analytical fragments into larger constructs.
Reference1. C.B. Seaman, “Qualitative Methods in Empirical Studies in Software Engineering,” IEEE
Trans. Software Eng., vol. 25, no. 4, 1999, pp. 557–572.
Study Method
Requirements engineering and project management
Arguably, Evo’s most important attribute isproject management guided by impact-basedeffort prioritization and formalized require-ments engineering. In Evo, impact quantifiesthe software qualities that give customers thegreatest value, and goals denote measurablerequirements. In Evo jargon, a goal isn’t sim-ply a statement of functionality, such as “Thesystem should …,” but a statement of the per-formance the software system must or shouldhave. This approach explicitly helps distin-guish requirements from designs.1
To keep track of the goals and monitorprogress, each project uses impact estimationtables. An IET is a spreadsheet that lists theproject’s planned iterations horizontally andthe current goals vertically. The goals are sup-plemented by a tolerable metric, which is a nu-meric “worst-acceptable case” constraint levelfor the quality requirement and helps makethe goal’s sensitivity explicit. For example, infigure 1, Usability.Productivity is a goal meas-ured as “the time in minutes for the system togenerate a survey.” The goal is 25 minutes, butthe tolerable metric is 35 minutes. In Evo, de-sign ideas are proposed solutions to reach thegoals. In the figure, the design idea for itera-tion 9 is “Recoding.”
CompNN produces two product releaseseach year. Each release is broken down intotwo to five parallel time-boxed projects withan R&D team of two to five developers each,including one appointed project manager(PM). Each project focuses on specific
ProdNN modules or types of functionality.The product management team (PMT), com-posed of five senior employees, is responsiblefor creating and maintaining product develop-ment plans in close cooperation with cus-tomers. Input from the CTO, executive man-agement, and customers (for example, fromyearly surveys) is important in this work.
Product development plans can span multi-ple releases, and the PMT together with theCTO are responsible for ensuring that theIETs conform to them. Before release initia-tion, selected customers are invited to supple-ment the plans with additional requirements.During the initiation phase, which spansroughly two weeks, the PMT together withthe respective PMs capture the most promisingdesign ideas to address the initial goals and en-ters them in the IET for future iterations. AsCompNN’s CTO explained, “This plan helpsus ensure adherence to an overall technicalstrategy for product development.”
Process schedulingEvo is a highly scheduled process. Figure 2
presents the Evo week, a schedule thatCompNN iterates each week to define tasks forthe project’s defined roles. Most planning andcoordination work occurs on Friday, preparingthe development team for a quick start onMonday morning. In each iteration, the projectteams implement the planned design idea, makea new version of the software system availableto selected stakeholders, and compare the ef-fort’s results to the goal. In weekly project lead-ers’ meetings, QA collaborates with the PMs
Figure 1. Impact estimation table fromProdNN 8.5. Currentgoals are listed vertically, and their impacts in planned iterations appear horizontally.
9 8 I E E E S O F T W A R E w w w. c o m p u t e r. o r g / s o f t w a r e
and CTO to review the stakeholders’ feedbackand decide whether the work meets the require-ments (to either the tolerable or the goal level).When the work meets specified goal levels, theIET is updated with new goal and tolerable re-quirement levels, according to stakeholders’feedback. This concludes an iteration.
At the start of a new iteration, the PM re-views each requirement according to how theiteration’s design idea impacts it. The IET sim-plifies the consideration of a design idea’s sideeffects on the range of current goals. Occa-sionally, the project team might suggest revi-sions to design ideas at this point. IETs func-tion both as the overall development plan anda progress record. Using metrics for each goalensures that any person within the companycan easily determine the development’s status.The QA manager and CTO jointly fill the roleof manager for the release.
CompNN established a technical frame-work for continuous integration to supportmore frequent builds and tests. CI integrates atoolset that almost fully automates the soft-ware build process and the execution of par-tial tests of the software several times a day. Atpresent, CI includes Starteam (from Borland)for source control, NUnit for unit testing(open source), FxCop for code analysis (opensource), NAnt for build management (opensource), and CruiseControl.Net for coordina-
tion and reporting (open source). CI is config-ured to automatically upload the product totest servers each Friday.
Adhering to the Evo week requires disci-plined teamwork. The PMT appoints a stake-holder to represent important project goals. Athree-person QA group is responsible for de-ploying testable product versions to the stake-holders. The QA group also ensures follow-upon any reported discrepancies from CI. Its ac-tivities run in parallel with the unit testing per-formed by R&D. QA is responsible for re-viewing the test plans and results from thetests run by R&D. Thus, QA’s role is to ensurethat all issues detected during unit testing areeventually resolved.
The transition experiencesThe Evo implementation for each of the
four product releases involved a different con-figuration and yielded different lessons.
The Evo pilot: ProdNN release 8.5CompNN established the PMT a few weeks
before initiating the 8.5 release of ProdNN.Collaborating with customers, PMT created alist of important improvements. The 8.5 re-lease plan was divided into four parallel proj-ects focusing on distinct product modules. Thisresulted in small, easily manageable R&Dteams. Each project followed the Evo week,
R&D team
Stakeholder
PM: Attend project mgmt meeting:12:00-15:00
PM: Send version N detail plan toCTO + prior to project mgmt meeting
R&D team: Focus on generalmaintenance work, documentation
CTO
QA
• Approve/reject design & step N • System architect to review code and test code
• Run final build and createsetup for version N-1
• Install setup on test servers (external and internal)
• Perform initial crash test and then release version N-1
• Attend project mgmt meeting: 12:00-15:00
R&D team: Develop test code and code for version N
• Use version N-1
• Follow up Cl
• Review test plans, tests
• Review test plans and tests
• Follow up Cl
• Review test plans and tests
• Follow up Cl
• Review test plans and tests
• Follow up Cl
R&D team: Develop test code for version N
R&D team and PM: Meet withstakeholders todiscuss actiontaken
• Meet R&D team to give feedback and dis-cuss action taken from previous actions
R&D team: Develop test code and code for version N
R&D team: Com-plete test code and code for version N
Friday Monday Tuesday Wednesday Thursday
Figure 2. The Evoweek. Planning and coordination of work,according to projectroles, occurs on Fridayto jumpstart work onMonday morning.
M a y / J u n e 2 0 0 7 I E E E S O F T W A R E 9 9
1 0 0 I E E E S O F T W A R E w w w. c o m p u t e r. o r g / s o f t w a r e
For R&D, the change
to Evosignificantly
improved the comfort
level andmotivation.
employed IETs, and used the CI framework.Additionally, each project included one PMTmember who gave feedback to the iterations,simulating customers.
Although the change to Evo might appeardramatic, many interviewees commented thatthey had expected a change from managementfor some time (“something had to happen”).A PMT member went so far as to say, “What-ever changes we implemented, it would prob-ably be a good thing.” Because CompNN wasexperiencing a difficult period, the organiza-tion was generally enthusiastic about the pilotproject, which got off to a flying start.
CompNN observed many positive effects asit adopted the new, highly structured develop-ment approach. One benefit, applying primarilyto R&D, was a reduction in the intense over-time work at a project’s end. Reviewing sug-gested improvements iteratively resulted occa-sionally in postponements to a later release. ForR&D, the change to Evo significantly improvedthe comfort level and motivation. Throughnightly builds, supported by the CI framework,the company always had a running versionready for testing and feedback, leaving R&Dmore time and quiet to develop code.
Maintaining CI, a premise for Evo collabo-ration and control, required substantiallymore effort than anticipated. This related pri-marily to maintaining the tests. Furthermore,establishing quality metrics in cooperationwith PMT succeeded only partially. Welearned that some developers continued theirold “feature counting” practice. We weren’toverly surprised by these findings—establish-ing good requirements and associated metricsis inherently hard.7 For the goals that did in-clude quality metrics and effective measure-ments, developers appreciated the added cer-tainty. They expressed this in the context of“interest in what we do on a weekly basis—not in a four-monthly basis as before.”
The general experience with the Evo pilotwas positive. It ran for five months, requiredonly modest preparations, and gave the CTO,R&D, QA, and PMT confidence with the newprocess.
Inviting the customers: ProdNN 9.0CompNN decided to expand the use of Evo
practices in the following 9.0 release, com-mencing May 2004. PMT carefully selectedthree customers to participate as collaborating
stakeholders on the basis of three criteria: eco-nomic value, experience with the product, andcomplementary interests in ProdNN. The cus-tomers received no compensation for partici-pation, so PMT felt certain that their collabo-ration was based on interest and willingness toshape the product. One of the customers said,“Since 2001 we had discussed the possibilityof closer collaboration with development.When CompNN turned to Evo, it became areality.” This customer reported having spenttwo person-weeks on preparing product planinput and two person-weeks on evaluationand feedback.
The three customers were assigned to indi-vidual projects. PMT’s role changed corre-spondingly—from surrogate customers to fa-cilitators of the collaboration between thecustomers and R&D. Stakeholders gave feed-back in project meetings via telephone confer-encing. Because the project teams were locatedin Norway and two stakeholders were locatedabroad (US and Canada), establishing thetechnical infrastructure for software sharingposed a technical challenge. CompNN decidedto make the software versions available for thestakeholders through an application serviceprovider (ASP). CompNN supplemented themeetings with screen sharing and occasionaltechnically focused meetings to discuss com-plex issues. R&D soon accepted the customerdialogue as the most significant mechanismfor refining goals and doing functional testing.This interaction generated valuable knowl-edge used to update the IETs and subsequentlyto dynamically control the R&D efforts.
Collaborating directly with the customersincreased the developers’ confidence comparedto collaborating with PMT. One developersaid, “It’s the feedback that works much better.Now we get feedback very fast.” Another PMTmember commented on the explicit focus onsoftware qualities: “… because we now focuson qualities as opposed to features we now arecapable of giving the customers what theyneed, not necessarily what they want.” Thetighter relationship with customers enabledCompNN to better understand the underlyingbusiness problems. A developer stated: “It wasa very satisfying situation because we managedto define measurable goals that we knew wereimportant to the stakeholder.” The collabora-tion provided a fruitful environment in whichvaluable knowledge was externalized and
transformed into the evolving product. APMT representative with additional documen-tation and training-materials responsibilitiescommented: “Our job got easier because weknew what was coming at an early point in therelease. For the first time, we were finishedone week before the release date.” As a result,the sensation of being in control improved thecomfort for both R&D and PMT and furtherincreased their motivation.
The rapid iterations in Evo made some re-quested quality goals unrealistic. One customerhad business requirements related to the per-formance of certain administrative tasks in theproduct, such as installation of new databasesand other system configuration duties. Theproject couldn’t include these goals for practi-cal reasons. Giving feedback on such goalswould have required the customer to test on alocal installation as well as longer iterations.
As the release progressed, cracks started toappear. The weekly schedule became too fast-paced and R&D complained that too littletime was left for testing, resulting in increasederrors and discomfort in R&D. This exposeda fragility in the process, primarily its depend-ence on timely deliveries and effective collabo-ration. We define fragility as the level of aprocess’s sophistication as given by the num-ber of preconditions and dependencies thatmust be met. This problem was intolerable,and roughly midway through the release,CompNN decided to try a suggestion fromGilb—namely, the green week. Rather thanspending each Friday for corrective mainte-nance, R&D now dedicated a full week to thistask. The project could add no new function-ality during this green week. The PMs ad-justed schedules so that stakeholders didn’t doanything at this time.
The experiences from introducing the greenweek were mixed. First, it helped address theproblem of too little time for testing. The in-terviewed developers agreed that the greenweek enabled them to catch up with the test-ing and increased their confidence in the code.However, they also felt the cost was too high.We were puzzled and asked why. One devel-oper said reluctantly, “Our testing tool (FX-Cop) is very cumbersome. It is not integratedwith our other development tools. So theprocess is slow and appears as a diversion.”This illustrates the importance of efficienttools. We believe that the near-perfect vision
of a fully integrated development environ-ment, created by CI, was severely broken bythe testing tool and that this gave R&D a verynegative attitude towards testing. This, inturn, explains the increasing number of errors.
Picking up some broken pieces: ProdNN 9.1Release 9.0—and Evo—was at this point
considered a success with respect to customercollaboration. However, the extreme focus onresponding rapidly to feedback led to manynew code errors. The green week helped, butnot enough. Many of the reported problemsweren’t malfunctioning features but poorlyperforming features that hadn’t been testedproperly because of lack of time. In an effortto counter this problem, CompNN decided torun an interim release: “The objective was toenhance performance, handle change requests,reduce number of bugs, document code andimprove the tests.” The release was dividedinto four parallel projects. PMs were told tofollow the Evo practices, but only one project,called “system configuration,” was allocatedan external stakeholder. The other three proj-ects simulated stakeholders in the projectgroup meetings with respect to the interim re-lease’s quality goals.
The system configuration project was com-pleted successfully with disciplined adherenceto the Evo-week and a fruitful collaborationwith the appointed stakeholder (CompNN’sASP). Results for the other projects weremixed, primarily because of PMT and othermanagement’s lack of engagement. This, inturn, created a sense of vacuum among the de-velopers. A developer stated: “Driving theprocess alone is very demanding. You sit therewondering what is happening.” Confidencesuffered accordingly. What they so easily hadbecome accustomed to during the two previ-ous releases was now taken away. Conse-quently, the Evo process appeared as a burden.They missed valuable feedback and interac-tion with a clearly defined stakeholder. Somedevelopers also reported that motivation suf-fered in this release. The PMs didn’t explicitlydecide to drop the Evo measurement and feed-back practices—it “just happened” becausefollowing Evo in detail seemed irrelevant withno external stakeholders. However, withoutthe direct, instructive collaboration withstakeholders, developers eventually movedaway from the practice of performing weekly
M a y / J u n e 2 0 0 7 I E E E S O F T W A R E 1 0 1
measurements, and these projects partly fellback on past practices.
Experiencing the fragility: ProdNN 10.0In release 10.0, CompNN’s management
again set a stronger focus on employing theEvo principles throughout all projects. Thecommitment to recreate the spirit of the 9.0 re-lease was strong. The 10.0 release was dividedinto three projects, each to be run according toEvo principles and to include the green week.Again, it started well. R&D quickly regainedtheir motivation through stimulating collabo-rations with customers. PMT also did a goodjob in ensuring active stakeholder engagement.Occasionally, some customers couldn’t partici-pate in the feedback meetings. For the mostpart, PMT was prepared for this event and hadgathered enough customer input beforehand togive the developers good information. Thus,PMT functioned as the buffer for unpre-dictable customer availability.
But problems started to arise during thesummer holidays, roughly halfway throughthe development period. One customer with-drew from its allocated project because thegoals had been reached successfully. Othercustomers left and reappeared arbitrarily,causing an unpredictable supply of feedbackto steer the projects. The company took littleaction to correct the situation, and the sum-mer holidays also reduced the availability ofkey people. Lacking steering and able decisionmakers, the 10.0 projects experienced growinguncertainty and mistrust.5
Evo’s dependency on frequent feedbackand fluent decision making contributes to itsfragility. Release 10.0 exposed this fragility.Without management’s attention to keep theprocess components going, Evo showed signsof collapse. We weren’t surprised to find acombination of reasons behind this. First,Evo’s key mentors in CompNN had becomeoverloaded with other tasks and thus failed tonotice the process’s gradual decay. Addition-ally, as this problem became evident, manage-ment failed to take corrective action. Insteadof protesting loudly, R&D quietly took this re-sponsibility themselves and ended up asprocess owners. Lacking other directions, theyfocused on adding functionality that they be-lieved would be useful to customers. Paradox-ically, the 10.0 release impressed customerswith the introduction of a large number of fea-
tures. But R&D failed to understand theirown limitations. They were too ambitious.The result was an inevitable all-out effort asthe release date approached, pushing many ofthe R&D team members to exhaustion.
DiscussionEvolutionary software development pro-
motes a collaborative environment where sup-plier and consumer can generate the knowl-edge to gradually create the optimal product.This knowledge also supports control of thesoftware development process. CompNNreaped such benefits from adopting Evo. Inaddition, participants reported increasing con-fidence and comfort with the developmentprocess.
Companies can only exist in relationshipswith customers. Evo has made these relation-ships more collaborative for CompNN. Fur-thermore, constantly focusing on deliveringcustomer value increases the synergetic valueof these relationships. This becomes a signifi-cant competitive advantage, enabling more ef-ficient innovation transfer between CompNNand the collaborating customers. Ideally, allcollaborating customers should representstrategically, technically, and economically op-timal product requirements, but this is—ofcourse—impossible to achieve. The cost of in-cluding an unresponsive stakeholder is moder-ate in a responsive organization, which can ac-knowledge the problem in a few iterations andthen begin steps to correct it. Furthermore,during the time it takes to replace the stake-holder, vigilant PMT and project teams canstill reap benefits for the organization. In thefriendly, collaborative environment fosteredby Evo, CompNN extracted useful knowledgeabout how customers used the product. Thecompany also used this time to try out newideas with the customers.
And here lies a key lesson from this study:maintaining a watchful, vigorous stakeholdermanagement capability is paramount for Evo’ssuccess.
Evo increased the product’s exposure andresource usage compared to the originalprocess. One PMT member, working abroad,noted that “Evo makes ProdNN and the de-velopment process more visible throughoutthe company and also among our key stake-holders.” More people can observe or criticizeprioritizations. This can help maintain focus
1 0 2 I E E E S O F T W A R E w w w. c o m p u t e r. o r g / s o f t w a r e
Evo’sdependency on frequent
feedback and fluent
decision makingcontributes
to its fragility.
M a y / J u n e 2 0 0 7 I E E E S O F T W A R E 1 0 3
on generating value, but it also influences thenature of the demands on the product. ACompNN representative said, “I think we haveimproved our ability to think in terms of thereal business problem of the stakeholders.”This follows implicitly from the increase in di-rect stakeholder feedback. However, R&D saw“a lack of attention to software engineeringprinciples.” Finding the balance in this dynamicenvironment is a well-known challenge.5 There-fore, companies should select stakeholders witha particular interest in the product as an engi-neering artifact (called “internal stakeholders”in Evo) at intervals to counter architectural ero-sion and subsequent excessive costs of imple-menting improved quality.
We also learned that short cycles of specifi-cation, design, development, and testing re-duce tolerance toward inefficient tools. Poten-tial adopters should keep this in mind.
The continuous stream of knowledge fromthe collaboration with stakeholders is a mainfactor guiding the development process. We be-lieve these rapid, direct feedback loops explainmost of the increase in employee motivationand enthusiasm, particularly in R&D. In thisrespect, the IETs functioned as lightweight yeteffective decision-making tools. Rapid feedbackon developed code, appropriate metrics, and ef-fective decision making help reduce uncertaintyand thus increase developer motivation.CompNN has embraced user involvement, as-suming it’s a key source of innovation.8
The positive experiences reported by stake-holders stemmed mainly from a feeling of beinglistened to and able to affect the development.
Evo is a sophisticated process, highly de-pendent on diligent fulfillment of the specifiedroles and activities. This sophistication is alsoEvo’s Achilles’ heel. Evo is very vulnerable toirregularities. This process fragility increaseswith the frequency of iterations, which puthigher demands on the timeliness of role ful-fillment and activities. Unresponsive stake-holders expose this fragility. Our observationsfrom the ProdNN 10.0 release demonstratethis. CompNN’s experiences with weekly iter-ations showed this schedule to be too de-manding. An adopting organization must takemultiple factors into account when determin-ing the frequency. Evo strongly recommendsiterations consuming less than 5 percent of thetotal project funding.1 CompNN’s projectsused between 10 and 20 iterations, each con-
suming 5–10 percent of the total project fund-ing. Even this is too frequent, as the experi-ence from the 9.0 and 10.0 releases showed.CompNN plans to experiment with iterationsspanning two weeks instead of one in the com-ing release, hoping to contain fragility andleave more time for development by reducingthe coordination overhead.
Potential adopters of this process must alsoconsider their available resources for managingstakeholder involvement when determining theiteration frequency. Additionally, if the risk ofmissing stakeholders’ real concerns is consid-ered low or moderate, companies might want tostart with longer iterations and then shortenthem to train the organization for Evo. If therisk is considered high, they can use more fre-quent iterations but then with a strong attentionto effective stakeholder management. Success isimportant not least because the consequences ofabandoning this highly motivating way ofworking can be devastating.
E vo certainly isn’t a process for thefaint of heart. As we’ve seen, imple-menting and maintaining the organi-
zational and technical capability to supportweekly iterations is expensive and requires ahigh degree of discipline at many organiza-tional levels. Roles other than those we’vestudied might have been affected, thus consti-tuting additional costs to adopting Evo. Orga-nizational changes take time and effort. Dur-ing the two years of our study, CompNNimplemented and matured a new development
About the AuthorsTor Erlend Fægri is a researcher on software process improvement and software archi-tecture at SINTEF ICT, a research division of Norway’s largest independent research institution.He has an MSc in computing science from the University of Glasgow. Contact him at SINTEF ICT,No-7465 Trondheim, Norway; [email protected].
Geir Kjetil Hanssen is a researcher on software process improvement and methodolo-gies at SINTEF ICT, a research division of Norway’s largest independent research institution. Hehas an MSc in informatics from the University of Trondheim. Contact him at SINTEF ICT, No-7465Trondheim, Norway; [email protected].
1 0 4 I E E E S O F T W A R E w w w. c o m p u t e r. o r g / s o f t w a r e
process. However, a company should con-sciously and regularly consider whether it’snecessary to adapt their development pro-cesses. CompNN has done this by introducingthe green week in the 9.0 release and by ex-tending the iterations to two weeks in the com-ing release. Other adaptations are likely in fu-ture releases.
As Frederick Brooks so eloquently put it,there are no silver bullets in software engineer-ing.7 Nevertheless, improvement through evo-lution is undeniably a powerful practice.
AcknowledgmentsThe Research Council of Norway sponsored this
study through the SPIKE research program under grant156701/220. We thank CompNN for their kind co-operation.
References1. T. Gilb, Competitive Engineering: A Handbook for Sys-
tems Engineering, Requirements Engineering, and Soft-ware Engineering using Planguage, Elsevier Butterworth-Heinemann, 2005.
2. T.Gilb, “Evolutionary Development,” ACM SoftwareEng. Notes, Apr. 1981, p. 17.
3. P. Abrahamsson et al., “Agile Software DevelopmentMethods—Review and Analysis,” VTT Publications,no. 478, VTT Electronics, 2002.
4. B.W. Boehm, “A Spiral Model of Software Develop-ment and Enhancement,” Computer, vol. 21, no. 5,1988, pp. 61–72.
5. B. Boehm, “Get Ready for Agile Methods, with Care,”Computer, vol. 35, no. 1, 2002, pp. 64–69.
6 W.W. Royce, “Managing the Development of LargeSoftware Systems,” Proc. IEEE Westcon, 1970;reprinted in Proc. 9th Int’l Conf. Software Eng. (ICSE87), IEEE Press, 1987, pp. 328–338.
7. F.P. Brooks, “No Silver Bullet: Essence and Accidents ofSoftware Engineering,” Computer, vol. 20, no. 4, 1987,pp. 10–19.
8. E. Carmel and S. Becker, “A Process Model for Pack-aged Software Development,” IEEE Trans. Eng. Man-agement, vol. 42, no. 1, 1995, pp. 50–61.
For more information on this or any other computing topic, please visit ourDigital Library at www.computer.org/publications/dlib.
TOPICS OF INTEREST:•Practical tools and methods for detecting
or preventing security-relevant defects•Practical approaches to incorporating security
as part of the software development process•Attacks and vulnerabilities—common ways that
security fails in modern industrial software•The economic motivation for creating secure
software
For author guidelines and submission details, contact the magazine assistant at [email protected] or go to www.computer.org/software/author.htm. Specify that you are submitting for the “Security for theRest of Us” special issue. A detailed call is at www.computer.org/software/cfp.htm.
C A L L F O R A R T I C L E S
Security for the Rest of Us:An Industry Perspective
on the Challenge of Secure Software
GUEST EDITORS:•Konstantin Beznosov, University of British
Columbia; [email protected]
•Brian Chess, Fortify Software; [email protected]
IEEE WWW.COMPUTER.ORG/SOFTWARE
The public need for good software security becomesmore acute every day. Typical activities such as pur-chasing products and services, conducting business,and holding elections increasingly depend on securesoftware. While security was once a specialty of interestto only a small number of developers, it’s now critical foralmost all software developers, project managers, anddecision makers. The worldwide software industry in-cludes thousands of vendors, from massive enterprisesto one-person shops, and the industry as a whole has toface the software security challenge.
PUBLICATION: January/February 2008SUBMISSION DEADLINE: 1 July 2007
Security for the Rest of Us:An Industry Perspective
on the Challenge of Secure Software
Paper 3
Hanssen, G. K. and Fægri, T. E. (2008). Process Fusion - Agile Product Line Engineering: an Industrial Case Study. Journal of Systems and Software 81: 843-854.
Available online at www.sciencedirect.com
www.elsevier.com/locate/jss
The Journal of Systems and Software 81 (2008) 843–854
Process fusion: An industrial case study on agile software productline engineering
Geir K. Hanssen a,b,*, Tor E. Fægri a
a SINTEF ICT, NO-7465 Trondheim, Norwayb Norwegian University of Science and Technology, Department of Computer and Information Science, NO-7491 Trondheim, Norway
Received 27 March 2007; received in revised form 19 July 2007; accepted 10 October 2007Available online 7 November 2007
Abstract
This paper presents a case study of a software product company that has successfully integrated practices from software product lineengineering and agile software development. We show how practices from the two fields support the company’s strategic and tacticalambitions, respectively. We also discuss how the company integrates strategic, tactical and operational processes to optimize collabora-tion and consequently improve its ability to meet market needs, opportunities and challenges. The findings from this study are relevant tosoftware product companies seeking ways to balance agility and product management. The findings also contribute to research on indus-trializing software engineering.� 2007 Elsevier Inc. All rights reserved.
Keywords: Software product development; Software product management; Software product line engineering; Agile software development
1. Introduction
Software engineering is a complex task that involvesfast-paced product development, competitive market con-ditions, knowledge management, organizational factors,rapidly evolving technologies, etc. As a means of handlingthis complexity, establishing control and using resourcesefficiently, development organizations normally adopt amethod that defines how their software engineering activi-ties should be carried out within the given context. Amethod may cover a broad range of issues; typically esti-mation, design, and development, among others. A greatmany methodologies are available and they vary a lot; fromstrict plan-based approaches to agile approaches and anyvariant in between. Regardless of the method being used,most software projects strive to reach a balance between
0164-1212/$ - see front matter � 2007 Elsevier Inc. All rights reserved.
doi:10.1016/j.jss.2007.10.025
* Corresponding author. Address: SINTEF ICT, NO-7465 Trondheim,Norway.
E-mail addresses: [email protected] (G.K. Hanssen), [email protected] (T.E. Fægri).
three basic goals: satisfactory software quality (scope),the right cost and timely delivery. Attempting to satisfythese requirements causes further complications to arise,particularly with respect to long-term product managementissues.
This paper discusses two popular development approaches;software product line engineering (SPLE) and agile soft-ware development (ASD). It describes how these have beencombined to improve the ability to achieve the three basicgoals. These two approaches can, in their most radicalforms, be placed in each end of a plan-based/agile spectrum(Boehm, 2002). The former is based on planning and prep-arations for efficient software development based on rapidconstruction by assembling predeveloped assets, while thelatter aims at efficient change response instead of extensiveup-front planning. They may correspondingly be catego-rized as proactive and reactive approaches to softwareengineering. The principles of SPLE have been in use fora long time and industrial experience shows that theapproach has substantial advantages when it comes tocost-efficient development, product quality and the ability
1 We say that a product is derived from the platform if a significant partof the product is reused from it.
2 http://www.sei.cmu.edu/productlines/framework.html (accessedDecember 2006).
844 G.K. Hanssen, T.E. Fægri / The Journal of Systems and Software 81 (2008) 843–854
to deliver on time (Birk et al., 2003; Linden, 2002; Bosch,2000; Clements and Northrop, 2002). ASD is a more recenttrend within the software engineering community and hasattracted wide interest in both industry and academia. Ini-tial evidence from the results of ongoing research suggeststhat using ASD is advantageous, given the right context(Boehm, 2002; Erickson et al., 2005). Both approacheshave the same overall objective; that of improving softwaredevelopment efficiency.
Recently, there has been interest in investigatingwhether, and if so how, SPLE and ASD can be combinedto complement each other (Carbon et al., 2006; Tian andCooper, 2006), but little empirical research on the topichas been conducted. In this paper, we present our findingsfrom a case study of a medium-sized, Norwegian softwareproduct company called CompNN (real name made anon-ymous). CompNN has used and matured the ASD methodEvo, short for Evolutionary Project Management (Gilb,2005) for three years, covering four releases of their mainproduct.
Our study is based on a well established industry–researcher relationship that began several years ago. Thestudy expands upon a previous one, in which we investi-gated the introduction and preliminary effects of Evo(Hanssen and Fægri, 2006). This previous study gave usinsights into several important aspects of the use of an agileprocess in a product development context, with respect tobenefits, problems and prerequisites.
The aim of the study reported herein was to describe andanalyze an industrial case to understand how SPLE andASD can be combined and to clarify associated costs andgains.
Next, we provide an overview of SPLE and ASD andexplain how they both relate to, and conflict with, eachother (Section 2). Then we explain the context of the casestudy (Section 3). Then, we present the study method (Sec-tion 4). Subsequently, we present the findings from thestudy (Section 5), discuss the results (Section 6) and offerour conclusions (Section 7).
2. State of the practice
SPLE is an approach to software engineering in whicha set of products is built on a common set of core assets(Bosch, 2000; Clements and Northrop, 2002). SPLE drawsupon principles similar to those that have been used fordecades in most other industries, in which productionincludes the assembly of prefabricated parts. SPLE isclaimed to have numerous benefits: large-scale productiv-ity gains, decreased time-to-market, increased productquality, increased customer satisfaction, more efficientuse of human resources, and improved ability to applymass customization, maintain market presence and sustainunprecedented growth. However, achieving these claimedbenefits requires careful attention to a wide range of issues(Linden, 2002; Knauber et al., 2000; Kakola and Duenas,2006).
The basic assumption underpinning SPLE’s claimedadvantages is that there is sufficient commonality amongmultiple products to justify investment in their systematicreuse. Reusable assets, of various kinds, are said to consti-tute the product line platform. Examples include require-ments, architectures, designs, code and test cases.Domain engineering consists of the set of activities neces-sary to define, extend and maintain the platform. The plat-form, the key to an SPLE approach, bears the cost ofdomain engineering, which can, presumably, be amortizedacross multiple, derived products.1
Multiple preconditions must be satisfied if an SPLEapproach is to be profitable, but the most fundamental ofthese is that the market for the products be predicted accu-rately. If prediction of market trends is inaccurate, SPLEwill inhibit the business potential of the organization byenforcing wasteful effort related to adherence to referencearchitectures, organizational overhead, reuse strategies,etc., essentially slowing down the software production. Inaddition, product management is more complicated dueto complex dependencies between product variants andthe scope of the platform (Bosch, 2006; Helferich et al.,2006). SPLE has an influence on most practices in softwareengineering in an organization. Thus, SPLE activities mustbe based on the strategic ambitions of the organizationand, in the service of such ambitions, careful planning.SPLE demands long-term investment and effort. Neverthe-less, given the right processes and a suitable product plat-form, new products can be derived faster, at lower cost,and with more predictable quality (Birk et al., 2003). SPLEhas already become economically necessary for some com-panies (Linden, 2002).
There are more or less rigorous ways to apply SPLEpractices. For example, product derivation can be imple-mented using proactive engineering or reactive engineering.In a proactive approach, the reusable assets are developedbefore the product is built. In a reactive approach, theproduct is built first and subsequently the reusable assetsare recovered from the product and integrated back intothe platform. Further, variability models are used to repre-sent variation among the products that the platform shouldsupport, i.e. which variation defines the scope of the prod-uct line? A vast number of different techniques and sup-portive artifacts exist for doing SPLE, but theirdiscussion lies beyond the scope of this paper. For furtherdetails, please refer to, e.g. (Bosch, 2000; Clements andNorthrop, 2002). It is useful to consider SPLE as a spec-trum of approaches to software engineering.2 SPLE mayhave a wide impact on business, organization, processand technological factors (Sugumaran et al., 2006). There-fore, organizations that adopt SPLE must choose their ownpath by introducing a set of practices that suit the
G.K. Hanssen, T.E. Fægri / The Journal of Systems and Software 81 (2008) 843–854 845
objectives and capabilities of the organization (Lorge Par-nas and Clements, 1986). Naturally, as with any otherinnovation in software processes, SPLE practices shouldbe reviewed and revised continually to ensure learningand improvement.
Agile software development (ASD) lies at the other endof the plan-based/agile spectrum. ASD focuses little onplanning and expects limited predictability. Further, it doesnot address issues of reuse. Hence, it contrasts sharply withthe plan-centric SPLE approach. While SPLE supports thestrategic objectives of the organization, ASD primarilybenefits tactical ambitions, for example, addressing theimmediate demands of customers or targeting smaller-scaledevelopment efforts.
ASD is a common name for a set of defined methods forsoftware development. Some of the best known and usedare Extreme Programming (Beck, 2000) and Scrum (Schw-aber and Beedle, 2001). Although these agile methods varyin focus and presentation, they are all based on a few sim-ple guidelines for ASD, defined by the Agile Manifesto3
which states four basic values: (1) Individuals and interac-
tions over processes and tools; ASD emphasize self-organiz-ing teams, close cooperation with the user(s) and informaland direct communication. (2) Working software over com-
prehensive documentation; the main goal, and main proof ofprogress, is working software, not models or demonstra-tors. Documentation is kept on a strictly need-to-havebasis. (3) Customer collaboration over contract negotiation;the customer is given an active role in development andthus has a direct responsibility to state requirements andcorrespondingly to verify both incremental and finalresults. (4) Responding to change over following a plan;development is a dynamic and creative endeavor, so plansare kept on a minimum with the focus being on flexibilityand response to change. In addition to these four values,the Agile Manifesto also defines a handful of principlesthat form rules for development. These again are used toform a set of practices in the various agile methods. Somemethods emphasize practice and technique descriptions(such as Extreme Programming), while others place moreemphasis on management activities (such as Scrum) (Abra-hamsson et al., 2002).
As part of the interest in, and uptake of, agile methodsin industry a variety of experiences have been reported.Most of these can be defined as simple, single-case studies,but there are some reports that have investigated more dee-ply and have contributed to a growing knowledge base onthe costs and benefits of agile methods. The agile methodExtreme Programming has attracted the most attention.Within this method, the practice of pair programminghas been the most investigated by far (Erickson et al.,2005). In this regard, a recent large controlled experimenthas clarified the conditions under which pairs of developersperform better than individuals and vice versa (Arisholm
3 www.agilemanifesto.org (accessed November 2006).
et al., 2007). Task complexity and level of experience wereshown to influence the outcome. Another relatively well-investigated agile practice is test-driven development(TDD) (Muller and Hagner, 2002; George and Williams,2004; Erdogmus and Morisio, 2005). The results are some-what contradictory. Erdogmus and Morisio (2005) showedthat TDD improves productivity but not quality, yet theirresults also indicated that TDD programmers achieve moreconsistent quality results. With respect to Extreme Pro-gramming as a whole, Erickson et al. (2005) has reviewednumerous studies on the method and found no clear evi-dence as to its benefits. A possible reason is that ExtremeProgramming, as along with other agile methods, is consti-tuted by a set of diverse guidelines, each of which should beinvestigated individually.
SPLE and ASD have significant differences due to thefundamentally different challenges they are intended tosolve. SPLE addresses longer-term strategic objectivesrelated to life-cycle product management, whereas ASDaddresses short-term tactical objectives, such as single pro-jects developing one specific product. In practice, an orga-nization will face both strategic and tactical challenges.Thus, it makes sense to investigate whether, and if sohow, these two approaches can complement each other.The table below summarizes how the SPLE approach cor-responds to the twelve principles of the Agile Manifesto(see Table 1).
This comparison shows that ASD and SPLE have fewconflicts. They are virtually independent of each other.On most counts, they possess complimentary properties(Carbon et al., 2006; Tian and Cooper, 2006). WhereasSPLE attempts to introduce large-scale cross-product eco-nomic benefits through reuse, ASD focuses on deliveringsingle products with the best possible quality for the cus-tomer. Thus, there is, prima facie, no reason why thesetwo approaches should not be combined. On the contrary,there is a clear motivation to combine them.
SPLE as such is agnostic regarding processes of softwaredevelopment. One can use product line engineering with dif-ferent software process models. However, intuitively, thereare a number of interesting issues to be addressed when con-sidering the use of agile methods in a SPLE setting:
– SPLE advocates a conscious adherence to architecture.Software reuse beyond the most trivial level demandsthat numerous constraints and guidelines be respectedregarding both the reused asset and the consumer ofthe asset. Agile methods have a reputation for paying lit-tle attention to architecture, and occasionally beingdownright damaging to it. Naturally, architecture ero-sion will result if constraints and guidelines are broken.Hence, it is useful to discuss the practices used by agileproduct companies that enable long-term architectureconformity and prosperity of the product line.
– Agile methods are (arguably) best suited to projects ofmodest to moderate size (Boehm and Turner, 2004).The different products (or variants) in a product line
Table 1Comparing ASD and SPLE
Agile principle SPLE correspondence
Priority on early and continuous delivery of valuable software Dependent upon SPLE approach. A reactive model supports early delivery ofvaluable software (no particular bias with respect to continuous delivery)
Welcome changing requirements, even late in development. Agileprocesses harness change for the customer’s competitive advantage
Foreseen changes covered by platform and variability models are cheap.Other changes become increasingly expensive closer to domain engineeringartifacts as they may participate as elements in multiple products, and thusincrease the cost of maintenance
Deliver working software frequently, from a couple of weeks to a coupleof months, with a preference to the shorter timescale
An SPLE approach does not reject or inhibit this principle. Product deliveryis, nevertheless, somewhat slowed down compared to agile development, dueto overhead in maintaining integrity with the product platform
Business people and developers must work together daily throughout theproject
An SPLE approach neither rejects nor inhibits this practice
Build projects around motivated individuals. Give them the environmentand support they need, and trust them to get the job done
At the product level, an SPLE approach neither rejects nor inhibits thispractice. At the platform level, however, more formalism is required
The most efficient and effective method of conveying information to andwithin a development team is face-to-face conversation
SPLE, due to increased technical and/or organizational complexity,introduces a need for more explicit, formal and disciplined communication
Working software is the primary measure of progress SPLE incorporates significant value in the platform that is more difficult tomeasure However, for product development, an SPLE approach neitherreject nor inhibits this principle
Agile processes promote sustainable development. The sponsors,developers, and users should be able to maintain a constant paceindefinitely
SPLE promotes sustainable development at a larger scale
Continuous attention to technical excellence and good design enhancesagility
SPLE platforms encode technical excellence and design to support rapidproduct derivation
Simplicity – the art of maximizing the amount of work not done – is –essential
The main motivation for SPLE is the same – but by using reuse as vehicle foreliminating work. In a reactive SPLE approach, the risk of doing unnecessarywork is reduced
The best architectures, requirements, and designs emerge from self-organizing teams
The assumption of SPLE is that investments in requirements, architectures,and designs can be reused successfully across multiple products
At regular intervals, the team reflects on how to become more effective,then tunes and adjusts its behavior accordingly
An SPLE approach neither rejects nor inhibits this practice. In fact, due toinvestments in the platform, most SPLE organization promotes experiencegathering and learning
846 G.K. Hanssen, T.E. Fægri / The Journal of Systems and Software 81 (2008) 843–854
can be considered as semi-autonomous projects withinthe line. In this regard, SPLE may be considered as away of partitioning the work of software product com-panies and subsequently assisting the use of agile meth-ods in larger-scale development efforts.
How can potential benefits and challenges be managedin practice?
3. Study context
CompNN is a medium-sized Norwegian software com-pany that develops, maintains and markets a product linecalled ProdNN (real name made anonymous). The prod-ucts derived from the product line are aimed at the high-end market of market and customer surveys. CompNNhas a wide customer base that includes some of the world’slargest market research agencies. The company was estab-lished by three friends in 1996 and has since grown steadily,such that it now has about 90 employees and offices in Nor-way, the UK, Vietnam, and the USA. There has been agradual shift from building custom-made applications toa product line. Today, licenses for the ProdNN productsconstitute most of the revenue.
The bulk of research and development is done at themain office in Norway supported by developers in Viet-
nam. The other offices are mainly devoted to marketing,sales, and customer support. However, as we will showlater, they have a clear role to play in providing importantinput to the development process. Physical presence nearthe largest markets abroad is important both to serve cus-tomers and to include them in the development of theproduct.
ProdNN can be defined as a single product, but thereare many ways to use it. The system is modular. It has fivemain modules (with numerous submodules): authoring toassist the planning of surveys, panel to support panel sur-veys, a survey engine that serves most use cases, reporting
to produce survey reports, and data transfer to feed thedatabase for analysis. The use of these modules variesaccording to the use case. Some modules are central andalways in use, while the use of the others depends on thesituation. As CompNN’s customers vary greatly in sizeand operation, the ProdNN tool is built to be customiz-able. CompNN operates with a set of predefined configura-tions for the most common use cases, but there is alsobuilt-in support for detailed customization to support morevariants. However, in some cases, a customer has needsthat go beyond the functionality available in the productplatform. In these cases, CompNN may provide a cus-tom-developed solution for the customer, with the provisothat the result may be included in the platform. If this is
G.K. Hanssen, T.E. Fægri / The Journal of Systems and Software 81 (2008) 843–854 847
not acceptable to the customer CompNN will deny therequest. This strategy of restrictive scoping is chosen toavoid too heavy a burden with respect to single-case devel-opment, because that would hamper the continuous devel-opment of the platform.
The adoption of Evo was a deliberate choice to move onfrom an inefficient waterfall-like process (Fægri and Hans-sen, 2007). In this case, SPLE was not introduced deliber-ately, as prescribed by common guidelines, e.g. theSoftware Engineering Institute’s guidelines.4 However, thiscompany’s development process clearly coincides withsome of the fundamental principles of SPLE. Thus, wewere able to use this case to understand how the companyhas combined SPLE and ASD and to identify the experi-enced effects of the combination. We believe that this isvaluable input to the process of shaping an agile productline engineering practice, which is in its early stages.
Several roles cooperate at different levels in the develop-ment of the product platform. At a tactical level, the devel-opers in the R&D department handle the softwaredevelopment, guided by the Evo process. The developersare organized in small teams, each of which has three tofive persons. To ensure that focus is maintained, each teamis usually dedicated to the development of just one of themain modules in the platform. In addition, each teammay also cooperate directly with a selected stakeholder.The R&D department has a Chief Technical Officer(CTO), who is responsible for the platform architectureand for ensuring technical alignment with the businessdomain, and who oversees overall product planning.CompNN puts great emphasis on new and improved qual-ities which leads to a high pace in the development projectsand an unsatisfactory error density in released products.This issue is now being addressed by the adoption oftest-driven development.
To handle the diverse processes towards the market atboth strategic and tactical levels, CompNN has a ProductManagement Team (PMT). This function was establishedat the same time as Evo was introduced, in 2004. ThePMT has six members, one of whom acts as the PMT man-ager. The PMT plays an important role at several levels.They are responsible for marketing initiatives, sales, andkey account management. The product roadmap, togetherwith supporting business cases, are the main assets devel-oped by the PMT. Two main channels of information areused to develop these. First, there is the PMT member’sknowledge of the domain, market and specific customers.Secondly, the ProdNN Advisory Board, was establishedrecently. This is a group of the most important customers,who provide their ideas and state their requirements forfuture development. The members of this forum are CEOs(Chief Executive Officer), board members and other repre-sentatives from top management.
4 http://www.sei.cmu.edu/productlines/adopting_spl.html (accessedDecember 2006).
One of the core principles in the adopted Evo process isthe close cooperation with selected stakeholders (selectedrepresentatives of important requirements). When a prod-uct roadmap has been approved by the executive manage-ment, the PMT appoints stakeholders, who agree toparticipate actively in the release projects. Usually, onestakeholder per project is appointed, which again placesthe focus on one of the modules or a well-defined feature(Fægri and Hanssen, 2007). The criterion for appointinga stakeholder is that the development focus or goal ofthe project addresses a module or feature in which thestakeholder has a special interest, which in turn providessufficient motivation for investing resources in develop-ment. In some cases, internal stakeholders are appointed,typically when the project addresses issues that are notdirectly visible to the end users. External stakeholders arenot given any compensation for participating. The sole rea-son for cooperating is the opportunity to affect and directthe development.
The roles and their participation in product develop-ment activities are summarized in Fig. 1. Their interplayis further elaborated in Section 5.
4. Study method
4.1. Rationale
As this study sought to investigate the diverse and com-plex topic of the fusion of product-line engineering andagile software development, which touches on a varietyof factors, such as technology, architecture, process, mar-ket development, innovation, and management and organi-zational issues, we chose a qualitative approach. Our aimwas to present a broad view and to explain how this fusionhas worked in practice in actual industry, with a view tounderstanding how SPLE and ASD can be adopted andused to complement each other. We believed that weneeded to understand the practicalities and opportunitiesof this fusion better before we can plan more specific stud-ies on, for example, the effect on time, cost and quality.Thus, we performed a single-case study (Yin and Camp-bell, 2002) to obtain detailed knowledge about a completeproduct development organization to be able to under-stand how internal and external roles interplay at variouslevels.
4.2. Data collection
Our main source of data collection was a series of inter-views. We used existing interviews from a previous study(Hanssen and Fægri, 2006). There were nine interviews intotal, which three roles: three customers (different compa-nies) from the development of a single release (three inter-views), all members of a product management team (fiveinterviews) and a group of six developers (group interview)where all of these had recent experience from Evo-projects.To acquire a deeper understanding of the strategic and
AcPro
Roles
PMT
CTOCEO
Adv.board
Proj. m
gmt.
R&DStak
eholde
rs
Users
Support
ArtifactsActivitiesProcess level
Strategic Product planning X X XX X
FeedbackMarket knowledgeRoadmapBusiness case
Release development X X X X X Roadmap
Proj. Management X XIETSoftware
OperationalSoftware useCustomer support XX X
SoftwareFeedbackChange req.
Tactical
Fig. 1. Roles, activities and artifacts.
848 G.K. Hanssen, T.E. Fægri / The Journal of Systems and Software 81 (2008) 843–854
tactical processes and how the development process andthe product line system architecture relate to each other,we extended the interview material with thorough semistructured interviews with the PMT manager and theCTO. These two roles are central to the planning and man-agement of the product development process. We definedan interview guide for each of the PMT manager andCTO, but the interviews were semi-structured so that addi-tional information, which was not originally covered in theinterview guides, could be acquired. All interviews exceptthe group interview with the developers were done via tele-phone. Furthermore, all interviews were recorded using aDictaphone. In sum, we thus base our case study on eleveninterviews covering all roles involved in the product lineengineering and the applied agile process.
As a secondary data source, we used documents such asproduct roadmaps and business cases, which are importantassets in long- and short-term development of the productline. These documents were used to understand the overallpractice of software development and market-related activ-ities and how these relate to each other.
4.3. Data analysis
All recorded interviews were transcribed to form a basisfor a thorough and detailed textual analysis. In cases ofinterviews that covered more than one individual per role,for example the members of the product managementteam, we analyzed the material according to the principlesof constant comparison (Seaman, 1999) using the NVivoTM
tool for textual encoding. The transcribed interviews andthe documents were used to make a description of theproduct development (see Section 5).
4.4. Bias and limitations
Several factors restrict the credibility and generalizabil-ity of the study. (1) We collected and analyzed data fromonly one company. However, if we had involved morecompany cases, the study would have been much more dif-ficult to perform. In addition, we only know of, and have
access to, one company that has adopted and combinedboth SPLE and ASD practices; the one that we studied.Despite this limitation, a reader may, by considering thecontext information, be able to identify experiences thatmay be transferred to another software development orga-nization. (2) There is a threat with respect to the complete-ness of our study. We did not interview all parties involvedin the total process of developing the product platform, butwe believe we have selected the most central; those thatcould give us a good basis for reflection. To strengthenthe study further, we incorporated the analysis of impor-tant documentation.
5. Findings – a holistic view of software engineering at
CompNN
We now describe, from a holistic point of view, howCompNN develops and maintains their product platform.We have identified three distinct software processes anddescribe how they interact. Then we describe the productplatform architecture and related services followed by adescription of how the organization has shifted to thisnew development regime.
5.1. A symbiosis of three interacting customer-centric
software processes
Organizations of some size need well defined structureand operational formalism to ensure that important func-tions are implemented and maintained in a professionalmanner. CompNN has matured their approach to softwareengineering over the last 10 years. As new challenges haveappeared, CompNN has responded by making differentadjustments and innovations in their processes. BecauseCompNN operates in a highly competitive market, theymust continually look for opportunities to improve theway they do business. Thus, the description presented hereis merely a snapshot of how things are done currently.
The Fig. 2 illustrates how CompNN, through a symbio-sis of three different software processes, connects softwaredevelopment and customer collaboration. The embedded
Stratetic (1-2yr horizon)
Tactical (2wk-4mnth horizon)
Operational (day-to-day operations)
Function Strategic,technologicalinnovation
Tactical (reactive) innovation
Systematize experience
Process elements SPLE practices Evo, IET Request driven, prioritization
Deliverables Productroadmaps, business cases
Software
Technical focus ProdNN software platform
Release projects Fixes, updates, upgrades
Key roles Management, PMT, CTO
R&D, PMT, CTO Support, QA
Customercollaboration
Advisory Board Key stakeholders End users, ASP operations personel
Fig. 2. A customer-centric product development process.
G.K. Hanssen, T.E. Fægri / The Journal of Systems and Software 81 (2008) 843–854 849
table highlights the various issues of particular attention inthe three processes. As a result, CompNN is able toembrace the market all the way down to individual users,and thus improve their ability to respond and interact withtheir customers. We believe this works well because thethree processes serve different but important needs:
Strategic (long-term, SPLE): Evolution at the SPLElevel addresses the need to implement long-term strategicplans. SPLE supports CompNN’s need to introduce tacti-cal innovation, which is fuelled, for example, by innova-tions in technology, changes in the business models, ornew business opportunities. ProdNN is a product platformthat contains a set of building blocks that can be tailored toprovide different product offerings (ProdNN solutions).This platform can be considered as the vehicle used bythe CTO and PMT to implement long-term visions. Moreoften than not, these innovations are costly in terms ofeffort and risk. Thus, they should be planned and intro-duced with great care, in order to exploit the organization’sdevelopment capability. SPLE-style development inher-ently supports this ambition.
Tactical (medium-term, ASD): Evolution at the projectlevel (note that a release typically includes 4–6 projects)
addresses the need to embrace customers and listen to theirmore down-to-earth needs. Close cooperation ensures thata customer’s ideas and experience with the software istranslated into what we may call tactical innovation, i.e.reactive innovation that seeks to polish, improve, adjustor otherwise make small to moderate adjustments to theproduct. As the customer’s requirements mature duringthe project, ASD-style development with moderate-sizeprojects works well to accomplish this aim.
Operational (short-term, day-to-day operations): Cus-tomers rely on the software to help them to implement vitalbusiness functions in their own organizations. Thus, it is ofgreat importance that CompNN is able to sustain a goodlevel of satisfaction with the software in its day-to-dayuse. During operation, the products are exposed to moreusers, which exposure increases the potential amount offeedback to the innovations implemented at the strategicand tactical levels.
These three processes are concurrent. They overlap interms of activities, people and artifacts. All three embracethe customers and give CompNN the opportunity tobuild strong, cross-level, collaborative relationships withthem.
850 G.K. Hanssen, T.E. Fægri / The Journal of Systems and Software 81 (2008) 843–854
5.1.1. The strategic process elaborated
The market process is at a strategic level and providesvital input to the work on the biannual releases and thedevelopment projects. This strategic work is operated bythe product management team (PMT), who are engagedboth in product development and other general market-ori-ented activities, such as sales, marketing, and key accountmanagement. Thus, they act as a link between market (stra-tegic) and development (tactical) activities.
The central activity in the market process is the develop-ment of a roadmap. This is a strategic document whereneeds, ideas and opportunities for future development aredocumented. In practice, the roadmap is a set of Power-Point slides that defines growth opportunities, main chal-lenges and strategies. Each issue is described at a highlevel, to prevent the view of features and functionality frombecoming too detailed. On the basis of the suggested direc-tion for developing the product platform, the PMT alsoindicates how existing or new resources should be used;for example, a need to extend the R&D department. Theroadmap also defines what is called ‘Commitments andmust haves’, which is a high-level list of improvementsand extensions that presumably, will meet the identifiedopportunities and challenges. To operationalize the road-map, the PMT also suggests how to set up a handful ofdevelopment projects to reach the defined commitmentsand must haves. This includes the development focus perproject and staffing (number of developers per project).To establish a long-term view on development, e.g. toexpose resource needs, this part of the roadmap character-izes the release projects for both upcoming and subsequentreleases, though subsequent releases are characterized moreloosely. The last part of the roadmap looks even furtherinto the future and defines potential new areas for the fol-lowing two years. This overview is developed further in thefollowing year’s roadmap.
The PMT develops a new roadmap once a year. A pro-posal is given to executive management in the autumn forapproval, and is then used as a guide to decide formallywhich parts of the product platform will be addressed inthe upcoming releases. The goal descriptions in the road-map are deliberately kept at a high-level. Thus, the possi-bility is left open that projects may suggest refinements tothe platform to facilitate achieving the goals. This is impor-tant, because the development projects will utilize coreagile principles, such as short iterations and close customerengagement as defined by the Evo development process, tocapture detailed requirements for finding the best possiblepractical solution for achieving the high-level goals definedin the roadmap.
The roadmap relates to a set of business cases, which aredocuments that illustrate the assumed effect of achievingthe high-level goals. A business case usually addressesone of the modules in the product platform to obtain amanageable focus and follows a standard structure: itdefines the stakeholders, the scope of the project (what isto be done), and its rationale. A business case also contains
a discussion on the return on investment, identified risksand threats, and associated costs.
To provide management with the best possible founda-tion for making their decisions what is proposed in theroadmap, it also contains descriptions of cases that arenot a part of it, thus illustrating trade-offs that must bemade by executive management due to limitations of timeand resources; the total quantity of ideas always exceedsthe quantity that can be developed for one release. ThePMT thus keep track of ideas that are excluded from theroadmap and these are revised each time a new roadmapis developed.
5.1.2. The tactical process elaborated
Gilb (2005) is the agile method that CompNN adoptedat the tactical level, i.e. the level where software is devel-oped. It is not as well-known as Extreme Programming(Beck, 2000) and Scrum (Schwaber and Beedle, 2001), yetit is clearly an agile method, because it is customer-centricand strongly emphasizes short iterations and the frequentdelivery of working software. In Evo, the product evolvesiteratively as new requirements are discovered, constructedinto the code, and subsequently tested. Stakeholders (cus-tomers or internal) are expected to give feedback on eachof the iterations of the product in order to guide the devel-opment of the next iteration. Finally, suites of integrationtests are performed before the final system test. Evo is a rig-orous process that draws upon principles from the risk-dri-ven Spiral model (Boehm, 1988) but is ultimately driven bythe highest possible stakeholder value in each iteration,rather than by risk reduction.
The core process of Evo is requirements management,which serves two purposes per iteration. Firstly, selectedstakeholders evaluate the achievements in the previous iter-ation in terms of a hands-on test or some other type ofreview. This helps to ensure that progress is in line withstakeholders’ expectations. Secondly, on the basis of stake-holders’ feedback, it is decided what existing requirementsneed to be improved and what new requirements should betargeted for the next iteration. Requirements are not spec-ified as functions or features, but as product goals thataddress real business concerns of the customer businessconcerns. Great emphasis is placed on identifying metricsand associating them with each goal. CompNN usesImpact Estimation Tables (IET) as its key project manage-ment tool. Each of the projects within a release has an IET,which is essentially a spreadsheet that lists goals verticallyand project iterations horizontally. Each of the iterationsis broken down into one or two solutions (design propos-als) that are believed to address the goals. The structurehelps to separate requirements from solutions. By docu-menting both requirements and project progress (measuredresults) per iteration, the IET becomes both a plan for, andhistorical record of, the project. Before release of the itera-tion is initiated (which initiation spans roughly two weeks),selected customers are invited to give input to productplans with initial goals for the release. During initiation,
G.K. Hanssen, T.E. Fægri / The Journal of Systems and Software 81 (2008) 843–854 851
the most promising ways of addressing the initial goals arenoted and laid out for a number of future iterations in theIET. CompNN’s CTO explained: ‘‘This plan helps usensure adherence to an overall technical strategy for prod-uct development’’. Evo at CompNN is a tightly scheduledprocess in which biweekly iterations follow a fixed schedule(called the ‘Evo-week’) that defines each role’s responsibil-ities per day.
In addition to playing a vital role in the strategic pro-cess, the PMT also have responsibilities in the tactical pro-cess, in connection with the software development in therelease projects. Because they have the best access toknowledge of the market and the customers (Keil and Car-mel, 1995), the PMT are responsible for appointing cus-tomer stakeholders in the development projects and formaintaining the link between CompNN and these stake-holders. If a customer stakeholder leaves a project forany reason, it is vital that the PMT take action to appointanother stakeholder or to fill the gap in some other way.
During the second Evo-release, CompNN introduces anadditional concept: the green week. This is a week dedi-cated to error correction and improvements, withoutengaging with customers directly.
Another great aid in the development process (while notbeing a part of Evo) is the Continuous Integration frame-work (CI). This is a set of well-integrated tools for codemanagement, automated tests and builds that enable swiftproduction of product versions ready for testing bystakeholders.
5.1.3. The operational process elaboratedThe operational process consists of the day-to-day activ-
ities in connection with the customer’s use of the productsbased on the product platform. As CompNN provides upto two releases per year, which surpasses the update fre-quency of most customers, there are several versions ofthe platform in use at any one time. In addition, thereare many variants based on the product platform in use,all of which may be considered advanced applications. Insum, this means that there is great variance in the supportrequests to CompNN, which are handled by a dedicatedsupport department. Depending on the situation, supportmay involve guidance in use of the product, scripting, pro-gramming, quality assurance of customers’ use, or issuesrelated to maintenance. Some requests also address errorsand flaws in the software, even ideas for future develop-ment. Such requests are recorded in a support system andare used as input to the strategic process that is aimed atupcoming releases. In addition to assisting in the use ofthe software and recording issues that pertain to potentialimprovement, the support department develops userdocumentation.
5.2. Product platform architecture and services
The ProdNN architecture is deliberately kept simple toachieve flexibility. From a technical point of view, it is a
traditional three-tiered application with a data layer, abusiness layer and a web-based (ASP) interface layer. Thewhole platform is based on the Microsoft.Net platform.Some old COM-components are present and constitute alegacy from the first years of shaping the platform.Recently, the R&D department did so-called interimreleases, in which no external stakeholders were involved,and concerning which the only purpose was to port oldcode, fix errors and resolve pressing open issues. Now,these types of task are performed using free developmentresources in between the ordinary releases.
All end-user interaction with the software is done via across-platform web-based interface and two alternativedelivery models; for sophisticated use or large organiza-tions, CompNN offers ProdNN as a stand-alone serversolution, which allows the customer to operate the serverlocally. For less demanding users, CompNN offers anASP alternative, in which the application is hosted byCompNN.
From a functional point of view, the platform consistsof five main modules (with submodules), each of whichsupports specific features or tasks in the survey process.Some modules are essential in all use cases, while othersare optional and depend on the situation. Each module isdesigned to be configurable to achieve the benefit of devel-oping one common component that can serve many situa-tions of use. Variability in customers use is handled simply;a license file is defined for each customer, which file deter-mines the modules that the customer can use. This solutionis the same for both the local server and web-access vari-ants. Working on the next release, development is usuallyorganized such that one development team (defined as aproject) is responsible for one of these modules. R&D will,in the future, establish dedicated teams for the modules,which teams will specialize in the features and the relatedsub domain. ProdNN may be extended by a customer ora provider of third-party solution providers by the use ofa set of API’s (offered as web services), there being onelibrary for each of the five modules.
ProdNN is not provided as a single standard product,but is defined as a product platform that forms the basisfor many variants. As a part of this paradigm, CompNNoffers a set of services to assist each customer in decidingupon a variant that matches their specific needs in eachcase. In most cases, this is done through the ‘customizedsolution’ service, where the basic modules are composedand adjusted. This service may also include documentation,training, support and maintenance. Examples of custom-ized solutions in use are help-desk quality feedback systemsand claim management systems. In cases where a customerhas requirements that are not supported by the productplatform, CompNN offers the ‘custom development’ ser-vice which, besides customization, also involves the devel-opment of new features. This service may be offered ifthe new features under consideration are thought to bereusable in a future version of the product platform. It isan absolute requirement that the customer agrees to allow
852 G.K. Hanssen, T.E. Fægri / The Journal of Systems and Software 81 (2008) 843–854
the new features to become available to other customers aspart of the product platform. If this is not feasible for thecustomer, CompNN will not develop the features. This pol-icy contributes to keeping the product platform within amanageable scope.
Due to the nature of the product platform, customersoften desire to integrate it with other software systems.CompNN offers the ‘system integration’ service to supportthese cases. Typical systems integrated with ProdNN areCRM-systems, help-desk systems, and call-centre systems.
5.3. The shift to the new approach
From the start of the company, ten years ago, the devel-opment process matured from a more or less ad hoc type ofprocess to a well-defined waterfall-inspired process. Thiswas a non iterative process with emphasis on extensiveplanning and up-front design. Though this process helpedthem to structure the development of the product platform,it became apparent that market needs and trends were notbeing met to their satisfaction. Customers also started toraise questions regarding close collaboration, as the prod-ucts became an important part of their core business. Bychance, some representatives from the company met TomGilb at a workshop and became interested in the conceptof iterative and evolutionary development. After an intro-duction, they decided to try Evo in the upcoming release.At the same time, they introduced the PMT function.These two changes were not originally planned together,but experience quickly showed that they fitted well togetherand formed the basis for the combination of SPLE andASD (Johansen, 2005).
6. Discussion
The integrated software processes at CompNN supportthree key virtues of product development. (1) Technicalexcellence: an open and modular platform architectureimplemented using industry-standard technology enablessimple development and maintenance of the product line.(2) Market knowledge and relevance: the well organized,yet nimble strategic process provides adequate decisionsupport for company management and guidance for thedevelopment projects. (3) Agility: the adoption of Evo,and agile principles, enables fast response to changes instakeholder requirements and accurate delivery of desiredfeatures and qualities to the users. This process configura-tion has been developed and matured within the organiza-tion to provide a useful balance between discipline andagility. We believe this case is an example of how to build‘your home ground’ (Boehm and Turner, 2004) for agileSPLE.
An important observation from our findings is thatCompNN’s integrated development process is not just anaccidental collection of three processes. Rather, the threeprocesses play distinct, supplemental and important rolesat three different levels in CompNN’s overall software
development business. All three are lightweight to reducewaste, that is, unnecessary work (Imai, 1997). Taking abroad view of the strategic, tactical and operational pro-cesses, we see a complete process that has the same func-tion and effects as the more general Deming improvementcycle (Deming, 2000). This is also known as the PDCAcycle, from its four main activities: Plan (plan how to sat-isfy improvement requirements); Do (accomplish plannedactions); Check (monitor the actions and verify the out-come with respect to the planned effects); and Act (imple-ment actions for improvement based on the acquiredinformation). This improvement process, simple in princi-ple, comes in many variants and is central in, for example,Total Quality Management (TQM) (Mathiassen et al.,2001). In CompNN’s case, the Plan and Act steps coincidein the strategic process where business cases, technologytrends and other high-level objectives are used to createthe product roadmap. Both SPLE and ASD allow andencourage continuous learning. The fact that the cycleshave different durations poses no noticeable problems:strategic planning is intrinsically a long-term process. Theroadmap is a stable artefact during the development oftwo released versions. Thus, the tactical process can bemanaged using the roadmap as a guide to the prioritiesand selection of stakeholders in the Evo process. Thereleased products may be amended with patches duringtheir lifetime. However, this is not a problem for the sup-port department in charge of the operative software pro-cess, because feedback from customers is associated withthe patch level of their software. Indeed, the issuing ofpatches enables the support department to verify whetherpatches actually help solve the immediate problems thatcustomers may experience. All feedback to the supportdepartment is available for the PMT when preparing thenext roadmap. In this way, the three cycles feed each otherwith up-to-date, accurate knowledge from the customers.
It is worth noting though, that this approach has itscosts; running the strategic process and working closelywith stakeholders entails a considerable extra overhead,resources that could have been invested in development(as was the previous practice). Despite this drawback,CompNN holds the benefits of these additional activitiesto be more valuable than spending the majority ofresources purely on development activities.
Arguably, the most important benefit of this processconfiguration is that it helps CompNN to exploit both stra-tegic long-term ambitions for innovation and smaller-scaletactical innovations, such as refinement of the software tomeet more detailed end-user requirements. In order toimplement this scheme, the PMT plays a crucial role inmediating and facilitating processes (Fægri and Hanssen,2007). Further, the process configuration is a potent foun-dation for further process innovations. As long as thePDCA cycle is maintained, any component in the configu-ration can be further refined to improve performance. Ifthe company experience the expected growth, the modular-ization gives extra organizational flexibility. For example,
G.K. Hanssen, T.E. Fægri / The Journal of Systems and Software 81 (2008) 843–854 853
introduction of separate roles for platform and productdevelopment within R&D can readily be supported becausethe objectives of strategic and tactical development arealready defined and well established. Additionally, theexplicit engagement of primarily external stakeholders inthe tactical development process ensures that the companyis able to supplement strategic planning by selecting stake-holders based on long-term as well as short-term interest.This is an extra benefit for use by the PMT.
The conscious engagement of market representatives isused as an important driver in the development process asa whole. It is interesting to see the differences in how thisis done in practice at the strategic and tactical levels. Duringthe development of roadmaps, knowledge obtained byPMT and CTO via any kind of market- or technology-ori-ented activities is exploited. In addition, key customer rep-resentatives influence the process directly through theProdNN Advisory Board. Input is deliberately kept to ageneral level to establish a sufficiently wide scope. A firstrelease is presented in detail, less detail is captured for thesecond, and only a rough outline is delivered for the 2-yearvision, which allows for less prescriptive plans into thefuture. Having documented the main ambitions in the road-map, customer representatives are given a much moredetailed and direct role in the Evo projects. Selecting andinvolving the right stakeholders is critical however (Fægriand Hanssen, 2007). Engaged stakeholders provide require-ments and feedback on a detailed level but it is the respon-sibility of the development projects to suggest practicalsolutions that will meet the quality goals stated by theengaged stakeholders. This close cooperation with a fewselected stakeholders promotes valuable creativity (Hans-sen and Fægri, 2006). If process innovation at CompNNhad a more revolutionary style, replacing all existing prac-tices with a pure and very formal SPLE approach, thispotential could have been lost.
The present practice in CompNN was not defined andintroduced from scratch as a major initiative for improvingprocesses. However, the organization, the product plat-form, and the processes and their interplay, have eachdeveloped and matured over time. This has been a naturalevolution, based on experience and needs (Dyba, 2000).Our overall impression from this case is that CompNNhas been successful in combining and balancing practicesfrom SPLE and ASD, essentially adopting those practicesthat are valuable to them, as a company.
Referring back to the figure, the strategic SPLE prac-tices creates a fertile, controlled environment in which thetactical ASD practices can exploit the creative potentialof the developers. Cockburn (2002) defines five dimensionsto describe ‘an agile home ground’. These are: size (numberof personnel), criticality (loss due to impact of defects), per-
sonnel (level of software method understanding and use),dynamism (percent requirement change per month) and cul-ture (percent thriving on chaos versus order). The dimen-sions size and criticality are not significantly affected bythe SPLE/ASD synergy. Looking at the personnel dimen-
sion, we see that their SPLE practices, combined with therather disciplined IET planning tool, give the organizationa rigid framework for requirement specifications. However,due to the freedom of the solution concept in Evo, develop-ers are still encouraged and motivated to creative problemsolving. Thus, the discipline enforced by SPLE and IETenables agility and creativity. Considering the dynamismdimension, much of the long-term planning is performedwithin the strategic SPLE process, leaving a comfortableand controlled amount of dynamism to the individual pro-jects. Last, regarding the culture dimension, we see that ahigh level of order does not inhibit agility, it enables it.
7. Conclusions
The aim of this study was to describe and analyze anindustrial case to understand how SPLE and ASD can becombined and to clarify associated costs and gains.
We found that these two approaches, which at first sightmay seem contradicting, in fact complement each other.SPLE and ASD work together, supporting strategic andtactical objectives, respectively. At CompNN, an opera-tional process constitutes the experience-bearing link fromthe tactical to the strategic process, thus completing theimprovement circle. We found that this, as a whole, consti-tutes a framework that balances discipline and agility(Boehm and Turner, 2004).
The overall process described and discussed hereinvolves multiple disciplines, such as product planning,knowledge management, organizational aspects, and inno-vation. We believe that our conclusions constitute a valu-able contribution to determining the practicalities andeffects of agile SPLE, being an interesting approach toindustrialized software engineering.
Acknowledgements
The authors would like to thank CompNN for theirtime and valuable inputs. We are also grateful to ChrisWright for proofreading. This work was done as part ofthe EVISOFT project funded by the Research Council ofNorway under Grant 156701/220.
References
Abrahamsson, P. et al., 2002. Agile software development methods –Review and analysis. VTT Electronics, 112.
Arisholm, E. et al., 2007. Evaluating pair programming with respect tosystem complexity and programmer expertise. IEEE Transactions onSoftware Engineering 33 (2), 65–86.
Beck, K., 2000. Extreme Programming Explained: Embrace Change.Addison-Wesley.
Birk, A. et al., 2003. Product line engineering, the state of the practice.Software, IEEE 20 (6), 52–60.
Boehm, B., 1988. A spiral model of software development and enhance-ment. IEEE Computer 21 (5), 61–72.
Boehm, B., 2002. Get ready for agile methods, with care. IEEE Computer35 (1), 64–69.
854 G.K. Hanssen, T.E. Fægri / The Journal of Systems and Software 81 (2008) 843–854
Boehm, B., Turner, R., 2004. Balancing Agility and Discipline – A Guidefor the Perplexed. Addison-Wesley, p. 266.
Bosch, J., 2000. Design and Use of Software Architectures – Adopting andEvolving a Product-Line Approach. Addison-Wesley.
Bosch, J., 2006. The challenges of broadening the scope of softwareproduct families. Communications of the ACM 49 (12), 41–44.
Carbon, R. et al., 2006. Integrating product line engineering and agilemethods: flexible design up-front vs. incremental design. In: Workshopon Agile Product Line Engineering.
Clements, P.C., Northrop, L., 2002. Software Product Lines: Practices andPatterns. Addison-Wesley.
Cockburn, A., 2002. Agile Software Development. In: Cockburn, A. (Ed.),The Agile Software Development Series. Addison-Wesley.
Deming, W.E., 2000. Out of the Crisis. The MIT Press.Dyba, T., 2000. Improvisation in small software organizations. IEEE
Software 17 (5), 82–87.Erdogmus, H., Morisio, M., 2005. On the effectiveness of the test-first
approach to programming. IEEE Transactions on Software Engineer-ing (31), 3.
Erickson, J., Lyytinen, K., Siau, K., 2005. Agile modeling, agile softwaredevelopment, and extreme programming: the state of research. Journalof Database Management 16 (4), 88.
Fægri, T.E., Hanssen, G.K., 2007. Collaboration and process fragility inevolutionarily product development. IEEE Software 24 (3).
George, B., Williams, L., 2004. A structured experiment of test-drivendevelopment. Information and Software Technology 46 (5 SPEC ISS),337–342.
Gilb, T., 2005. Competitive engineering: A handbook for systemsengineering, requirements engineering, and software engineering usingP language. Elsevier Butterworth-Heinemann, p. 480.
Hanssen, G.K., Fægri, T.E., 2006. Agile customer engagement: alongitudinal qualitative case study. In: International Symposiumon Empirical Software Engineering (ISESE). Rio de Janeiro,Brazil.
Helferich, A., Schmid, K., Herzwurm, G., 2006. Product management forsoftware product lines: an unsolved problem? Communications of theACM 49 (12), 66–67.
Imai, M., 1997. Gemba Kaizen: A Commonsense Low-Cost Approach toManagement. McGraw-Hill.
Johansen, T., 2005. Using evolutionary project management (Evo) tocreate faster, more userfriendly and more productive software.Experience report from FIRM AS. In: International Conference onProduct Focused Software Process Improvement. Springer Verlag,Oulu, Finland.
Kakola, T., Duenas, J.C., 2006. Software Product Lines: Research Issuesin Engineering and Management. Springer Verlag.
Keil, M., Carmel, E., 1995. Customer–developer links in softwaredevelopment. Communications of the ACM 38 (5), 33–44.
Knauber, P. et al., 2000. Applying product line concepts in small andmedium-sized companies. Software, IEEE 17 (5), 88–95.
Linden, F.V.D., 2002. Software product families in Europe: The ESAPSand CAFE projects. In: IEEE Software, pp. 41–49.
Lorge Parnas, D., Clements, P.C., 1986. A rational design process: howand why to fake it. IEEE Transactions on Software Engineering 12 (2),251–257.
Mathiassen, L., Ories-Heje, J., Ngwenyama, O., 2001. Improving softwareorganizations: From principles to practice. In: The Agile SoftwareDevelopment Series. Addison-Wesley Professional, p. 368.
Muller, M., Hagner, O., 2002. Experiment about test-first programming.Software, IEE Proceedings 149 (5), 131–136.
Schwaber, K., Beedle, M., 2001. Agile Software Development with Scrum.Prentice Hall.
Seaman, C.B., 1999. Qualitative methods in empirical studies in softwareengineering. IEEE Transactions on Software Engineering 25 (4), 557–572.
Sugumaran, V., Park, S., Kang, K.C., 2006. Software product lineengineering. Communications of the ACM 49 (12), 28–32.
Tian, K., Cooper, K., 2006. Agile and software product line methods: arethey so different? In: 1st International Workshop on Agile ProductLine Engineering (APLE’06). IEEE Computer Society: Baltimore,Maryland, USA.
Yin, R., Campbell, D.T., 2002. Case Study Research. Sage PublicationsInc.
Geir Kjetil Hanssen works as a researcher on software process improve-ment and methodologies at SINTEF ICT, Norway’s largest independentresearch institution. He has a M.Sc. degree in informatics from the Uni-versity of Trondheim and 3 years of industrial experience. Currently healso holds a position as Ph.D. student at the Norwegian University ofScience and Technology.
Tor Erlend Fægri works as a researcher on software process improvementand software architecture at SINTEF ICT, Norway’s largest independentresearch institution. He has a M.Sc. degree in computing science from theUniversity of Glasgow and 6 years of industrial experience. Currently healso holds a position as Ph.D. student at the Norwegian University ofScience and Technology.
Paper 4 Hanssen, G. K., Yamashita, A. F., Conradi, R. and Moonen, L. (2010). Software Entropy in Agile Product Evolution. In proceedings of 43d Hawaiian International Conference on System Sciences (HICSS'10), Hawaii, USA, 4-7 January. 1-10.
Software entropy in agile product evolution
Geir Kjetil Hanssen NTNU IDI / SINTEF ICT
Aiko Fallas Yamashita Simula Research Laboratory
Reidar Conradi
NTNU IDI [email protected]
Leon Moonen Simula Research Laboratory
Abstract
As agile software development principles and methods are being adopted by large software product organizations it is important to understand the role of software entropy. That is, how the maintainability of a system may degrade over time due to continuous change. This may on one side affect the ability to act agile in planning and development. On the other side, an agile process may affect growth of entropy. We report from a case study of a successful software product line organization that has adopted the agile development method Evo, showing how agility and entropy are negatively related. We conclude this study by suggesting a two-step approach to manage entropy while maintaining process agility. First, the system needs to be restructured to establish a level of manageable entropy, and then, that the agile process must be complemented with continuous semi-automated quality monitoring and refactoring support.
1. Introduction Agile software development methods have over the
past decade become a preferred approach to many project organizations and are now also finding their way into the more complex arena of packaged software product development and evolution [1]. This comes along with several challenges as the complexity of the development context increases dramatically as compared to the development of a single stand-alone solution for a single customer �– which in many respects was the target of agile methods initially. One of the challenges when adopting agile software development practices to large-scale product development is the problem of software entropy, which refers to the gradual decrease in maintainability of a system as it evolves over time. With the core principles of agile software development in mind we suspect that
development process agility and system entropy may negatively and naturally affect each other. That is, the rapid, incremental and iterative approach of agile development methods may actually aggravate entropy, and the other way around; that an increasing system entropy may hamper the ability to act agile.
To investigate this potential relationship and improvement actions we have done a case study of a successful software product organization that have developed their product line for over a period of 14 years and are now established in the top segment of their domain, still with a strong need and urge for further development and improvement. Five years ago the R&D department experienced severe problems related to a heavily plan-based development process and adopted the agile method Evo [2] which over the past years have proved to fit well into a hectic software product development scheme with constant releases of new versions into a competitive market [3]. However, for each release of their product, the internal structure of the product have grown more and more complex and are now a major concern as it dramatically threatens their ability to be agile in terms of reduced analyzability, modifiability and testability of the system as well as problems in separating areas of concern for the development teams. In sum this leads to a reduction in productivity and product quality.
This motivates our research questions: 1) How may system entropy and agile processes
mutually negatively affect each other? 2) Can code smell analysis and refactoring be a
viable solution? We have collected data from three sources. (1) We
have done an extensive interview with members of a dedicated product architecture team �– a group of four expert developers serving the development teams and being responsible of improving the architecture of the product line as well as optimizing the development infrastructure. (2) We have held a workshop with an external consultant analyzing the system using a
1
Proceedings of the 43rd Hawaii International Conference on System Sciences - 2010
978-0-7695-3869-3/10 $26.00 © 2010 IEEEAuthorized licensed use limited to: Norges Teknisk-Naturvitenskapelige Universitet. Downloaded on April 20,2010 at 10:06:24 UTC from IEEE Xplore. Restrictions apply.
comprehensive tool called NDepend . (3) We have interviewed members from one of the development teams in the R&D department. The interview data are analyzed according to the principles of constant comparison [4] revealing a set of problems related to the state of entropy of the system as well as the agile process as it is implemented in this case.
Based on the findings from this qualitative study and on an overview of relevant literature on code smell analysis and refactoring decision-making we discuss how these problems potentially can be resolved in a two-step process. First, the state of entropy must be reduced to benefit from the agile process. Then, having established a level of manageable entropy, proper actions (changes in process) must be taken to avoid entropy from growing again.
This paper extends an initial short-paper reporting from the study of CSoft [5]. In the remainder of the paper we first describe our research approach in chapter two. Chapter three gives an overview of relevant literature covering software entropy, code-smell analysis and refactoring decision-making. Chapter four present our findings from the case study, which is then being discussed in chapter five. Finally, we answer the research questions and give some concluding remarks in chapter six.
2. Study context and methodology
2.1. Case study context and background CSoft (anonymized name) is a medium-sized
Norwegian software company that develops, maintains and markets a product line having the same name. They serve the high-end segment of their market and have a wide international customer base. Despite having considerable challenges CSoft is now one of the market leaders. The company was established in 1996 and has grown steadily since, currently employing about 260 people, including 60+ developers. They have several development locations across Europe and Eurasia and there has been a gradual shift from building custom-made applications to a software product line. CSoft can be seen as a highly modular product line that allows many configurations and ways to use it. It contains five main modules (with numerous sub-modules). The use of these modules varies per customer and per case. Some central modules are used in any configuration, while the use of others depends on the usage situation. CSoft comes with a set of predefined configurations for the most common usage scenarios, but there is also built-in support for detailed customization to support more variants. From the start of the company, fourteen years ago, the development process matured from a more or less ad-hoc type of
process (creative chaos) to a well-defined waterfall-inspired process (plan-based and non-iterative).
About five years ago the development process had become too slow and inefficient. Out of necessity CSoft changed to a radically different process: Evo [6]. This change was guided by Tom Gilb, who originally defined the process [2]. Evo is an agile method comparable to the better-known Scrum-method [7], although the terminology differs. At CSoft, work is done in two-week iterations (equivalent to sprints in Scrum), working software is deployed on test servers by the end of every iteration and invited customers evaluate the latest results and give corrective feedback to the development teams [3, 8]. Although similar to other agile methods, the perhaps most distinct characteristic of Evo is the strong focus on product qualities and the definition and use of metrics for expressing evaluating quality level goals. As adopted at CSoft, Evo conforms to the four basic values in the Agile Manifesto (see www.agilemanifesto.org): interaction is highly valued, there is a strong emphasis on delivering working software after every iteration, invited lead users participate in development, which is open to changes in requirements and design.
2.2. Study method We have collected empirical data in three ways:
First, the company had a workshop with Patrick Smacchia, an external consultant who analyzed the CSoft source code using his own commercial tool called NDepend 1 which was used to do a live analysis of the code �– numerous metrics were generated on the fly, presented graphically and the code was browsed alongside a discussion in the workshop group. Based on this �“live�” analysis of the system the participants (system architects, developers and researchers) developed through discussions a common understanding of the state of entropy of the system. A brief summary of this discussion is given in [9]. Secondly, we conducted an in-depth interview with two members from the four-person product architecture team. This group has two main responsibilities: (a) to ensure and improve the architecture of the system, that is, to make it easy and safe to add and improve features and to ease deployment of the product, and (b) to improve the system�’s development infrastructure (testing framework, code management, automated builds etc.). The interview lasted for 3,5 hours and was recorded and transcribed. This transcription (30 pages of text) was analyzed using NVivo , a tool for tagging fragments of text with information about context, meaning etc. Finally, we also did interviews with the
1 See http://www.ndepend.com/
2
Proceedings of the 43rd Hawaii International Conference on System Sciences - 2010
Authorized licensed use limited to: Norges Teknisk-Naturvitenskapelige Universitet. Downloaded on April 20,2010 at 10:06:24 UTC from IEEE Xplore. Restrictions apply.
leader and one developer from one of the module teams.
2.3. Threats to validity Our case study covers only a single software
development organization and the interviews were done with a few representatives from the central architecture team of four and just a few of the members of R&D. Also, the situation described and discussed here only reflects the present state and not details on how the situation have developed over time. However, findings from interviews are supported by data obtained in the workshop with the external consultant and the dynamic analysis of the code, together this gives us valuable insight into the complex problem of entropy and its reciprocal influence with the agile development process. We believe that the results and the discussions in this paper may be of practical value as well as a pointer to further research.
3. Background
3.1. Agile development and code entropy As early as 1976 Belady and Lehman [10] defined
what they call a set of laws of program evolution dynamics. Their second law �– �‘The law of increasing entropy�’ say: The entropy of a system (its unstructuredness) increases with time, unless specific work is executed to maintain or reduce it. This notion of the entropy problem has been followed up by later research on software evolution and maintenance. One notable example is a much-sited study by Eick et al., which investigated a long backlog of change history of a very large software system for telephony. On one hand, it demonstrated decay/entropy as a natural process. On the other hand it identified a set of useful symptoms or predictors of decay [11]. The entropy-problem have recently also been related to agile software development �– one of the most notable ideas to software engineering over the past decade [12]. For example, Martin Fowler discusses evolutionary design as a possible cause for software entropy �– if not managed [13]. As a countermeasure he promotes simple design and refactoring. As we will show by our case study �– this is truly necessary but not easy in complex situations such as the constant evolution of a software product line. This view is supported by other studies, for example by Oizka that summarizes that �“..refactoring proved to be much more difficult and time-consuming than expected.�” [14]. Rajlich defines the industry�’s interest in, and shift towards, agile methods, as a paradigm shift in software engineering and that attention to actively managing software entropy is a vital success factor [1]. Missing this focus
may actually shorten the lifetime of a software product as a decaying system is eventually impossible to evolve, consequently moving into a phase-out stage. Neill and Laplante develop the refactoring concept further and define what they call strategic refactoring [15]. This is an approach, starting out with looking for code smells, where also macro- and domain-level architecture is being evaluated (and refactored).
3.2. Code smell analysis and refactoring In a thorough search for relevant and rigorous
empirical research we have identified 11 papers addressing code smell analysis and refactoring covering, broadly, four sub categories:
(a) Subjective evaluation of code smells. In [16] Mäntylä et al. report from an empirical study of subjective evaluation and detection of code smells and compare it with automated metrics-based detection. The study was done in an industrial setting and showed that subjective evaluations by developers were not uniform. However, in cases with a low level of problems, the conformance was higher than cases with a high level of problems. When investigating the demographics of the evaluators they saw that experienced developers were better at spotting structural problems in the code than regular developers who could spot problems mainly at the code level. Also, developers that had worked with the code for a long period of time tended to see fewer smells than developers with shorter experience. Finally, when comparing subjective evaluation of code with automated metric-based detection of code smells, they discovered that developers�’ evaluations of complex code smells did not match the results of the metrics based detection. Based on these findings they conclude that subjective evaluations and metrics based detection should be used in combination.
Mäntylä et al. also reports on a student experiment for evaluating subjective evaluation for code smells detection and refactoring decisions [17]. He observed the highest interrater agreements between evaluators for simple code smells. When the subjects were asked to make refactoring decisions he observed low agreement, thus questioning the reliability of such.
(b) Refactoring and refactoring decisions. Counsell et al. investigated refactorings done in seven open-source Java systems to see which types of refactorings were most common and which effects they had in solving code-smells [18]. The study used fifteen refactorings from the classification by Martin Fowler and Kent Beck [19]. The analysis showed that a group of six refactorings were more commonly used: Pull Up Method, Move Method, Add Parameter, Move Field, Rename Method and Rename Field. Surprisingly, these
3
Proceedings of the 43rd Hawaii International Conference on System Sciences - 2010
Authorized licensed use limited to: Norges Teknisk-Naturvitenskapelige Universitet. Downloaded on April 20,2010 at 10:06:24 UTC from IEEE Xplore. Restrictions apply.
most common refactorings were not addressing inheritance or encapsulation. Two of the refactorings, Move Method and Move Field, seemed to solve several code smells. In another study by Counsell et al. [20] the same code smells were used on the same data-set to investigate indirect and composite refactorings. Three refactorings were found to have large chains of following refactorings (Encapsulate Downcast, Extract Subclass and Extract Superclass). Refactorings inducing long chains tended to be used relatively infrequently by developers as opposed to refactorings inducing short chains. In a third study, Counsell at al. [21] used the same refactorings and the same empirical data set to investigate how refactorings affected the testability of a system. That is, to what extent the fifteen refactorings affect the re-usability of a test suite, i.e. having to update the tests is a spin-off cost of refactoring. The main conclusion from this study is that while semantically preserving refactorings may be ideal for preserving tests sets, they are not necessarily always the right refactorings to choose. (c) General applicability of code metrics. We identified one study investigating the applicability of code metrics across different software systems. Bakota et al. [22] collected code metrics for four software systems: an OSS system vs. a closed source system and an office application vs. a telecommunication system. They found that the systems could be differentiated from each other pretty well based on the metric values, but remark that two metrics �“Response For A Class�” and �“Weighted Methods Per Class�” behaved very different on the systems analyzed. Somewhat related, Li and Shatnawi investigated how well code smells can predict post-release class defects [23], results showing that the Shotgun Surgery, God Class and God Methods bad smells were positively associated with the class error probability. (d) Code cloning. Two studies address problems related to code cloning. Aversano et al. [24] studied code clone evolution by combining clone detection and co-change analysis, concluding that either for bug fixing or for evolution, most of the cloned code is consistently maintained during the same co-change or during temporally close co-changes. This finding seems to somewhat demystify the image of code clones being bad design. In the same line of research, Lozano and Wermelinger [25] report the results from an experiment to investigate the effects of code clones on maintenance. Their analysis suggests that existence of code clones does increase maintenance efforts, at times significantly, depending on code characteristics. However, they were unable to identify characteristics that systematically revealed a significant relation between cloning and maintenance effort increase.
4. Findings from the Case Study This section presents an overview of data collected
during the case study. It describes (i) the structure and the complexity of the product being developed by CSoft (ii) the problems that this structure imposes and (iii) ideas that the architecture team themselves has for improving the situation.
4.1. Complexity of the system The system has been under constant development
for the last fourteen years and is based on several technologies that have emerged over those years. Aging solutions from years ago are still part of the system, such as older ASP solutions, COM+ components, VB6 code and other legacy technologies. Today, most new code is developed in C#, and is spread over approximately 160 .Net assemblies. The complete product is best described as a traditional three-tier system with an MS SQL Server driving the data layer, a business layer and a presentation layer based on a dozen ASP.Net applications. There is a clean separation between the presentation- and the business layer. However the most obvious problem in the software is what the architects refer to as �“the Blob�”: a very large assembly (aptly named Core) consisting of approximately 150K lines of code in 144 namespaces. The NDepend tool was used to visualize and generate code metrics for the internals of this assembly. One of the results was a so-called dependency structure matrix, which showed an extremely entangled structure, where most namespaces refers to most namespaces, thus creating a lot of cyclic dependencies. NDepend also provides means for additional analysis through CQL (Code Query Language) [26] which in name and function is inspired by SQL. The code is the database and structured queries can be defined to investigate the code and its internal relationships. CQL retrieves instances of classes, which display certain characteristics from a code metrics perspective.
Figure 1. Overall CSoft architecture
4
Proceedings of the 43rd Hawaii International Conference on System Sciences - 2010
Authorized licensed use limited to: Norges Teknisk-Naturvitenskapelige Universitet. Downloaded on April 20,2010 at 10:06:24 UTC from IEEE Xplore. Restrictions apply.
The query language can be used for detecting code smells, and we will give a small example with the God class [19] code smell. Table 1 explains the abbreviations being used. Table 1. Metric abbreviations AOFD Access Of Foreign Data WMPC1 Weighted Methods Per Class 1 Cyclomatic Complexity
Class-level cyclomatic complexity
LCOM Lack of Cohesion Of Methods TCC Tight Class Cohesion TypeCe Efferent coupling
A God class is a class with too many
responsibilities, delegating only minor details to a set of trivial classes and using data from other classes. To detect God classes in NDepend, we adopt the detection strategy proposed by Marinescu [27] to the set of available metrics in the NDepend CQL. The original query of Marinescu is the following:
AOFD top 20% and AOFD higher 4 and WMPC1 higher 20 and TCC lower 33
We use the following query for detecting instances of god class with the NDepend CQL:
SELECT TOP 20 TYPES WHERE TypeCe > 25 AND CyclomaticComplexity > 20 AND LCOM > 0.77
Where TypeCe was used instead of AOFD, CyclomaticComplexity was used instead of WMPC1 and LCOM was used as the counterpart of TCC. As a result from this query, NDepend returned a set of business management classes, all from a module called Reporting, which is also the section of CSoft�’s Core with the highest defect rate and performance problems.
4.2. Development problems We can summarize the acknowledged problems
during development by distinguishing four aspects: (a) Analyzability and comprehensibility. Due to the
high complexity of the system, it is very hard for developers to get an overview of the code and its structure. Especially the central component has grown extremely large and has many internal references (each namespace depends directly or indirectly on another namespace), making it difficult to understand how it really works. This was clearly not by design, but the result of years of intense development. The system is intended to be structured as vertical modules, but as it is now there are too many relationships between the verticals �– changing one will inevitable affect many others. New developers joining R&D have a steep learning curve and require close follow-up over a long period of time by more experienced developers. There exists no documentation or models that explain the
structure of the system, even though this clearly would be highly useful both to existing and new developers. Even worse, having problems understanding how the code is structured leads to a fear of changing the code, both for adding new features and for improving existing code. The unclear internal structure creates a cognitive overload and a common (unfortunate) way to deal with this is code duplication: instead of modifying existing code, developers create their own copy over which they have full control. This leads to a larger cognitive overload for other developers, only making the problem worse �– a self-reinforcing effect.
(b) Modifiability and deployability. As a result of the duplication and entanglement of code, developers frequently need to perform so-called shotgun surgery, meaning that even the modification of a small detail forces them to identify and change code in many places. These problems slow down the development process and the potential for errors increases due to the high chance of overlooking one or more locations. Having to deal with bad code is frustrating to the developers as they in some ways in practical terms are enforced to build bad code on bad code as there is no room to actually resolve the problem. Besides development and maintenance, also deployment of the product suffers from its structure: The current core component aggregates features and functionality for every possible configuration of the product and it has to be released as a whole, even though only a fraction of the functionality may actually be needed for a particular configuration.
(c) Testability and stability. Due to the size of the code and the many cross-references, there are too many paths through the code to test them all systematically. The test coverage is not high enough and existing tests have shown to be unstable and inconsistent. For example, the same tests run on similar systems may produce different outcomes that are hard to explain. Also, a lot of the existing tests are extremely large, meaning that they too are hard to maintain and use. When a test fails, it often takes a lot of time to locate and fix the actual problem that triggered the failure. Although such tests are supposed to act as a safety net and give developers the courage to make changes they are not trusted. This increases the fear or at least reluctance to change existing code �– since the effects of a change are hard to foresee and errors can have considerable negative effects. Nevertheless, regression testing is done, albeit with a lower than desired quality.
(d) Organization and process. As both the business domain and the system are highly complex, each of the development teams (4-6 developers in each) has an expert (the so-called guru). This guru has high technical skills and extensive experience with the code, which is vital for the team to solve its tasks.
5
Proceedings of the 43rd Hawaii International Conference on System Sciences - 2010
Authorized licensed use limited to: Norges Teknisk-Naturvitenskapelige Universitet. Downloaded on April 20,2010 at 10:06:24 UTC from IEEE Xplore. Restrictions apply.
Consequently, this organization represents a considerable vulnerability; losing just a few of these gurus would have devastating effects on the development. The development process is based on two-week iterations and it is a strong focus on delivering working software by the end of each iteration. A negative effect of this focus is that delivering quality software is at times traded in for creating a working version. Each iteration ends with a review, but the high velocity typically does not give enough time to catch all issues. This causes extra work close to a release when the system is thoroughly tested as a whole, yet entropy is allowed to grow from release to release. The development teams are set up to have separate areas of concern, each team being responsible for a part of the total product, e.g., the reporting solution or the data storage. The idea is to build competence around a well-defined part. Unfortunately, the structure of the system does not reflect this organization in practice, because functionality is spread throughout the code. This forces the teams to operate outside their area of concern, which has shown to negatively affect their ability to produce enough new and improved features of the product in their releases. The total request for improvements from the market is constantly higher than what actually is delivered, thus indicating a need to improve development efficiency.
4.3. Ideas for improving As part of the discussions with the architects, we
also collected several of their high-level ideas to further improve the product and development process:
(a) Process automation. Currently too much testing is done manually and more automation is desired. In addition, to establish an efficient and trustworthy safety net for the developers, tests need to become more stable and trustworthy. With this in place, the architects can introduce what they call �“pain-driven development�”. That is, when a developer introduces or changes code that breaks the tests, he or she will get notified immediately to correct it.
(b) Restructuring and refactoring goals. The architects feel that components of the software need to be de-coupled from the core and the overlapping and duplicated code has to be removed. They also agreed that the system should have a clearer separation of concerns were vertical modularization should reflect business segments and horizontally, the system should better separate business and platform related code.
(c) Continuous monitoring of quality. The architects proposed a principle that they refer to as �“quality-from-now�”, meaning that any change to the code should be analyzed at development time, to check that it does not conflict with defined rules of good
design. This can, for example, be achieved using a tool like NDepend, by defining CQL rules to detect code smells and monitor potential problems nearly constantly during development. The architects believe that this approach would considerably reduce the fear of changing the code.
5. Discussion In this section, we discuss each of the problem
areas identified in the case study. We analyze their implications in the agile process and propose potential solutions. We conclude with some avenues for future research that follow from our literature review.
5.1. Analyzability and comprehensibility In [28], van Deursen analyses the effects of various
agile practices on program comprehension, concluding that pair programming, unit testing and refactoring are the practices that support comprehension. We observe that most agile methods assume that development starts from scratch and ends with a release �– post-release maintenance is not covered. In our case, the system was already very complex when Evo was adopted, and although agile methods promote communication over documentation, the lack of adequate documentation holds back the comprehension of such a system. Although XP states that �“the code is the documentation�”, there is no guarantee that the code can serve this purpose if the system was originally developed using different methods, in this case, a changing mix of approaches from ad-hoc via waterfall to agile. The limited number of �‘experts�’ of the system, the high number of new coming developers, and urgent demands on new functionality, makes pair programming a not very practical solution for spreading knowledge. Visualization tools could help new developers to understand the code while refactoring, and additionally generate adequate models and documentation for the system, but the challenge here is to understand which visualizations are better for which purposes.
In addition to visualisation tools, the application of refactorings to untangle crosscutting concerns will improve the comprehensibility [29]. Such migrations will better distinguish the code for various business segments and separate business and platform related code, allowing newcomers to explore the code more intuitively Semi-automated tools for this kind of refactoring has recently become available [30, 31].
Finally, to overcome �“the fear of change�” and cope with the time pressure, we suggest semi-automatic code inspections (cf. [32]), potentially extended with advanced visualization and analysis tools such as [33-35]. Tools will help developers to get a better
6
Proceedings of the 43rd Hawaii International Conference on System Sciences - 2010
Authorized licensed use limited to: Norges Teknisk-Naturvitenskapelige Universitet. Downloaded on April 20,2010 at 10:06:24 UTC from IEEE Xplore. Restrictions apply.
understanding of non-trivial refactorings, and can even automate the more trivial ones. Dedicated tools have been developed to eliminate code clones [36], although the negative effects of code clones are still being investigated [25].
5.2. Modifiability and deployability According to Martin [37], dependency problems,
like the ones observed in our case study, largely relate to two design smells: rigidity and immobility. Rigidity means that a change in the system implies a cascade of changes in other modules. Immobility refers to the inability of the system to encapsulate components that can be reused, because it implies too much effort or risk. If the smells are all over the system, high-level restructuring is needed to get rid of unwanted dependencies. One immediate consequence of these dependency issues is the violation of the Interface Segregation Principle [37], explaining most of the difficulties in the deployment stage. The analysis of module dependency [38, 39] could represent a feasible strategy for �“leveling the code�”. In [40], Bourqun and Keller proposed the analysis of code smells alongside with architectural violations for achieving high-impact refactorings, and presented a comprehensive case study where they describe how they combined several tools and techniques, the resulting architecture and the refactoring process. Our findings lead us to believe that an approach along the lines of this work will be very beneficial to improve modifiability and deployability in the context of our case study.
5.3. Testability and stability Unit testing is one of the important components of
agile methods. In the context of our case, the considerable code size combined with a large amount of dependencies in the code makes it hard to define unit tests and achieve high levels of coverage. Due to the high pace of development, there is little room for regression, integration and system testing during the iterations and CSoft relies on the feedback from external stakeholders as quality checks. Recent work has focused on methods and techniques for improving unit test suits [41-43], alongside with empirical studies on defects prediction [23] that aid planning. However, there are still various challenges to agile testing that go beyond unit testing that are not completely understood [44, 45]. Although we consider visualization and analysis tools to be useful, we know that non-trivial refactorings are risky and time consuming due to the unstable characteristic of the system. The current lack of understanding of the effects of given code smells and refactorings makes this task very challenging [21]. The usage of multiple criteria and goal-centered
indicators could be a feasible solution for focusing on the relevant aspects within a project (see [46, 47]).
5.4. Organization and process The strong focus on rapid and continuous delivery
of features at CSoft has lead to the construction of teams with defined areas of concern. In the same spirit of �“inspect-and-adapt�” from Scrum, CSoft has deviated slightly from certain agile practices in order to adapt agile practices to their context. As mentioned before in the Analyzability and comprehensibility section, when the system and organization become too complex, the use of practices such as pair programming and team rotation seems not to provide the same advantages as in small teams. We also conjecture that an important reason for delays on the incorporation of new features is due to the system not reflecting the same separation of concerns as the development tasks. This entanglement of crosscutting concerns is a common problem with software maintenance. Refactoring towards an aspect-oriented version could help to restructure the existing code according to the areas of concerns [29], but this area is relatively new and tools have only recently been presented [30, 31]. The lack of adequate information to perform the planning could be another reason for delays. Planning of iterations could be enhanced by considering additional information, such as complexity analysis of the tasks to improve on estimates obtained from planning poker. However, such complexity analysis may still have limited effect in practice, as there is not enough empirical evidence on the impact of different refactorings [21]. These uncertainties could be compensated by continuous quality monitoring, for example by combining evolution monitoring [48-51] and semi-automatic code inspections [32] to analyze metric-based characterizations and code smells of the system, e.g. using a tool like NDepend. Such a combination could be incorporated to the development flow to detect problematic areas and decide upon refactoring strategies. There still is the challenge on deciding on the prioritization of refactorings. One prioritization approach could be to use detectors of defect or performance issues in the system [52, 53].
5.5. Initial improvement actions In the interview, discussing improvement actions
with the system architects, two types are relevant. First, on a short term basis the situation must be improved to actually release the potential in the agile process. This includes both a restructuring of the system by breaking it up, removing dependencies, to make areas of concerns possible as well as automation of costly manual activities, typically testing. This requires a
7
Proceedings of the 43rd Hawaii International Conference on System Sciences - 2010
Authorized licensed use limited to: Norges Teknisk-Naturvitenskapelige Universitet. Downloaded on April 20,2010 at 10:06:24 UTC from IEEE Xplore. Restrictions apply.
major effort by the whole development organization and in the case studied here it has been decided that approximately 80% of all effort in a whole release period (approximately one calendar year) will be dedicated to plan and execute a massive restructuring. Tool support, like the use of NDepend in this case, will anyhow be invaluable but it is clear that this will require a massive amount of manual work. However, this is a one-time effort and investment for future development �– an expensive yet inevitable treatment. Secondly, and on a long-term regular basis the development process and the evolving software system must be continuously monitored and controlled to prevent entropy. Here a tool for evaluating continuous system change according to a set of rules or guidelines will be invaluable. However, there is still a challenge in defining these rules and defining proper responses to violations.
5.6. Avenues for future research From the identified literature, we see that most of
the work reported on methodological aspects and tools are on the development stage. More relevant case studies and better evaluations of the available tools are needed, especially studies following development over time, evaluating actions taken and their potential effects. This could permit practitioners to evaluate the different solutions and adopt the most appropriate ones to their context. In that sense, the use of evaluation frameworks like the one suggested by Maletic et al. [54] could be useful. Mealy et al. [55] have also suggested a set of usability requirements for refactoring tools. We have seen many examples of tools supporting Java, but we have scarcely seen tools supporting other widely used platforms or languages such as C#, C++ and others. Consequently, more research on integrated, cross-platform or cross-compiler, frameworks is of interest. One of our major findings is that there is relatively little empirical evidence and methods available that supports refactoring decision making. Code smells themselves are suggestions for refactoring, but when we analyze code smells, we also need additional information to drive refactoring in a cost-effective way. Detection focuses on answering: �“where are the code smells?�” and analysis should focus on answering �“which code smells should we refactor?�” or �“which refactorings should we apply for this code smell?�” Moreover, knowledge and methods for assessing the cost-benefits of different refactoring are still largely an open area for research [21].
6. Concluding remarks In this paper, we have presented some of the
problems agile practitioners face when dealing with software entropy in the long-term evolution of a software product line in general. We have also emphasized how the agile development process affects, and are being affect by, the system entropy. We have consulted relevant literature addressing the problem of system entropy and code smell analysis and refactoring as a viable solution.
Through our case study we found that, to keep agile responsiveness in the presence of entropy, the agile workflow needs better support for understanding, planning and evaluating the impact of changes. We have proposed a combination of two strategies to address this issue: (1) Short term: progressive high-level restructuring by untangling crosscutting concerns, which will help to improve comprehensibility, modifiability, testability and deployability of the system �– all important enablers for efficient agile development. And (2) Long term: semi-automatic quality monitoring and improvement during development, which will ensure that the above qualities are kept and make the development process more predictable and thereby easier to plan. Based on our findings and discussion we revisit our two research questions: 1) How may system entropy and agile processes
mutually negatively affect each other? We have exemplified and discussed several cases of system entropy negatively affecting the ability to act agile in the development process, e.g. how complexity hampers productivity and quality. Also, we have seen cases where the velocity of the agile process with short iterations does not give time to resolve problems. One possible solution would be to add time to solve issues; however stretching iteration length is not desirable. 2) Can code smell analysis and refactoring be a
viable solution? Based on our study of relevant literature and previous studies as well as the ideas coming from the case company itself along with the discussions of our findings it seems that code smell analysis and refactoring may help to resolve the problem of entropy on a short term and also to establish a manageable structure suitable for the speed and flexibility of the agile process.
Finally, based on identified literature, we pointed out two promising research directions were solutions are currently lacking, and where practitioners need advice: (1) refactoring decision support, and (2) task complexity analysis.
8
Proceedings of the 43rd Hawaii International Conference on System Sciences - 2010
Authorized licensed use limited to: Norges Teknisk-Naturvitenskapelige Universitet. Downloaded on April 20,2010 at 10:06:24 UTC from IEEE Xplore. Restrictions apply.
As part of our own future work, we will continue the study at CSoft by evaluating the effects of the extensive ongoing refactoring project.
Acknowledgments Our thanks to the architecture team at CSoft and to
the other staff members that have been involved in this study. We would also like to thank Patrick Smacchia for his valuable insight and ideas. This work was partly funded by the Research Council of Norway under grant # 179851/I40.
7. References [1] Rajlich, V., Changing the paradigm of software
engineering, in Communications of the ACM. 2006. p. 67-70.
[2] Fægri, T.E. and G.K. Hanssen, Collaboration and process fragility in evolutionarily product development. IEEE Software, 2007. 24(3): p. 96-104.
[3] Hanssen, G.K. and T.E. Fægri, Process Fusion - Agile Product Line Engineering: an Industrial Case Study. Journal of Systems and Software, 2008. 81: p. 843-854.
[4] Seaman, C.B., Qualitative methods in empirical studies in software engineering. IEEE Transactions on Software Engineering, 1999. 25(4): p. 557-572.
[5] Hanssen, G.K., et al., Maintenance and agile development: challenges, opportunities and future directions, in proceedings of International Conference on Software Maintenance. 2009, IEEE Press.: Edmonton, Canada. p. 487-490.
[6] Gilb, T., Competitive Engineering: A handbook for systems engineering, requirements engineering, and software engineering using Planguage. 2005: Elsevier.
[7] Schwaber, K., Beedle, M., Agile Software Development with Scrum. 2001: Prentice Hall.
[8] Hanssen, G.K. and T.E. Fægri. Agile Customer Engagement: a Longitudinal Qualitative Case Study. in International Symposium on Empirical Software Engineering (ISESE). 2006. Rio de Janeiro, Brazil.
[9] Smaccia, P. Getting rid of spaghetti code in the real-world: a Case Study. 2008 [cited; Available from: http://codebetter.com/blogs/patricksmacchia/archive/2008/09/23/getting-rid-of-spaghetti-code-in-the-real-world.aspx.
[10]Belady, L.A. and M.M. Lehman, A model of large program development. IBM Systems Journal, 1976(3): p. 225-252.
[11]Eick, S.G., et al., Does Code Decay? Assessing the Evidence from Change Management Data. Transactions on Software Engineering, 2001. 27(1): p. 1-12.
[12]Dybå, T. and T. Dingsøyr, Empirical Studies of Agile Software Development: A Systematic Review. Information and Software Technology 2008. 50(9-10): p. 833-859.
[13]Fowler, M., Is Design Dead?, in Extreme Programming Explained, G. Succi and M. Marchesi, Editors. 2001, Addison Wesley Longman: Reading, Mass.
[14]Pizka, M., Straightening spaghetti-code with refactoring?, in Proceedings Of The International
Conference On Software Engineering Research And Practice. 2004. p. 846-852.
[15]Neill, C.J. and P.A. Laplante, Paying Down Design Debt with Strategic Refactoring, in IEEE Computer. 2006, IEEE Computer Society. p. 113-116.
[16]Mäntylä, M. and C. Lassenius, Subjective evaluation of software evolvability using code smells: An empirical study. Empirical Software Engineering, 2006. 11(3): p. 36.
[17]Mäntylä, M., An experiment on subjective evolvability evaluation of object-oriented software: explaining factors and interrater agreement, in Intl Symp. on Empirical Softw. Eng. (ISESE). 2005, IEEE.
[18]Counsell, S., et al., Common Refactorings, a Dependency Graph and some Code Smells: An Empirical Study of Java OSS, in proceedings of International Symposium on Empirical Software Engineering. 2006, ACM: Rio de Janeiro. p. 288-296.
[19]Fowler, M., et al., Refactoring: Improving the Design of Existing Code. 2000: Addison-Wesley.
[20]Counsell, S., Is the need to follow chains a possible deterrent to certain refactorings and an inducement to others?, in proceedings of Second International Conference on Research Challenges in Information Science. 2008. p. 111-122.
[21]Counsell, S., et al., The Effectiveness of Refactoring, Based on a Compatibility Testing Taxonomy and a Dependency Graph, in proceedings of Testing: Academic and Industrial Conference - Practice And Research Techniques. 2006. p. 181-192.
[22]Bakota, T., et al., Towards Portable Metrics-based Models for Software Maintenance Problems, in proceedings of IEEE International Conference on Software Maintenance. 2006. p. 483-486.
[23]Li, W. and R. Shatnawi, An empirical study of the bad smells and class error probability in the post-release object-oriented system evolution. JSS, 2006. 80(7): p. 1120-1128.
[24]Aversano, L., L. Cerulo, and M. Di Penta, How Clones are Maintained: An Empirical Study, in proceedings of European Conference on Software Maintenenace and Reengineering. 2007. p. 81-90.
[25]Lozano, A. and M. Wermelinger, Assessing the effect of clones of changebility, in proceedings of IEEE International Conference of Software Maintenence. 2008, IEEE. p. 227-236.
[26]Smaccia, P. Code Query Language. 2009 [cited; Available from: http://www.ndepend.com/Features.aspx#CQL.
[27]Marinescu, R., Measurement and quality in object-oriented design, in Intl Conf. on Softw. Maintenance (ICSM). 2005, IEEE. p. 701-704.
[28]van Deursen, A., Program comprehension risks and opportunities in extreme programming, in Working Conf. on Reverse Eng. (WCRE). 2001, IEEE. p. 176-185.
[29]Moonen, L., Dealing with Crosscutting Concerns in Existing Software, in Intl Conf. on Softw. Maintenance - Frontiers of Softw. Maintenance (ICSM/FoSM 2008). 2008, IEEE. p. 68-77.
[30]Binkley, D., et al., Tool-Supported Refactoring of Existing Object-Oriented Code into Aspects. IEEE
9
Proceedings of the 43rd Hawaii International Conference on System Sciences - 2010
Authorized licensed use limited to: Norges Teknisk-Naturvitenskapelige Universitet. Downloaded on April 20,2010 at 10:06:24 UTC from IEEE Xplore. Restrictions apply.
Transactions on Software Engineering, 2006. 32(9): p. 698-717.
[31]Marin, M., et al., An integrated crosscutting concern migration strategy and its semi-automated application to JHotDraw. Automated Software Engineering, 2009. 16(2): p. 323-356.
[32]van Emden, E. and L. Moonen, Java quality assurance by detecting code smells, in Working Conf. on Reverse Eng. (WCRE). 2002. p. 97-106.
[33]Parnin, C., C. Görg, and O. Nnadi, A catalogue of lightweight visualizations to support code smell inspection, in ACM Symposium on Software Visuallization. 2008: Ammersee, Germany. p. 77-86.
[34]Trifu, A. and U. Reupke, Towards Automated Restructuring of Object Oriented Systems, in European Conference on Software Maintenance and Reengineering. 2007. p. 39-48.
[35]Van den Brand, M.G., et al., Using The Meta-Environment for Maintenance and Renovation, in proceedings of European Conference on Software Maintenance and Reengineering. 2007. p. 331-332.
[36]Nasehi, S.M., G.R. Sotudeh, and M. Gomrokchi, Source code enhancement using reduction of duplicated code, in Conference on IASTED international Multi-Conference: Software Engineering 2007, ACTA Press. p. 192-197.
[37]Martin, R.C., Agile Software Development, Principles, Patterns and Practice. 2002: Prentice Hall.
[38]Arevalo, G., S. Ducasse, and O. Nierstrasz, Discovering Unanticipated Dependency Schemas in Class Hierarchies, in proceedings of Conference on Software Maintenance and Reengineering. 2005. p. 62-71.
[39]Leitch, R. and E. Stroulia, Assessing the Maintainability Benefits of Design Restructuring Using Dependency Analysis, in proceedings of International Symposium on Software Metrics. 2003. p. 309-322.
[40]Bourqun, F. and R.K. Keller, High-impact Refactoring Based on Architecture Violations, in Conf. on Softw. Maintenance and Reengineering (CSMR). 2007. p. 149-158.
[41]Guerra, E.M. and C.T. Fernandes, Refactoring Test Code Safely, in proceedings of International Conference on Software Engineering Advances. 2007. p. 44-50.
[42]van Deursen, A., et al., Refactoring test code, in eXtreme Programming Perspectives, M. Marchesi, et al., Editors. 2002, Addison-Wesley: Reading, Massachusetts.
[43]Van Rompaey, B., et al., On The Detection of Test Smells: A Metrics-Based Approach for General Fixture and Eager Test. IEEE Transactions on Software Engineering, 2007. 33(12): p. 800-817.
[44]Pettichord, B., Agile Testing Challenges, in proceedings of Pacific Northwest Software Quality Conference. 2004. p. 481-517.
[45]Talby, D., Agile Software Testing in a Large-Scale Project. IEEE Software, 2006. 23(4): p. 30-37.
[46]Mäntylä, M., Developing New Approaches for Software Design Quality Improvement Based on Subjective Evaluations, in proceedings of International Conference on Software Engineering. 2004. p. 48-50.
[47]Walter, B. and B. Pietrzak, Multi-criteria Detection of Bad Smells in Code with UTA Method, in proceedings of
extreme programming and agile processes in software engineering (XP). 2005. p. 154-161.
[48]D'Ambros, M., Supporting software evolution analysis with historical dependencies and defect information, in International Conference on Software Maintenance. 2008. p. 412-415.
[49]Jermakovics, A., M. Scotto, and G. Succi, Visual identification of software evolution patterns, in International Workshop on Principles of Software Evolution: in Conjunction with the 6th ESEC/FSE Joint Meeting. 2007, ACM: Dubrovnik, Croatia. p. 27-30.
[50]Kiefer, C., A. Bernstein, and J. Tappolet, Mining Software Repositories with iSPAROL and a Software Evolution Ontology, in proceedings of International Workshop on Mining Software Repositories. 2007. p. 10-18.
[51]Xing, Z., Analyzing the Evolutionary History of the Logical Design of Object-Oriented Software. IEEE Transactions on Software Engineering, 2005. 31(10): p. 850-868.
[52]Chaabane, R., Poor Performing Patterns of Code: Analysis and Detection, in proceedings of International Conference on Software Maintenance. 2007. p. 501-502.
[53]Wasylkowski, A., A. Zeller, and C. Lindig, Detecting object usage anomalies, in Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering. 2007, ACM: Dubrovnik, Croatia. p. 35-44.
[54]Maletic, J.I., A. Marcus, and M.L. Collard, A Task Oriented View of Software Visualization, in proceedings of International Workshop on Visualizing Software For Understanding and Analysis. 2002. p. 32-40.
[55]Mealy, E., Improving Usability of Software Refactoring Tools, in Australian Softw. Eng. Conf. (ASEC). 2007. p. 307-318.
10
Proceedings of the 43rd Hawaii International Conference on System Sciences - 2010
Authorized licensed use limited to: Norges Teknisk-Naturvitenskapelige Universitet. Downloaded on April 20,2010 at 10:06:24 UTC from IEEE Xplore. Restrictions apply.
Paper 5
Hanssen, G. K. (2010). Opening up Software Product Line Engineering. In proceedings of 1st International Workshop on Product Line Approaches in Software Engineering, in conjunction with the 32'nd International Conference on Software Engineering (ICSE), Cape Town, 2 May. 1-7.
Opening Up Software Product Line Engineering
Geir Kjetil Hanssen SINTEF ICT
S.P. Andersens vei 15
NO7465 Trondheim, Norway +47 92492454
ABSTRACT
The software industry is experiencing a shift towards more open processes, a globalized market and more active and engaged customers and end users. This change seems natural and inevitable, imposing necessary changes in how software product line organizations plan and drive the development of their
products. This paper gives insight into some recent developments in a product line organization and discusses how their efforts have helped them in improving their development processes and their product line. Based on this experience, this paper provides some preliminary guidelines to both industry and research, indicating that software product line organizations should exploit open innovation, engage customers, build communities and simplify processes and organization.
Categories and Subject Descriptors D.2.9 [Management]: Life cycle, Software process models; D.2.13 [Reusable Software]: Reuse models
General Terms Management, Economics, Human Factors.
Keywords
Software Product Line Engineering, Open processes
1. INTRODUCTION Software product line engineering (SPLE) is a comprehensive and organization-wide approach to developing and evolving lines of
software products over time. In this context, a line is a set of related software products that share commonalities such as software components, architecture, data structures etc., but that still show distinct and different characteristics. Thus, the main concept of SPLE is to prepare for and support this anticipated reuse of core assets as well as the variability in the characteristics of products in a line in order to improve time to market performance, to reduce development costs, and to improve
quality. SPLE as an explicit software engineering practice have been developed, practiced and researched for a long period of time, clearly demonstrating usefulness and relevance
However, an ongoing change in the software industry today;
towards open processes, closer customer involvement, volatile domains and intra-organizational collaborations calls for some rethinking of how we view and practice SPLE.
This paper investigates this change and some of it’s consequences based on recent and relevant insights from an SPLE organization (called CSoft1 here); which have developed and matured over fourteen years, now being one of the market leaders within their application domain. CSoft develops one product line (having the same name). It is highly modular, allowing many configurations and ways to use it. It contains five main modules that serve various business-related activities. In addition, it offers an
Application Programming Interface (API), implemented as standard web services. The composition and use of the modules varies according to customer and case. Some modules can be used in any configuration, while the use of others depends on the situation. The CSoft software comes with a set of predefined configurations for the most common usage scenarios and it provides built-in support for detailed customization to create more variants.
The product line, the development processes and the development organization have changed greatly since the gentle start. The approach back then could be characterized as ad-hoc and as a
creative chaos. The recent uptake of agile development principles has affected the whole organization and their operations, helping the organization to better meet the requirements from the market. Based on these insights, the aim of this paper is to:
1) explain how the SPLE process can be opened up and why this may be a necessary development,
2) illustrate some practicalities, and
3) provide some preliminary guidelines to industry and indicators for future research.
2. BACKGROUND Various comprehensive and detailed descriptions of SPLE practices and principles are available, for example, the frequently quoted books by Clements and Northrop [11], and Pohl et al. [40]. These descriptions vary somewhat in their use of terminology, yet at a conceptual level, they describe SPLE as two complementary processes. The first process deals with the development and
maintenance of reusable core or domain assets, which typically are reusable pieces of software, but can also requirements, design, documentation, etc. The second process deals with the
1 Due to a non-disclosure agreement with CSoft, domain-specific details are omitted to maintain the anonymity of the case organization.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. PLEASE’10, May 2, 2010, Cape Town, South Africa. Copyright © 2010 ACM 978-1-60558-968-8/10/05 … $10.00.
1
development of software products, or applications, using these core assets for rapid and efficient composition of software products adjusted to the need of the customers. In addition, other supportive processes and techniques are described, for example management processes and software architecture practices.
There exists a considerable body of research on SPLE, covering its various aspects. The general impression when studying this
research is that SPLE is a complex practice that can be, and is, carried out in quite different ways [1, 2], yet the general opinion is that this approach holds a potential for “ensuring quality,
economic efficiency, and manageability of software system families.” [3].
One implicit premise though for this structured approach to software product development is that the business domain being targeted and the scope of the product line are relatively stable. Stability in business processes, technology, competition etc. makes the idea of preparing for, and investing in, future reuse and composition reasonable. However, when this stability diminishes
the risk of doing bad upfront investments increases and we may have to rethink the idea of SPLE as we know it today.
During a relatively short time period, the software industry has
experienced several paradigm shifts [4], e.g. the shift in the 80/90’s from computer centrals to a “PC on every desk and in
every home”2, the shift in the 90’s to the internet as the main domestic and global infrastructure and the recent shift from stationary to mobile units. Another shift is how the software development practice has changed from plan-based to more agile approaches [5]. A part of this recent development is how the various actors in software development relate to each other.
Traditionally, development of software products has been organized as a value chain, with well-defined patterns of interaction and responsibilities (Figure 1).
Figure 1. The traditional value chain.
These patterns, very much suitable to the core principles of SPLE because relationships between actors are relatively fixed, are now breaking up, being arranged into new structures. Customers and users demand faster feedback and stronger influence on the technology development. Emerging Web 2.0 deployment models and service-oriented usage patterns changes the relationship between suppliers and customers, which can select and compose
software as services more easily than before. In short, the power balance is shifting, from the suppliers to the customers, which have better ways of influencing the development. This leads to more open relations and faster innovation processes crossing organizational borders, but consequently also less predictable and controllable processes – meaning that traditional and plan-based processes may have to be rethought. The term ‘networked organizations’ [6] is useful to describe this emerging landscape.
2 Microsoft’s famous vision in the 90’s.
Figure 2 illustrates how actors now are related in networks, in contrast to the more traditional value-chain relationships.
Figure 2. The new networked paradigm.
This emerging situation will most likely affect strategic planning, marketing, cross-organizational collaborations, control mechanisms, innovation processes, IPR, economical mechanisms, and others business-vital issues.
Such technology-driven networks have already in short time become familiar to most of us through social networking services such as Facebook, Twitter, Wikis, Mashups and other similar solutions. Now we see this trend affecting the professional
software industry as well in that customers and end-users expect and are empowered to get involved in technology development more than ever. Also, we see that customers and third’s parties can relate in new ways, beyond the direct control of the software supplier.
This type of change is not new at all. In the post-WWII period researchers from the Tavistock institute in England did several thorough studies of the relationship between social structures (humans, teams, work-processes etc.) and technology, and developed the socio-technical theory [7], which then was radical in the way that social structures and technology are viewed as a
system of mutual influence. This strongly contrasted the mechanistic view of humans simply as parts of a machinery, well known through the ideas of Frederic Taylor and his principles of scientific management [8]. This new view and the increased interest in how humans and technology interact and develop were matured during the following decades and lead to the concept of ‘organizational ecology’ [9, 10]. Here, the focus is expanded from how organizations work internally to how organizations exist
together as an ecosystem. This is particularly relevant to organizations operating in complex and unstable domains – a suitable description of the software engineering industry. Viewing the organizational environment as an ecosystem means that it is considered to be an open system (as opposed to a closed one), organizational borders are permeable and organizations relate (dynamically) to other organizations in the same field. ‘Ecosystem’ is the alternative to ‘technocratic bureaucracy’,
which have been the common organizational form in western industry, including many software engineering organizations. However, this type of system fails to adapt to conditions of persistent and pervasive environmental turbulence. ‘Ecosystem’ is
2
a relevant alternative as it promotes self-regulation (as opposed to centralized control) and that it has a networked character.
According to Trist [9] (p. 172), such systems, lacking formal structure, exist yet through the use of technology. Further, they also need shared values, such as for example the 60/70’s youth-culture had. Unlike the micro-level- (the single organization) and the macro-level systems (society), the intermediate level
(organizational ecosystems) is hard to see, understand and describe due to weak structuring. They are also the most recent type, meaning there is less experience with them. This yields especially to software engineering ecosystems, which despite decades of development of software engineering as a practice, is a new but rapidly shaping concept [11, 12], driven by the Internet as a rich and speedy collaborative platform and a growing interest in, and acceptance of, open processes. Agile methods like XP and Scrum are, at least in principle, appropriate in this context.
Socio-technical theory, and the concept of organizational ecology are derived from the field of sociology, but may become
particularly useful as research on software ecosystems progresses and may serve both as 1) a mean to understand new aspects that are observed in the study of software organizations opening up their products and development processes and, 2) as a tool to systematize and relate results from studies (theory building).
3. BACKDROP: AN AGILE SPLE CASE This chapter briefly presents an example of an organization that is moving from a traditional value-chain way of relating to other actors to a more open and networked way.
From the start-up of CSoft about 14 years ago, the development process has matured from a more or less ad hoc type of process
(creative chaos) to a well-defined waterfall inspired process (plan-based and non-iterative). About five years ago, the development process had become too slow and hard to manage with very long periods between the collection of requirements and feedback from actual users of the product – it could take nearly a year before the development teams got any external feedback on the software they were developing. In addition, the organization and its product had grown in size and complexity. Out of necessity, CSoft changed to a radically different development process: Evo [5].
Evo is an agile method comparable to the better-known Scrum method [6], although its terminology differs. Work is done in
fortnightly iterations that are equivalent to the sprints in Scrum [13]. Working software is deployed on test servers by the end of every iteration and stakeholders then evaluate the latest results and give corrective feedback directly to the development teams [7, 8]. Stakeholders may be both internals and externals (customers). As adopted at CSoft, Evo conforms to the four basic values in the Agile Manifesto [14]: interaction is highly valued, there is a strong emphasis on delivering working software after every
iteration, and invited lead users participate frequently with the development teams, which is open to change in requirements and design until near the release of a new version.
The overall SPLE process at CSoft can be described as three interacting main processes with different time horizons [15]. First, a long-term strategic process is running continuously, producing product roadmaps based on input from nearly all parts of the organization as well as several external sources. These roadmaps are high-level plans, or visions, for the product line looking one to two years ahead. They typically present business cases, key stakeholders, prioritized product qualities etc. This information is
made visible external to the organization through various meetings with customers and partners, at conferences and other channels. Roadmaps do not describe specific design decisions but rather high-level guidelines, which are elaborated when detailed plans are laid out for the development projects. This process is
managed by a dedicated product strategy group (PSG) of five product managers, each having a particular responsibility of one of the core components in the product line. In some cases, customers or related third parties visit the R&D department to have close discussions directly with one or more of the development teams to elaborate ideas.
The second main process is the short-termed tactical process, which is the agile development process the R&D department follows to manage the approximately one-year long development projects, leading towards the next main release of the product line (all components are released at the same time). Each core
component is developed/maintained by a fixed component team of developers. Each such team is running an Evo-project, meaning that development is done in two-weeks iterations and that each iteration delivers new working software. Each iteration ideally starts with a meeting with an external stakeholder (a customer), which explains needs and requirements. At the end of the iteration, the team meets with the stakeholder again to get feedback on the outcome (new or improved software) from the
iteration. As CSoft has customers all over the planet they use the web meeting solution WebEx! to communicate as effectively and
closely as possible. This is a radical change compared with the previous waterfall approach where feedback was rare.
The third and last main process is the operational process, which encompasses the day-to-day operations such as support, training, sales and marketing, and high-level maintenance. These activities are valuable sources for input to the tactical process as they represent a wide, diverse and continuous interface with customers.
The product line itself consists of five main modules that together support the core business operation of their customers - a value chain of planning, data collection, analysis, and reporting of results. Due to CSofts request for anonymity, the descriptions here
are kept general and high-level. The composition of a specific product based on the product line can be varied in several ways:
Variability in task: Different modules offer various ways of doing
data collection. For example, there is one module for collecting data via telephone and one for collecting data via the Internet. All variants of the product use the same core component for storing and analyzing data, and have the same functionality for reporting. It is possible to select dynamically which features to include according to the license, in order to satisfy the greatly varying needs of the customers.
Variability in application domain: The product line is being used in three different but related domains.
Variability in feature richness: The product comes in two main variants, the full professional suite and a simple (much less used) version, which simply is a derivation from the full version offering only the basic features and is low in cost and not
supported (mostly used by students). Further, a set of web-service API’s are offered such that externals can add features and use the product line as a platform. As a consequence of this, a relatively large third-party community of more than 60 external organizations has emerged during the past few years, offering additional products and services.
3
Variability in operation: The product is a three-tier application with a web-browser front-end only. This allows for a dual operational model. A customer may buy the product entirely as a service provided by CSoft, meaning that the solution runs within a remote hosted environment together with all data, and all access is
done through an ordinary web browser (software as a service). Alternatively the total solution can be hosted and managed locally. This is more suitable for customers who use the product extensively and need to retain maximum operational control.
According to Bosch’s software ecosystem taxonomy [11] CSoft and their related actors constitutes a ‘web/application’ type, being comparable to more known cases such as Amazon, eBay and SalesForce.
4. LESSONS LEARNED The adoption of Evo about five years ago with the resulting
processes and organizational set-up represents a major change at CSoft. This chapter looks into the actions they have taken, some of the following effects and some enabling and contextual factors.
Figure 3. Actions, factors and effects
Actions taken: The initial waterfall-like approach with extensive up-front planning required a considerable amount of resources and blocked input from customers and other important stakeholders
during the development phase. Through the adoption of core agile development principles, fewer resources are spent on making detailed up-front plans and controlling plan conformance during development. The iterative process opens for continuous corrections during development based on a close dialogue with stakeholders, including invited customers [16]. This enables a re-active approach to SPLE where no assets are changed or added to the line unless they are explicitly founded on concrete needs.
Observed effects: The resulting SPLE process at CSoft helps addressing some of the inherent risks that comes with this approach. McGregor [17] describers that, for a pro-active
approach, there is a risk of developing assets that will become obsolete and not used in later applications. For a re-active approach, there is a risk of missing short-termed business opportunities due to increased time-to-market and that it might require considerable effort to, re-actively, prepare assets for later reuse. CSofts incremental and iterative approach meets these risks as a compromise between the pro-active and re-active approaches. Continuous corrections by involved stakeholders ensure that
changes and additions to the product line actually will be used. The process also opens for corrections close to the release, if found necessary. In addition, the proximity of representatives from the business domain and the extensive use of direct and
conversational exchange of information improves the ability to catch both explicit and implicit (tacit) requirements [18]. One potential effect on the negative side is a reduced maintainability of the product line. The agile development process emphasizing a continuous focus on short termed goals seems to reduce the focus
on maintaining the code and architecture properly, thus causing escalating system entropy which seriously hampers the organizations ability to improve and develop the existing code base [19].
Enabling factors: The opening of the SPLE process at CSoft has been supported by some additional measures. One of the most noteworthy is that a community of third party organizations has emerged, partly supported by CSoft and partly as an effect of the accessibility to the product line via the API’s. Approximately 60 organizations are now offering software products and/or services partly or completely based on the CSoft product line. This means
that CSoft may keep full focus on developing the core product line and that customers are offered a large number of specializations and applications of the product line. The emergence and growth of this community come as a result of not only the technical accessibility via the API’s but also the sharing and external visibility of product plans and roadmaps. Key clients are given insight into these plans; some are even visiting the R&D department to elaborate business needs and ideas, and to discuss
potential solutions directly with the development teams. CSoft has also (once so far) organized a large product conference where strategies were announced, new features introduced and third parties got to expose their solutions.
Contextual factors: This change in CSofts approach to developing their product line is moderated by a set of contextual factors. First of all, the changes done to the development process is a reaction to the volatility of the business domain being addressed. This particular domain is still shaping, its boundaries still being determined. This is an alternating process; the technology being developed and offered through the product line is creating new
business opportunities, which in return leads to new requirements. In this context, it is more important to be able to respond quickly to the market than to produce reusable components that might be useful at a later moment. Enabling an agile and responsive development organization gives it control over how the domain and its supportive technology develops – a clear advantage in a competitive business. Another factor affecting the shaping of the SPLE process is the strongly emerging software-as-a-service
(SAAS) delivery model. Software is not procured as a commodity, installed and managed locally, but rather as a service accessed over the Internet. This simplifies the delivery of software and consequently also the development process. For example, this model makes it practical to engage selected stakeholders, which frequently and with a minimum of effort can access the latest increments to assess and provide corrective feedback.
5. DISCUSSION The study of CSoft shows an organization that has started to open up. This has influenced, among other matters, the information flow, the innovation processes and the interface to the product line.
Information: The software development process prior to the adoption of Evo was to a large extent based on the development and use of detailed project plans. The idea was that these plans, with detailed requirements would provide the developers the information they needed to develop the next release of the product
4
line. Likewise, information collected externally, typically requirements from customers, was collected in advance of and during the planning phase and constituted a limited time slot of information flow which to a large degree dried out as the development phase initiated. In addition, this information
exchange was pretty much one-way and not based on experience and dialogue. This negative pattern changed considerably when Evo was adopted. Especially the emphasis on short iterations and continuous feedback from involved customers has opened up the information flow dramatically. This change in information flow and communication has added value to the development process. Takeuchi and Nonaka [20] uses the term multilearning which works in two dimensions; across multiple levels (individual,
group and corporate) and across multiple functions. Equally important, this new information flow is not based on plans and formal documents but on frequent and direct communication (face to face, telephone or web-meetings), thus increasing the richness of communication [21].
Innovation: Closely related to this more open information paradigm is how innovations now are managed. The new process at CSoft resembles what has been defined as open innovation, a term initially coined by Henry Chesbrough [22]. In a nutshell, open innovation is about opening up organizational borders to bring in ideas and innovations from external sources and to profit
from them. This as opposed to the earlier implicit approach of what can be called closed innovation where the generation, elaboration and implementation of new ideas were kept as internal processes, mainly as an act to maintain maximum control of the process and to protect the results and the intellectual properties. This (traditional) approach has often resulted in a separation between research and development, often as separated organizational units - somewhat like the explicit division between
core asset development and product development in (traditional) SPLE [23]. At CSoft, these two high-level activities are brought together through the Evo process. Consequently, also gathering the innovation process and the development process into one where stakeholders, external to the organization, are a driving innovative force.
Interface: Another source of innovation, indirect and external to CSoft, is enabled by the API’s, which partially opens up the product line itself. These interfaces nurtures the community of third parties, which together represents an enormous amount of knowledge, competence and creativity – far larger than CSoft
themselves represents. Together, this openness radically increases the innovation capacity related to the CSoft product line. The benefit of this policy is twofold. First, the customers get access to a well of specialized solutions and services, which CSoft are not able to offer - the extent of applicability increases. Secondly, this preserves a maximum focus on the development and evolution of the core product line. On the other side, the drawback is that CSoft does not own and benefit directly from these associated
solutions and services. If one such added solution should develop to an extent that it would naturally fit into the core product line it self, the only option would be a merger or an acquisition. The product line company, their customers and the third-parties shapes a kind of virtual and networked enterprise or what can be defined as a software ecosystem [12]. Originally, descriptive frameworks, the use, and research on SPLE have focused on organization internal matters. However, as Jan Bosch says: “..,there is no
reason why a software product line would have to stop expanding
at the organizational boundary” [11]. This development towards
more collaborative and open arrangements will most likely affect industrial SPLE practice in the near future.
On the whole, the ongoing change at CSoft seems to improve the organizations ability to serve a market that is still shaping and hard to plan for as well as being highly demanding. Also, establishing collaborative connections with other, non-traditional, partners in planning and development strengthens the innovation.
Although an open approach to SPLE seems like a natural development there is also some related risks and costs. First, the level of control is reduced – the organization is subject to actions of others and the development in the market in general. However, this is often the case no matter approach taken, it is thus a better strategy to develop an organization and processes to deal with external change and even benefit from it. This is very much the core motivation for agile development methods in the first place [24] – favoring the ability to respond to change over strictly
following plans. Another potential risk, over time, is the negative effects of over focusing on short-termed goals and needs. This is what James March explains as exploitation; that the development organization mainly refines the products it already has (exploiting the existing) and thus risks forgetting to make use of new opportunities (exploring the future) [25]. However, CSoft seems to be aware of the need to balance exploitation and exploration by – respectively – engaging external stakeholders both in short-termed development as well as long-termed strategic planning.
6. CONCLUSIONS The lessons learned from CSoft draws an interesting picture. Organizational boundaries and traditional business patterns have less meaning than before. Customers and even third-party organizations demand and take a much more active role in product development, driving innovation and intensifying competition. Software product organizations are not isolated units but will have
to partner with other actors to provide the best products and services to their customers. Some of the major leaders in the ICT industry have already adapted to the new landscape, for example IBM3 who have established a ‘partner ecosystem’ to actively support a community of external actors that creates added value based on their products. Somewhat similar, SAP4 actively drives a partner ecosystem to facilitate innovation across their own organizational boundaries, benefiting from being open towards
customers and other related partners. CSoft is an example of a medium-sized organization and there is no reason why these changes should not apply to smaller ones too.
The lessons learned from CSoft shows how they, through an opening of the organization and their product line engineering process enabled by the adoption of the agile method Evo, have contributed to the emergence of a software ecosystem. This change in the way they as a software product line organization relate to other actors in the ecosystem, both customers and third parties, creates new business opportunities, but new challenges as well. Relating this to Trist [9], we see that 1) the ecosystem is
enabled through the use of technology (API’s, web meeting tools and rapid sharing of information) and 2) shared values (a mutual benefit in the growth of the product line as a platform between CSoft, customers and third-parties).
3 http://www-01.ibm.com/software/tivoli/governance/servicemanagement /ecosystem/partner-ecosystem.html
4 http://www.sap.com/ecosystem/communities/index.epx
5
Predicting future development is always though, but out of the experience presented in this paper we give four guidelines to other SPLE organizations:
a) Exploit open innovation.
Relying solely on internal resources for bringing the product line(s) forward is risky. External stakeholders may directly or indirectly provide turning-point ideas. Internet as an integration platform with external organizations opens for new and unforeseen opportunities that should not be missed. Also, employees shift workplace more often than before, especially
knowledge workers tend to be less faithful, seeking new challenges and opportunities. This is of particular importance to software organizations where the knowledge and competence of the workers is the key asset.
b) Bring the users into the planning and development.
Enabling customers and end-users of the software to become
participants in the planning and development is beneficial in several ways. Internally, this naturally may become a good source of requirements and ideas. It has also shown to contribute to internal motivation in development teams – having such close connection with the real users is simply motivating [26]. From the customers’ perspective, such close involvement creates opportunities to affect the direction of development, being particularly valuable in cases where the
software product(s) are fundamental to the operation of the business. Lastly – customer-supplier proximity will also most likely have positive effects to marketing and sales as the selected actors in the market are given detailed insight into plans and products.
c) Build and nurture communities.
It is quite common to software product organizations to maintain communities of customers and users. Now, it may also become equally important to build and maintain communities of third party actors, external organizations that base their business on the product line(s). This creates additional value to the product line(s) and may become an extremely valuable source of innovation. One key investment in maintaining such a community is to provide insight into
strategies and plans and to provide interfaces to the software, either strictly controlled ones (like CSoft) or even open interfaces by offering parts of the software under open licenses.
d) Simplify processes and products.
To enable the changes described here, both development processes and the technology need to be kept simple. Unneeded complexity makes planning hard and development slow. Agile principles can improve flexibility and responsiveness, if applied wisely [15]. Also, proper refactoring of existing software may make it easier to improve and evolve [19].
Most product line organizations are probably taking some of these actions, more or less, already – but continued focus is vital to stay competitive in an increasingly more open and globalized software business. Other actions or variants of these may very well also be
appropriate, and both academia and industry should pay more focus to these topics in the future.
7. ACKNOWLEDGMENTS Thanks to the people at CSoft for sharing their precious time,
providing highly valuable insight into the work they do. This paper was written as part of the EVISOFT project (grant 156701/220) and the Agile project (grant 179851/I40), both partly funded by the Research Council of Norway.
8. REFERENCES [1] Bosch, J. Software product lines: organizational
alternatives. in 23d International Conference on
Software Engineering (ICSE'01). 2001. Toronto, Canada IEEE Computer Society.
[2] Mannion, M. Organizing for software product line
engineering. in 10th International Workshop on
Software Technology and Engineering Practice
(STEP’02). 2002. Montreal, Canada: IEEE Computer Society.
[3] Birk, A., et al., Product line engineering, the state of the
practice. IEEE Software, 2003. 20(6): p. 52-60.
[4] Rajlich, V., Changing the paradigm of software
engineering, in Communications of the ACM. 2006. p. 67-70.
[5] Cockburn, A., Agile Software Development. The Agile Software Development Series, ed. H.J. Cockburn A. 2002: Addison-Wesley.
[6] Van Alstyne, M., The State of Network Organization: A
Survey in Three Frameworks. Journal of Organizational
Computing and Electronic Commerce, 1997. 7(3): p. 83-151.
[7] Trist, E.L. and K.W. Bamforth, Some social and
psychological consequences of the longwall method of
coal-getting. Human Relations, 1951. 4(1): p. 3-38. [8] Taylor, F.W., The Principles of Scientific Management.
1911: Newton Library Harper & Row. [9] Trist, E.L., A Concept of Organizational Ecology.
Australian Journal of Management, 1977. 2(2): p. 161-175.
[10] Trist, E.L., Referent Organizations and the
Development of Inter-Organizational Domains. Human Relations, 1983. 36: p. 269-284.
[11] Bosch, J. From Software Product Lines to Software
Ecosystems. in 13th International Software Product
Line Conference (SPLC'09). 2009. San Fransisco, USA:
IEEE Computer Society. [12] Messerschmitt, D.G. and C. Szyperski, Software
Ecosystems, Understanding an Indespensable
Technology and Industry. 2003, Cambridge, Massachusetts, USA: The MIT Press. 424.
[13] Schwaber, K., Beedle, M., Agile Software Development
with Scrum. 2001: Prentice Hall. [14] Agile Manifesto. 2009: http://www.agilemanifesto.org/. [15] Hanssen, G.K. and T.E. Fægri, Process Fusion - Agile
Product Line Engineering: an Industrial Case Study. Journal of Systems and Software, 2008. 81: p. 843-854.
[16] Hanssen, G.K. and T.E. Fægri. Agile Customer
Engagement: a Longitudinal Qualitative Case Study. in 5th International Symposium on Empirical Software
Engineering (ISESE'06). 2006. Rio de Janeiro, Brazil: IEEE Computer Society.
6
[17] McGregor, J.D., Agile Software Product Lines,
Deconstructed. Journal of Object Technology, 2008. 7(8): p. 7-19.
[18] Grunbacher, P. and R.O. Briggs. Surfacing tacit
knowledge in requirements negotiation: experiences
using EasyWinWin. in 34th Hawaii International
Conference on System Sciences (HICSS'01). 2001. Hawaii, USA: IEEE Computer Society.
[19] Hanssen, G.K., et al. Software entropy in agile product
evolution. in 43d Hawaiian International Conference on
System Sciences (HICSS'10). 2010. Hawaii, USA: IEEE Computer Society.
[20] Takeuchi, H. and I. Nonaka, The New New Product
Development Game. Harward Buisiness Review, 1986. [21] Daft, R.L., R.H. Lengel, and L.K. Trevino, Message
Equivocality, Media Selection, and Manager
Performance: Implications for Information Systems. MIS Quarterly, 1987. 11(3): p. 355-366.
[22] Chesbrough, H., Open Innovation: The New Imperative
for Creating And Profiting from Technology. 2003, Boston: Harvard Business School Publishing
Corporation. [23] Clements, P.C. and L. Northrop, Software Product
Lines: Practices and Patterns. 2002: Addison-Wesley. [24] Boehm, B. and R. Turner, Balancing Agility and
Discipline - A Guide for the Perplexed. 2004: Addison-Wesley. 266.
[25] March, J.G., Exploration and exploitation in
organizational learning. Organization Science, 1991.
2(1): p. 16. [26] Fægri, T.E. and G.K. Hanssen, Collaboration and
process fragility in evolutionarily product development. IEEE Software, 2007. 24(3): p. 96-104.
7
Paper 6 Hanssen, G. K. (2010). A Longitudinal Case Study of an Emerging Software Ecosystem: Implications for Practice and Theory Journal on Systems and Software. Under review.
1/24
A Longitudinal Case Study of an Emerging Software Ecosystem: Implications for Practice and Theory∗
Geir K. Hanssen a, b.
a Department of Computer and Information Science, Norwegian University of Science and Technology (NTNU), Sem Sælands vei 7-9, NO7491 Trondheim, Norway
b SINTEF ICT, NO7465 Trondheim, Norway
Abstract
Software ecosystems is an emerging trend within the software industry implying a shift from closed organizations and processes towards open structures, where actors external to the software development organization are becoming more involved in development. This forms an ecosystem of organizations that are related through the shared interest in a software product, leading to new opportunities and new challenges to the industry and its organizational environment. To understand why and how this change occurs, we have followed the development of a software product line organization for a period of approximately five years. We have studied their change from a waterfall-like approach, via agile software product line engineering, towards an emerging software ecosystem. We discuss implications for practice, and propose a nascent theory on software ecosystems. We conclude that the observed change has led to an increase in collaboration across (previously closed) organizational borders, and to the development of a shared value consisting of two components: the technology (the product line, as an extensible platform), and the business it supports. Opening up both the technical interface of the product and the organizational interfaces are key enablers of such a change.
Keywords: software ecosystems, software product line engineering, agile software development, longitudinal case study.
1 Introduction A recent development within software engineering is the emergence of software ecosystems [1, 2]. This new concept and its implicit reference to ecology imply a shift of focus from the internals of the software organization (the individual organism) towards its environment and the relations and actions within (the ecosystem). Viewing the software industry and the market it serves as an ecosystem may introduce a set of new challenges and opportunities [3], for example new business models, open innovation, collaborative development, issues of ownership, strategic planning, and variability management. This seems to be a part of a general development in the software industry [4], where customers expect to be more involved in the shaping of the technology they use, where innovation is no longer an internal matter, where time to market is decreasing, and adoption rates are increasing.
*) Submitted to the Journal of Systems and Software - Under review
2/24
To understand some of this ongoing development we have studied CSoft, a medium size software product line organization for a period of approximately five years. During this time they have moved from a closed and plan-driven approach to a practice of agile software product line engineering (SPLE) and now further towards a software ecosystem. Based on three recent studies of this organizational development we have designed and conducted a final study, collecting new qualitative data to investigate in more detail this development in general, and how the organization relates to its environment in particular. Thus, the research question for this study is:
Why and how is software product line engineering developing towards a software ecosystem?
Our study will give insight into three aspects of this question: 1) how this ecosystem has emerged, 2) how the present organization works in terms of its structure, its processes, and its product line, and 3) how the organization relates to actors in its external environment.
To help develop this understanding and to contribute to the growing research on software ecosystems, we relate our findings from this case study to the general theory of organizational ecology [5, 6], which is derived from socio-technical theory. This theoretical platform has matured for decades and describes how organizations in general relate to their external environment. We have found that this knowledge is relevant, applicable and beneficial to the software engineering domain, and that existing literature on software ecosystems lacks this theoretical connection. Several recent studies have suggested definitions and developed important concepts but the terminology and connectivity between these concepts are still vague. We believe that this field of research can benefit from the development of an empirically grounded theory of software ecosystems. The results obtained from this study are used to propose such a theory, as a starting point. We also discuss implications for practice as well as providing advice for further research.
2 Background
2.1 Socio-technical theory and organizational ecology Socio-technical theory is a view of the organization as the sum of and interplay between a social system and a technical system. That is 1) the people, their relations, their knowledge, and how they work together as a whole, and 2) the tools and techniques being used to perform the work. A fundamental principle of socio-technical theory is the natural interdependence between these two systems, meaning that to improve the performance of the organization (productivity, quality of work etc.) both subsystems have to be considered at the same time. Changing one affects the other. This principle was first drafted by Trist and Bamforth in 1951, based on their studies of changes of work processes in coalmines [7].
Through studies that followed, the theory has been developed to consider how organizations relate to their external environment [5]. This implies that an organization has to be understood as both 1) the internal interplay between a social subsystem and a technical subsystem (the socio-technical system), and 2) the interplay between the organization and its external environment.
3/24
Figure 1 - The socio-technical system and the organizational ecosystem
Emery and Trist [ibid.] developed a simple classification system of four types of organizational environments – forming a series, in which the degree of causal texturing is increased. Thus, understanding and ordering the types of environments is useful in understanding socio-technical systems beyond the limits of a single organization. The first, and the simplest type, is ‘the placid, randomized environment’ where “goods” and “bads” are unchanging, and are randomly distributed in the environment [ibid., p.7]. The optimal strategy is to ‘do one’s best’ on a purely local basis – there is no difference between strategy (planning) and tactics (execution). The second type is ‘the placid, clustered environment’ where “goods” and “bads” are not randomly distributed but band together in certain ways. Strategy is different from tactics, and survival becomes critically linked with what an organization knows about its environment. Organizations in this environment tend to become hierarchical, with a tendency towards centralized control and coordination [ibid., p. 8]. The third type, ‘the disturbed-reactive environment’, is an environment where there is more than one organization of the same kind. The existence of a number of similar organizations becomes the dominant characteristic of the environmental field. These organizations compete, and their tactics, operations and strategy are distinguished. The flexibility encourages a certain decentralization, and it also puts a premium on quality and speed of decisions at various peripheral points [ibid., p 9]. The fourth, and the most recent type is ‘turbulent fields’. This type implies that significant variances arise from the field itself, not simply from the interaction of the component organizations. Three trends contribute to the emergence of these dynamic field forces: i) the growth to meet type-three conditions, ii) the deepening interdependence between the economic and other facets of the society, and iii) the increasing reliance on research and development to achieve the capacity to meet competitive challenge. A change gradient is continuously present in the field [ibid., p. 10].
4/24
Figure 2 Types of organizational environments
This interplay, inherent in turbulent organizational environments, has been further studied, leading to the development of the concept of organizational ecology [6]. It is particularly relevant to organizations operating in complex and unstable domains. Viewing the organizational environment as an ecosystem means that it is considered to be an open system as opposed to a closed one, organizational borders are permeable, and organizations relate dynamically to other organizations in the same field.
Developing the ecology concept further, Trist describes three classes of organizations within the turbulent environment. In a Class 1 system member organizations are linked to a key organization among them. The key organization acts as a central referent organization, doing so even though many of them are only partially under its control, or linked to it only through interface relations. Interface relations are as basic to systems of organizational ecology as superior-subordinate relations are to bureaucratic organizations. Interface relations require negotiation as distinct from compliance – a basic difference between the two types of system. In a Class 2 system the referent organization is of a different kind. It is a new organization brought into being and controlled by the member organizations rather than being one of the key constituents. A Class 3 system has no referent organization at all.
Technocratic bureaucracies have been the natural organizational form for disturbed-reactive environments (up to the 1960’s or so), and this is a form that has been applied to many software engineering organizations. However, this type of system fails to adapt to conditions of persistent and pervasive environmental turbulence, mostly because it is constructed and optimized to work well in stable environments. This leads to the emergence of the new ecologically oriented systems, which show clear differences in that they promote self-regulation (as opposed to centralized control), and that they have a networked character (as opposed to segregated organizations). According to Trist [ibid. p. 172], such systems, lacking formal structure, exist through the use of technology. Further, they also need shared values. Trist used the example of the 60/70’s youth-culture that had a set of distinct (political) values. A more appropriate example from a business perspective is a shared value in growth and profit.
Unlike the micro-level (the single organization) and the macro-level systems (society), the intermediate level systems (organizational ecosystems) are hard to see, understand, and describe due to their weak structuring. They are also the most recent
5/24
type, so there is less experience with them. This relates especially to software engineering ecosystems, which is a new but rapidly advancing concept [1, 2] despite decades of development of software engineering as a practice and business. This approach is driven by the Internet as a rich and speedy collaborative platform (the technology), and a common interest in the product line (the shared value).
2.2 Software ecosystems Software ecosystems is a more recent term, that refers to a networked community of organizations, which base their relations to each other on a common interest in a central software technology. Some other definitions of this emerging concept have been proposed, for example by Jansen et al. [8]: “a set of businesses functioning as a unit and interacting with a shared market for software and services, together with the relationships among them.” (p. 2). Another definition by Bosch [1], focusing more on the common interest in the software and its use, is: “the set of software solutions that enable, support and automate the activities and transactions by the actors in the associated social or business ecosystem and the organizations that provide these solutions.” (p 2).
Well-known examples of communities that may be seen as software ecosystems are Apples iPhone/Appstore platform, and the open-source development environment Eclipse. The first is an example of a partially closed and controlled ecosystem, and the latter is an example of an open ecosystem allowing more flexibility in use and development. This simply illustrates that the ecosystem concept may refer to a wide range of configurations. Yet, they all involve two fundamental concepts: 1) a network of organizations and, 2) a common interest in central software technology. These organizations may have different relations to the central software technology, and for this reason, different roles in the ecosystem. There are three key role types. First, one organization (or a small group) acts as the keystone organization, and is in some way leading the development of the central software technology. The second key organizational role is the end-users of the central technology, who need it to carry out their business, whatever that might be. The third key role is third party organizations that use the central technology as a platform for producing related solutions or services. In addition to these key roles, various other related roles might be part of the ecosystem [9], for example standardization organizations, resellers, and operators.
A fundamental property of the central software technology is that it is extensible beyond the keystone organization [10]. Extensibility can be achieved in various ways, for example by providing an application programming interface (API) or a software development kit (SDK), by supporting exchange of open data formats, or by offering parts of the technology as open source. Opening up the technology in these, and potentially other ways, enables external organizations to use the central software technology as a platform where existing services or data can be used and extended. Bosch proposed a Software Ecosystem Taxonomy [1] that identifies nine potential classes of the central software technology, according to classification within two dimensions. The first one is the category dimension, which is ranging from operating system to application, and to end-user programming. The second one is the platform dimension, ranging from desktop to web, and to mobile. The case discussed in this paper is an application-web type.
The keystone organization has a special position in the ecosystem as it controls, strictly or loosely, the evolution of the central software technology. This may include
6/24
various responsibilities, for example typical software product development activities such as strategic planning, R&D, and operational support. These responsibilities come in addition to activities specific to ecosystems such as enabling efficient external extensibility, provision of insight into planning and development, and supporting ecosystem partners in various other ways.
One potential benefit of being a member of a software ecosystem is the opportunity to exploit open innovation [11], an approach derived from open source software processes where actors openly collaborate to achieve local and global benefits. External actors and the effort they put into the ecosystem may result in innovations being beneficial not only to themselves (and their clients) but also to the keystone organization, as this may be a very efficient way of extending and improving the central software technology as well as increasing the number of users. Closer relationships between the keystone organization and the other actors may drive both an outside-in process as well as an inside-out process, as described by Enkel et al. [12]. Also, the proximity between the organizations in an ecosystem may enable active engagement of various stakeholders in the development of the central software technology [13].
The ultimate objective for investing in and working towards an ecosystem is that all members will gain more benefits from being a part of it, as compared to the more traditional approach for software product development with segregated roles, a low level of collaboration, and closed processes. A well functioning ecosystem is, in summary, a complex configuration with collaboration across traditionally closed organizational borders. Such multi-organizations are probably not established as a deliberate, planned effort. Rather, they emerge as a result of many congruent factors such as technology development, globalization, new collaborative patterns, and clients becoming more and more accustomed to participating in the shaping of the technology they use.
3 Study method The case study reported in this paper is the last in a series of four consecutive studies of the software product line organization CSoft, constituting a longitudinal study started in 2004, and now covering five years of the organization’s history [13-16]. In addition, two earlier papers written by other authors provide background information on the historical development of the organization [17, 18]. Together, this provides a valuable insight into the longitudinal development of a software product line organization.
The name of the case organization and its product is kept anonymous due to a non-disclosure agreement that has been signed by the author.
7/24
3.1 Study type The study can be classified as a longitudinal single case study. We have applied a set of principles for interpretative field studies defined by Klein and Myers [19]:
Table 1 – Application of Klein and Myers seven principles of interpretive field research. Principles (from Klein and Myers [16], p. 72) Practiced in the case study
1. The Principle of the Hermeneutic Circle
Data are collected through repeated interviews with actors playing various roles. The data collection is supported by observations and as collection of relevant documentation. The growing knowledge of the case has guided the data collection.
2. The Principle of Contextualization
The study of the case is conducted from two viewpoints – the present organization and its activities, and how this organization has emerged over time.
3. The Principle of Interaction Between the Researchers and the Subjects
A large part of the collected data is based on semi-structured interviews [17] that followed open interview guidelines to ensure a balance between thematic focus and room for reflection, correction, and discussions. This allows for unplanned but relevant topics to be addressed.
4. The Principle of Abstraction and Generalization
Findings are related to the concept of organizational ecology [5], derived from socio-technical theory. Key principles from this theoretical background are applied to the studied case. Some are adopted, some are adjusted, and some are added.
5. The Principle of Dialogical Reasoning
The theory applied to the case was not used to plan and guide the data collection. The applicability of the theory became evident through the analysis after the data had been collected.
6. The Principle of Multiple Interpretations
This principle is followed by collecting data from both external actors and people with various roles in the product line organization.
7. The Principle of Suspicion
The data have been collected and analyzed by the author, who is external to the organization, having no formal responsibilities, interests or agenda, except to create an unbiased view of the organization and its development.
8/24
3.2 Data sources During 2008, 2009 and 2010 new data were collected to investigate the agile software product line organization, its processes, and in particular how they relate to external actors. Data are of three types: interviews, observations, and collected documents. Table 2 shows the list of content for each type.
Table 2 - Collected data
Interviews R&D manager (semi-structured interview) Manager of Professional Services (semi-structured interview) Product Strategy Group manager (semi-structured interview) Product Strategy Group members (3 semi-structured interviews) Technical Account Manager (semi-structured interview) Team leader (semi-structured interview) Team member/developer (semi-structured interview) Product Strategy Group manager (follow-up interview after observation of the review meeting) 2 (of 3) members from the Architecture Team (group interview)
Observations Product conference (various presentations and ad-hoc conversations) Customer review meeting (one R&D team + customer team + sales) Webinar presentation of the new API
Documents Component A-E project roadmaps Chief Strategy Officers keynote at a product conference Chief Executive Officers keynote at a product conference Vice President Product marketing – presentation at a product conference Customer’s presentation at a product conference
3.3 Sampling and collection The focus of this study has been to investigate how CSoft relates to external actors such as customers and third party organizations. This has guided the sampling of interview respondents, selection of events for observation and documents to be collected. Interview respondents have been asked to recommend other respondents, based on their understanding of the study (snowball sampling). A single-person interview lasted approximately 30-40 minutes. Group interviews lasted up to 3 hours.
All data have been collected and stored in a database for later analysis. Interviews were recorded using a digital voice recorder and then transcribed.
3.4 Analysis Data has been analyzed in two steps:
Step1 – All data were first examined to produce an intermediate analysis report, which documents the development process in terms of roles, activities, and artifacts, in addition to high-level concepts, necessary to understand how product planning and development is conducted. This analysis created a structure by grouping information coming from the various data sources. Examples of such concepts are teamwork, planning, and innovation. The objective of this report was to establish a broad understanding of the context, i.e. the organizational set-up and its processes. The report has been used in the description of the study context (3.5), as well as a preparation for step 2.
9/24
Step 2 – All data, in textual format, were analyzed using a tool for qualitative data analysis, NVivo™. Data were coded, meaning that fragments of text, for example statements, facts, comments, concerns, and ideas, were tagged with nodes describing the data fragment. Examples of such nodes, which emerged from the analysis, are ‘co-creation’, ‘finding the right stakeholders’, ‘learning of business processes and domain’ etc. These are detailed in chapter 4 (Results). This way of analyzing the data develops a meaning and an interpretation of the data, and relates fragments from different locations in the data material to concepts, which may be grouped into categories. These results can then be used as the basis for formulating a theory explaining some of the findings. This approach resembles ‘grounded theory’ in that a theory is developed, and that it is grounded on data [20]. The theory being developed may be new, but it can also be related to an already established theory. In the case of the CSoft study, the analysis is related to the organizational ecology concept explained in section 2.1, and it seeks to apply this theory in a software product-engineering context. Implications for theory are discussed in section 5.1.
3.5 Study context
3.5.1 The organization, processes and the product line The organization. CSoft is a medium-size software company that develops, maintains, and markets a single product line under the same name. They have now become the market leader in the high-end segment of the market. Currently CSoft employs about 260 people, including more than 60 developers. The main office is located in Oslo, which houses the main section of the development department as well as top management and various support services such as operations, technical support, sales, training and others. The rest of the organization is distributed internationally with development departments, sales and other support services in Eurasia and in the USA.
The development department is organized as a set of teams, each responsible for one of the main modules in the product line. A team is mostly a fixed group of 4-6 developers and a team leader, who is experienced in the domain and the technology. The teams share some supportive services such as the Chief Technical Officer (CTO), a system architecture team, and QA-services.
Being a product line organization means that strategic planning is a natural and important activity. This used to be a side activity of some of the supportive roles but has now developed into a fulltime prioritized internal service. A Product Strategy Group (PSG), consisting of five product managers, is responsible for developing a product roadmap for each of the main modules, and supports their development.
The processes. The overall SPLE process at CSoft can be described as three interacting main processes, each with a different time horizon [13]. First, the PSG drives a continuous long-term (1-2 years) strategic process, creating product roadmaps based on input from nearly all parts of the organization as well as several external sources. These roadmaps are high-level plans, or a vision, for the product line looking one to two years ahead. They typically present business cases, key stakeholders to participate in development, and prioritized product qualities, instead of functional requirements or feature descriptions. The main content of these plans is made visible externally to the organization through various meetings with customers and partners, at conferences, and through other channels. Roadmaps do not describe
10/24
specific design decisions but rather high-level guidelines, which are elaborated when detailed plans are laid out for the development projects. In some cases, customers or related third parties visit the R&D department to have close meetings directly with one or more of the development teams to elaborate ideas and discuss needs.
The second main process is the agile development process Evo [21], which the R&D department follows to manage the approximately one-year long development projects, leading towards the next main release of the product line (all components are released at the same time). Each component team runs an Evo-project, meaning that development is done in fortnightly iterations, and that each iteration delivers new working software. Each iteration ideally starts with a meeting with an external stakeholder to explain and discuss needs and requirements. At the end of the iteration the team meets with the stakeholder again to get feedback on the outcome (new or improved software) from the iteration. Customers come from all over the world, using a web meeting solution (WebEx™) to communicate as effectively and closely as possible. This is a radical change compared with the previous waterfall approach where feedback was rare.
The third and last process is the operational process, which encompasses the day-to-day operations such as support, training, sales and marketing, and high-level maintenance. Apart from being common functions in a product organization they are also highly valuable sources of input to both the strategic process and the Evo development processes as they represent a wide, diverse, and continuous interface with customers.
The product line consists of five main modules, which together support the core business operation of the customers: a value chain of planning, data collection, analysis, and reporting of results. The composition and use of the modules varies according to customer and case. Some modules can be used in any configuration, while the use of others depends on the situation. The software comes with a set of predefined configurations for the most common usage scenarios.
The product line offers an Application Programming Interface (API), which is implemented as standard web-services. Most modules offer an API, which enables clients to integrate the product with other systems, and which is extensively used by other third-party organizations to offer additional software solutions and/or services. More than 60 such partners now base their business partly or completely on using the CSoft product line as a platform through these APIs.
11/24
3.5.2 From creative chaos to an emerging software ecosystem CSoft was established in 1996 and has grown continuously since then. This development has gone through three phases, and has now entered a fourth. This section presents a summary of these phases of development, and it indicates some important milestones in the development of the organization.
The timeline in figure 3 shows the main events [22] in the development of the organization, the approximate increase in staff, and the studies of the organization.
Figure 3 - Timeline of the development of the case organization
1996-1999: “Creative chaos” The company initially grew out of a small business providing manual services to very few clients. A simple homemade software tool grew into a solution that could be sold as a stand-alone software product. The main focus of the company changed, and the development of this product became the main objective. At the start, in 1996, there were only a few employees providing the product to a handful of clients. The process can best be described as ad-hoc since the main drivers were almost daily interactions with and feedback from customers. A customer request was literally routed directly to the developers. This start-up phase was a ‘creative chaos’ – that is, it had nearly no plans and no control, but it was undoubtedly extremely creative and productive. The product grew rapidly, not only in terms of features and functionality, but also in terms of defects and complexity. Work became stressful, with little control, and a lot of overtime.
1999-2003: Waterfall As the number of customers increased the organization formalized the development process, mostly according to the principles of the waterfall model [23]. This somewhat disreputable approach to organizing software development emphasizes upfront detailed planning of requirements, design, and development. The development is divided in consecutive phases, where requirements are developed into a design, and the design is developed into a software system, which is tested close to deployment. Prior to this, the R&D department was extended with a QA-manager. This structured approach established a certain level of control, and helped the organization in the continuing development of their product, which grew alongside the customer base. After a few years, several problems arose, clearly related to the waterfall approach [17]. Testing and verification was postponed to the final stages leading to late identification of problems, which in turn caused much rework. Also,
12/24
requirements were almost solely focused on functionality, leaving out the quality perspective. To support a growing R&D department, and to spread knowledge about the formalized development process, an electronic process guide (EPG) was developed, and made available via the intranet [18]. The product expanded and eventually became a product line, capable of serving various usage scenarios. To manage this increasing complexity a Product Management Team (PMT) was formed – a group of experienced employees with other main responsibilities, who were supposed to spend part of their time in strategic product planning. In addition various specialized functions, beyond software development, were introduced such as Technical Account Managers (TAM), the operations department, and training services.
2004 – approx. 2009: Agile product line engineering Due to a critically declining process performance, management of R&D was looking for a way to improve the situation. At a software engineering conference, a few representatives from the company learned about evolutionary development and the Evo method [21]. As it seemed to address some of their concerns they initiated a three-month testing period of this radically different development approach in one of the release projects. Instead of an extensively prepared process adoption, they started out with a few principles, focusing on requirements management, where functional requirements were replaced by explicit expression and evaluation of product qualities, preferably stated by customers involved in the development process [15]. Early experience showed that the number of issues near release was reduced, and that the delivered product matched customer expectations better than before. After this initial process trial, Evo was adopted on a permanent basis [13, 14]. Alongside the growth in the organization and the product line the PMT group was re-established as a full-time Product Strategy Group managed by a Chief Strategy Officer (CSO). Another supportive service, the architecture team, was established, originally with three full time members. Their task was to handle the excessive level of system entropy [16], and to support R&D in architectural issues. In 2006 CSoft acquired a former competitor (PSoft), and boosted the number of employees to 260. Adding new offices, for both R&D and marketing, was a considerable challenge. Through extensive internal training in the following year, the new organization was mostly using Evo as the development process.
4 Results This section structures and summarizes the results obtained from the recent study of CSoft. First, we look at how the present product line organization relates to its clients. Then we describe the recent emergence of a community of third party organizations. Together, these results show how CSoft relates to its external environment, constituting a software ecosystem.
4.1 Engaging customers An important aspect of the continuous change over the past years is how CSoft now relate to their customers. The shift from a plan-driven approach to an agile approach has included an increased proximity to the customers. The initial experience from collaborating with customers as external stakeholders in development projects [14, 15] showed positive effects such as better management of requirements, and improved motivation among developers. It also showed that the relationships with the
13/24
stakeholders were fragile, and that it takes continuous and careful management to maintain their motivation to participate. These initial lessons inspired CSoft to further develop and actively exploit close relationships with their customers.
The main motivation for customers to spend time participating in the development projects is the ability to affect development: no payment or any other compensation is provided. One of the product managers explained:
“…they see their wishes or their requirements or whatever in the product at the end. And then you get very nice feedback like ‘I can see that I said this and that, and in the next release you did it’”.
This collaboration forms a self-regulating system where the supplier and the stakeholders mutually adapt to each other through their shared interest in developing the software product line. This usually works well, but there is always a risk of having external stakeholders, which do not provide the necessary input, as explained by the manager of the PSG team:
“Everybody has busy jobs and projects that need to be on time etc. It happens quite often that we have to cancel these meetings or that they haven’t done anything since the last time. Then we can only show them what we’ve done and get some ad-hoc feedback...”
The PSG manager also explained that it is relatively easy to discuss ideas, but that it is more of a challenge when they are included in the development process:
“There’s no problem to get them to discuss high-level plans, but it varies when it comes to the development process”
Maintaining the motivation for participating is an important task for the PSG.
Interestingly, large and leading customers tend to expect and demand to be more and more involved at both planning and development stages. One example is a product conference keynote given by the VP from one of the large customers. He stated several ‘requirements’ (this was the word he used) for being involved, for example:
“Regular meetings with product development teams”, “To work as a stakeholder on new software developments that are key to us.”, “Help to guide product strategy.”, and others.
Finding the “right” stakeholders for participation in the development projects is not done through a formal and structured process, but is mostly based on the collective knowledge about the customers. The PSG manager says,
“We don’t have a formal process for selecting stakeholders. We have internal discussions, listen to sales people etc. We know which customers have asked for certain features or improvements or those that are heavy users of a particular type of functionality.”
In addition, experience from previous participation is also useful as explained by the R&D manager:
“We have become better at selecting [external stakeholders]. Those that have disappointed us are never asked again. You end up with a pool of persons that you know you can trust.”
14/24
An important part of recruiting customers as external stakeholders is to communicate to them clearly the opportunity, which is given to them. The PSG manager explains:
“..quite often we talk about our development process. We do it in sales situations because it tells that our goal is to solve business problems for our customers. This level of interaction and the way we try to listen has been well received, and now some customers insist on being involved in development.”
In the very start when Evo and collaboration with external stakeholders were at an experimental stage it took quite some effort to recruit and stakeholders to the development projects and to keep them active [14]. After some releases where collaboration with external stakeholders have become an integrated part of the development process the situation is turned upside-down. When asked to explain this relationship, one of the product managers told us:
“It's almost a problem because as soon as you offer the capability of being a stakeholder, the hardest part is rejecting people, turning them away from actively participate. So people are very keen on participating.”
Co-creating the product line is one of the most significant effects of engaging and communicating with external stakeholders. The rationale is simple, CSoft have the most up to date knowledge of the technology, and the ability to make use of it in the development. Likewise, customers hold the most up to date knowledge of their own business domain, and how it seems to develop. These two pools of knowledge and competence are joined in several ways. One important arena for sharing and gaining knowledge is the product conference where management, strategists, developers and other internal actors get to meet externals from various customers and third parties. Equally important – customers can meet other customers, third parties can meet customers or internals etc. This shows the networked character of the ecosystem that is shaped around the product line. Some examples from the product conference in 2008:
A former customer of a competing solution was seeking experienced customers to discuss the product line and share experience.
Several providers of third party products and services were having stands at the conference, communicating with both existing and potential customers and developers from CSoft.
Another major event, which is more directly focused on the development of the product line, is the annual Advisory Board meeting. Top management from some of the largest and most demanding clients meet with the PSG and other actors who are involved in the shaping of the product strategy. A PSG member explains that they meet to:
“…discuss high level product strategy and how the demands of their companies and the market are developing.” Bringing together major competitors like this was a daring thing to do according to the PSG manager: “The first time we did this it was a bit exciting – would they discuss issues openly, and would they open up? It turned out that they did very fast. They have many concurrent needs, and even though they are competitors they see the value of doing this.”
From a practical viewpoint, we see that tools and infrastructure for collaboration are important enablers for co-creating the product line. Especially the Webex online
15/24
meeting solution lowers the threshold for having frequent and detailed meetings with stakeholders:
From our observation of one of the customer review meetings we saw a lot of very detailed discussions that were made possible by on-the-fly demonstration of the software through the screen sharing solution. This sparked detailed discussions both on the customer side and among the development team. The meeting resulted in a list of clear actions points to be addressed in the next development iteration.
Close corrective feedback in the Evo development projects is another approach to co-creating the product line, but on a tactical level. One of the developers describes the meeting with the external stakeholder at the end of the two-week Evo iteration:
“What you get during a meeting is often very valuable. Especially when you are about to move in the wrong direction, which you can adjust. We get feedback saying that our solution is not quite what they had in mind or what they need.”
This demonstrates one important function of the agile process; the development teams get nearly immediate (within two weeks) and detailed feedback. This closeness to a few selected customers means that CSoft must also consider the needs of other customers, as they are the referent organization, which always has the last word in the development of the product line. This is partly achieved through Evo’s focus on product qualities instead of product features, which are typically emphasized in plan-driven development methods. This is a useful abstraction, and it turns the focus from predefined design (features) to effect and impact (qualities). Both the product roadmaps and the evaluation meetings at the end of the Evo iterations evaluate the product qualities. This means that both the development teams and the external stakeholders have to consider why something is needed, leaving the how to the developers. The PSG manager explains:
“..we take one step back, and try to think about why our stakeholder needs this, and then rethink other ways of solving their problem. It is in this type of process that the smart things can turn up – that your thinking is totally new and that you come up with a solution which may be a totally different way of doing it, maybe faster...”.
Catching and following up on customer ideas on an ad-hoc basis is equally important as involving customers in regular processes such as roadmapping and the Evo development projects. At the product conference:
A customer representative told about a case where his company gave input to CSoft on some changes they would have liked to see. This led CSoft to invite a delegation from the (abroad) customer to the R&D department in Oslo. Ten CSoft people spent the whole day discussing the solution with four representatives of the customer. This was perceived very positively, and in the end actually affected the software.
Some of the largest customers may also request dedicated workshops to discuss needs and ideas. The PSG manager talks about this:
“…for some of our largest customers, mostly by their initiative, we organize workshops once a year, usually on a strategic level. They want to know what the roadmaps may bring for the next couple of years, and talk a lot about their needs etc.”.
16/24
The close contact with customers is also a valuable source of learning about competing solutions. One of the team leaders talks about customers visiting the R&D department:
“In these meetings they demonstrated the solution they used today, and actually demonstrated how they used the competing solutions – what worked well and what needed improvements, as well as ideas they might have. These meetings gave the team a wealth of details, and it was quite clear what to deliver to the stakeholder.”
The PSG manager also explains the value of learning of the use of competing solutions:
“…alternatively they do it using other tools today when not using our solution. The option to work more closely with them and to get that knowledge made us more capable to meet their needs better than before when the development was more of the black-box type”.
A phrase from one of the roadmaps illustrates the business impact this may have:
“Through a client we have been given a thorough demonstration of the competing solution NN, and by implementing support for [some advanced functionality] and a couple of small features, the X-module will by far exceed their corresponding functionality. These improvements alone will ensure we win one [sales] deal, and have also been brought up by several other clients/prospects.”
Learning the business processes and domain is another valuable outcome from the direct contact with selected stakeholders. One of the developers talks about one of the stakeholder meetings in an Evo project:
“…we have tried to solve a task in a way we believed would be reasonable, but to people who actually use this it is obvious that we have misunderstood the process. This gives us guidance as early as possible.”
A PSG member tells about another case:
“They [the customer] were here in a workshop for two days. We presented the roadmap [for module X] and they presented their wishes and their business, what they are doing.”
This illustrates the shared interest that the customers and the supplier have – customers want to learn about the product line and its development. Correspondingly, CSoft learns about the business that their product line is supporting.
4.2 An emerging third party community At present around 60 external organizations base their business completely or partly on CSoft as a platform. This can be value-adding solutions or products, related services, and consulting. Examples are solutions for data visualization or voice data capture technology, assistance in using various components in the product line, and training. This networked community [24] has not been planned and deliberately established by CSoft, it has emerged spontaneously over the past years. This emergence is mostly driven by customers’ need for additional features and services on one hand, and the opportunity to extend and use the product line as a platform on the other hand. Also, building solutions and providing services based on the product line means that external organizations get immediate access to a large group of established users of the product line.
17/24
Providers of third-party solutions are considered to be important external stakeholders, and are included in the development of the product line in very much the same way as customers.
During a product conference, a representative from a third party company, delivering an integrated product, explained that when they needed to improve the integration with the CSoft platform they took on the role of an Evo stakeholder. Communication was mostly done by phone, supported by web meetings with screen sharing.
Offering an efficient integration technology enables a third party community. Over the past few years a set of simple APIs have been offered to enable external actors to make extensions to the product line. The development of these APIs have followed the development of the product line, where each new release has improved existing and offered new APIs due to requests from external actors. This means that there is a long (a year) connection time between a request for an interface and its actual release. As more and more externals have made use of this connection point to the software it has been given increasingly higher priority in the development of the product line. An excerpt from one of the roadmaps exemplifies this:
“We are in dialogue with some clients/prospects who are building their portal in a Content Management System, and need to integrate content from [module X] into it. Some competitors seem to have APIs that are easier to use than our SOAP1 based APIs, making it easier to integrate with other portals/communities. It is therefore an ambition to provide an easier API for including module X content into an external portal.”
Due to the extensive use of the API’s by externals and their increasing demand for integration with the product line it became clear that the simple web-service based interface had become obsolete. This has led CSoft to develop and offer a new API called FlexibilityFramework (FF), which enables a closer integration to core services in the product line than the previous (and still existing) simple messaging-based APIs offer. A recent webcast, where the CSO presents FF explains further the motivation for this improved interface:
“CSoft is like a supertanker. It is large, can take huge loads, travel far, and take heavy weather. These are all very positive things, on the other hand, the consequence of that approach is that we are quite careful at looking after the supertanker. That means various procedures, on policy, on quality assurance and so forth. And that means that we get less nimble than we would like. The question we posed ourselves is how can we behave like a speedboat while having all the benefits of the supertanker? I’d like you to think of FF as the speedboat. The tanker is still there. It will still take heavy loads and perform extremely well, but in order to be nimble we can build a few speedboats. And they have independent lives from the supertanker and can run on different development schedules.”
The last argument is worth a comment; with this new interface to the product line external actors are disconnected from the long release cycles of the product line, and can develop value-adding solutions independently. This is likely to further drive the growth of the third party community.
1 Simple Object Access Protocol, http://www.w3.org/TR/soap12-part1/
18/24
Actively supporting the community has become a regular activity in addition to the continuous development of the product line. As this community has emerged and grown, CSoft have seen its value, and started to actively support it. In 2007 a dedicated web-portal was launched to make this community visible and each partner is listed and presented. There are five types of partners, those offering technology that is integrated with the product line, those offering value adding services, some can prepare the use of the product line, some can use it on behalf of clients and some offer consultancy services.
5 Discussion We have now described the CSoft case, emphasizing the present organizational set-up, its processes and the product line, the development timeline of the organization (section 3.5.2), and how the present organization relates to its external environment (section 4). Using this insight we now seek to provide answers to our research question: Why and how is software product line engineering developing towards a software ecosystem?
First of all, the initial motivation for changing the waterfall-like development process by adopting Evo in 2004/2005 was that CSoft struggled with unstable requirements incurring high costs due to little flexibility in the process. Much emphasis was given to extensive and thorough requirements engineering upfront, but with little effect [15]. The immediate experience from involving stakeholders in the short Evo development iterations was that developers felt more comfortable and secure by having this close and continuous dialogue on requirements and results [ibid]. However, in the first release projects using Evo, it became a considerable challenge to maintain the motivation of the external stakeholders throughout the project. The new process was fragile [14].
(Change 1) From the recent study we see that this has clearly changed; now external stakeholders are keen to participate – CSoft actually have to turn down candidates. This change is the result of a learning process that has progressed during the first years of using Evo – customers have gotten to know of this practice and some have gained experience as stakeholders. The engagement of customers and users is generally considered to be an important success factor in any kind of software development [25, 26].
(Change 2) We can also observe another change that took place internally at CSoft. The first experimentation with Evo was only done as an R&D-internal matter, like a kitchen experiment. However, as this turned out to be an improvement of the development practice, this way of working eventually became adopted in the rest of the organization. Now, all parts of the organization, from operational support to the top management, are supporting this practice. An example is the CEO explaining the software development process Evo and its strategic importance in his keynote at a large product conference. Another example is the strengthening of the PSG, which has a liaison function between customers and development teams. This tells us that changing a product line organization takes effort and time, and that both internal and external actors need to learn from practice to accept this opening of the organization and its work processes.
19/24
(Change 3) Another change we can see from the results is an increasingly higher external visibility of plans and strategies. Initially this kind of information was kept internal, but it is now more and more openly communicated through various channels. It has turned out that doing this does not introduce the presumed risk of leaking vital information to competitors, but that it is rather an advantage as external actors see what might be coming, they can relate it to their own business, and potentially respond to it.
(Change 4) Another related change is the opening of the product line at the technical level, first with the SOAP-based APIs and now the recent and more efficient Flexibility Framework. Initially this represented a minimal and very limited opportunity for extending the product line, but it quickly grew to a considerable extent as it represented tangible business value. This aspect has eventually been given more attention, and has been designated as strategically important in some of the roadmaps. We see several benefits from allowing externals to use the product line as a platform. Firstly, it increases the variability of the product line – it can be used in more specialized ways, serving more needs. Secondly, existing users represent a great opportunity to the third parties (being the second component of a symbiosis-like relationship). Thirdly, letting externals deal with specialization and minor extensions enables the product line organization itself to maintain focus on developing the core product line. This may be the most important effect [27].
To recap the research question - this change and the organization it has resulted in explains why and how software product line engineering at CSoft has developed towards a software ecosystem.
Why 1) Customers expect and have learned to value to be involved in development and in product strategy making. 2) A plan-based development approach is unfit when serving a volatile domain where the product line is under continuous and extensive development. 3) The total demands and requirements from customers can become too high for one product line organization to manage alone.
How 1) CSoft learns about the business it serves through active collaboration with customers and third parties. 2) CSoft makes strategy and plans visible externally. 3) The technical interface of the product line is opened. 4) Both customers and value-adding third parties are considered as external stakeholders. 5) CSoft actively support and assist the community of third parties.
5.1 Implications for theory Software ecosystems, as a concept, have the potential of becoming an important field of practice and research in the years to come. We propose to shape a theoretical platform for this research. Just like the taxonomy suggested by Bosch [1], a theory of software ecosystems is valuable and useful to generalize the concept and bring together results from more empirical studies. It may over time develop towards a unified and empirically justified understanding of the concept.
Fortunately, the theory of organizational ecology [6], briefly presented in the background section, seems to fit well as a starting point. It concerns organizations operating in complex and unstable domains, in principle a suitable description of software ecosystems – and certainly of CSoft. Using this general theory of
20/24
organizational ecology, we derive a set of theoretical propositions suitable to software ecosystems:
1. Member organizations in a software ecosystem are linked to a key organization among them, which acts as a central referent organization, doing so even though many of them are only partially under its control or linked to it only through interface relations. (This is the Class 1 system according to Trist’s classification). CSoft is an example of such a referent organization. None of the external organizations are formally controlled by CSoft. However, all activity in the ecosystem is related to the product line, which is controlled by CSoft.
2. Software ecosystems promote self-regulation. Our study of CSoft show that the collaborative approach can be seen as a self-regulating system in that the referent organization to a large degree adapts to its external environment, and that the external environment adapts to the referent organization. This is in contrast to the previously centralized control that was applied in the development of the product line.
3. Software ecosystems have a networked character. CSoft’s and its external environment constitutes a network of customers and third party organizations. Even competitors may be considered a part of this network, although this aspect has not been studied in particular here.
4. Software ecosystems exist through the use of technology. The ecosystem, which CSoft is a part of, relies on the use of technology to enable collaboration. Examples are web-meetings, web-casting, and the software-as-a-service deployment model.
5. Software ecosystems have shared values. In the CSoft ecosystem the software (product line) is this shared value. For CSoft, the value is revenue from licenses and services, for the customers the value is improved business operations, and for the third parties the value is revenue from sales of value-adding solutions. This common interest in the shared value creates motivation to collaboratively care for the shared value.
These five propositions constitute a start of a theory for software ecosystems. In addition to these principles adopted from Trist’s work [ibid.] we also propose two extensions:
6. The shared value of a software ecosystem is both the software product and the business domain. Through the increased proximity to the external environment, CSoft have an interest in both the product line and the business it serves. Likewise, customers have an interest both in their business and the technology they use to drive it.
7. As a software ecosystem emerges, control moves from the supplier of the software to its users. An opening of the product line process, with external stakeholders participating in development and with external visibility of plans and strategies, means that some of the control move towards the users. Users in this case are both customers and third-parties. This affects the motivation to collaborate, and is of benefit to all members of the ecosystem.
21/24
5.2 Implications for practice From the analysis of our findings we derive a set of implications for practice, relevant to other software product line organizations similar to CSoft.
• Support the external environment by sharing information on plans and strategies – this opens a channel for valuable input and enables collaboration with externals.
• If appropriate, encourage and support a third party community; it can be a valuable extension to the normal development of the product line.
• Establishing and benefiting from a software ecosystem takes time. A successful development relies on repeated cycles of experimentation and learning. This learning process needs to involve all types of actors.
Based on our findings we can derive a strategy that can be of practical value to other product line organizations. The shared interest in the product line (the shared value) is a key enabler for driving the collaboration between the actors in the ecosystem. Thus, a viable strategy would be to 1) make the product line supplier more involved in the development of the business domain and 2) make external actors more engaged in the development of the technology.
22/24
5.3 Limitations The case study of CSoft is subject to three limitations.
Firstly, this is a single case study, which naturally affects the generalizability of the conclusions. Yet there are good reasons for selecting such an approach. First of all, the number of relevant cases is still low. In addition, focusing on a single case means that the study can be more thorough than a study of multiple cases, with respect to available resources. Yin discusses the single case study design [28] (p. 38-41) and presents several arguments in favor of choosing such a design. One of these is particularly applicable to CSoft, namely that it is a unique case, at least for the researcher who conducted the study. According to Yin, such a study may act as a prelude to further studies of a relatively new topic, such as software ecosystems in this case.
The second limitation concerns the completeness of the study. Only a subset of the employees was contacted. Likewise, relatively few samples of all available documentation were collected and analyzed. This is naturally due to the relatively long duration of the study.
The third limitation concerns the applicability of the findings and conclusions of this study. The organization investigated is a medium-size product line organization and a web/application type of ecosystem (according to the taxonomy proposed by Bosch [1]). Thus, results do not necessarily apply to all other types of software ecosystems.
6 Conclusions Over a period of approximately five years we have studied a software product line organization and its external environment, showing and explaining an emerging software ecosystem.
We conclude that the development that has occurred has produced effects both internally in the product line organization and in its external environment. The change has led to an increase in collaboration across (previously closed) organizational borders, and it has developed a shared value consisting of two components: the technology (the product line) and the business it supports. Opening up both the technical interface of the product and the organizational interfaces are key enablers of such a change
We propose two directions for further research. First, like Jansen et al. also point out [9], we need to see more empirical studies of various types of software ecosystems, how they develop and what effects they produce. Secondly, more studies should contribute to the shaping of a theory of software ecosystems.
7 Acknowledgements We are grateful to the people at CSoft for sharing their precious time, and providing highly valuable input to this study. This work was done as a part of the EVISOFT project (grant 156701/220) and the Agile project (grant 179851/I40), both partly funded by the Research Council of Norway.
23/24
8 References 1. Bosch, J., From Software Product Lines to Software Ecosystems, in 13th
International Software Product Line Conference (SPLC'09). 2009, IEEE Computer Society: San Fransisco, USA.
2. Messerschmitt, D.G. and C. Szyperski, Software Ecosystems, Understanding an Indespensable Technology and Industry. 2003, Cambridge, Massachusetts, USA: The MIT Press. 424.
3. Jansen, S., et al., Introduction to the Proceedings of the First Workshop on Software Ecosystems, in First International Workshop on Software Ecosystems. 2009, CEUR-‐WS.
4. Qualman, E., Socialnomics: how social media transforms the way we live and do business. 2009: John Wiley & Sons.
5. Emery, F.E. and E.L. Trist, The Causal Texture of Organizational Environments. Human Relations, 1965. 18: p. 21-‐32.
6. Trist, E.L., A Concept of Organizational Ecology. Australian Journal of Management, 1977. 2(2): p. 161-‐175.
7. Trist, E.L. and K.W. Bamforth, Some social and psychological consequences of the longwall method of coal-getting. Human Relations, 1951. 4(1): p. 3-‐38.
8. Jansen, S., A. Finkelstein, and S. Brinkkemper. A sense of community: A research agenda for software ecosystems. in 31st International Conference on Software Engineering (ICSE'09). 2009. Vancouver, Canada: IEEE Computer Society.
9. Jansen, S., S. Brinkkemper, and A. Finkelstein. Business Network Management as a Survival Strategy: A Tale of Two Software Ecosystems. in First Workshop on Software Ecosystems. 2009.
10. Alspaugh, T.A., U.A. Hazeline, and W. Scacchi, The Role of Software Licenses in Open Architecture Ecosystems, in First International Workshop on Software Ecosystems, S. Jansen, et al., Editors. 2009, CEUR-‐WS.
11. Chesbrough, H., Open Innovation: A New Paradigm for Understanding Industrial Innovation, in Open Innovation: Researching a New Paradigm, H. Chesbrough, W. Vanhaverbeke, and J. West, Editors. 2006, Oxford University Press: Oxford. p. 1.
12. Enkel, E., O. Gassman, and H. Chesbrough, Open R&D and open innovation: exploring the phenomenon. R&D Management, 2009. 39(4): p. 311-‐316.
13. Hanssen, G.K. and T.E. Fægri, Process Fusion - Agile Product Line Engineering: an Industrial Case Study. Journal of Systems and Software, 2008. 81: p. 843-‐854.
14. Fægri, T.E. and G.K. Hanssen, Collaboration and process fragility in evolutionarily product development. IEEE Software, 2007. 24(3): p. 96-‐104.
15. Hanssen, G.K. and T.E. Fægri. Agile Customer Engagement: a Longitudinal Qualitative Case Study. in 5th International Symposium on Empirical Software Engineering (ISESE'06). 2006. Rio de Janeiro, Brazil: IEEE Computer Society.
16. Hanssen, G.K., et al. Software entropy in agile product evolution. in 43d Hawaiian International Conference on System Sciences (HICSS'10). 2010. Hawaii, USA: IEEE Computer Society.
24/24
17. Johansen, T. Using evolutionary project management (Evo) to create faster, more userfriendly and more productive software. Experience report from FIRM AS. in 6th International Conference on Product Focused Software Process Improvement (PROFES'05). 2005. Oulu, Finland: Springer Verlag.
18. Moe, N.B., et al., Process guides as software process improvement in a small company, in EuroSPI. 2002: Nuremberg, Germany.
19. Klein, H.K. and M.D. Myers, A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Quarterly, 1999. 23(1): p. 67 -‐ 93.
20. Glaser, B.G. and A.L. Strauss, The Discovery of Grounded Theory: Strategies for Qualitative Research. 1967, New York: Aldine Transaction. 271.
21. Gilb, T., Competitive Engineering: A handbook for systems engineering, requirements engineering, and software engineering using Planguage. 2005: Elsevier Butterworth-‐Heinemann. 480 pages.
22. Pettigrew, A.M., Longitudinal Field Research on Change: Theory and Practice Organization Science, 1990. 1(3): p. 267-‐292.
23. Royce, W.W. Managing the development of large software systems. in IEEE WESCON. 1970.
24. Fricker, S., Specification and Analysis of Requirements Negotiation Strategy in Software Ecosystems, in First International Workshop on Software Ecosystems, S. Jansen, et al., Editors. 2009, CEUR-‐WS.
25. Chiasson, M.W. and L.W. Green, Questioning the IT artefact: user practices that can, could, and cannot be supported in packaged-software designs. European Journal of Information Systems, 2007(16): p. 542-‐554.
26. Keil, M. and E. Carmel, Customer-Developer Links in Software Development. Communications of the ACM, 1995. 38(5): p. 33-‐44.
27. Zook, C., Profit from the core: a return to growth in turbulent times. 2010: Bain & Company, Inc.
28. Yin, R. and D.T. Campbell, Case Study Research. 2002: Sage Publications Inc.
Geir Kjetil Hanssen works as a researcher on software process improvement and methodologies at SINTEF ICT, Norway’s largest independent research institution. He has a M.Sc. degree in informatics from the University of Trondheim.
Declarations on co-author consensus Papers P1, P2 and P3: Tor Erlend Fægri Paper P4: Aiko Fallas Yamashita, Reidar Conradi and Leon Moonen
All publications
Hanssen, G. K., Yamashita, A. F., Conradi, R. and Moonen, L. (2010). Software Entropy in Agile Product Evolution. In proceedings of 43d Hawaiian International Conference on System Sciences (HICSS'10), Hawaii, USA, 4-7 January. 1-10. Hanssen, G. K. (2010), Agile Software Product Line Engineering: Enabling Factors, accepted (minor revision) for publication in Software: Practice and Experience (Wiley). Hanssen, G. K. (2010). Opening up Software Product Line Engineering. In proceedings of 1st International Workshop on Product Line Approaches in Software Engineering, in conjunction with the 32'nd International Conference on Software Engineering (ICSE), Cape Town, 2 May. 1-7. Hanssen, G. K., Yamshita, A. F., Conradi, R. and Moonen, L. (2009). Maintenance and agile development: challenges, opportunities and future directions. In proceedings of 25th International Conference on Software Maintenance (ICSM'09), Edmonton, Canada, 20-24 September. 487-490. Hanssen, G. K. and Haugset, B. (2009). Automated Acceptance Testing Using Fit. In proceedings of 42d Hawaiian International Conference on System Sciences (HICSS'09), Hawaii, USA, 5-8 January. 1-8. Stålhane, T. and Hanssen, G. K. (2008). The application of ISO 9001 to agile software development. In proceedings of Product Focused Software Process Improvement (PROFES 2008), Frascati, Italy, 23-25 June. 371-385. Haugset, B. and Hanssen, G. K. (2008). Automated Acceptance Testing: a Literature Review and an Industrial Case Study. In proceedings of Agile Conference, Toronto, Canada, 4-8 August. 27-38. Hanssen, G. K. and Haugset, B. (2008). Automated Acceptance Testing Using Fit. In proceedings of EuroSPI 2008, Dublin, Ireland, 3-5 September. Hanssen, G. K. and Fægri, T. E. (2008). Process Fusion - Agile Product Line Engineering: an Industrial Case Study. Journal of Systems and Software 81: 843-854. Hanssen, G., Bjørnson, F. and Westerheim, H. (2007). Tailoring and Introduction of the Rational Unified Process. In proceedings of European Systems & Software Process
Improvement and Innovation (EuroSPI 2007), Potsdam, Germany, 26-28 September. 7-18. Fægri, T. E. and Hanssen, G. K. (2007). Collaboration and Process Fragility in Evolutionarily Product Development. IEEE Software 24(3): 96-104. Dybå, T., Dingsoyr, T. and Hanssen, G. K. (2007). Applying Systematic Reviews to Diverse Study Types: An Experience Report. In proceedings of Proceedings of International Symposium on Empirical Software Engineering and Measurement (ESEM), Madrid, Spain, 20-21 September. 10. Westerheim, H. and Hanssen, G. K. (2006). Extending the Rational Unified Process with a User Experience Discipline: a Case Study. In proceedings of European Systems & Software Process Improvement and Innovation (EuroSPI 2006), Joensuu, Finland, 11-13 October. 3.11-13.19. Hanssen, G. K. and Fægri, T. E. (2006). Agile Customer Engagement: a Longitudinal Qualitative Case Study. In proceedings of 5th International Symposium on Empirical Software Engineering (ISESE'06), Rio de Janeiro, Brazil, 21-22 September. 164-173 Dingsøyr, T., Hanssen, G. K., Dybå, T., Anker, G. and Nygaard, J. O. (2006). Developing Software with Scrum in a Small Cross-Organizational Project. In proceedings of European Systems & Software Process Improvement and Innovation (EuroSPI 2006), Joensuu, Finland, 11-13 October. 2-12. Westerheim, H., Hanssen, G. K. (2005). The Introduction and Use of a Tailored Unified Process - A Case Study. In proceedings of 31st EUROMICRO Conference on Software Engineering and Advanced Applications (Euromicro 2005), Porto, Portugal, 31 August - 2 September. 196-205. Hanssen, G. K., Westerheim, H., Bjørnson, F. O. (2005). Using Rational Unified Process in an SME - A Case Study. In proceedings of European Systems & Software Process Improvement and Innovation (EuroSPI 2005), Budapest, Hungary, 9-11 November. 142-150. Hanssen, G. K., Westerheim, H., Bjørnson, F. O. (2005). Tailoring RUP to a defined project type: A case study. In proceedings of Product Focused Software Process Improvement (PROFES 2005), Oulo, Finland, 13-15 June. 314-327.
Hanssen, G. K., Dingsøyr, T. (2003). A Comparison of Automated and Manual Functional Testing of a Web-Application. In proceedings of European Systems & Software Process Improvement and Innovation (EuroSPI 2003), Graz, AUstria, 10-12 February. 1-10. Hanssen, G. K., Westerheim, H. (2003). Extending Lightweight Postmortem Analyses for Use in Software Process Improvement. In proceedings of 2nd Workshop in Workshop Series on Empirical Software Engineering, Rome. Westerheim, H., Dingsøyr, T. and Hanssen, G. K. (2002). Studying the User Experience Discipline extension of the Rational Unified Process and its effects on Usability - The design of a case study. In proceedings of First International Workshop on Empirical Studies in Software Engineering, Rovaniemi, Finland. 69 - 74. Hanssen, G. K., Dybå, T., Stålhane, T., Westerheim, H. (2002). SPI - easy in theory, hard in practice. In proceedings of European Systems & Software Process Improvement and Innovation (EuroSPI 2002), Nuremberg, Germany. 327-336. Dingsøyr, T., Hanssen, G. K. (2002). Extending Agile Methods: Postmortem Reviews as Extended Feedback. In proceedings of Workshop on Learning Software Organizations (LSO 2002), Chicago, 6 August. 4-12. Stålhane, T., Hanssen, G. K., Westerheim, H. (2002). Improving the Software Estimation Process. In proceedings of Quality Week Europe, Brussels, Belgium, 11-15 March. Stålhane, T., Dingsøyr, T., Hanssen, G. K. (2001). Post Mortem - An Assessment of Two Approaches. In proceedings of European Systems & Software Process Improvement and Innovation (EuroSPI 2001), Limmerick, Ireland, 10-12 October. 129-141.