04.04.2007anna ruokonen/tut open source and empirical studies an empirical study of open-source and...

04.04.2007 Anna Ruokonen/TUT

Open source and empirical studies

An Empirical Study of Open-Source and Closed-Source Software Products

by Paulson, J.W.; Succi, G.; Eberlein, A.


Empirical studies (1)

• Empirical research

– Empirical observations or experiments are run in order to answer particular research questions

• A research design: the logical sequence connecting the empirical data to initial research questions and conclusions

1. The research questions (who, what, why, how, where)

2. Propositions (->what should be studied)

3. Unit(s) of analysis

4. The logic linking the data to the propositions

5. The criteria for interpreting the findings[Jin2003]


Empirical studies (2)

Construct validity

1. Select specific types of changes that are to be studied2. Demonstrate that selected measures of the changes do reflect the

selected types of changes

Internal validity

• Address rival explanations• Are the results due to circumstances only (the null hypotheses)?

[Jin2003]


The Beliefs

• The 5 common beliefs of OS: – Defects are found and fixed more rapidly in OS projects

– Hypothesis of more prevalent creativity, faster project growth, modularity and simplicity

• The beliefs often lack empirical evidence

• Switch from CS to OS an attempt to win market share, expand products growth, improve market penetration

• This study investigates claims of OS SW community

• Chosen metrics compare OS and CS project characteristics


Project characteristics

Some factors affecting a software growth (OS or CS):

• the application domain

• the SW development process

• the type of product

• the management

• marketing dynamics

• organizational dynamics


Analyzed projects

The most well-known and succesful OS projects

• Linux

• Apache

• GCC, GNU Compailer Collection

Three CS projects

• Representative of particular CS domain

• SW protocol stacks of wireless telecommunications products


GROWTH

HYPOTHESIS: OS development fosters faster system growth METRICS: • Overall project growth in functions over time• Overall project growth in LOC over time

FINDINGS:• Growth rates over time were similar and constant for all analyzed

projects• Does not support hypothesis


Linear approximation of total project size


SIMPLICITY

HYPOTHESIS: OS projects succeed because of their simplicity

METRICS:

• Overall project complexity

• Average complexity of all the functions

• Average complexity of the functions added

FINDINGS:

• All 3 metrics showed that OS projects are more complex

• Possible reasons: embedded SW, personal style, language

• Does not support hypothesis


Complexity


DEFECTS

HYPOTHESIS: OS projects have fewer defects as they are found and fixed more rapidly

METRICS: • Functions modified over time

• Modified functions as percentage of total functions

FINDINGS:

• In OS more function was initially changed

• In OS more defects are found and the product becomes more stable over time

• OS showed significantly higher percentage of modified functions

• Supports hypothesis


Functions modified


CREATIVITY

HYPOTHESIS: OS foster more creativity

METRICS:

• Functions added over time

FINDINGS:

• No commercial project showed any positive growth rate over time

• The growing rate of OS projects was greater than the CS projects

• Supports hypothesis


Growing rate over time


MODULARITY

HYPOTHESIS: OS projects are more modular

METRICS:

• Correlation between functions added and functions modified

FINDINGS:

• There was a strong corrrelation between the growing rate and changing rate in OS projects

• No correlation in CS projects

• Does not support hypothesis


Corrrelation between the growing rate and changing rate


Summary

• OS foster more creativity

• OS projects have fewer defects as they are found and fixed more rapidly

• External global factors and feedback mechanisms have a greater impact on the SW evolution than the development metod

• What are the global factors shared by both OS and CS projects?

• Before move to use OS development method, all of the goals to be achieved should carefully considered

• Ensure that proper metrics are collected and monitored


References

• [Jin2003] Case Study Research: Design and Methods Third Edition (Applied Social Research Methods): Robert K. Yin. 2003, Sage Publications.

04.04.2007anna ruokonen/tut open source and empirical studies an empirical study of open-source and...

Documents