isern open issues, grand challenges or have we made any progress and where are going? vic basili
DESCRIPTION
ISERN Open Issues, Grand Challenges or Have we made any progress and where are going? Vic Basili. 2001. ISERN Areas of Contribution and Influence. What have been our contributions to Software Engineering: Experimentation Design and Implementation Replication Research Knowledge - PowerPoint PPT PresentationTRANSCRIPT
ISERNOpen Issues, Grand Challenges
orHave we made any progress and where are going?
Vic Basili
2001
ISERN Areas of Contribution and Influence
What have been our contributions to Software Engineering:
ExperimentationDesign and ImplementationReplication
Research KnowledgeLevels of UnderstandingDepth of Hypotheses and Questions
Software Engineering PracticeDomains of KnowledgePracticality of Information
And what has been our influence on:the ISERN communitySE researchSE Practice
An example: Reading Experiments
Before ISERN (1986): Code Testing vs. Reading Experiment (UMD)…
First ISERN 1992: First discussion of the requirements reading (DBR)(UMD)
ICSE 1996: Basili presented a talk on the state of experimentation and discussed reading experiments to that point in time.
Set some goals for the maturing of an experimental discipline
The next 6 slides come from that talk
The Maturing of the Experimental Discipline
How is experimentation maturing?Sign of maturity in a field:
level of sophistication of the goals of an experimentunderstanding interesting things about the discipline
For software engineering that might mean:
Can we build models that allow use to measure and differentiate processes and products?
Can we measure the effect of a change in a particular process variable on the product variable?
Can we predict the characteristics of a product (values of product variable) based upon the model of the process (values of the process variables), within a particular context?
Can we control for product effects, based upon goals, given a particular set of context variables?
The Maturing of the Experimental Discipline
How is experimentation maturing?
Sign of maturity in a field: a pattern of knowledge built from a series of experiments
Does the discipline build on prior (knowledge, models, experiments).
Was the study an isolated event?
Did it lead to other studies that made use of this information Have studies been replicated under similar or differing conditions?
Does the building of knowledge exist in one research group or environment, or has it spread to others - researchers building on each other's experimental work?
For example, inspections, in general, are well studied experimentally
However, there has been very little combining of results, replication, analysis of the differentiating variables
The Maturing of the Experimental Discipline
How is experimentation maturing?There is some evidence that researchers appear to be
- asking more sophisticated questions- studying relationships between processes/product characteristics - using more studies in the field than in the controlled laboratory
- combining various experimental classes to build knowledge
On such example of evolving knowledge over time, - based upon experimentation and learning is - the evolution of the SEL knowledge - of the effectiveness of reading techniques and methods
Software Engineering Laboratory is a consortium (established in 1975)
- NASA/Goddard Space Flight Center - University of Maryland - Computer Sciences Corporation
Evolution of Knowledge over Time Reading Technology Experiments
This example - shows the combination of multiple experimental designs - provides insight into the effects of different variables on reading - demonstrates replication by other researchers
The experiments start with the early reading vs. testing experiments to various Cleanroom experiments to the development of new reading techniques currently under study to replications at other groups
The experiments are based upon the ideas that Reading is a key technical activity for improving the analysis of all kinds of software documents and we need to better understand its effect
Early experiments (Hetzel, Meyers) showed very little difference between reading and testing
But reading was simply reading, without a technological base
The Maturing of the Experimental DisciplineHow is experimentation maturing?
Several of these experiments have been replicated - under the same and differing contexts
The original analysis technique comparison has been replicated University of Kaiserslautern
Scenario-based reading study variations University of Bari, Italy University of New South Wales, Australia Bell Laboratories, USA University of Trondheim, Norway Bosch, Germanyto better understand the reading variable
ISERN organized explicitly to share knowledge and experiments has membership in the U.S., Europe, Asia, and Australia represents both industry and academia supports the publication of artifacts and laboratory manuals
Its goal is to evolve software engineering knowledge over time, based upon experimentation and learning
What will our future look like?
Experimentation can provide us with - better scientific and engineering basis for the software engineering - better models of - software processes and products - various environmental factors, e.g. the people, the organization - better understanding of the interactions of these models
Practitioners will be provided with - the ability to control and manipulate project solutions - based upon the environment and goals set for the project - knowledge based upon empirical and experimental evidence - of what works and does not work and under what conditions
Researchers will be provided laboratories for experimentation This will require a research plan that will take place over many years - coordinating experiments - evolving with new knowledge
Continuing: Reading Experiments
Since then: new variations and replications
Code Reading vs. TestingUMD (USA), UKL (Germany), U Strathclyde (Scotland), U Sao Paolo (Brazil),
Defect Based Reading UMD, Lucent, U Bari (Italy),
Perspective Based ReadingUMD, IESE, FC-MD, U Bari, UMBC (USA), U Sao Paulo, NTNU (Norway), USC (USA), COPPE/UFRJ (Brazil), Lund U (Sweden)
Object Oriented ReadingUMD, NTNU, U Rio deJaniero, USC, IESE
Other ReadingUMD, UNSW (Australia), U Strathclyde, TU Vienna (Austria), IESE
Reading Experiments: Contribution and Influence
ExperimentationDesign and ImplementationSeveral designs form full fractional factorial to constant comparison approachesAre learning the limits of controlled experimentsReplicationSeveral at universities, often with support fro original experimentersCombining ResultsIdentifying common factors (Goal Template)
Need to build a stronger body of experimental designbe able to combine the results of experiments of all kindsmake “controlled” type experiments cheaper
And what has been our influence on:the ISERN communitywork on the combined experiment for inspectionsSE researchtwo books on experimentation from ISERN participantsSE Practicenot much so far but some nation wide programs have begun, e.g., CeBASE, ViSEK, …
Reading Experiments: Contribution and Influence
Research KnowledgeLevels of Understandingevolution of process based upon empirical studyuse of experiments as learning rather than validationDepth of Hypotheses and Questionsasking questions about meta issues, such as context variables
Questions from ICSE:Can we build models that allow use to measure and differentiate processes and
products? Yes
Can we measure the effect of a change in a particular process variable on the product variable? Yes
Can we predict the characteristics of a product (values of product variable) based upon the model of the process (values of the process variables), within a particular context? Some, but not really
Reading Experiments: Contribution and Influence
Research KnowledgeLevels of UnderstandingDepth of Hypotheses and Questions
Questions from ICSE:Does the discipline build on prior (knowledge, models, experiments)?
More than before
Was the study an isolated event? Few isolated events by ISERN personnel
Did it lead to other studies that made use of this information? YesHave studies been replicated under similar or differing conditions? Yes
Does the building of knowledge exist in one research group or environment, or has it spread to others - researchers building on each other's experimental work? More building together
Reading Experiments: Contribution and Influence
Research KnowledgeLevels of UnderstandingDepth of Hypotheses and Questions
And what has been our influence on:the ISERN community
More working together, sharing experimentsSE research
More experimentalists joining ISERNSE Practice
Some interest in national projects like CeBASE, ViSEK, ESERNET,..
What results do we have so far? For example …
Combining Results from multiple Reading Experiments
We have generated useful empirical results for technique definition guidance
For a reviewer with an average experience level, a procedural approach to defect detection is more effective than a less procedural one.
Procedural inspections, based upon specific goals, will find defects related to those goals, so inspections can be customized.
A set of readers of a software artifact are more effective in uncovering defects when each uses a different and specific focus.
Reading Experiments: Contribution and Influence
Software Engineering PracticeDomains of Knowledge
have been primarily limited to inspection areahave opened up to areas like COTS have attracted empirical modelers which include areas such as cost, quality, and schedule estimation
Practicality of Informationnational experience base activities are trying to reach out to the development communitynot too many members other than researchers
And what has been our influence on:the ISERN communitySE researchSE Practice
What can be packaged for practice, research, education?
Examples of Using Empirical Results for development, research, education
Technique TailoringIs tailoring the reading process associated with an inspection worth the effort?
Procedural inspections, based upon specific goals, will find defects related to those goals, so inspections can be customized. (UMD)
Implications for empirically based software development process: The better you can articulate your goals, the more effectively you can choose and tailor process.
Implications for software engineering research:It is important to empirically study the effects of processes on product
Implications for software engineering education:Don’t teach that there is a one size fits all process; teach how to tailor processes
Examples of Using Empirical Results for development, research, education
Technique Selection GuidanceWhen should you use a procedural approach to code reviewing? For a reviewer with an average experience level, a procedural approach to defect detection is
more effective than a less procedural one (UMD, USC)
Implications for empirically based software development process: Experts might be more effective working on their own but most people should apply a procedural
approach. Novices need training.
Implications for software engineering research:How can we improve document reading procedures based upon how experts analyze documents?
Implications for software engineering education:Effective procedures that can be taught for reviewing documents
Examples of Using Empirical Results for development, research, education
Technique Selection GuidanceWhen are peer reviews more effective than functional testing? Peer reviews are more effective than functional testing for faults of omission and incorrect specification
(UMD, USC)
Implications for empirically based software development process: If, for a given project set, there is an expectation of a larger number of faults of omission or incorrect facts
than use peer reviews.
Implications for software engineering research:How can peer reviews be improved with better reading techniques for faults of omission and incorrect fact?
Implications for software engineering education:Teach how to experiment with and choose the appropriate analytic techniques
ISERN Areas Challenges
ExperimentationHow do we define more effective and efficient experimental techniques?How do we combine results from experiments?How do we deal with (define, study) meta data? How doe we package lab manuals, experience bases?
Research KnowledgeHow do we define the next levels of research questions?What other processes should we study? Other analytic (testing), creative (design)? What product properties should we be studying? Mostly have looked at the relationship between process and product characteristics. What else should we be studying?In our grand inspection experiment, should we be studying ourselves?How do we get more researchers interested in empirically studying their techniques?
Software Engineering PracticeHow do we ,package our results for practice?How do we get more industrial members involved?
Example of Specific Experimental Issues
In looking at the relationship between process and product, there are many questions that need to be covered:
Process variable (Independent variable) issues:How do we define/specify the process? How do we account for process conformance?
Effectiveness of Product (Dependent variable) issues:How do we select good criteria for effectiveness?
Context Variables (meta data) Issues:What subjects are performing the process? What characterizes their experience, environment,…?
Questions associated with the variables need to be further specified and documented for replication
Varying the values of these variables allow us to vary the detailed hypothesessupport validity of study results