ginger2: an environment for computer- aided empirical ...€¦ · aided empirical software...

Ginger2: An Environment for Computer-Aided Empirical Software Engineering

Ginger2: An Environment for Computer-Aided Empirical Software Engineering

Koji Torii, Ken-ichi Masumoto, Kumiyo Nakakoji, Yoshihiro Takada, Shingo Takada, Kazuyuki Shima

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING 1999

20062006--0808--1616Kim, SuKim, Su--hyunhyun

Software Engineering Lab, KAIST 2/22

ContentsContents

IntroductionBackgroundCAESE FrameworkGINGER2 System- Data Collection- Data & Control Integration- Data Analysis

Case StudyConclusion & Future WorkDiscussion


Empirical software engineering

The study of software related artifacts for the purpose of characterization, understanding, evaluation, prediction, control, management, or improvement through qualitative or quantitative analysis [Harrison and Basili]

A series of actions to obtain knowledge and better understanding about some aspects of software development given a set of problem statements in the form of issues, questions or hypotheses.[Koji Torii et al.]

Introduction (1/2)Introduction (1/2)

The study of software related artifactsfor the purpose of characterization, understanding, evaluation, prediction, control, management, or improvementthrough qualitative or quantitative analysis [Harrison and Basili]

In vivo (naturalistic)

A series of actions to obtain knowledge and better understanding about some aspects of software development given a set of problem statements in the form of issues, questions or hypotheses.[Koji Torii et al.]

In vitro (laboratory)

Actual software development project Consist of controlled Experiments

* These two approaches complement each other

Relevance of real, ill-defined problems that emerge in actual S/W development

Develop and test hypothesis about particular aspect of human behavior

Develop generalized model of behaviorProposed techniques and methods

Make “knowledge” available and reusable for other projects and members


Computer-Aided Empirical Software Engineering (CAESE)- Supporting in vitro study

Involving in empirical activitiesDealing with many types of data

- FeatureEmpirical study process modelData collection modelEmpirical study tools

Introduction (2/2)Introduction (2/2)


Background (1/2)Background (1/2)

Architecture of GINGER [IST’96]GINGER automatically collects software process and product data, and performs analysis to provide feedback to developers


Background (2/2)Background (2/2)

History of GINGER [IST’96] (since 1985)PerformanceQuality Software artifacts

Programmer Performance Model


Evaluation ofProgrammer Performance


Evaluation ofTeam Performance

Evaluation ofTeam PerformanceEvaluation of Quality

(automated testing)Evaluation of Quality(automated testing)

Evaluation ofData Flow Diagram(from CASE tool)

Evaluation ofData Flow Diagram(from CASE tool)Evaluation of

Programmer debugging Performance

Evaluation of Programmer debugging

Performance

Off-line feedback(size, change)

Off-line feedback(size, change)

On-line feedback(size, change)

On-line feedback(size, change)

Estimation residual faults based on change

Estimation residual faults based on change

Estimation residual faults based on petrinet model

Estimation residual faults based on petrinet model Evaluation of process

and product base on change history

Evaluation of process and product base on

change history








CAESE

Empirical studyProcessModel

KnowledgeConcerning

SoftwareDevelopment

DataCollection

Model

Empirical StudyTools

(Hypothesis, Issues, Questions)

ProblemStatements

Experimenter

CAESE Framework (1/4)CAESE Framework (1/4)

Illustration of CAESE with CASE

Customer Requirements CASE

DevelopmentProcessModel

ProductModel

DevelopmentTools Software

Artifacts

Software Developer

Empirical studyProcessModel

DataCollection

Model

Empirical StudyTools



Empirical study process model

NeedsAnalysisNeeds

Analysis

ExperimentDesign

ExperimentDesign

ExperimentationExperimentation

DataAnalysisData

Analysis

PackagingPackaging

Identifying problem, specifying goal, formulate hypothesis

Detailed specifications for the experiment. Experimental tools, types of subjects, analysis methods and tools

Experimenters analyze data collected

Experiments are monitored and data are collected

Experimenters compile “knowledge” into a “package” available as an output from the study.



Data collection model- Three entities & Five viewpoints

Same collection tool for different aspect of dataData analysis needs to be conducted on the five types of data

SubjectSoftwareArtifact

DevelopmentTool

States ofsubject

Input from subjects to software engineering tools

Output from software engineering tools to subjects

States of software engineering tools

States of software engineering artifact repositories


Empirical study toolCAESE


NeedsAnalysis

Tool

NeedsAnalysis

Tool

ExperimentDesignTool

ExperimentDesignTool

Data CollectionTools


Data AnalysisTools

Data AnalysisTools

PackagingTools

PackagingTools

Data Integration ToolsData Integration Tools

Control Integration ToolsControl Integration Tools

Empirical Study Management ToolsEmpirical Study Management Tools



Data AnalysisTools

Data AnalysisTools



Managing the empirical study itself(Same way as project management tools in CASE)

Empirical Study Management ToolsSynchronizing the use of different toolsControl Integration ToolsIntegrating the various types of dataData Integration Tools


Data Collection (1/2)Data Collection (1/2)

Considerations- Developers are engaged in many types of activities

simultaneously- CAESE should support continuous automatic

collection of dataFine-grained dataThe load on the subject is lessenedNecessary data will not be left out by mistakeThe chances of mis-informing would be minimized


Data Collection (2/2)Data Collection (2/2)

Implementations

XProgram changes

XTool usage logGinger1 System

XSkin resistance levelSkin Resistance level Measurement

X3D motion data3D Motion Measurement System

XEye-traceEyetrackd

XXChanges in window(displaying on monitor)

XWindow operations

XMouse

XKey strokeMon, Kterm-mon, Emacs-mon

(window monitoring tools)

XAudio data

XXXVideo dataAudio & Video Recorder

Data collection model category

Data TypeData Collection Tools

subj

ects

from

use

rto

use

r

tool

s

artif

acts

(more)



Data & Control IntegrationData & Control Integration

System architecture

Data Collection Data Collection ToolsTools

Data Analysis Data Analysis ToolsTools


Audio & VideoRecorder


OtherData Collection

Tools

OtherData Collection

Tools

AV-MonitordAV-Monitord

MonitordMonitord

AV-Database

Data CollectionServer

Database

Audio & VideoPlayer

Audio & VideoPlayer

OtherData Analysis

Tools

OtherData Analysis

Tools

Data RecorderData Recorder Data PlayerData Player

MEPG1

MON language


Data Analysis (1/2)Data Analysis (1/2)

Two analysis activities- Observational analysis (Replaying)

During and after experimentsData display and visualization tool

- Computational analysisHeavily depends on computational means

ObservationalObservational

Image retrievalImage retrieval

Video scannerVideo scanner

Video indexingVideo indexing

ComputationalComputational

MetricsMetrics StatisticsStatistics

VisualizationVisualization SimulationSimulation


Data Analysis (2/2)Data Analysis (2/2)

AnnotatorTool

DataDisplay

MultidataDisplay

MovementSimilarityAnalyzer

SRLDisplay

VideoSearchTool

Audio &VideoPlayer

StatisticalAnalysis

GINGER2GINGER2

Experimenter may make while observing and experiment

Keystroke and Gaze point

Data Screen,3D Motion,and Event

Measuring “Stress”

Data Analysis tools

StatisticalAnalysis


Case Study (1/3)Case Study (1/3)

Modeling Debugging ProcessExperiment DesignExperiment Design

Identifying which of the 15 modules contained a bug Collecting DataCollecting Data

Eyetrackd

Audio & Video Recorder:(To record interviews)

Need AnalysisNeed Analysis

Experts vs. novice users

Data AnalysisData Analysis

Analyzed how often gaze points shifted among modulesHow long the gaze point stayed within a single module

Visualized the trace of gaze points



Modeling Debugging Process (cont’d)- Analysis results

Difference in the frequency of shifts of gaze points• Experts gradually focused on one or two modules

• Novice subjects kept shifting their gaze points



Two-person Debugging process

Collecting DataCollecting Data

Kterm-mon

Audio & Video Recorder:(communications)

Need AnalysisNeed Analysis

How effective multiple programmer can work together in debugging

Data AnalysisData Analysis

Experiment DesignExperiment Design

Two subjects find single bug

How knowledge is communicated between the subjects

How strict roles between the two subjects

Analysis resultsAnalysis resultsDebugging was more effective

- with a “shared buffer”: communicated asynchronously

- separating role:understanding the program / identifying the location of bug


Conclusion (1/2)Conclusion (1/2)

Computer-Aided Empirical Software Engineering- Benefits

Integration• Synchronization of simultaneously occurring data

• Analyzing data from multiple perspectives

Sharing experiment designs• Allowing to develop “knowledge base”

(Experiment design requires lots of experties and experience)

Extensibility• Reducing the cost of introducing new data collection

equipments and new analysis methods and tools

(Just make an addition to MON language)


Conclusion (2/2)Conclusion (2/2)

Future Work- Support for Needs Analysis

Applying GQM [Basili, TSE’88]

- Support for Experiment DesignAfter accumulating “know-how”

• Focus on analysis by simulation

- Support for PackagingDeveloping representation standardPublishing data on the web

- Distributed CAESEExtending transmission from LAN to Internet


Discussion (1/2)Discussion (1/2)

Critiques- Weak support for computational analysis

Lessons learned- Data and control integration mechanism

MON LanguageSynchronized collection of simultaneously occurring data


Discussion (2/2)Discussion (2/2)

My approach

CASECASE

Effort

SoftwareArtifacts

Type of activities

DevelopmentTools

DevelopmentTools

DevelopmentTools

Size, defects

Data CollectorData Collector

Developer

Database

Data RetrieverData RetrieverM. Server

ManagerFeedbackInformation

Counting Rule

IntegrationRepresentation


Q & AQ & A

Thank you!!!


Data Collection of GINGER2Data Collection of GINGER2

Mon, Kterm-mon, Emacs-mon(window monitoring tools)



MON languageMON language


Data DisplaysData Displays

Data Screen Display

3D Motion Display

Event Display


Multidata DisplayMultidata Display

inside

outside


Movement Similarity AnalyzerMovement Similarity Analyzer

Base movement


Skin Resister Level DisplaySkin Resister Level Display

More stress

ginger2: an environment for computer- aided empirical ...€¦ · aided empirical software...

Documents