Evaluating the impact of a MDWE Approach on the Productivity and

the Satisfaction of Software Development Teams

Yulkeidi Martínez, Cristina Cachero and Santiago Meliá

Universidad Máximo Gómez Báez de Ciego de Ávila, CubaUniversidad de Alicante, Spain

Summary

Introduction

Empirical Evidences

Productivity and Satisfaction ExperimentObjectiveSubjectsDesignAnalysis Data

Conclusions and Future works

Context

Web Engineering community advocates the use of models in order to improve software development processes for Web Applications

Many MDWE approaches have proposed the use of models to obtain automatically (MDD) or only to define (MBD) a Web Application (e.g. WebML, UWE, OOHDM, OOH4RIA, etc.)

ContextThese MDWE approaches claims the

following benefits of MDD:Short and long term productivity gains

(process)Improved the efficiency of maintainability tasks

(process)Defect and rework reduction (better quality of

the resulting product)

What explains the low level of adoption of modeling practices by the Web developer’s mainstream?

Context

The MDE (MDWE) research community have not provide a sufficient body of practical evidence that soundly backs these benefits of the modeling practices with respect to code-centric approaches

The percentage of empirical studies that illustrate the impact of MBD and MDD over productivity or satisfaction is still very low, which hampers the generalizability of the results

Empirical EvidencesA systematic review carried out in Dec 2010 on 600

papers [Martinez, 2011] that claimed to provide empirical evidence sustaining in any way MDD assertions about impact on product quality and process productivity produced the following results:

Empirical Evidences

Some experiments show how when projects grow larger, the use of MDD increases productivity (results ranging from 2 up to 9 or even 20 times)[Mellor 2003], [Alfonso, 2006], [Mellega, 2010]

An industrial experiment [Mohagheghi, 2008] contradicts previous works where MDD productivity gains and losses depending on the particular study. They consider a strong dependency of the maturity level of the tool

Objective

The aim of this study is to augment the repository of empirical data that contributes to giving a scientific answer to the following research question:

What is the impact of the development method on the productivity and satisfaction of junior developers while developing Web 2.0 applications?

Our experiment

The study analyses three representative development methods based on Fowler’s modeling madurity classification:Code-Centric: models are only used as sketches,

without tools, the modeling activity is performed in whiteboards or blank pages to discuss complex or unclear aspects of the system

Model-Based Development (MBD): diagrams show most of the details of a system in order to foster its understanding. They are usually based on standards as UML. We use the class diagram of UML with the Rational Software Modeler tool

Our experiment

Model-Driven Design (MDD): fully-fledged models that can be used to characterize partially or completely an application

To represent the business logic, the subjects have used the OOH4RIA Domain Model that permits to obtain the business logic and the persistence of a Web application using a Object-Relational Mapping (such as NHibernate)

OOH4RIA Model-Driven Development

OOH4RIA (Meliá et al. 2008) proposes a MDE process based of a set of functional models and transformations to obtain a RIA implementation

Navigation ModelUser Interface Models

Domain Model

Problem-Space Models

OOH4RIA IDE: http://suma2.dlsi.ua.es/ooh4ria

Domain Model: Business Logic and Persistence

Domain Model Characteristics

Extended Class Diagram notation (a small learning curve) based on a DSL language

Automatic synchronization of arguments of CRUD operations (reduced maintainability)

Model constraints based on Hibernate QL rules

Wizards to accelerate the modeling (creating the OID and CRUD operation by default)

Data types based on Hibernate types (generic for any database)

Synchronization with manual code (custom operations and customized operations)

Degree of automation of different paradigms

Discipline Code-Centric

(VS 2010)

Model-Based(RSM)

Model-Driven (OOH4RIA)

MODEL Sketch or Absent

Blueprint Fully-fledged (DSL)

IMPLEMENTATION Manual Semi-automatic Automatic

TEST Manual Manual Semiautomatic

Disciplines during the construction phase of the Agile UP [Ambler, 2005] development process

Subjects of the study 26 Mc S. Students of a Master in Web Software Development which

are divided into 6 teams from 4 to 6 people

75% of them had more than 2 years of professional experience as developers of Web applications

The mean age was 25.6 years old and all of them were Computer Engineering graduates

Subjects Background (prior knowledge questionnaire to the Master): 81% knew UML, and that 12% had a high-level of knowledge of it 76% had programmed with VS.NET and 12% had applied it in the

industry 0% had previous practical of MDD, although 56% of them were aware

of the existence of it

All of them had received more that 50 hours of training in the Master about the .NET programming, UML and OOH4RIA modeling

Projects

Each of the six groups developed a social network application for a different domain: Trips, Events, Hospitals, Academic, facework and Automobile

Each group implements the same three subsystems of their social networks: Events: people can invite their friends or colleagues to

attend to a place Groups: a community of users to create contents and

relationships among people Organizational Page: subjects (companies, celebrities,

etc., depending on the particular application) can publish content, photos, etc. in a unidirectional way to the social network

Experiment Design The experiment applies a factorial design eliminating

any possible order effect. Teams were randomly assigned to each treatment order.

Team/Module

Application

Group Events Organization

1 Travel Code-centric

MBD MDD

2 Events Code-centric

MDD MBD

3 Hospital MBD MDD Code-centric

4 Academics MDD MBD Code-centric

5 Facework MBD Code-Centric

MDD

6 Automobile

MDD Code-Centric

MBD

Experiment Design All projects must implement the same Architecture: an object-oriented business

logic based on a relational-mapping framework as Hibernate

Business Logicimplemented in the Experiment

Experiment Design

Independent Variables or Factors (IV) are:Meth: Method (code-centric, MBD, MDD)Mod: Module (Group, Events, Organization)

Dependent Measurable Variables (DV) are:P(Meth, Mod): a ratio variable that measures the

productivity of a team with a method and module

S(Meth, Mod): an interval variable, based on a 7-point Likert scale, that measures the satisfaction of developers with each method and module

Productivity Hypotheses

These measures have been used to test the following testable hypotheses, which are based on the research questions and the existing empirical evidence:Productivity Hypothesys (PH):

Prod (MDD) > Prod (MBD) > Prod (code-centric)Productivity-Module Interaction Hypothesys

(PMIH):

P(Module*Meth) < 0.05

The effect on Productivity of the particular module is insignificant compared to the effect of the method

Empirical Measures: Productivity

We measured both the development time and the size of the modules developed by each team. Development time: The student had to document

the time of each development activity through the JIRA tool. They report the development time of each task (modeling, programming and testing)

Module size: We measure the size of the code produced by students in source lines of code (SLOC). SLOCs come in to express the size of software among programmers with low levels of experience [Gollapudi]. We automated the obtainment of this measure through the Line Counter tool

Analysis of DataRQ1: Impact of the method on the team

productivityProductivity Means:• P(MDD) = 4.6• P(MBD) = 2.3• P(Code-Centric) = 0.23

MDD is 2 Times more productive than MDB and 20 Times more productive than Code-centric (Adhoc)

The productivity is significantly affected by the method used, and regardless of the particular module being developed

Empirical Measures: Satisfaction

Satisfaction Hypothesis (SH): S(MDD) > S(MBD) > S(Code-centric)

Satisfaction-Module Interaction Hypothesis (SMIH):

S(Module*Meth) < 0.05

The effect on Satisfaction of the particular module is insignificant compared to the effect of the method

Empirical Measures: Satisfaction

To measure Satisfaction we defined a satisfaction scale made up of eleven items, where each one was based on a 7-point Likert rating scale

Subjects responded to a Post-experiment questionnaire that includes a semantic-differential scale that required developers to judge each method describing the developer's overall satisfaction

Empirical Measures: Satisfaction

Example of Ease of Use:

Analysis of DataRQ2: Impact of method on developer's satisfaction

Productivity Means:• P(MDD) = 4,76• P(MBD) = 3,48• P(Code-Centric) = 4,17

MDD is 36 per cent more satisfactory than MDB and 14 per cent more satisfactory than Code-centric (Adhoc)

The satisfaction also is significantly affected by the method used, regardless of the particular module being developed

Satisfaction Results H1:Perceived usefulness of the development approaches

H2:Perceived ease of use of the development approaches

H3:Perceived compatibility of the development approaches

Final Results H4: Intention of Adoption (to develop the Final Master Project)

Conclusions

These results show that model-driven techniques improves the productivity and also the satisfaction among developers when they are accompanied by a generation environment

The satisfaction can decrease even below code-centric practices when the modeling activities are used exclusively as blueprints to improve the understanding, and the developers must implement manually almost all the final code

Future Work

This is the beginning of a family of experiments in which replicate this analysis with practitioners in industry and more complex Web application models (e.g. navigation and UI models)

Further experimentation is need to separate the effect of methods from their tools (VS, RSM and OOH4RIA Tool)

We must generalize to different population, methods and application types and sizes

Thanks!

Questions?

Contact:Yulkeidi Martínez yulkeidi@informatica.unica.cu Cristina Cachero ccachero@dlsi.ua.es Santiago Meliá santi@dlsi.ua.es