whittle modeling wizards 2012

The Truth about Model-Driven Development in Industry

– and Why Researchers Should Care

Jon Whittlejoint work with John Hutchinson, Mark Rouncefield

School of Computing & Communications

Lancaster University, UK

j.n.whittle@lancaster.ac.uk

2001 AD

 I'm sorry, Dave. I'm afraid I can't do that.

model-driven architecture

aka.model-driven developmentmodel-driven engineeringmodel-based software developmentmodel-based designmodel-integrated computingdomain-specific modeling

productivity

maintainability

portability

interoperability

all is well with the world

except for…

those darned naysayers

2012 AD

who was right?

talk outline• Project EAMDE

– Social/organizational/technical factors

• Discoveries– Some observations– Some lessons– Some tips

Project EAMDE

• Widely-distributed questionnaire on how MDD is used– 35 questions pertaining to MDD application– 449 responses

• In-depth interviews with MDD practitioners– 22 interviews; 17 different companies– 150,000 words of transcribed data– >360 years of cumulative experience

• On-site ethnographic studies (ongoing)– 2 done (by Steinar Kristoffersen); more planned

What EAMDE is not

• an attempt to quantify the penetration of MDD in industry

• a study on UML– deliberately broad view of MD*

• an attempt to evangelise or promote MDD– interested in failure as much as success

Diversity

Examples of MDD

Examples of MDD

“the broader the domain you try and cover, the less the productivity increase… with DSM, you’re not looking at building a modeling language for embedded applications … you’re not looking at building one for mobile applications… mobile embedded, or even mobile phones or even Brand X mobile phones, but a particular product family of Brand X mobile phones…”

First discovery: a lot of MDD success is hidden

UML BPMN VendorDSL

In-houseDSL

SysML Matlab/Simulink

Which modeling languages do you use? (tick all that apply)

DSLs favored over general-purpose modeling

• mostly, companies write their own code generators for very specific tasks

– In contrast, companies often ditch commercial tools because they cannot modify them the way they want or because they “don’t do everything”

– Multiple references to the fact that off-the-shelf tools could have killed an MDD effort

• Generation of whole systems is not widespread

Second discovery: code generation is a red herring

Code generation

“I guess at the end of the day, this dream of code generation from models doesn’t exist – I mean everything ends up being done by hand because either we don’t trust code generators or they just don’t generate the code we need… it’s actually impossible to get in non-functional requirements into code generators – it’s too difficult”

Offsetting gains without realising it

• “sometimes the code generated makes it necessary to use a larger CPU which costs more money than the efficient code of an experienced programmer”

• 8x more expensive to certify generated code

Don’t obsess about productivity

• 65-100% code generated

• Figures on productivity gains differ– 20-800% gain– 27% loss

Third discovery: the real benefits of MDD are holistic

So if not productivity, then what?

• The real benefits of MDD are quality, architecture, reuse

• “whenever you name any single advantage…you can always achieve the same advantage with another approach…”– “it tends to be that a model-driven approach is more

likely to have a well articulated design and architecture”

Example

• An organisation finds itself developing lots of little (modeling) languages over time:– “we were generating 70% of the system off these little

XML languages… we would try to separate out the pieces that were generateable and the pieces that weren’t… it motivated us to have better separation of concerns”

Fourth discovery: MDD must enable new things, not just speed up old things

Doing things faster and better is not enough

• If the status quo works (or is perceived to work), there will be insufficient buy-in to change

– MDD should be sold not based on how it can do things (slightly) better, but in terms of how it can fix things that are broken

– “software is no longer a bottleneck”

People/Organizations

The Psychology of MDD

Architects love MDD

The code guru hates it

as does the hobbyist developer

It’s bad news for offshoring

Middle-managersare usually the

bottleneck

The MDD guru is likelyto be a developer anddomain expert

MDD works best in companiesthat are not in the software business

Current Business PracticeDoesn’t Support MDD

• “lead architects need to understand that just because their models are simple, they weren’t going to be put out of a job…”

• Developers are defined by their expertise for a technology not a domain; so it is not in their interests to innovate

Business Practice Doesn’tSupport MDD

• domain experts already model– “they already have an established way to design in

Powerpoint”– “I think they are more open than a company that has very

long years of experience in software development”

MDD Works Best in Non-Software Companies

The Good, The Bad, and The Ugly

The Good

• mature domain• non-software context• ground-up effort• real business driver

“we put it directly in the line of the product”

“we are not allowed to fail”

“the benefit was not only because we introduced model-driven design”

“we also started a re-use group”

“if you didn’t introduce model-driven design, the reuse initiative would have failed”

“at this moment, embedded software is not bottleneck in any project”

“56% of all our code is from re-used building blocks, but in the beginning it was only 5 or 10%”

“normally, decisions are made as low in the organisation as possible

The Good

• critical path• non-software context• real business driver• MDD an enabler for reuse

The Bad-Ass

“the same electrical design in a full size truck as in a Cadillac”

“somebody would be writing the spec”

“they were actually outsourcing”

“there was an order of magnitude difference in terms of number of people involved from generation to generation”

“you couldn’t find a computer scientist if you went on a search party”

“vendors try to push out-of-the-box code generation”

“very, very challenging to do… so we wrote our own code generator”

“if we did modeling just for code generation”

“you’ll likely not get the right abstraction”

The Bad-Ass

• mature domain• non-software context• ground-up effort• real business driver

The Ugly

“it was a totally new concept of switching system”

“he made this huge decision ”

“50 developers all using this CASE tool”

“if there is a problem, they just contact the [CASE tool] engineers.. That was kind of their strategy”

“and suddenly the tool doesn’t do something expected”

“they try to contact the vendor but they don’t really know what’s going on”

“they couldn’t optimize the generated code”

“so the way they had to do it was asking the hardware guys to have more hard disc, more memory, because of the tool”

“it was a nightmare to them to add all the features”

The Ugly

• immature domain• top-down effort• no real business driver• lack of control

What would 449 MDD practitioners say?

Disclaimers

Snowball sampling

Not everyone answered every question

We encouraged only those with actual MDE experience to participate…

…and discouraged those with opinion but no experience

Aims of the Survey

Real use of MDE in practice

Unraveling points of contention related to MDE

Technical factors• Code will write itself!• BUT needs to be

integrated with legacy systems

People factors• Easier to train new hires• BUT new hires require

more training

Organisational factors• Encourages organisation-

wide standards• BUT requires heavy-

handed, rigorous management

Key findings

The “Community” of Respondents

Do you consider MDE to be a good thing?

84%

• Significant software engineering experience– Over 1000 years total experience– 70% involved in software development for 5 years or

more

• Employed in a range of different roles– Developer 17%– Modeler 21%– Team leader 19%– Project manager 16%– Domain expert 4%– Researcher 23%– Architect 7%

• Good spread of size of company– Roughly a third each had:

• 1-100 employees• 100-1000• >1000

• A significant number of employees are not engaged in software development

Experience with MDE

• Initial exploration: 10%• Prototyping: 10%• First major project: 18%• ~ 1/3: extensive experience of MDE on many

projects and/or over many years

What are models used for?

“Do not use” percentages for MDE activities

UML BPMN VendorDSL

In-houseDSL

SysML Matlab/Simulink

Which modeling languages do you use?

Use of multiple languages

• 62% of those using custom DSLs also use UML• Almost all users of SysML and BPMN also use

UML• UML is the most popular ‘single use’ language

– 38% of all respondents

• UML used in combination with just about every combination of modeling languages– 14% of UML users combine with vendor DSL– 6% with both custom and vendor DSL

Which diagrams are used?

19 different diagram types are used regularly

Points of contention – paired questions

Effect of MDE on Training Costs

Q: Does using MDE allow you to employ developers with less software engineering experience?

Q: Does using MDE require you to carry out significant extra training in modeling?

46% agree, 34% disagree

74% agree, 9% disagree

Benefits of code generation

Q: Is your use of code generation an important aspect of your MDE productivity gains?

Q: Is integrating generated code into your existing projects a significant problem?

75% agree, 10% disagree

36% agree, 40% disagree

Changes on model or code?

Q: Do you mainly make updates on the model rather than the code?

Q: Do you spend a lot of time synchronizing the model and the code?

70% agree, 15% disagree

35% agree, 45% disagree

MDE and Agility

Q: Does MDE make you faster at implementing new requirements?

Q: Does MDE prevent you from responding to business opportunities?

75% agree, 12% disagree

32% agree, 38% disagree

Complexity of UML

Q: Is UML too complex?

Q: Is UML powerful enough?

44% agree, 32% disagree

52% agree, 31% disagree

Promoting understanding

Q: Does your use of MDE lead to better understanding between stakeholders?

Q: Does your use of MDE result in unexpected confusion and/or misunderstandings between stakeholders?

66% agree, 16% disagree

25% agree, 49% disagree

Tooling

Q: Are MDE tools too expensive?

Q: Do organizations attempt to deploy MDE using inappropriate and/or cheap tools?

45% agree

55% agree

Tools used

ATL, Acceleo, Agile4R, AndroMDA, ANTLR, Apple XCode, ArgoUML, Artisan Studio, ASCET, ASF+SDF, AToM3, Blu Age, Borland Together, BridgePoint, Eclipse (EMF, GMF, M2T), Enterprise Architect, Rhapsody, Rational Technical Developer, Rational Rose, IdealXML, Influx, Intalio, Intelligent Software Factory, iUML, Kermeta, Lisp, LTS, M2Flex, Magic Draw, Matlab/Simulink, Maven, Mendix, MetaEdit+, MetaSketch, MiaGeneration, Microsoft DSL Tools, Microsoft Visual Studio, Visio, Omnigraffle, MicroTool Objectif, MID Innovator Object, ModelBus, ModelPro, MOFScript, NexJ MDE Framework, Obeo, Objecteering, Oliva Nova, OpenArchitectureWare, OpenEmbeDD, OptimalJ, Oracle Case, OWLtoAspectJ, Papyrus, Popkin SA and BPWIN, Poseidon, PowerDesigner, Protégé, QVT-Operational (Eclipse Implementation), Rascal, Rational Software Architect, Scheme, SketchiXML, starUML, Stratego, TAYLOR, Telelogic Tau, Tibco Business Studio, Together Architect, Topcased, T-VEC, Umbrello, UModel, URDAD, Visual Paradigm, VisualState, WEBRATIO, WebSphere Integration Developer, IBM XDE

Over 100 tools

Key findings

• Productivity gains from code generation tend to outweigh losses from integration with existing code

• MDE allows for faster turn-around on new requirements– But there is a significant risk that it may prevent

organizations from responding to new business opportunities

Key findings

• MDE increases overall training costs• UML is not yet universally accepted as the

modeling language of choice– Indeed, DSLs are far more prevalent than anticipated

Education, education, education

• Are we teaching modeling the wrong way?– bottom-up versus top-down– combine teaching modeling

& compilers/code optimizationoptimization

Top 10 Tips for Practitioners

Keep Domains Tight and Narrow

1

Target well-documented, well-understood domains

2

Put MDD on the Critical Path

3

MDD works best when drivenfrom the ground-up

4

Be Careful About Gains Offset Elsewhere

5

Don’t Obsess About Code Generation

6

Not everyone can think abstractly

7

Most projects fail at scale-up

8

Match tools/processes to the way people think; not the

other way around

9

OK, there were only 9…

10

Bedtime reading…

• Model Driven Engineering Practices in Industry, ICSE 2011

• Empirical Assessment of MDE in Industry, ICSE 2011

• Mismatches between Industry Practice and Teaching of Model-Driven Software Development. MODELS 2011 Educators’ Symp.

• A Survey of Practitioners’ Use of MDE (ask me for a copy)

Why should researchers care?

Match tools/processes to the way people think; not the

other way around

Dataset

http://www.comp.lancs.ac.uk/~eamde/site/content_qres.php

These Slides

http://www.slideshare.net/jonathw/whittle-modeling-wizards-2012