The Future of Testing

James A. WhittakerPrincipal ArchitectVisual Studio Team TestMicrosoft

blogs.msdn.com/james_whittaker

toerris

HUMAN

nothing

is

PERFECT

WEall

make

MISTAKES

ITcouldhave

happenedto

ANYONE

We have many expressions to capture the fact

that humans are fallible

Our creations and inventions echo this unnerving tendency

toFAIL

if there was an Olympics for FAILURE

Or a World Cup for that which sucks

surelySOFTWARE

wouldreign

supreme

It’s enough to make one wonder …

… what we’re doing wrong

Software … … stocks supermarket shelves… delivers electricity and water… stores our personal information… runs nuclear power plants… controls doomsday weapons…

Are these failures typical?

How many of you have seen software fail?

How many users share this same experience?

Any reason to think things will improve?

Let’s take a look at the future and find out

How will we test the applications of tomorrow?

Virtualization

Information

Visualization

Virtualization

Information

Visualization

Virtualization

This prediction is 20 years out, but the technology to achieve it is here today

Virtualization

• Users are effective testers• Let’s leverage their power– They have the environments we seek– They have the data we lack– They run the tests we don’t think of

• “Crowd-sourcing” is one answer I like• Virtualization is the next step

Virtualization

• The story today:– A user (or tester) finds a bug, developer can’t repro it– A tester wants to reuse test cases, but can’t get them to run

• Virtualization solves this– User creates a VM at the point of failure

• These VMs have a market value• Developers could have access to the sum total of all the

environments in which their application can run

– A tester creates VMs for every test suite • We now have reusable, fully transferable testing assets • An automated historical record of all testing on our app

• If we can store every video in the world, why not every test case/test data/user environment?– It’s not been possible before but it is now

Virtualization

• What does the virtualized testing world look like?– Think crowd-sourcing on steroids– Test VM’s have great value, businesses emerge owning

and managing these environments– Testers analyze an app, license the appropriate Test VM’s,

conduct the test and report results

• Companies no longer need QA departments• Why test a subset of possible usage when (in the

future) we can test them all?• Software is important, it deserves this kind of

coverage

Virtualization

Information

Visualization

Virtualization

Information

Visualization

Information

This prediction is 1-3 years out, people are working on it today

Information

Information

Information about me (Status, Health, etc.)

Quick Access to oft-used abilities.

Information about my target. Information about my

location in the world.

Information

• What is the information we need?– Architects see Visio diagrams and flow

charts– Program Managers see PowerPoint decks

and Word storyboards– Developers see Visual Studio– Testers see binaries and interfaces

• In this model, testers see only what the user sees … I object!

Information

What the user sees

What the tester sees

Information

• Testers need more information– The architecture, design, code … they

are all useful to testers in the right context

– I want to hover over a UI element and see code, data types, value ranges, previous bugs, test history …

– ‘Cheating’ is not wrong when you are trying to build a great product!

Virtualization

Information

Visualization

Virtualization

Information

Visualization

Visualization

This prediction is happening today among the best test teams, soon it will be pervasive

Visualization

What does software look like?

Visualization

• Well, there’s:– Input– Output– Data flow– Control flow– Modules– Dependencies– Environment variables– Files– Interfaces– Bugs

Visualization

Conclusion

• Software will continue to run our economy, our businesses and our lives

• Software will be a critical part of the solutions that will grow food, cure disease and produce sustainable energy

• Remove software from the equation and humankind’s problems get a lot harder

• It is imperative that we get software right!

Parting thoughts

• 20 years from now– Software testing will look fundamentally

different– The tools will be more powerful and

allow us to be more impactful– Software testers will contribute more

fundamentally to the development process

Parting thoughts

• 20 years from now–Will the quality of software be taken for

granted? Will users be genuinely surprised when it fails?

–Will researchers look back in wonder that there was ever even a need for dedicated bug finders?

–What hard problems will simply cease to be hard because the software industry of this time did what needed to be done?

James A. Whittaker, PhDBlog: blogs.msdn.com/james_whittakerEmail: jw@microsoft.comURL: msdn.microsoft.com/testercenter