fostering long-term test automation success

BT9 ConcurrentSession11/12/151:30pm

“Fostering Long-Term Test Automation Success”

Presented by:

Carl Nagle

SAS Institute, Inc.

Broughttoyouby:

340CorporateWay,Suite300,OrangePark,FL32073888-268-8770·904-278-0524·[email protected]·www.techwell.com

Carl Nagle SAS Institute, Inc.

At SAS Institute Carl Nagle is a principal software developer, focused for the past seventeen years on the development, integration, and evolution of enterprise software test automation tools and the open source SAFSDEV framework. For more than thirty years, Carl has been responsible for engineering both hardware and software testing systems including the design of advanced electronics for analog and digital subsystem testing with embedded software for high-end power generation and communication systems. This naturally evolved into the development of advanced software testing systems and frameworks. Carl and his team thrive on solving “impossible” test automation scenarios across multiple processes, platforms, and technologies.

Fostering*Long,term*Test*Automa2on*Success

!�Sustainable�!Test!Automa0on!

!with!Carl%Nagle,!!SAS!Ins0tute!

• Why%are%YOU%here?%

• Why%am%I%here?%(Has%it%really%been%9%years?)%

•  How%did%we%achieve%ini@al%success?%

• Where%are%we%now?%

•  Q%&%A%%!

In%A%Nutshell%

If!these!apply!to!YOU:!!•  NEED!or!WANT!easy!test!automa0on!NOW,!•  For!MY!current!team!or!project,!•  Have!NO!concern!about!others,!or!even!future!projects,!•  Hope!to!�fix�!a!failing!automa0on!strategy…!

Then,!you!might!be!looking!at!the!wrong!presenta0on!!!(!But!stay!!!We!might!be!able!to!change!your!mind!;Q)!

Why%Are%YOU%Here?%

If!these!apply!to!YOU:!!•  NEED!test!automa0on!processes!that!last!•  To!be!shared!and!leveraged!across!projects!•  That!don�t!change!with!technology!or!tools!•  That!might!�replace�!a!�struggling�!automa0on!strategy…!

Then,!you!might!be!looking!at!the!right!presenta0on!!!!

Why%Are%YOU%Here?%

!!!!!!!!!!!!!Over!18!years!ago,!SAS!Ins0tute!tasked!me!to:!!•  Provide!a!common!test!automa0on!frameworkQQeliminate!silos!•  Make!test!automa0on!accessible!to!developers!*and*!testers!•  Handle!all!product!and!everQchanging!technology!domains!•  Sustain!test!automa0on!success!over!years,!not!cycles!•  Develop!it!in!an!openQsource!environment!

!!!!!!Today!I!report!that!this!prospers!s0llQQeven!aWer!18!years!!

Why%am%I%Here?%

1997% 2002% 2004% 2006% Today%

1.  We!listened!to!the!industry!experts!�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�!

2.  We!listened!to!poten0al!users!�We#need#to#leverage#and#share#our#test#exper4se#across#current#and#future#projects#regardless#of#the#technologies#or#tools#in#use.�!

3.  We!collaborated!with!professional!peers!�Focus#the#test#automa4on#framework#to#deal#with#the#common#components#that#make#up#all#our#unique#applica4ons.�!!

4.  We!implemented!it,!!shared!it,!!and!it!con0nues!to!evolve.!

How%did%we%achieve%ini@al%success?%

The!headlines!read:!!

!“In%today’s%environment%of%plumme@ng%cycle%@mes,%test%automa@on%becomes%an%increasingly%cri@cal%and%strategic%necessity.%%%Assuming%the%level%of%tes@ng%in%the%past%was%sufficient%(which%is%rarely%the%case),%how%do%we%possibly%keep%up%with%this%new%explosive%pace%of%webUenabled%deployment%while%retaining%sa@sfactory%test%coverage%and%reducing%risk?%%%The%answer%is%either%more%people%for%manual%tes@ng,%or%a%greater%level%of%test%automa@on.%AVer%all,%a%reduc@on%in%project%cycle%@mes%generally%correlates%to%a%reduc@on%of%@me%for%test.”%

The!headlines!read:!!!

%“With%the%onset%and%demand%for%rapidly%developed%and%deployed%web%clients,%test%automa@on%is%even%more%crucial.%%%Add%to%this%the%cold,%hard%reality%that%we%are%oVen%facing%more%than%one%ac@ve%project%at%a%@me.%%%Perhaps%the%team%is%finishing%up%Version%1.0,%adding%missing%features%to%Version%1.1,%and%prototyping%some%new%technologies%for%Version%2.0!”%

IBM!Ra0onal!Robot,!Func0onal!Tester!

HP!WinRunner,!QuickTest!Pro!

AutoIt!

“We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.”%

DIFFERENT!Technologies!

HTML!3.2,!4.0,!5,!SAP,!DOJO,!GWT!

Flex!

Java!

iOS,!Android!

VB,!.NET,!WPF,!Office!

DIFFERENT!Tes0ng!Tools!

SmartBear!TestComplete,!TestExecute!

EggPlant,!Sikuli,!ImageQBased!Tes0ng!

Mobile!SDKs,!Robo0um!

Selenium!1.0,!WebDriver!

Require%

C/C++!

•  Separate!the!test!design!process!and!assets:!!•  Completely!independent!of!the!test!execu0on!“engines”!•  DataQdriven,!ac0onQbased,!keywordQdriven,!behaviorQdriven!•  Support!testQfirst!principles!where!desirable!

!

�We#must#minimize#the#impact#caused#by#changes#in#the#applica4ons#we#are#tes4ng,#*and*#changes#in#the#tools#we#use#to#test#them.�%

PseudoQTypical!Example!Test!Records:!!

Google%%%%SignIn%%%%%%%%%%%Click%Explorer%%FolderTree%%Click%%"Coords=50,200�%%%%(tsk,%tsk,%tsk)%Explorer%%FolderTree%%Click%%My%Documents%%%%%%%%(beher)%

•  Separate!the!test!design!process!and!assets:!!•  Keep!vola0le!test!data!in!Maps!for!singleQpoint!of!maintenance!


[Google]%SignIn=�id=gb_70�%%[Explorer]%FolderTree=�Type=Treeview;Index=1�%%[FolderTree]%My%Documents=�Coords=50,200�%


The!Map:!

The!Test:!

Keep!vola0le!test!data!in!Maps!for!singleQpoint!of!maintenance.!

•  Separate!the!test!design!process!and!assets:!!•  Common%API!despite!years!of!evolving!technologies!and!tools!


•  Doesn’t!maler!what!the!run0me!execu0on!engine!is:!!SQA%Robot,%IBM%Ra@onal%Robot,%XDE%Tester,%Func@onal%Tester%HP%WinRunner,%QuickTest%Pro,%SmartBear%Test%Complete,%AutoIt,%EggPlant,%Sikuli,%IBT,%Selenium,%Mobile%SDKs,%Robo@um%…%

Google%%%%SignIn%%%%%%%%%%Click%Explorer%%FolderTree%%Click%%My%Documents%


hlp://safsdev.sourceforge.net/sqabasic2000/SAFSReference.php!

•  Separate!the!test!design!process!and!assets:!!•  Test!IDE!can!be!different,!depending!on!skills!and!preferences!•  CSV%Text%Files,!Spreadsheets,!Test!Composer!(CSV!Exporters)!•  Java!IDE,!Eclipse,!SeleniumPlus!(Java!API!Drivers)!•  Custom!IDE,!or!programs!(COM,!Ruby,!Perl…)!invoking!the!API!


Java!API!Examples:!!

Click(Map.Google.SignIn);%Click(Map.Explorer.FolderTree,%�My%Documents�);%

Click(Map.Google.SignIn);%Click(Map.Explorer.FolderTree,%�My%Documents�);%

Java!API!!Test:!

CSV!Test:!

The!Map:!


Then%do%safs%ac@on%Click%on%SignIn%in%Google%Then%do%safs%ac@on%Click%on%FolderTree%in%Explorer%using%�My%Documents�%

Cucumber!Test:!

•  We!need!to!be!able!to!move!our!people!around!easily.!•  A!person!tes0ng!.NET!can!easily!switch!to!tes0ng!HTML.!

!

•  They!can’t!be!bogged!down!constantly!learning!new!tools.!•  We!just!want!to!focus!on!test!design!using!the!Common%API.!

!

•  They!want!to!support!each!other!across!divisions.!•  Experience!applies!to!all!current!and!future!projects.!

“We#need#to#leverage#and#share#our#test#exper4se#across#current#and#future#projects#regardless#of#the#technologies#or#tools#in#use.”%

Applica@on%Map(s)%%(Vola0le!SPOM)!

SAFS%Services%and%Engine(s)!!

Java,!Web,!Flex,!Android!

Applica@on%Under%Test%%Java,!Web,!Flex,!Android!

Logs%

Func@onal%Tester%TestComplete%SeleniumPlus%Robo@um%ImageUBased%Tes@ng%

Executable%%Test%Designs%

%(CSV,!Cucumber,!

!Java,!SeleniumPlus)!

What%this%looks%like%in%a%complex%graph?%

•  Implement!independent!Engines!for!test!execu0on:!!•  Completely!independent!of!the!applica0on!being!tested!•  Engines!interpret!and!implement!the!Common%API%

•  Whether!the!domain!is!Java,!.NET,!Flex,!HTML,!SAP,!DOJO…!•  Whether!the!tool!is!Adobe,!IBM,!Selenium,!SmartBear…!

•  Engines!can!be!services!available!to!any!machine!process!•  Engines!don�t!know!and!don�t!care!what!is!driving!them!•  Mul0ple!engines!can!run!concurrently,!on!the!same!machine!•  Support!engines!centrally!for!ALL!users,!and!ALL!applica0ons!

“Focus#the#test#automa4on#framework#to#deal#with#the#common#components#that#make#up#all#our#unique#applica4ons.”%

What%are%those%internal%Services?%

Logging!

Maps!

Variables!

Drivers!

Engines!

(But%you%really%don’t%have%to%care%about%this.)%

Where%are%we%now?%•  We!have!grown!to!4!fullQ0me!inQhouse!framework!developers.!•  Internally,!our!userQbase!grows!in!the!US,!Canada,!India,!and!China.!•  We!just!completed!development!migra0on!to!SAFSDEV!on!GitHub.!•  New!discussion!forums!at!safsdev.freeforums.net.!•  Older!forum!archives!at!sourceforge.net/p/safsdev/mailman/!

!•  We!have!doc!and!updated!binaries:!SAFSDEV!on!SourceForge.!

!•  We!have!always!welcomed!contributors!for!new!engines!or!

features.!•  Frankly,!we!have!been!VERY!BUSY!since!our!last!public!talk!in!2006.!

Fostering*Long,term*Test*Automa2on*Success

!�Sustainable�!Test!Automa0on!

!with!Carl%Nagle,!!SAS!Ins0tute!Q*&*A

fostering long-term test automation success

Software