how to test the new fashioned way

How to Test the New-Fashioned Way

Chris Kraus

DevOps: Continuous Delivery

CA Technologies

Product Manager

DO4X80E

@ChrisKraus3

#CAWorld

2 © 2015 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD

For Informational Purposes Only

Terms of this Presentation

© 2015 CA. All rights reserved. All trademarks referenced herein belong to their respective companies. The presentation provided at CA

World 2015 is intended for information purposes only and does not form any type of warranty. Some of the specific slides with customer

references relate to customer's specific use and experience of CA products and solutions so actual results may vary.

Certain information in this presentation may outline CA’s general product direction. This presentation shall not serve to (i) affect the rights

and/or obligations of CA or its licensees under any existing or future license agreement or services agreement relating to any CA software

product; or (ii) amend any product documentation or specifications for any CA software product. This presentation is based on current

information and resource allocations as of November 18, 2015, and is subject to change or withdrawal by CA at any time without notice. The

development, release and timing of any features or functionality described in this presentation remain at CA’s sole discretion.

Notwithstanding anything in this presentation to the contrary, upon the general availability of any future CA product release referenced in

this presentation, CA may make such release available to new licensees in the form of a regularly scheduled major product release. Such

release may be made available to licensees of the product who are active subscribers to CA maintenance and support, on a when and if-

available basis. The information in this presentation is not deemed to be incorporated into any contract.


Abstract

Manual testing got you down? It's time to step into the

application economy and get rid of slow, sloppy and

ineffective testing methods. Learn from this introductory

presentation what's new and how CA Application Test can

expand test scenarios and allow reuse of test assets to

catch defects earlier in the SDLC. Learn how to build

portable, executable test cases that are easy to extend,

modify and maintain.

Chris Kraus

CA Technologies

How to Test the New Fashioned Way


Agenda

HOW HAVE APPLICATIONS AND TESTING APPROACHES CHANGED?

WHY MOVE THE TESTS CLOSER TO THE APPLICATIONS

DON’T FORGET TO INVOKE AND VERIFY

SHIFT TESTING TO THE LEFT

1

2

3

4


3270/CICS Data Streaming Testing

User-interface Oriented Testing

Presentation Testing vs. Backend Testing

Device Testing vs. API Testing

Mainframe Client Server Apps 3 Tier/N TierSOA Apps

Mobile Apps Customer Facing

Apps

Testing has Evolved With Development

1980 1985 1990 1995 2000 2005 2010 2015 2020


Remember When…

Mainframe testing was simple– UI, calculations, and persistence

all in 1 layer

– All errors reported back to the 3270 Screen

– Trained employees were the users of application


Then…

Client server testing got more complicated, we moved CUI to GUI– UI and calculations inside the GUI

– RDA (Remote Data Access) means the persistence was external

– Most business logic was in GUI so testing GUI ran logic (sans. Stored procedures)

– Employees were trained on how to click and select

– Errors were still presented in the UI –better business errors presented


Now... We Start with the Layers – 3-Tier, N-Tier, to SOA

Originally HTML web pages provided simplistic markup of static and dynamic data

The logic of the application was decoupled from the presentation layer

Presentation testing was look and feel – users struggled to test the business logic

SOAP and SOA services meant we had to change to testing data on the wire – no UI to test

Introduction of hiding error messages – a clever response “sorry something went wrong”

Load testing of applications became an exercise in putting data on the wire


Now… Mobile Applications and the Consumer Rule

Mobile applications have created the shift of the user –users are no longer trained on the application

Mobile applications are similar to client servers there is business logic within the mobile app

RDA was replaced with services and API’s, so business logic in backend systems must be tested separate of the device

The layers of the application are blurring - Enterprise Service Buss

A new approach is needed to test API’s to ensure function and performance

The application running on the device needs to be treated as a SUT (System Under Test)

100’s of testing options may have to be created to test mobile devices

ESB


Why Move the Tests Closer to the Applications?

Moving the test closer to the code it is validating enables you to:– Distribute the tests across the architecture

– Remove layers of complexity by testing as close to logic as possible

– Start functional testing before development is complete

– Decompose big response times into little pieces


Key Considerations When Looking at the Layer

What business rules and validation is at the Mobile layer?

Are the API/Service end points changing data?

ESB all the data on the wire?

Backend systems –what is old is new or is it new?

ESB


Considerations for Mobile and HTML5 Layers

Need to validate the data on the screen for:

– Formatting– Alignment – visual representation– Input validation– Enabling functions - business rules X and Y

but not Z

Performance testing – how is the code on the device performing

Traditionally brittle small changes in UI’s cause test case maintenance

ESB


Considerations for API and Service Delivery Layers

Where’s the UI? We need a new approach!– SOAP XML

– REST JSON

– HTTP & Message Queue’s

Testing orchestration of multiple service calls to accomplish a requirement

API calls are less brittle than UI testing

Nonfunctional load testing to understand composite response times

ESB


Enterprise Service Buss – it is all the Wires in the House

Always the simple central point of blame

Hardest system to test– Data on the wire is needed to test – stimulate the system

– Need specific dependent data to test the transformations – simulations to match required transformations

– Thousands of combinations of data

Small changes to contracts consistently requested

Shared services mean small changes can break multiple consumers

Changes in performance break everyone

ESB


Systems of Record – What’s Old Is New and More New

Business functionality turns into composite services or API –service decomposition becomes important

Legacy systems behind the ESB have there own set of testing challenges– Data on the wire is needed to test – it may not be pretty

– Cobol Copybooks, CSV and EJB’s custom layouts

– Non human readable data – hello java object or CORBA

– Business logic and persistence are combined functionality

Building out regress for undocumented system

Failures this low in the stack can be gobbled up by a asynchronous process

ESB


So Many Layers – Where Do I Automate?

Everywhere you can! And YES there are some exceptions:– Look and feel is just hard –

you need eyes on the glass

– Graphical diff’s of images is not great but okay

CA Application Test– Applications on mobile devices

– Browser based UI’s on PC’s and services

– Modern API’s and services

– Legacy technologies

ESB


CA Application Test for Mobile and UI’s

Write your test once and run anywhere– Locally attached devices

– Cloud attached devices

– Local and remote simulators

Focus on testing the specific code in the mobile program– Enable/disable buttons, sliders, input

– Formatting of data

– Navigation

– Application to application dependencies

ESB


CA Application Test Testing the Services

Service testing is all about the data on the wire– SOAP XML

– REST JSON

– HTTP POST and GETS

– Message Queue’s with XML

There is no UI so you need CA Application Test to put data on the wire and get it off

One test can be used for functional, regression and non-functional

ESB


CA Application Test Supporting the Complexity of ESB

CA Application Test support industry standards and custom protocols for ESB– webMethods Broker

– webMethods IS

– Tibco Activities

– Tibco EMS

– MQ Series and JMS providers

The secret sauce is CA Service Virtualization to stabilize the environment

Regression is a huge problem – test generationvia CA Continuous Application Insight creates

the volume of tests needed with velocity

ESB


CA Application Test – Good Thing it was Born in 1999

System behind the ESB may be modern stateless services or not

Or the backend systems are the museum of technology– Databases

– Java objects, Corba, EJB’s

– MQ Series

– FTP files, read files, check directories

– Non XML data formats like Swift, FIX, Fin, EDI, CSV, Cobol Copybook

ESB


Invoke and Verify With a Test

Don’t trust the response– Transaction path can traverse

multiple boxes

– Asynchronous processes that happens fast enough looks synchronous

– Never trust the System of Record is updated until you verify that the update happened

– CA Continuous Application Insight automatically generates Database Validation to ensure SOR has been updated

ESB


Why “shift left”? Discover Defects Earlier in SDLC

Find defects

hereNot here

UNIT SYSTEM PRODUCTIONUATINTEGRATION PERFORMANCE STAGING

$977

$2,517

$4,057

$5,596 $7,136

$14,272

$-

$2,000

$4,000

$6,000

$8,000

$10,000

$12,000

$14,000

$16,000

1 2 3 4 5 6

Source: Lyon, Dan; Systems Engineering: An Essential Role for Cost-Effective Development of Secure Products.


How Does the Secret Sauce Help - “Shift left” the SDLC

Without SV

WITH SV

Dev1

Dev2

Dev3

System Test

Integration

Performance Test

UAT

SystemDev1

Dev2

Dev1

Dev3

Dev2

Dev3 Integration Performance UAT…

Uncertain delivery schedule – defects persist until UAT

Typical composite app today waits for whole

assembly to beginWith CA Service Virtualization quality effort moved earlier in the lifecycle

Reduction in SDLCFaster Delivery

“Went from finding 40% of our defects in the 3rd and final stage of testing to often finding ZERO. Service Virtualization turned this final test phase into a dress rehearsal for production.”

- AutoTrader http://bit.ly/AutoTrader-SV

http://bit.ly/AutoTrader-SV


Q & A


Recommended Sessions

SESSION # TITLE DATE/TIME

DO3T06SVision & Roadmap: The CA Roadmap for Service

Virtualization and the Dev and Test Portfolio11/18/2015 at 2:00 pm

DO3X118SThe Why, Where and How of Service Virtualization

Adoption11/19/2015 at 2:00 pm

DO3X114SFreeing the World from Slow: How Service Virtualization

and the Concept of S.P.E.E.D. 11/19/2015 at 3:45 pm


Must See Demos

Application Test DemoCreate Better Application Tests

Theater 4

DevOps Sim

DevOps Simulation Experience

Theater 3

Continuous App InsightAutomate Tests for Better Apps

Theater 4

Service VirtualizationDeliver Better Apps Faster with SV

Theater 3


Follow On Conversations At…

Smart Bar

Theaters 3 & 4

Tech Talks

Theaters 3 & 4


For More Information

To learn more, please visit:

http://cainc.to/Nv2VOe

CA World ’15

http://cainc.to/Nv2VOe

http://bit.ly/1wbjjqX

how to test the new fashioned way

Technology