emerging strategies in software application testing · 44% are hampered further by limited access...

–––

Copyright ©2019 Gatepoint Research. All rights reserved. This report is the sole property of Gatepoint Research and may not be used, reproduced or redistributed in any form including, but not limited to, print & digital form without express written consent of Gatepoint Research.

Summary Results | June 2019

Emerging Strategies in

Software Application Testing



Survey Summary▶ Between March and May 2019, Gatepoint Research invited selected executives to participate in a survey

themed Emerging Strategies in Software Application Testing.

▶ Candidates were invited via email and 100 executives have participated to date.

▶ Management levels represented are predominantly IT and application development senior decision makers:

6% hold the title CxO, 4% are VPs, 42% are Directors, and 48% are Managers.

▶ Survey participants represent firms from a wide variety of industries including business services,, consumer

services, financial services, healthcare, manufacturing (general, primary and high tech), retail trade,

telecom services, transportation, utilities, and wholesale trade.

▶ Responders work for firms with a wide range of revenue levels:

▪ 53% work in Fortune 1000 companies with revenues over $1.5 billion;

▪ 17% work in Mid-Market or Large firms whose revenues are between $250 million and $1.5 billion;

▪ 30% work in Small companies with less than $250 million in revenues.

▶ 100% of responders participated voluntarily; none were engaged using telemarketing.



Executive OverviewSoftware testing on the mainframe is not without challenges. Accurate test data can be difficult to

create, difficult to mask, difficult to access. Test teams often spend valuable time resolving quality

issues, cost overruns, and delivery delays. What are organizations doing to overcome these problems?

This survey asks respondents to report:

▶ How much time are test teams spending testing on the mainframe, and what challenges do they face in the process?

▶ What tools have they employed to improve testing on the mainframe?

▶ What enhancements to data or mainframe services would provide the greatest benefit?

▶ Have they considered a continuous testing solution?



How much of your mainframe test team's time is spent testing software?

45% of respondents say software testing on the mainframe is not a major work element for the mainframe test team—it only consumes 20% of testers’ time.

45%

23%

16%

10%

4% 2%0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Less than 20% 21% to 40% 41% to 60% 61% to 80% 81% to 90% More than 90%



Do you have any of the following challenges testing software applications on the mainframe?

Testing software apps on the mainframe is a manual process, report more than half of respondents, and 44% are hampered further by limited access to the mainframe for testing. More than a quarter of respondents complain they have poor test data and/or are delayed waiting for test data to be provided.

55%

44%

29%

27%

17%

0% 10% 20% 30% 40% 50% 60%

Manual testing processes

Limited/no access to the mainframe for testing

Insufficient or flawed test data

Test provisioning delays/wait times

Masking data for GDPR and other

personal data requirements/regulations



What do you wish you could improve in your software application testing process?

Automation! 65% of respondents are adamant about needing automated test execution. 38% or more respondents say desired improvements include optimized test coverage, better quality of the apps that are delivered, time to market speed and faster test data generation. All of which might result in lower testing costs, wanted by 29%.

65%

43%

39%

39%

38%

29%

0% 10% 20% 30% 40% 50% 60% 70%

Automate test execution

Optimize test coverage

Improve the quality of

delivered applications

Accelerate time to market

Generate test data more quickly

Lower overall testing costs



What tools or strategies have you employed to improve application delivery as it pertains to the mainframe?

Improvements in application delivery are being realized with the use of test automation, report 62% of respondents. 47% use load and performance testing to improve app delivery. Half as many (23%) employ service virtualization and/or API management.

62%

47%

36%

26%23% 23%

0%

10%

20%

30%

40%

50%

60%

70%

Test automation Load andPerformance testing

Test DataManagement

Release automation Service virtualization API management



What enhancements to testing on the mainframe would reap the greatest benefits?

Of two main frame testing categories, data and services, respondents say enhancements to test data is most beneficial. Access to the right data (50%), and automatic data creation (42%) are most desired enhancements. The same number of respondents say having app performance and stability testing is the top mainframe services enhancement.

50%

42%

26%

42%

34%

24%

0% 10% 20% 30% 40% 50% 60%

MAINFRAME TEST DATA

Easy access to the right data

Automated synthetic test data creation

Automated data masking & sub-setting

MAINFRAME SERVICES

Hosted app performance and stability testing

Performance testing and othe virtualized services

Developer access to virtualized systems



At what stage in the software development lifecycle would a continuous testing solution on the mainframe yield the greatest benefit?

Two thirds of respondents agree having continuous testing on the mainframe during integration testing, over all other stages of the lifecycle, would be of most benefit.

52%

43%

66%

44%

37%

18%

0%

10%

20%

30%

40%

50%

60%

70%

Development Unit test Integration test Performance test Acceptance test Production



How helpful would accessing open testing frameworks using a client-side command line interface (CLI) be to your mainframe testing?

All but 20% of those surveyed agree there is “a little bit” to extraordinary benefit in having access to open testing frameworks using a CLI.

Not helpful

20% A little bit

helpful

16%

Somewhat helpful

42%

Very helpful

19%

Extraordinarily helpful

3%



What steps should be taken when considering a better solution for continuous testing on the mainframe?

50% of respondents insist ensuring sufficient resources, human and otherwise, is paramount if a new solution for testing on the mainframe is to be properly implemented. 45% say mainframe testing improvements must be mapped out and validated, and, of course, a budget must be defined (42%).

50%

45%

42%

36%

32%

0% 10% 20% 30% 40% 50% 60%

Sufficient resources to implement

the solution and train staff

A validated roadmap for improvement

A defined budget

Available time or staff to assess the options

Purchase approval across

multiple teams/departments



Profile of Responders:

Industry Sectors

Responders represent a wide variety of industries.

Business Services

27%

Financial Services

25%

Mfg - High

Tech

14%

Healthcare

8%

Mfg - General

4%Public Administration

4%

Retail Trade

4%

Utilities

4%

Consumer

Services

3%

Telecom

Services

3%

Education

1%

Mfg - Primary

1%

Transportation

1%

Wholesale Trade

1%Other

10%




Revenue

53% of those surveyed work in Fortune 1000 companies with revenues over $1.5 billion.

>$1.5billion

53%

$500 million -

$1.5 billion

10%

$250 - 500 million

7%

<$250 million

30%




Job Level

52% of those surveyed hold director or executive level positions in their organizations.

CxO

6%

VP

4%

Director

42%

Manager

48%



Broadcom is a global technology leader that designs,

develops and supplies semiconductor and

infrastructure software solutions, as well as mainframe

testing and quality solutions that reduce test cycle

times and costs.

Learn more at broadcom.com

https://www.broadcom.com/applications/ca-technologies-infrastructure-software-solutions/

emerging strategies in software application testing · 44% are hampered further by limited access...

Documents