p_1 col template
DESCRIPTION
TRANSCRIPT
Borland® Agile Testing Survey Results 2009 -Update
W h i t e Pa p e rJ u ly 2 0 0 9
Borland Agile Testing Survey Results 2009 - Update
2
Contents
Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Delegate Responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Agile Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
The Challenges of Tooling for Agile Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
The Role of QA in Agile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3
Borland Agile Testing Survey Results 2009 - Update
Executive SummaryThis survey contains primary research data collected at the EuroStar Software Testing event held in Den Hague, Holland, on
November 11-13, 2008, and at the STAREAST event held in Orlando, Florida, on May 5-8, 2009. The survey was given to
attendees regardless of whether they used, or were aware of, Agile software delivery methods. This is an update to the 2008
Survey from November 2008 to include data surveyed at the STAREAST event.
In total, 394 respondents completed the survey from which this combined and updated report has been produced. Respondents
represented many countries, including those outside Europe and the United States, and worked in a wide range of industry
sectors and company sizes, from retail, such as eBay; telecommunications, such as T-Mobile; and large scale IT providers,
like Cap Gemini and Microsoft. As you will see, the additional data captured further enhances our findings from the
previous report.
The driver for this study was to understand in greater detail the needs of Quality Assurance (QA) organizations in Agile
software delivery projects. Borland is managing its own Agile transformation and has discovered challenges that appear to be
consistent with other enterprises who have embarked on a company–wide transformation to Agile.
As part of our own transformation, the Borland development team in Austria, who develop our software testing suite known
as Borland® Silk™, found it necessary to adapt the automated testing applications so that they could scale to meet the needs of
testers integrated into the Agile process. As a result, Borland has now produced a unique set of products to fulfil the needs of
Agile and traditional testing, Borland SilkCentral® Test Manager™, Borland SilkTest® with Silk4J® and Borland SilkPerformer®.
The following sections discuss the findings of the surveys and reflect the needs of genuine QA practitioners from all aspects of
the industry, whether current Borland customers or not.We have summarized what we believe to be the key ‘takeaways’ from
the findings of our survey and provided comparisons to the earlier report where differences are highlighted.
If you wish to learn more about Borland Lifecycle Quality Management (LQM) solutions, visit www.borland.com.
Delegate ResponsesDelegates were asked to give the title that closest described their function within the organization. The majority of responses
came from test management positions and quality practitioners.
Agile ExperienceDelegates were then asked about the level of Agile adoption within their organization. Recent trends have indicated an increase
in Agile in the IT industry from 18% to 30% in the last 3 years.
4
Borland Agile Testing Survey Results 2009 - Update
Figure 1: Roles of respondents
The survey indicates a much larger growth trend as perceived by this audience of quality practitioners, indeed 61% of
respondents declared some experience of Agile adoption. Furthermore, the largest portion, 42%, was currently in the pilot
phase of Agile Transformation, and now considers it a viable alternative to current development techniques. The survey also
shows that only 23% of respondents had not considered Agile: no knowledge of Agile adoption (7%) or were in an
organization where it was not being considered (16%).
This indicates that the level of Agile adoption within established IT organizations is gaining pace more quickly than previous
predictions. However, it should also be noted that the numbers that have full adoption is low, only 5%, even though this is an
increase in the figure from the November report (2%). This reflects two key findings from Borland;
� Agile adoption is not a destination, it is a journey. It is in the journey where the benefits are realized and amongst theseis finding a toolset to support the Agile organization.
� A pragmatic approach to Agile transformation is required, since Agile and traditional projects are going to have to operatein an integrated, collaborative way
Agile transformation is a continuous and iterative process, including the adoption of true Agile testing into Agile development
to create Agile delivery teams.
5
Borland Agile Testing Survey Results 2009 - Update
Figure 2: Agile experience
The Challenges of Tooling for Agile TestingWe asked users what they saw as the biggest challenge for testing in Agile delivery today. The options we gave were based upon
challenges that we saw during our own Agile adoption, and also those of our customers.
The chart shown reflects only those users that have had actual experience of Agile adoption, i.e. those that responded ‘c’, ‘d’ and
‘e’ as seen in Figure 2. Almost one third (32%) responded that they did not see how the testing cycles could possibly cope with
the short timescales of Agile iterations. This is indicative of teams that are reliant upon manual testing, inherently slower than
automated testing, or a dependence upon tools that are incapable of supporting short iterative delivery. In order for QA to
deliver, they require test automation, and they need it to be fast.
This is somewhat validated by another 25% of respondents who expressed dissatisfaction with the capability of their tools. It
follows that over half the respondents with experience in Agile (57%) identified significant hurdles to Agile adoption because of
6
Borland Agile Testing Survey Results 2009 - Update
Figure 3: The Challenges to testing in Agile
tooling limitations for automated Agile testing, specifically with the short timescales involved. This indicates a general lack of
flexibility, or capability in automating testing processes in Agile today, and the speed with which tests could be generated and
executed.
When looking at responses from Agile users as to which tools they actually were using, 42% were not using any form of actual
automation for their testing (18% nothing and 24% basic office products). However, the numbers using the traditional
commercial toolset (19%) and Open Source tools (13%) was comparable.
This indicates that the Agile testing community has considerable exposure and knowledge of current automated testing tool
offerings, but still leaves the question as to why still so little adoption of genuine test automation, and a reliance instead on
manual, office products. So when asked further about test automation in Agile (Figure 5), it was interesting to see the results
and why so many felt they were unable to use automated testing tools.
7
Borland Agile Testing Survey Results 2009 - Update
Figure 4: Tools used
37% of the Agile testing community felt that the tools they had did not provide a satisfactory capability for their needs.
This is from a community using current testing tools from both the traditional and Open Source community. A further 37%
believed that process and test automation should be done, but could not see a way to do it, potentially indicative of why so
many users have not made a choice of tool; are relying on simple office tools like spread sheets and word processors or have
developed an in-house solution.
Looking at these specific results in isolation, we see a total of 74% of Agile community respondents felt either ‘short-changed’
or were given a lack of viable options when selecting a tool to automate their testing. In fact, only 12% were actually happy
with the tools they had. Again, indicating a widespread dissatisfaction amongst the Agile testing community with the
offerings available.
So in order to understand greater what these issues or short-comings are, Borland asked respondents what it was that they
really needed from a tool as their top priority. In compiling the questionnaire, Borland considered some of the issues they had
themselves been forced to confront, and highlighted the top five as options, with the ability for other answers to be entered.
8
Borland Agile Testing Survey Results 2009 - Update
Figure 5: I think Test Automation in Agile…?
The interesting aspect of this chart is that 91% of respondents agreed with Borland on what they consider to be the biggest
issues in tool provision. The biggest issues surrounded regression testing and metrics respectively.
Many of the issues around regression testing are the lack of use of tool automation which takes away the grind of repetitive
testing and allows testing automation to be run overnight without the need for user oversight or operation, in particular, the
need for speed.
The issue of metrics is a long standing one, giving test management the visibility and oversight of all testing and application
quality so that qualitative and quantitative judgements can be made on product quality.
15% of respondents were concerned about the level of effort required to ensure delivery of complete testing within required
timescales and the burdens that this then places on QA staff, the need for long working hours, typical of manual testing
operations or slower tool automation.
The next two issues were about having a tool with the flexibility to actually operate in an Agile way and not wasting valuable
time correcting environments and scripts instead of testing, and finally allowing quality to be seen as a team responsibility,
ensuring that quality is built into the applications from the start with the involvement of developers and testers
working together.
9
Borland Agile Testing Survey Results 2009 - Update
Figure 6: What is needed from a tool
The first three issues are typical of any testing environment, whether traditional or Agile in the need to ensure test coverage.
Of particular interest are the final two issues, cumulatively accounting for 30% of the respondents. These issues map
significantly to the unique demands and needs of an Agile development environment, that is, providing an environment to
sustain and support users, rather than hinder their operation in a fast, collaborative team manner.
We also asked users with Agile experience about programming languages used. It is widely considered appropriate, and even
important, that the test scripts for Agile testing be written in the same language as the application under test, so we asked which
languages was your organization using. There has been a long held assumption that Agile and Open Source go hand-in-hand,
principally because both were founded in small development shops wishing to keep costs to a minimum. There is therefore, a
high expectation that the use of Java® with Eclipse™ will be the highest used language and resource amongst these groups.
From the above chart, filtered to show Agile users only, it can be seen that Java is, as expected, the most widely used language
(49%), however, with the introduction of the latest .NET release, there has been a much greater uptake in this area (40%).
The economics of .NET programming resources have helped contribute to an obvious, but slow transition to .NET. In fact,
a large number of respondents (17%) indicated that they were using both languages.
10
Borland Agile Testing Survey Results 2009 - Update
Figure 7: Programming Language used
The Role of QA in AgileOur survey not only considered the tools, but also importantly, the people and their roles and responsibilities for quality in
an Agile organization.We asked the respondents if they believed that the concepts of a modern QA organization in IT
development would change and the result was emphatic.
56% considered that the traditional role of QA will change, and significantly so. These respondents were then asked further,
what will be the most important way that the role of QA will change.
11
Borland Agile Testing Survey Results 2009 - Update
By far the greatest number (39%) believed that the concept of an isolated QA team will disappear and that instead, QA will
become an integrated part of development.While not surprising, as this is considered a fundamental part of the Agile
manifesto, referring back to figure 3, the challenges to testing in Agile, 16% believe that developers and testers do not mix.
The cultural aspect of this change is a huge challenge to overcome.
Borland has experience in this, with testers becoming part of the Agile development teams, creating Agile engineering teams.
There was a concern that this would weaken the quality position, but the opposite was found. Instead, the perceived value of
quality was raised and became something that was proactively built into the software, rather than treated as an afterthought.
It was also interesting to note that the second largest response was that there would be a change in perspective on test
automation, from ‘nice to have’ to ‘absolutely necessary’ (23%). So despite the earlier reservations from respondents about
current tool offerings, the need for tool automation is still seen as a priority in order to ensure quality software.
An interesting variation, however, from the last report, is the increase in belief that testers are going to have to do more to
champion the cause of integration testing, user experience and sophisticated test scenarios. This is a reaction to the
continued isolation of QA teams from the Agile process and the need to introduce and maintain quality levels in the systems
being delivered.
12
Borland Agile Testing Survey Results 2009 - Update
Borland is the leading vendor of Open Application Lifecycle Management (ALM) solutions - open to customers' processes, tools and platforms –providing the flexibility to manage, measure and improve the software delivery process.
Copyright © 2009 Borland Software Corporation. Borland and all other brand and product names are service marks, trademarks,or registered trademarks of Borland Software Corporation or its subsidiaries in the United States and other countries. All other marksare properties of their respective owners. 26736
www.borland.com
ConclusionsIn a review of these study results we can come to three key conclusions;
Agile adoption is increasingly common in the Enterprise, but is still at a relatively immature stage –With 42% of respondents involved in Agile projects that are in pilot, it’s clear that more and more Enterprises are recognizing
the business advantages of shifting to Agile, but have yet to still fully realize all of the potential benefit. Of particular concern is
the issue of integrating true functional and performance testing into an Agile organization.
Automated Agile Testing is recognized as an area of need, but one that is unfulfilled –57% wished to introduce automated testing into their Agile projects but felt that either the tools were not satisfactory or could
not see how testing could be run in such short timescales. 23% believed that test automation was in fact the most important
change that QA will have to face as part of Agile transformation. The need for speed is paramount.
The role of QA will change as a result of Agile adoption –39% see a fundamental change in the way that QA sits in the IT organization and therefore the way that testing will operate as
part of software development.
However, possibly the most important conclusion is that the transformation to Agile is happening everywhere and quality
assurance must adapt. QA must develop Agile testing processes that harmonize with current Agile development methods – this
will change the way we view QA forever.
As a result of Agile testing, QA will become truly about defect prevention, rather than defect cure.
Borland Agile Testing Survey Results 2009 - Update