developer productivity report 2015: java performance survey results preview

1All rights reserved. 2015 © ZeroTurnaround Inc.

JAVA PERFORMANCE SURVEY RESULTS

DEVELOPER PRODUCTIVITY REPORT 2015

Make your app do the Kessel Run in less than twelve parsecs!

PREVIEW


Figure 1.16

1Answers were mulitple choice, so the numbers don't add up to 100%. Deal with it :)

1


Another important metric to recognize is that there isn’t a single killer tool that does everything. In fact, if we look at the number of tools used by our respondents, almost half of those that took the survey claimed to use more than one performance profiler for their application. On average, 1.7 tools were used picked by each respondent.

Figure 1.13


The respondent, aka you!As we’d expect, the vast majority of those who responded to our survey labeled themselves software developers, as shown in figure 1.1. A further 27% sit in the Architect/Team Lead/Project Manager category. This constitutes well over 90% of those who responded, so we can rest assured that we’re talking to the techies! Interestingly, only 1.54% of those who responded were dedicated performance engineers, which either means there are very few performance engineers out there, or that perhaps we didn’t penetrate that specific market with the survey. It’s often difficult to get a survey out to an audience without experiencing bias, and this could be a result of bias from our market reach.

Figure 1.1


The ApplicationAs you’d expect, the vast majority of applications are going to be web applications, shown in figure 1.2 as taking over 70% of the audience. Desktop applications come in at a little over 11% with batch and mobile at just over 6% and 3.5% respectively. In the “Other” column, respondents mentioned other applications such as middleware or application servers. So we can now picture our typical respondent as a software developer who develops a web application. Should we name him/her? How about Sam? Actually, it’s probably better in the long run that we don’t get too attached to them, so we’ll continue calling them the respondent.

Figure 1.2


The OrganizationLet’s find out where our average respondent works. w We can start to understand the average organization a little better by understanding how big the teams that develop, test and support the application are. Figure 1.4 shows the distribution of people working on their application from design and development through to deployment and support. The average is 21.27 people. Remember, this is the arithmetic mean. There isn’t 0.27 of a person trying their best to develop an application using two of their available fingers and 3 toes on their one good leg! With over half of our respondents (55%) stating they work for a team of less than 9 people and 83% of respondents stating they work in a team of less than 25, it’s safe to say that the teams are mostly small. There will always be outliers which pull the average up as organizations turn enterprise, but these are the minority cases for this survey.

Figure 1.4


Next, we turn to responsibility. We can see straight away from figure 1.5 that as far as responsibility goes, the clear winner sits with the development teams. The data shows that at just over 55%, the most likely person to be responsible for performance testing is whoever wrote that piece of code. This could be for a couple of reasons. Perhaps we really do live in an agile world and the engineer who develops the code fully tests it - functionally, as well as for performance. Another possibility is that there isn’t any process at all. This would mean that performance testing is just dumped on the poor developer who has little support or knowledge about performance testing, so ultimately ignores it. While we all wish that would not be the case, it does creep into mind. Operations, Performance Teams and QA take up similar slices, as well as the answer ”Nobody” at around 11-14%.

Figure 1.5


While there is obvious value in performance testing against a production or staging environment, if we can shift as much as possible to the left, putting in effort earlier in the cycle, we should expect better applications that are cheaper to develop.

Figure 1.7 shows that 37% of respondents test their application for performance while it’s being developed. That’s a great statistic and shows how important people believe it is to test early. The most common phase to run performance tests was during system/integration testing.

Overall, there is a reasonable spread throughout the application lifecycle which is reassuring. As a multiple choice question, the average number of options selected by a respondent was 1.87.

Figure 1.7


Irrespective of who finds the performance problems, it’s clear from figure 1.8 who fixes the issues that emerge from performance testing. With almost 94%, it’s the developers that apply the fix. Yeh, the same people who wrote the code in the first place. As a result, it makes even more sense to performance test your code as you develop it, as the fix might as well be applied while the bug is being written!

Figure 1.8


Now we move to the question of test frequency. Profiling is an important part of performance testing, so that we can really understand execution paths, bottlenecks and so forth. However, figure 1.9 shows this as an extremely reactive activity. Over 40% of respondents state they profile their code only when issues arise, rather than something that’s typically done on a regular basis. Other than that answer, the results are largely spread across the other options with almost one in ten stating they never profile their code at all.

Figure 1.9


Sadly, our next data shows that one in four respondents (26%) spend no time on performance testing during a release. Perhaps this says something about the types or size of changes that tend to go into releases. Over 55% of respondents state they spend 1-5 days performance testing each release, which is much more encouraging. The remaining respondents test for 1 or more weeks each release.

Figure 1.11

The ToolsetThe proverb suggests that a good worker never blames their tools. We’ll not be looking into how effective each tool is just yet, but we will look at what is being used. VisualVM seemed an extremely popular choice as did JProfiler and Java Mission Control, the performance tool bundled with Oracle’s distribution of Java since version 7u40. Interestingly, 20% of respondents state they have their own custom in-house tools. Developers love to develop, right! XRebel is also worth mentioning with over 3% of the votes, particularly because it’s barely a year old!


I have Issues!Let’s now look at the issues themselves - the bugs behind all the complaints! First of all, how are they caught? Figure 1.14 shows the variety of paths a bug could take to become noticed by you. The most common (31%) way is via user reports and user feedback. In short, we as an industry are failing to test properly. Also, 20% of respondents say that some of their system faults or crashes are a result of performance issues. The vast majority of the remaining half (46%) of issues are found through tooling, including Performance monitoring tools, like APMs, Profilers and home grown software.

Figure 1.14


The FixWe know more about the issue now, so let’s try to understand the fix a little more. Different people will take different amounts of time to fix different issues in different ways. I guess I’m trying to say we’re all different, so let’s look at how long we tend to take for finding, fixing, and testing issues. Figure 1.18 shows that the most common answer was 0-1 days. This can be considered pretty fast, but of course it all depends on the complexity of the issue. Over half the respondents (52%) take less than half a week to fix an issue, but as you can see from the graphic, there’s a pretty long tail which pulls the average out to just under a week at 4.38 days. Just over one in four respondents (27%) claimed their fixes took a full week or more to diagnose, fix and test. It will be interesting to see who takes longer to fix the bugs later. *Spoiler* it is indeed interesting!

Figure 1.18


Report Author:Simon Maple (@sjmaple)

Report Contributors:Oleg Shelajev (@shelajev), Debbie Moynihan (@debdeb), Mart Redi (@martredi)

Report Designer: Ladislava Bohacova (@ladislava)

Contact us

EstoniaÜlikooli 2, 4th floorTartu, Estonia, 51003Phone: +372 653 6099

Twitter: @RebelLabsWeb: http://zeroturnaround.com/rebellabsEmail: [email protected]

USA399 Boylston Street, Suite 300, Boston, MA, USA, 02116Phone: 1 (857) 277-1199

Czech RepublicJankovcova 1037/49 Building C, 5th floor, 170 00 Prague 7, Czech RepublicPhone: +420 227 020 130

https://twitter.com/sjmaple

https://twitter.com/shelajev

https://twitter.com/debdeb

https://twitter.com/martredi

https://twitter.com/ladislava

https://twitter.com/RebelLabs

http://zeroturnaround.com/rebellabs

mailto:[email protected]

developer productivity report 2015: java performance survey results preview

Software

average respondent

application servers

average organization

web applications

typical respondent

vast majority of applications

desktop applications

performance profiler