april 25, 2008 application server quality: review your assumptions

Making Leaders Successful Every Day

April 25, 2008

Application Server Quality: Review Your Assumptionsby John R. Rymer and Jeffrey S. Hammondfor Application Development & Management Professionals

www.forrester.com

© 2008, Forrester Research, Inc. All rights reserved. Forrester, Forrester Wave, RoleView, Technographics, TechRadar, and Total Economic Impact are trademarks of Forrester Research, Inc. All other trademarks are the property of their respective companies. Forrester clients may make one attributed copy or slide of each figure contained herein. Additional reproduction is strictly prohibited. For additional reproduction rights and usage information, go to www.forrester.com. Information is based on best available resources. Opinions reflect judgment at the time and are subject to change. To purchase reprints of this document, please email [email protected].

For Application Development & Management Professionals

EXECUTIVE SUMMARYWhich application server has the fewest critical bugs? Is easiest to configure? Has the best quality overall? These questions play a key role in application development professionals’ evaluations of their options for existing and future applications. Experience with early app servers has created a canon of popular wisdom about the quality of app server products, but five long-held assumptions within this canon are out of date. Overall, our survey revealed that the quality differences among major platforms, including Microsoft Windows Server 2003 and two open source options, are narrower than many application development professionals assume. The result: Application development professionals have better platform options to choose from than they might think.

TABLE OF CONTENTSAs Application Platforms Evolve, Old Reputations Die Hard

Vendors Are Doing A Good Job Of Managing Quality Risks

Debunking Five App Server Myths

Myth No. 1: There Are Big Quality Differences Among App Server Products

Myth No. 2: Bugs Are More Important Than Configuration Errors

Myth No. 3: Microsoft Windows Server Is An Inferior Option To Java App Servers

Myth No. 4: All Except The Largest Vendors Are Doomed

Myth No. 5: Open Source Support Is Riskier Than Conventional Support Models

RECOMMENDATIONS

Use Four Practices To Guide Platform Reassessments and Selections

WHAT IT MEANS

This Snapshot Hints At The Next Generation’s Likely Quality Challenges

Supplemental Material

NOTES & RESOURCESThe January 2008 Global Forrester Research/BZ Media Application Platform Quality Online Survey surveyed 617 IT professionals to determine their perceptions regarding the quality of available application servers.

Related Research Documents“Java Platforms: Where The Market Is Headed” March 7, 2008

“IDE Usage Trends” February 12, 2008

“The Forrester Wave™: Application Server Platforms, Q3 2007”July 11, 2007

April 25, 2008

Application Server Quality: Review Your Assumptionsby John R. Rymer and Jeffrey S. Hammondwith Carey Schwaber, Mike Gilpin, Larry Fulton, Randy Heffner, David D’Silva

2

4

13

15

14

www.forrester.com

http://www.forrester.com/go?docid=45439&src=45729pdf




© 2008, Forrester Research, Inc. Reproduction ProhibitedApril 25, 2008

Application Server Quality: Review Your Assumptions For Application Development & Management Professionals

2

AS APPLICATION PLATFORMS EVOLVE, OLD REPUTATIONS DIE HARD

During the 10-year history of application servers, many application development managers have formed assumptions about the relative quality of products in the market.1 These assumptions have shaped vendors’ reputations and ultimately determined which platforms application development managers will and won’t use. For example, IBM WebSphere and BEA WebLogic earned reputations for scalability early in their histories, which helped to fuel the rise of those two application servers to prominence in large Web applications. By contrast, doubts that free, open source products could have similar quality to products that cost thousands of dollars have kept Red Hat’s JBoss and Sun’s GlassFish out of many large application projects.

The problem with popular wisdom is that it often outlives its usefulness. Products change, vendors merge, and innovations create a gap between assumptions and reality. We sometimes see this gap reflected in our inquiries with clients. Consider a recent example:

“I used to be a fan of [application server vendor X], but now I have begun recommending against using its product.”

For this client, critical bugs in the vendor’s latest release compromised a vital application project. Popular wisdom awards the same vendor a stellar reputation for software quality, but on the basis of good customer experiences with earlier versions of the product rather than on its actual quality at present. The client that we interviewed is representative of the most demanding consumers of application platform software: These companies push application platforms to the limits of their reliability and scale and quickly find a product’s quality gaps and reliability issues. But as more and more businesses come to depend on application servers and related platforms for their critical applications, just about everyone must manage the reality that platform quality can vary over time. As application platforms evolve, three factors can drive quality in the wrong direction:

1. Product evolution means greater maturity but may also introduce new risk. The third version of an application platform is usually of higher quality, and hence is more stable, than the platform’s first version. But when vendors revamp portions of their platforms instead of evolving an established code base, they may introduce new and risky code that affects overall system quality. For example, in recent releases, IBM and BEA Systems both introduced new JMS providers that are much less mature than other core features in their respective products. The result? Portions of otherwise stable products can be buggy. In the client case cited above, a new version of a key J2EE subsystem in the vendor’s latest release was the source of the problems.

2. Platform bundles often contain troublesome dependencies. An application platform stack that incorporates several platforms, such as a portal server and message broker, may be easier to install and configure, but it can also suffer from poor module integration. And if separate teams at the vendor manage each module, release coordination can break down, creating stability problems in the platform bundle or even slowing the release of the master product to the pace of the slowest-moving component.

© 2008, Forrester Research, Inc. Reproduction Prohibited April 25, 2008


3

3. Corporate changes may disrupt high-performance teams. An organization’s overall context affects its product development teams. Is the company losing or attracting key talent? Has a merger changed corporate leadership? Have market shifts caused changes in priorities? These kinds of changes — whether negative or positive — eventually affect the quality and innovation level of products. In recent years, the JBoss team has had to adjust to being part of Red Hat, and now Oracle and BEA product development teams are facing similar consolidation challenges.

Vendors Are Doing A Good Job Of Managing Quality Risks

We’ve long known that even as vendors release new versions of their application platforms, market perceptions of these products haven’t changed much. To find out the truth behind these perceptions, Forrester and BZ Media surveyed 617 IT professionals, 68% of whom hold application development roles in their organizations, in the January 2008 Global Forrester Research/BZ Media Application Platform Quality Online Survey. We found that:

· New product releases show steady improvements in product quality. Our respondents rated the latest versions of IBM WebSphere and Oracle Application Server higher than they did earlier versions. The investments that individual application server vendors have made in their latest products appear to have leveled out differences in perceived quality. There were differences evident in the experiences of respondents, but these differences were too granular to use when judging perceived overall quality. For example, Red Hat’s JBoss v4 met or exceeded quality expectations, yet users considered it harder to configure than other Java EE Servers. And while users were more likely to encounter new critical defects in the latest version of IBM WebSphere, this was also the product that they were most likely to say would perform and be reliable under their greatest workloads.

· Application servers are much more widely used than adjacent product categories. We asked respondents to tell us about their quality experiences with the platforms they know best. Their answers revealed that they had far more expertise working with application servers, such as BEA WebLogic Server and Microsoft Windows Server 2003, than they had working with portal servers, integration servers, and enterprise service buses (ESBs). This finding is surprising and suggests that the skills needed to implement advanced services, workflow, and information workplaces are not yet common in IT organizations.2

· Survey respondents are most familiar with Microsoft, IBM, Oracle, and BEA. Although respondents might have been familiar with half a dozen application server platforms, we asked them to tell us about the three they knew best. Nearly half of the time, Microsoft’s application platform products were among those three (see Figure 1). When it comes to Java platforms, our respondents were the most familiar with IBM (16%), outranked only by the soon-to-be-combined entity of BEA and Oracle (21%).



4

Figure 1 Survey Respondents Were Most Familiar With Products From Three Vendors

DEBUNKING FIVE APP SERVER MYTHS

Our research challenges five widely held beliefs, or “myths,” about the application platform market.

Myth No. 1: There Are Big Quality Differences Among App Server Products

Many application development managers believe that the big vendors’ products are much better than those of the smaller vendors, including open source providers. By “much better,” they mean

“scales better” and/or “is more reliable,” among other characteristics (such as “easier to develop for”). The availability of high-quality open source app servers began challenging these beliefs three years ago. In fact, there are small quality differences among the major application servers; our survey found that:

· Most vendors are meeting customer expectations. Overall, vendors’ products meet users’ expectations for reliability, availability, and serviceability. In particular, Red Hat, the parent company of JBoss, registered no “does not meet expectations” comments from respondents, a notable accomplishment since all other vendors registered at least some negative comments (see Figure 2). Overall, just 6% of respondents felt that their application server product did not meet quality expectations.

Source: Forrester Research, Inc.45729

Vendor Frequency

“Of the products you are able to discuss, please choose up to three products about which you know the most regarding reliability and support.”

Microsoft

IBM

BEA Systems

Oracle

Red Hat JBoss

Sun Microsystems

SAP

543

184

125

109

80

57

21

Base: 617 IT professionals(multiple responses accepted)

Source: January 2008 Global Forrester/BZ Media Application Platform Quality Online Survey

Java total Oracle-BEA total

576 234



5

· Just three application servers stand apart. While all application platforms vendors meet the majority of end users’ expectations, the latest versions of IBM’s and Oracle’s application servers and Red Hat’s JBoss Application Server v4 stand out. Respondents were significantly more satisfied with IBM WebSphere v6 and Oracle Application Server 10g R3 than with previous versions, in part because the newer versions have simplified installation and administrative workflows.3 Low bug counts are the likely cause of Red Hat’s JBoss Application Server v4’s high ratings.4

· Critical defects cluster around a few hot spots. Regardless of which products they were using, respondents cited memory leaks as the most important defects affecting application server quality. For Java servers, users were most likely to report errors with Java Message Service (JMS) and Java Management Extensions (JMX) subsystems, though they also commonly reported database connection pooling problems. On the other hand, users of Microsoft products reported frequent patches and updates as the most common cause of critical errors due to incompatible integrations and security issues.

Figure 2 App Dev Professionals Are Slightly More Satisfied With IBM And JBoss App Servers


“How would you characterize its quality now? By quality, we mean: Does the product work as advertised and meet your expectations for reliability, availability, and serviceability.”

Red Hat

SunMicrosystems

Microsoft

BEA Systems

IBM

Oracle

JBoss v4

WS 2003 w/ .NETFramework forBusiness Apps

Windows Server SP2

OAS 10g R3

WLS 8.1

WAS 6.x

WAS 5.x

OAS 10g R1

N=59

N=79

N=54

N=47

N=50

N=60

N=197

N=237

N=57

N=125

N=109

N=184

N=80

N=543

Exceeds Meets Does not meet

Vendor Product

33%

33%

28%

24%

23% 72%5%

70%

6%60%

12%57%

68%10%

5%20% 75%

34%

35%

15%

32%

22%

22%

23%

18% 77%

5%

74%3%

70%8%

62%

64% 14%

6%

76%

56%

9%

9%

66%


Base: IT professionals most able to discuss the quality of a given vendor’sproducts and their organizations’ experience with them

(multiple responses accepted)



6

Myth No. 2: Bugs Are More Important Than Configuration Errors

When application development professionals think about problems that decrease app server reliability, availability, and serviceability, they usually focus on product bugs as the primary culprit. Actually, configuration errors, not bugs, are the most common cause of project-busting problems. Specifically, our survey revealed that:

· Configuration errors are more numerous than bugs. About half the respondents reported that application platform configuration errors were a more frequent problem than bugs, while only about one in 10 reported the reverse to be true (see Figure 3-1). Why is this the case? Even modest applications require application development professionals to configure connections to security, directory, and identity management as well as other data sources. Large-scale applications raise the bar further by adding additional configuration challenges such as cluster configuration, failover support, and load balancing.

· Bugs aren’t a problem for most JBoss users . . . Only 10% of respondents reported that JBoss Application Server 4.x had introduced new critical bugs — the best result among the surveyed major platforms. JBoss Application Server 4.x also had the lowest reported rate of noncritical bugs that compromised an application project. Respondents were most likely to have a problem with new critical bugs when using Oracle Application Server 10g R1; 26% reported that this server introduced new critical bugs. Fortunately for Oracle users, the proportion of users reporting critical bugs in Oracle App Server 10g R3 is about half as high; only 15% reported that the R3 release introduced critical bugs (see Figure 3-2).

· . . . but JBoss is more likely to have configuration errors. While JBoss Application Server 4.x appears to be good quality code, users reported difficulty configuring the product: A whopping 45% of respondents reported reliability problems with JBoss Application Server 4.x due to configuration errors. Among the most up-to-date application platform options, only IBM WebSphere Application Server v6.x comes close to this level, with 39% of respondents reporting configuration errors (see Figure 3-3). So while JBoss is easier to configure than older products such as IBM WebSphere 5.x and BEA WLS 8.1, it’s falling behind Oracle Application Server 10g R3 and Microsoft Windows 2003 SP2 with the .NET Framework.

While many application development professionals have told Forrester that IBM WebSphere Application Server is notoriously difficult to configure, we haven’t heard as many complaints about JBoss Application Server. But JBoss clearly has some work to do to make its flagship product easier to configure.



7

Figure 3 Configuration Issues Cause More Reliability Problems Than Bugs Do


“Which is the more frequent problem: critical bugs or problems caused by installation/configuration issues?”

3-1

Respondents who had new releases of the platform introduce new critical bugs3-2

Respondents who had experienced reliability problems with the software that incorrectinstallation or configuration settings caused

3-3

BEA Systems

Red Hat

IBM

Microsoft

Oracle

Sun Microsystems

Installation/configuration issues Critical bugs Neither was a problem

N=125

N=89

N=184

N=543

N=109

N=57

57% 14% 29%

57%

55%

40%8%52%

43%

39% 14% 47%

16% 41%

15% 30%

1%42%

JBoss v4

JBoss v4

WS 2003 w/ .NET Framework for Business Apps

WS 2003 w/.NET Framework

for Business Apps

Windows Server SP2

Windows Server SP2

OAS 10g R3

OAS 10g R3

WLS 8.1

WLS 8.1

WAS 6.x

WAS 6.x

WAS 5.x

WAS 5.x

OAS 10g R1

OAS 10g R1

26%

23%

20%

19%

17%

16%

15%

10%

(N=50)

(N=50)

(N=53)

(N=53)

(N=60)

(N=60)

(N=80)

(N=80)

(N=197)

(N=197)

(N=237)

(N=237)

(N=47)

(N=47)

(N=58)

(N=58)

52%

51%

45%

40%

39%

38%

35%

23%


Base: IT professionals most able to discuss the quality of a given vendor’sproducts and their organization’s experience with them

(multiple responses accepted).



8

Myth No. 3: Microsoft Windows Server Is An Inferior Option To Java App Servers

Many application development managers believe that J2EE/Java EE application servers are better than Microsoft’s application server at meeting the demands of large enterprises and see Microsoft’s products as better suited to moderate workloads and departmental applications. The reality is not so clear: When we measure quality by looking at the proportion of users reporting critical bugs, the proportion of users reporting configuration errors, and overall customer satisfaction, Microsoft platforms fare well in enterprises and SMBs alike. And like IBM and Oracle platforms, Microsoft platforms are getting better as Microsoft releases newer versions. Just more than 14% of survey respondents rated multiple versions of Windows Server and the .NET Framework. They reported that:

· Windows Server 2003 SP2 shows improved fit and finish. Respondents reported few critical defects for Windows 2003 and even fewer for Service Pack 2 (SP2). Sixty-three percent of dual users encountered no critical defects while using the .NET Framework with Windows Server 2003. With SP2, this percentage rises to 70% (see Figure 4-1). Likewise, while 26% of the responding application development professionals reported that Windows 2003 contained a quantity of noncritical bugs that prevented their staff members from reaching their custom application goals, just 19% reported as much for SP2 (see Figure 4-2).

· Windows Server 2003 SP2 is easier to install and configure than Windows 2003. While 44% of users familiar with multiple versions of Windows Server experienced configuration problems with Windows 2003, only 31% encountered similar issues with Windows Server SP2 (see Figure 4-3). While only 36% of these respondents felt that neither defects nor configuration issues were a challenge with Windows 2003, 54% were able to say as much about SP2 (see Figure 4-4).

· .NET Framework also gets high quality marks. It’s no surprise that .NET Framework users reported that they felt satisfied with the product, considering how few of them reported encountering critical defects or configuration problems. In fact, a miniscule 2% thought that the quality of Microsoft products is declining. These dual users also show high confidence in both versions of Windows 2003 with the .NET Framework; for both products, almost 90% of respondents indicated that they felt confident that the product gives confidence that it can be reliable and perform well under the largest workloads (see Figure 4-5).

Users of multiple versions of Microsoft Windows Server expressed high confidence in the likely scalability of Microsoft’s platforms. This finding suggests that Microsoft’s application server is more scalable than its reputation says it is. But a note of caution: These results do not necessarily speak to the reliability of Microsoft’s platforms for the largest and most complex applications. It would be incorrect to assume that just because our survey respondents were satisfied with Microsoft Windows Server that others would necessarily feel the same way.



9

Figure 4 Microsoft Windows Server Got High Marks For Quality


Respondents who experienced a quantity ofnoncritical bugs that prevented their staff from reaching its goals with a custom application

4-2“How many critical bugs has your staffencountered during the time of use?”

4-1

“Which is the most frequent problem: criticalbugs or problems caused by configuration/installation issues?”

4-4Respondents who experienced reliabilityproblems that were caused by incorrectinstallation or configurations settings

4-3

Respondents who felt the quality of the product gives confidence that it can be reliable andperform well under the largest workloads

4-5

63%0

1

2-5

>5

70%

14%13%

19%12%

4%5%

26%

19%

Base: 86 IT professionals who were using Windows SP2 and Windows Server 2003 with .NET Framework

44%

31%

8%8%

56%38%

36%54%

Critical bugs

Configuration/installation issues

Neither was aproblem

89%

89%

Windows Server 2003 SP2

Windows Server2003 SP2



Windows Server 2003with .NET Framework









10

Myth No. 4: All Except The Largest Vendors Are Doomed

The largest vendors typically dominate mature product categories such as application servers, and our survey results mirror this trend: 527 out of 617 respondents (85%) evaluated at least one Microsoft, IBM, or Oracle/BEA product. But while application development professionals certainly know large vendor products well, there is still room in this market for small vendors. The big vendors are strong, but:

· SAP is a minor presence in the application platform market . . . Just 21 respondents indicated that SAP was one of the three app servers with which they were most familiar. To be fair, SAP primarily targets large enterprises, and our respondents hailed from a broader range of organizations. But the absence of SAP app server users in our survey indicates that SAP has not yet translated its position as the fourth largest software company in the world to a strong position in the market for application servers and related platforms for custom application development.

· . . . because SAP NetWeaver users know other platforms better. Our research indicates that SAP NetWeaver is a second choice app server platform. Forty-two of our 617 respondents felt qualified to discuss reliability and product support for SAP NetWeaver. But when it came time to select the products they knew the most about, these 42 respondents selected NetWeaver only one-fifth of the time (see Figure 5). Respondents were far more likely to select Microsoft products and almost as likely to select either IBM or BEA products than they were to select SAP NetWeaver.

· Sun and Red Hat users are satisfied with their choices. Red Hat and Sun Microsystems — the “smaller vendors” in our survey — earned fewer negatives responses than the major vendors.5 Almost half of the application development professionals that use Sun’s Java Application Server (the commercial version of Sun’s Glassfish open source product) told us that their opinion of Sun had improved as a result of their experiences with the product. And the low rate of reported experiences with bugs and configuration errors for Sun Java Application Server supports this finding.

Users of Red Hat’s JBoss were about three times less likely to form a negative opinion of their app server vendor than those who reported using application server products from the largest vendors. JBoss’ users reported that JBoss’ high product quality and strong support experience outweigh its issues with product configurability. The bottom line is that as long as Red Hat and Sun keep their users pleased, they are more likely to gain share than lose it.



11

Figure 5 The Three Biggest Vendors Are Strong, But Users Like Small Players, Too

Myth No. 5: Open Source Support Is Riskier Than Conventional Support Models

Many application development professionals and even more of their IT operations partners believe that using open source software in production is a lot riskier than using commercial products. Why? They believe a large vendor is more likely to provide a “single throat to choke” for timely support than is a small provider backed by an open source community. Our research shows that conventional support is not necessarily better than open source support. Users of Red Hat’s JBoss product line expressed satisfaction that is in line with the sentiments of people who use other commercial application platform products. In particular, our survey showed that:

· Timely, useful fixes to JBoss problems are the norm. Red Hat’s overall success rate at providing product fixes is in line the success rates of other vendors: Seventy percent of respondents reported that Red Hat provided timely fixes to their problems. Moreover, JBoss users are more likely to be critical of their experiences with other vendors when asked to compare them with their JBoss experience; only 57% of JBoss users felt that other vendors provided fixes on a timely basis (see Figure 6-1). And when it comes to fixes actually solving the reported problem, JBoss users were twice as likely (36% versus 18%) to report that a competing vendor was unable to fix a defect (see Figure 6-2).


28

18

15

10

7

3

21

Microsoft

Microsoft

IBM

IBM

BEA Systems

Oracle

OracleSun Microsystems

Sun Microsystems

Red Hat

Red Hat

SAP SAP

Base: 42 IT professionals able to discuss the quality ofSAP products and their organizations’ experience

with them(multiple responses accepted)

Base: IT professionals most able to discuss thequality of the respective vendor’s products and

their organizations’ experience withthem

(multiple responses accepted)

“Please choose up to three products* aboutwhich you know the most regarding liabilityand support.”

5-1 “How has your experience with the qualityof vendor(s) product(s) changed youropinion of that vendor?”

5-2

Improved No change Decline

46%

42%

39%

39% 46%

36%

37%

50%

52%

14%

15%

49%12%

56%3%

47%7%

(N=57)

(N=79)

(N=183)

(N=109)

(N=126)

(N=22)

(N=543)34% 57%9%

10%


BEA Systems

*Products rolled up into vendors in this graph.



12

· Most JBoss users are confident in the product’s ability to handle large workloads. Eighty-six percent of JBoss users are confident that it can handle their largest workloads. This puts JBoss squarely in the middle of the application server pack (see Figure 6-3).

Figure 6 JBoss’ Quality And Support Are As Good Or Better Than Those Of Conventional Firms


Respondents who felt that the quality of the product gives confidence that it can be reliable andperform well under the largest workloads

6-3

6-1 Respondents who felt the fix did not solve all the problems

6-2Respondents who felt the vendor provided the fix to their problem in a timely fashion

6-1

JBoss v4

WS 2003 w/ .NET Frameworkfor Business Apps

Windows Server SP2

OAS 10g R3

WLS 8.1

WAS 6.x

WAS 5.x

OAS 10g R1

70%

91%

90%

88%

87%

86%

84%

84%

83%

57%

Red Hat JBoss

Other appplatforms

18%

36%

Red Hat JBoss

Other appplatforms

Base: 74 IT professionals using at least oneRed Hat application server product.

Base: 74 IT professionals using at least oneRed Hat application server product.

Base: IT professionals most able to discuss the quality of the respective vendor’s products and theirorganizations’ experience with them.


(N=79)

(N=187)

(N=61)

(N=54)

(N=59)

(N=50)

(N=237)

(N=47)



13

R E C O M M E N D A T I O N S

USE FOUR PRACTICES TO GUIDE PLATFORM REASSESSMENTS AND SELECTIONS

Application development professionals periodically reassess their application server options for a variety of reasons, quality being one of those. In addition to reassessing their positions on the five myths, our research suggests four practices that will contribute to a successful selection process. When selecting application server products, application development professionals should:

· Look beyond the major vendor’s products. In our research, open source looks better than ever. Like the license cost and features of the two major open source options? You’ll find similar quality and support with both Red Hat on JBoss and Sun on Glassfish. JBoss has been a factor in the application server market for many years; Glassfish is an up-and-comer.

· Build communication bridges to the IT operations staff. Many application development professionals focus on development and put off planning for the realities of deployment until the end of the development cycle. As a result, it’s common for configuration errors to first surface when a new app version goes into production. Application development professionals would better serve their own interests by investing in coordination with IT operations to prevent installation and configuration errors before they compromise tight application development schedules.

· Budget extra time when implementing advanced platform features. If you plan to use advanced capabilities such as JMS and JMX, schedule extra time in your development schedule to deal with the defects and problems you will likely encounter. You should expect problems to subside as these platform subsystems mature over a series of product releases, but in the near term, using them will demand extra scrutiny and effort. In fact, the added quality risks of new, advanced features make them most appropriate for early adopters.

· Make the quality argument for software upgrades. Businesspeople sometimes question the value of product upgrades, but our research shows that application platform upgrades can provide clear quality benefits. The latest versions of IBM WebSphere, Oracle Application Server, and the .NET Framework scored better across the board than their predecessors. For Oracle and IBM, the differences were substantial. Make sure that the people objecting to the costs of an upgrade understand the potential quality benefits. On the other hand, user reports that Oracle Application Server 10g R3 was much better than 10g R1 suggest that it is better to wait for the “final” member of a rolling release program than to adopt the first release in a series.



14

W H A T I T M E A N S

THIS SNAPSHOT HINTS AT THE NEXT GENERATION’S LIKELY QUALITY CHALLENGES

It’s tempting to interpret our research as an indication that application server software has become a commodity for most buyers. But it would be a mistake to conclude that all future application server releases will also be commodities that introduce little new functionality and merely build on today’s quality levels. In fact, application server software is poised to enter a new era; expect big changes and new quality risks across application platform products. Much about the coming generation of application servers will be new and risky:

· New containers will support new programming models. All of the application platform vendors are moving to accommodate an explosion of new programming models and languages on their platforms. Some of the Java providers will accommodate new languages by adding new containers rather than using either J2EE Servlets or Enterprise JavaBeans (EJBs). Microsoft’s Oslo initiative is likely to provide a new and different technical foundation for .NET.

· New foundations promise improved configurability. The Open Services Gateway Initiative’s (OSGi) framework will form the foundation for next-generation Java application servers. OSGi offers a new life–cycle management model for software components, a service registry, and an execution environment designed for configurability and remote management. Some vendors incorporate parts of OSGi today, but not as their architectural cores. Converting to an OSGi technology foundation will require vendors to modify core subsystems, creating a potential risk of instability during the transition.

· New architectures will embrace SOA and RIA. The current generation of application servers is ideal for hub-and-spoke architectures featuring central control points and on-premise deployment. The new generation will need to support massively distributed architectures with federated control and deployment either on-premise, in a hosting center, or both. Why? These patterns are typical in emerging service-oriented architecture (SOA) and rich Internet applications (RIAs).

· New business models will embrace communities and cloud computing. The success of JBoss in particular and open source in general is changing business models, which will affect how vendors build products and provide production support. Our research suggests that Sun Microsystem’s embrace of open source has lit a fire under its formerly laggard application server products. And even Microsoft has embraced increased code transparency to add the Python and Ruby languages to its portfolio under a shared-source license model.

In short, the lessons of our survey apply to the application server software of today but serve only as a map of the new generation’s potential issues. As the new generation of application servers starts arriving, get ready to turn your product-quality filters to maximum strength.



15

SUPPLEMENTAL MATERIAL

Methodology

In the January 2008 Global Forrester/BZ Media Application Platform Quality Online Survey, we conducted a survey of approximately 617 IT professionals. Forrester surveyed 85 individuals with the following job titles: developer, project manager, director of application development, and architect. BZ Media provided the approximately 532 respondents. Roughly 83% of the respondents were located in the U.S., with the remainder spread across the globe in more than 130 countries. Additionally, respondents varied across IT functions, company sizes, and industries. Forrester motivated all 617 respondents by entering each participant in a raffle to win one of four $50 Hallmark Insight gift certificates as well as by promising to give each participant a copy of the research at its completion.

In addition to sampling error, one should bear in mind that the practical difficulties in conducting surveys can introduce error or bias into the finding of opinion polls. Other possible sources of error in polls are probably more serious than theoretical calculations of sampling error. These other potential sources of error include question wording, question ordering, and nonresponse. As with all survey research, it is impossible to quantify the errors that may result from these factors without an experimental control group, so we strongly caution against using the words “margin of error” in reporting any survey data.

These statements conform to the principles of disclosure of the National Council on Public Polls.

ENDNOTES1 The year 2008 marks a decade since the initial announcement of Sun Microsystems’ JPE project, which

directly led to J2EE 1.2 and the modern application server.

2 The sample sizes for the major application servers (except for SAP NetWeaver) were large enough to be representative of the broad market. For most of the portal servers, integration servers, and ESBs we included in our survey, the sample sizes were too small to be projectable to the broader market. The exceptions were IBM WebSphere Portal and two releases of Microsoft’s SharePoint portal technology.

3 IBM focused on making many installation and ease-of-use improvements in WAS 6.0. For more information about the improved deployment workflows and new administrative clients, see “What is new for administrators” on the IBM Web page (http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/welc_newadministrator.html). Likewise, one of the key new features in OAS 10g R3 is a set of simplified operational and administrative workflows. For more information see “Oracle Application Server 10g R3

(10.1.3.1) New Features Overview,” Oracle, October 2006 (http://www.oracle.com/technology/tech/java/oc4j/10131/OracleAS-NF-10131.pdf). Oracle rolled out the 10g release of its Application Server in three subreleases — 10g Release 1 (10g R1), 10g R2, and 10g R3 — each of which introduced a portion of the new features planned for the release.



16

4 Sixty-six percent of Red Hat JBoss v4 users reported that they encountered zero critical defects. The Java EE application server to come closest to this score was Oracle Application Server 10g R3, with 53% of users reporting no critical defects.

5 At more than $13 billion in annual revenue, Sun Microsystems is a large company. Sun does not report how much of this revenue it obtains from sales of application platform software, but we believe that this revenue is only a small portion of the total. In addition, most large corporate buyers of application server software consider Sun to be a second-tier supplier.

Forrester Research, Inc. (Nasdaq:

FORR) is an independent

technology and market research

company that provides pragmatic

and forward-thinking advice to

global leaders in business and

technology. For more than 24 years,

Forrester has been making leaders

successful every day through its

proprietary research, consulting,

events, and peer-to-peer executive

programs. For more information,

visit www.forrester.com.

Australia

Brazil

Canada

Denmark

France

Germany

Hong Kong

India

Israel

Japan

Korea

The Netherlands

Switzerland

United Kingdom

United States

Headquarters

Forrester Research, Inc.

400 Technology Square

Cambridge, MA 02139 USA

Tel: +1 617.613.6000

Fax: +1 617.613.5000

Email: [email protected]

Nasdaq symbol: FORR

www.forrester.com

M a k i n g L e a d e r s S u c c e s s f u l E v e r y D a y

For a complete list of worldwide locations,visit www.forrester.com/about.

Research and Sales Offices

45729

For information on hard-copy or electronic reprints, please contact the Client

Resource Center at +1 866.367.7378, +1 617.617.5730, or [email protected].

We offer quantity discounts and special pricing for academic and nonprofit institutions.

www.forrester.com