tool benchmarking 2013 - research2guidance€¦ · in terms of “design features”, air and...

© research2guidance 2013

Single User License: Access restricted to one specific user

A comparison of 10 leading tools for

multi-platform app development

October 2013

Detailed Cross PlatformTool Benchmarking 2013

©research2guidance | October 2013

Single User License

2

Detailed Cross Platform Tool Benchmarking 2013

2

About research2guidance

research2guidance is a strategy advisor and market research company. We concentrate on the

mobile app eco-system. Our service offerings include:

App Strategy: We help our clients in and outside of the mobile industry to develop their app

market strategy. Our consulting advisory projects are based on a set of predefined project

approaches including: App strategy development, App Evaluation, App Market Segment Sizing,

App Governance and App Marketing Spend Effectiveness.

App Market Reports: Our app market reports explore the major trends and developments

affecting the app markets. Separate research papers provide both general and specific coverage of

the market. The reports contain key insights for companies looking to enter or deepen their

engagement with the mobile applications market, providing data and analysis on all relevant

aspects of the market to ease investment decision-making.

App Market Surveys: We leverage our 70,000 app eco-system database to conduct surveys and

reports for our clients.

research2guidance , Berlin, Germany, +49 (0)30 609 89 33 60

www.research2guidance.com

http://www.research2guidance.com/


Single User License

3


3

Related reports and services:

Click on the covers to get more information

Need help with finding the right Cross-Platform Tool ? Use our standardized CP

Tool selection process to find the right tool for your app projects.

Contact the analyst Joachim Thiele-Schlesier: +49 (0) 30 609 89 33 60,

[email protected]

Cross-Platform Tool Benchmarking 2013: “The hidden champions of the app economy”

Marmalade Profile: “Service offering and user rating”

Xamarin Profile: “Service offering and user rating”

Corona SDK Profile: “Service offering and user rating”

Unity 3D Profile: “Service offering and user rating”

?

mailto:[email protected]

http://www.research2guidance.com/shop/index.php/cross-platform-tool-benchmarking-2013

http://www.research2guidance.com/shop/index.php/marmalade-profile

http://www.research2guidance.com/shop/index.php/xamarin-sdk-profile

http://www.research2guidance.com/shop/index.php/corona-sdk-profile

http://www.research2guidance.com/shop/index.php/unity-3d-profile


Single User License

4


4

CONTENT

1. Key takeaways ............................................................................................................................................................ 5

2. Benchmarking methodology ...................................................................................................................................... 8

3. The cross platform tool coverage of the benchmarking study .................................................................................. 9

3.2. Air by Adobe ............................................................................................................................................................. 10

3.3. JQuery Mobile by JQuery Foundation ...................................................................................................................... 11

3.4. Marmalade by Marmalade ....................................................................................................................................... 11

3.5. Unity 3D by Unity Technologies ............................................................................................................................... 11

3.6. Xamarin by Xamarin ................................................................................................................................................. 12

3.7. Titanium by Appcelerator ........................................................................................................................................ 12

3.8. Corona SDY by Corona Labs ..................................................................................................................................... 12

3.9. Sencha Touch by Sencha .......................................................................................................................................... 13

3.10. Construct 2 by Scirra ................................................................................................................................................ 13

4. User background comparison .................................................................................................................................. 14

5. Usage intensity comparison ..................................................................................................................................... 15

6. Comparison of targeted industries and app categories ........................................................................................... 16

7. Cross platform tool complexity comparison ............................................................................................................ 19

8. Support service quality comparison ......................................................................................................................... 22

9. Platform support comparison .................................................................................................................................. 24

10. App project duration and time savings comparison ................................................................................................ 27

11. User satisfaction with tecnology features ................................................................................................................ 29

11.1. Cloud API services .................................................................................................................................................... 29

11.2. Accessibility of device hardware features................................................................................................................ 31

11.3. Accessibility of pre-installed applications ................................................................................................................ 32

12. App Quality benchmarking ....................................................................................................................................... 33

12.1. Design feature comparison ...................................................................................................................................... 33

12.2. Usability comparison against native apps ................................................................................................................ 34

12.3. Performance comparison against native apps ......................................................................................................... 35

12.4. App security comparison ......................................................................................................................................... 37

12.5. App revenue comparison ......................................................................................................................................... 39

13. Cost-performance ratio comparison ........................................................................................................................ 40

14. Appendix .................................................................................................................................................................. 42

About the authors ............................................................................................................................................................. 42

List of figures and tables ................................................................................................................................................... 44

Contact research2guidance .............................................................................................................................................. 45


Single User License

5


5

1. KEY TAKEAWAYS

Over the course of the last five years, multi-app and multi-platform app publishing have become

common for developers who have become increasingly aware of second-tier platforms like

BlackBerry and Windows. The result has been an increase in complexity, as well as in development

& maintenance costs.

Cross Platform Tools (CP Tools) promise to reduce the complexity and costs it takes to develop

apps for multiple mobile, desktop and other platforms (in-car devices, TV, game consoles).

The CP Tool landscape has grown over the course of the last 2-3 years and it has become

increasingly difficult to select the right platform for an app project or even a portfolio of apps.

This report compares 10 of the most used CP Tools along a set of 16 criteria which span from

platform support to overall cost-efficiency.

The comparison is based on the results of the largest developer survey, with more than 1,000

participants, and was conducted in summer 2013, only addressing the strengths and weaknesses

of CP Tools in the market.

In summary, the results show that CP Tools have a significant impact on development time, costs

and app quality. But there are major differences between the CP Tools, of which every app

developer must be aware of before selecting a CP Tool for his project.

The majority of CP Tools have not made it into the enterprise world. Titanium, Unity 3D, Phone

Gap and JQuery Mobile have the highest share (10%-20%) of users coming from large companies.

CP Tools in the benchmarking can be generally divided into two groups: CP Tools with a focus on

the games industry and generalists. Corona SDK, Marmalade, Unity 3D, Construct2 and AIR have a

focus on game development. CP Tools that cater the games industry are in general of rather high

complexity and thus require a longer time to get used to.

Support channel usage does not show significant differences amongst CP Tools. Users seem to

have similar preferences. In contrast, the quality of the support services is rated very differently.

Xamarin, AIR and Sencha Touch offer the best support service quality.

Overall, CP Tools are most frequently used for developing mobile apps for Android, iOS and

Blackberry 10. App development for HTML targeting mobile and HTML targeting desktop follow

with distance.


Single User License

6


6

The highest time-savings for app development can be realized with Unity 3D, Corona SDK and

Xamarin. For all three tools more than 50% of their users were able to develop more than 50%

faster.

CP Tools differ significantly in how intensively developers are using or rating APIs, pre-installed

software and hardware features made accessible by CP Tools.

Across all users of CP Tools, the availability of cloud API services is rated of rather low importance.

AIR and Xamarin clients state the highest satisfaction with cloud API services. 30% of Xamarin

users and 18% of AIR users make extensive use of the services.

Across all CP Tools in the benchmarking, 60% of users regard the accessibility of device hardware

features as “critical”. Xamarin users are the most satisfied with the service of their tool: 69% are

satisfied or very satisfied with the accessibility of device hardware features.

Across all CP Tools, the accessibility of pre-installed applications is rated by 40% of users as

“important”. Xamarin users are the most satisfied with the service. Titanium users are the least

satisfied.

The results for all 3 benchmarked technology features indicate that Titanium users are the most

demanding developers and view technology features most critically. They make the most use of

APIs, device hardware access and pre-installed apps, but are the least satisfied with the offering of

Titanium. On the other hand, Xamarin users tend to rate their tool support the most positively.

In terms of “design features”, AIR and Xamarin apps are rated best by far: 92% of AIR users and

89% of Xamarin users describe their cross-platform apps as good as or even better than native

apps in terms of design.

CP Tools overall are being rated positively by their users for the usability quality of their apps.

Xamarin is leading in usability as well.

Across all CP Tools in the benchmark, performance is the major weakness of cross-platform apps.

Marmalade, Xamarin and Unity 3D apps are rated best. Especially Marmalade users (88%) do not

see their apps as being of a lower quality than apps created with an OS-specific native SDK.

App security is rated the highest for Xamarin and Construct2.

In terms of generated app revenues, all tools are being rated positively compared to native app

development. AIR users rate their apps the highest, followed by Unity 3D and Marmalade users.

Overall, the cost-performance ratio is very high: on average 69% or more of the users rate the

cost-performance ratio “okay” or “good value”. Construct2 and Marmalade cost-efficiency ratings

are the highest, with more than 50% of their users saying that the CP Tool has a good value.


Single User License

7


7

By comparing user ratings of 10 of the most used CP Tools, the report aims at providing valuable

insights for the selection of the best CP Tool for a specific app project or portfolio.


Single User License

8


8

2. BENCHMARKING METHODOLOGY

This benchmarking of cross-platform app development tools is based on a global survey conducted

between May and August 2013. More than 1,000 respondents participated and shared their views

on cross-platform app development tools (CP Tools).

Participants come mainly from EU (39%), North America (26%) and Asia (22%), The major CP Tool

user’s country of origin are the USA (15%), Canada (11%), India (11%), Germany (10%) and UK

(6%).

Figure 1: Geographical overview of cross-platform tool users

Tool user survey participants range from individual app developers to IT managers of multi-

national corporations.

39%26%

5%

22%

6%

2%


Single User License

9


9

Figure 2: Background of cross-platform tool users

3. THE CROSS PLATFORM TOOL COVERAGE OF THE BENCHMARKING STUDY

The benchmarking includes 10 of the most used CP Tools.

All listed CP Tools are integrated development environments (IDE) or toolkits for creation of web

apps.

CP IDEs allow multi-app/multi-platform app development. Most tools concentrate their output on

native apps but some also create web apps. These tools use their own SDK to develop a single

code faster and compile it to meet native requirements. Some CP IDEs specialize in certain genres

(mainly games) and/ or fewer platforms to get better results while others aim to cover as many

operating as possible.

Toolkits for creation of web apps support the development process of web apps and mobile

webpages. Developers can use classical web development language.

The comparison of the 10 CP tools is based on the ratings of their users. The accuracy and

informative value of the results depends on the number of user ratings of a particular CP Tool. The

number of user ratings varies between 10 and 85.

Survey participants by position in the company

Professional developer

58%

Business administrator

/ CXO / Management

21%

IT / telecom administrator

5%

Graphic designer

2%

Student / Learning

developer2%

Other12%

Survey participants by company size

Just me33%

Small business

41%

Medium business

13%

Enterprise13%

n=888


Single User License

10


10

Table 1: Benchmarked CP Tools and sample sizes

Even lower numbers of user ratings give a good indication of the strengths and weaknesses of a CP

Tool. Tools with user rating numbers below 30 are being marked in grey in the document to

facilitate interpretation of the results.

The following CP Tools have been included:

3.1. PHONE GAP BY ADOBE

Adobe made their initial steps in the CP Tool market by acquiring Phone Gap in 2011. Phone Gap is

a CP IDE and one of the most established CP Tools in the market.

Supported platforms: iOS, Android, Windows Phone, Blackberry 7.1 and lower, Symbian, Bada

App category focus: generalist

Supported features: accelerometer, camera, compass, contacts, file system integration, geo

location, media library, network, data storage, speaker, vibration

3.2. AIR BY ADOBE

Adobe Integrated Runtime (AIR) is a runtime environment for building rich internet applications

using Adobe Flash, Apache Flex, HTML and Ajax. AIR was released in 2008.

Supported platforms: iOS, Android, Kindle Fire, Windows (desktop), OSX (desktop), Facebook,

Blackberry Playbook

App category focus: games, generalist

Supported features: camera, file system integration, geo location, microphone, media library, NFC,

speaker, vibration

CP Tool Vendor Sample Size

Phone Gap Adobe 85

JQuery Mobile JQuery Foundation 57

AIR Adobe 55

Marmalade Marmalade 43

Unity 3D Unity Technologies 38

Xamarin Xamarin 31

Titanium Appcelerator 30

Corona SDK* Corona Labs 23

Sencha Touch* Sencha 14

Construct2* Scirra 10


Single User License

11


11

3.3. JQUERY MOBILE BY JQUERY FOUNDATION

JQuery Mobile is a touch-optimized web framework aiming for compatibility with a broad variety

of mobile devices. It was developed on top of JQuery by the JQuery foundation and first released

in 2006.

Supported platforms: iOS, Android, Windows Phone, Windows 8, Blackberry 7.1 and lower,

Symbian, Bada, HTML (targeting mobile)


Supported features: JQuery Mobile offers the full range of HTML5 standards for the supported

platforms

3.4. MARMALADE BY MARMALADE

Marmalade SDK, formerly known as Airplay SDK, was developed by Ideaworks 3D and released in

2008. The vendor is located in London (UK).

Supported platforms: iOS, Android, Windows Phone, Blackberry 10, Windows (desktop), OSX

(desktop), smart TVs

App category focus: games, news and media

Supported features: iOS and Android: accelerometer, geo location, camera, compass, landscape

orientation, microphone, speaker, multi-touch, image library, vibration, contacts, filesystem IO,

calendar, in-app email

3.5. UNITY 3D BY UNITY TECHNOLOGIES

Unity 3D by Unity Technologies, San Francisco (CA) is one of todays most used game engines

worldwide. It was first released in 2005 as an OSX supported game development tool. Today Unity

is a fully cross-platform capable IDE that caters mobile platforms, desktop platforms and consoles.

Supported platforms: iOS, Android, Windows Phone, Windows 8, Blackberry 10, Ubuntu, Windows

(desktop), Chrome, Facebook, Microsoft Xbox, Nintendo Wii, Sony PlayStation 3

App category focus: games

Supported features: Unity focusses on high performance 3D games that run on the supported

platforms.


Single User License

12


12

3.6. XAMARIN BY XAMARIN

Xamarin is a start-up, founded in 2011 and located in San Fransisco (CA). It was founded by the

developers of the Mono project. Mono was the parent of Xamarin Studio, part of the 2013-

released Xamarin 2.0.

Supported platforms: iOS, Android, Windows Phone, Windows 8, HTML (targeting mobile), OSX

(desktop)


Supported features: accelerometer, geolocation, vibration, camera, multitouch, landscape

orientation, compass, speaker, microphone, NFC, contacts, filesystem IO, calendar, image library,

in-app email, phone, SMS, MMS, speech recognition, maps

3.7. TITANIUM BY APPCELERATOR

Titanium was developed by Appcelerator and first released in 2008. Today it consists of Titanium

Mobile, an application framework for developing mobile apps with web technologies, Titanium

Studio, an open source SDK, Alloy, an Eclipse based IDE, and cloud services.

Supported platforms: iOS, Android, Windows Phone, Blackberry, Tizen, Windows (desktop), OSX

(desktop)


3.8. CORONA SDY BY CORONA LABS

Corona SDK is a cross-platform IDE first released in 2009. It was developed by Corona Labs, a start

up founded in 2008 and based in Palo Alto (CA).

Supported platforms: iOS, Android, Windows Phone, Windows 8, HTML (targeting mobile), OSX

(desktop)

App category focus: games, 2D apps

Supported features: accelerometer, geolocation, vibration, camera, multitouch, landscape

orientation, compass, speaker, microfone, NFC, contacts, filesystem IO, calendar, image library, in-

app email, phone, SMS, MMS, speech recognition, maps


Single User License

13


13

3.9. SENCHA TOUCH BY SENCHA

Sencha Touch is a Java Script library for building mobile web applications. Sencha, a 2008 start-up,

is based in Redwood City (CA). Sencha Touch is used by more than 50% of the Fortune 500 for

their app projects.

Supported platforms: iOS, Android, Windows Phone, Windows 8, Blackberry 10, Bada, Kindle Fire,

Chrome, HTML (targeting mobile)


Supported features: Sencha Touch offers the full range of HTML5 standards for the supported

platforms

3.10. CONSTRUCT 2 BY SCIRRA

Construct2 is a HTML5 based drag-and-drop editor for creating games without coding or with

usage of Java Script. It was first released in 2012 by Scirra Ltd.

Supported platforms: Construct 2 creates HTML5 apps that run in browsers, on desktops and web-

capable mobile devices.

App category focus: games


Single User License

14


14

4. USER BACKGROUND COMPARISON

CP Tool users reach from single developers to employees of multi-national enterprises. Single

developers and small businesses use all ten reviewed CP Tools.

9 out of 10 CP Tools are used by medium businesses as well. The share of medium businesses

using a particular CP Tool varies from 11% (Sencha Touch, Marmalade) to 22% (Phone Gap).

Marmalade has the highest share of users being single developers or coming from small

businesses (89%).

The majority of CP Tools have not made it into the enterprise world. Titanium, Unity 3D, Phone

Gap and JQuery Mobile have the highest share (10%-20%) of users coming from large companies.

Figure 3: CP Tool users by company size

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Phone Gap

Titanium

JQuery Mobile

Unity 3D

Corona SDK

Xamarin

AIR

Sencha Touch

Marmalade

Construct2

Single developer Small business Medium business Enterprise


Single User License

15


15

5. USAGE INTENSITY COMPARISON

The usage intensity of a CP Tool shows how often a developer uses a CP Tool to deliver an app

project. It is a good indicator for how the requirements of the applications can be realized with a

specific CP Tool.

Usage intensity of the respective CP Tool varies largely. On average CP tools are being used for

around 50% of all app projects.

AIR (ᴓ 69% of apps), Construct2 (ᴓ 47% of apps), Corona SDK (ᴓ 49% of apps), JQuery Mobile (ᴓ

69% of apps), Marmalade (ᴓ 68% of apps), Phone Gap (ᴓ 48% of apps), Sencha Touch (ᴓ 38% of

apps), Titanium (ᴓ 37% of apps), Unity 3D (ᴓ 58% of apps), Xamarin (ᴓ 55% of apps).

Marmalade (58%), AIR (55%) and Unity 3D (50%) have the highest share of users that develop

almost every app with the help of the CP Tool (70%-100%). Titanium (17%) has the lowest share of

users specializing on cross-platform development.

JQuery Mobile, Sencha Touch, and Titanium have the highest share of users which make use of the

tool only for a minor share of their app projects (less than 30% of app projects).


Single User License

16


16

Figure 4: CP Tool usage intensity by share of apps with CP Tool

6. COMPARISON OF TARGETED INDUSTRIES AND APP CATEGORIES

Selecting the right CP Tool also depends on the industry and app category the app publisher wants

to target. CP Tools sometimes offer industry or app category-specific features like design

templates or interfaces to standard software packages that are commonly used within an industry.

The comparison for which industries app developer make use of a CP Tool is a good indicator for

the quality of the industry-specific features of a CP Tool.

Currently CP Tools can be generally divided into two groups: CP Tools with focus on games

industry and generalists. Based on what the CP Tools have been used for, there is no CP Tool that

shows a strength for a certain industry.

The “gaming” tools are Construct2, Marmalade and Unity 3D with more than 60% of all app

projects being game projects. AIR and Corona SDK also seem to offer good app development

support for games as around 35% of all app projects resulted into games.

Generalists (with no industry focus) are JQuery Mobile, Phone Gap, Sencha Touch, Titanium,

Xamarin and JQuery Mobile.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

JQuery Mobile

Titanium

Sencha Touch

Corona SDK

Phone Gap

Unity 3D

Marmalade

Xamarin

AIR

Construct2

70%-100% 30%-70% up to 30% 0%


Single User License

17


17

Table 2: CP Tool users by industry focus

Which app category and tools developers use is also a good indicator of the performance of the

tools.

The CP Tools which have been mainly used for apps published in the game sections are AIR,

Construct2, Corona SDK, Marmalade and Unity 3D.

These CP Tools are also often used to develop apps for the education and entertainment category.

Generalist tools are used to develop for a number of different app categories with no focus

towards any particular category. This observation applies especially for Phone Gap, JQuery Mobile,

Sencha Touch and Xamarin (Table 3).

No

indu

stri

al f

ocus

Gam

es

Ret

ail

New

s /

Info

/ M

edia

pub

lishe

rs

Spor

ts

Food

/ B

ever

age

Hea

lthc

are

/ Ph

arm

a

Uti

litie

s

Aut

omot

ive

Fast

mov

ing

cons

umer

goo

ds

Fina

ncia

l

IT Oth

er

AIR 15% 35% 3% 9% 4% 3% 3% 13% 1% 0% 3% 1% 13%

Construct2 11% 78% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 11%

Corona SDK 19% 37% 7% 7% 0% 7% 7% 11% 0% 0% 0% 0% 4%

JQuery Mobile 29% 5% 1% 13% 5% 3% 5% 15% 3% 0% 3% 10% 9%

Marmalade 10% 71% 2% 2% 2% 0% 2% 5% 0% 0% 2% 0% 2%

Phone Gap 16% 7% 8% 8% 5% 5% 13% 13% 3% 1% 6% 8% 7%

Sencha Touch 18% 0% 11% 4% 4% 4% 11% 18% 4% 4% 18% 4% 4%

Titanium 26% 0% 6% 24% 0% 3% 6% 9% 0% 6% 9% 9% 3%

Unity 3D 2% 69% 4% 2% 0% 2% 2% 4% 4% 2% 2% 4% 2%

Xamarin 20% 6% 14% 9% 3% 3% 9% 14% 0% 0% 3% 9% 11%

No focus Moderate focus Strong focus

(<10% of app projects) (10-30% of app projects) (>30% of app projects)


Single User License

18


18

Table 3: CP Tool users by app category focus

Boo

ks

Bus

ines

s

Cata

logs

Educ

atio

n

Ente

rpri

se A

dmin

istr

atio

n

Ente

rtai

nmen

t

Fina

nce

Food

/ D

rink

Gam

es

Hea

lth

/ Fi

tnes

s

Life

styl

e

Med

ical

Mus

ic

Nav

igat

ion

Net

wo

rkin

g

New

s

Phot

o /

Vid

eo

Prod

ucti

vity

Ref

eren

ce

Spor

ts

Trav

el

Uti

litie

s

Wea

ther

Oth

er

Can´

t te

ll

AIR 3% 5% 2% 13% 1% 9% 0% 2% 27% 4% 2% 2% 2% 2% 0% 1% 5% 4% 0% 3% 1% 6% 2% 3% 5%

Construct2 7% 0% 0% 7% 0% 14% 0% 0% 64% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 7% 0%

Corona SDK 5% 5% 0% 21% 0% 12% 0% 5% 28% 2% 5% 0% 2% 0% 2% 0% 2% 5% 0% 0% 0% 2% 0% 2% 2%

JQuery Mobi le 3% 10% 2% 7% 5% 5% 1% 2% 4% 5% 8% 0% 1% 5% 2% 5% 3% 7% 6% 4% 3% 9% 0% 2% 5%

Marmalade 3% 2% 2% 2% 0% 14% 2% 2% 47% 3% 2% 0% 3% 2% 0% 2% 3% 0% 0% 2% 0% 5% 0% 3% 2%

Phone Gap 2% 9% 3% 7% 4% 4% 4% 2% 4% 7% 7% 5% 1% 3% 2% 5% 5% 4% 3% 2% 2% 5% 2% 2% 7%

Sencha Touch 5% 11% 5% 7% 5% 5% 5% 0% 0% 11% 9% 0% 0% 5% 9% 0% 2% 2% 7% 0% 5% 5% 0% 0% 5%

Titanium 0% 9% 3% 8% 4% 5% 4% 3% 0% 5% 8% 3% 4% 4% 4% 8% 4% 4% 3% 1% 5% 10% 1% 0% 5%

Unity 3D 3% 6% 1% 10% 3% 16% 1% 0% 43% 1% 4% 1% 0% 0% 3% 0% 1% 1% 0% 0% 3% 1% 0% 0% 0%

Xamarin 4% 8% 4% 6% 4% 6% 2% 4% 4% 4% 4% 2% 0% 0% 4% 4% 0% 10% 4% 0% 8% 12% 4% 2% 4%

No Focus Moderate Focus Strong Focus

< 7% of app projects 7%-20% of app projects >20% of app projects


Single User License

19


19

7. CROSS PLATFORM TOOL COMPLEXITY COMPARISON

An indicator for the complexity of CP Tools is the time to get familiar with the solution.

Besides Construct2 and Sencha Touch, the tools that are quickest to learn are JQuery Mobile and

Phone Gap. 39% of JQuery users master the tool within a couple of days and 64% within a couple

of weeks. 41% of Phone Gap users need just days to familiarize themselves with the tool and for

41% it took weeks.

Tools that require the longest familiarization time are AIR, Unity 3D, Marmalade and Corona SDK.

Between 3% and 15% of users of these tools need even years to learn.

Corona SDK, Marmalade, Unity 3D and AIR have a focus on game development. CP Tools that cater

the games industry are in general of rather high complexity and thus need a longer time to get

used to.

Across the generalists (JQuery Mobile, Phone Gap, Xamarin and Titanium) the necessary time to

learn tends to be shorter when compared to the CP Tools with a focus on game development.

However, Titanium is an exception: 24% of users need months to master the solution from

Appcelerator.


Single User License

20


20

Figure 5: Complexity benchmarking: time-to-familiarize

The overall complexity rating of a CP Tool is a broader indicator compared to the time it takes to

familiarize. It also includes the handling of the tool once the user has become acquainted with the

software.

The majority of users rate the complexity of the CP Tools comparable to other software tools

(average).

Marmalade is the tool with the highest complexity rating. 63% of its users rate the tool complexity

“high” or “very high”. Bearing in mind that 67% of Marmalades users claim to be professional

developers, this high complexity rating is clearly something to watch out for when selecting the

right CP Tool.

In contrast, Construct 2 (90%)1 and Corona SDK (38%) have been rated as the CP Tools with the

lowest overall complexity. This may be due to Corona’s programming language, which has a

reputation of being a rather comfortable programming language.

1 Small sample size might have influenced results for positive complexity rating of Construct 2

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

AIR

Unity 3D

Titanium

Marmalade

Corona SDK

Xamarin

Phone Gap

JQuery Mobile

Sencha Touch

Construct2

Days Weeks Months Years Can´t tell


Single User License

21


21

Figure 6: Complexity benchmarking: user rating complexity

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Marmalade

Titanium

Sencha Touch

Unity 3D

AIR

Phone Gap

Xamarin

JQuery Mobile

Corona SDK

Construct2

Very low / none Low Average High Very high


Single User License

22


22

8. SUPPORT SERVICE QUALITY COMPARISON

The most used service channel of the CP Tool users is online community support, established by

the vendors and external providers.

On site trainings and project support as well as online support are also channels that CP Tools

vendors offer and that are being used frequently.

Support channel usage does not show significant differences amongst CP Tools. Users seem to

have similar preferences.

Table 4: Support services usage

In contrast, the quality of the support services is rated very differently. Xamarin, AIR and Sencha

Touch offer the best support service quality.

Xamarin has the highest share of satisfied users. 93% of Xamarin users rate the support services as

“good” or “very good”. Their online community, on-site trainings or tutorials, as their most

popular support channels, seem to meet user demand very well.

Titanium, Marmalade and Phone Gap have the lowest share of users who are happy with the

support services.

Marmalade has the highest share of unsatisfied users. Marmalade is the only tool that has nearly

as many unsatisfied (34%) as satisfied (41%) customers. 9% of Marmalade users even rate the

support services as “very bad”. Marmalade support service channels do not seem to cope with the

On-

site

pro

ject

sup

port

On-

site

tra

inin

g /

tuto

rial

s

Phon

e su

ppor

t

Rea

l-ti

me

onlin

e su

ppor

t

Tim

e-de

laye

d on

line

supp

ort

Onl

ine

com

mun

ity

AIR 23% 23% 5% 2% 7% 73%

Construct2 25% 25% 0% 13% 0% 75%

Corona SDK 17% 28% 6% 6% 22% 78%

JQuery Mobile 17% 33% 6% 6% 6% 75%

Marmalade 24% 12% 9% 9% 45% 82%

Phone Gap 31% 20% 14% 5% 10% 66%

Sencha Touch 30% 40% 0% 20% 20% 90%

Titanium 12% 18% 6% 6% 12% 88%

Unity 3D 23% 35% 4% 4% 19% 85%

Xamarin 8% 31% 8% 15% 38% 92%


Single User License

23


23

help requests made by users. Companies that have limited access to app development expertise

should pay the highest attention to this point as Marmalade is also rated as the most complex CP

Tool.

Figure 7: Support services quality rating

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Titanium

Marmalade

Phone Gap

Construct2

JQuery Mobile

Corona SDK

Unity 3D

Sencha Touch

AIR

Xamarin

Very good Good Sufficient Insufficient Very bad


Single User License

24


24

9. PLATFORM SUPPORT COMPARISON

CP Tools offer a broader or narrower support for mobile, desktop and additional app platforms

(TV, game consoles, in-car devices). The intensity with which the developers use a specific CP Tool

to develop for one of the supported platforms can be used as a further benchmark for the quality

of the tools.

Overall, CP Tools are most frequently used for developing mobile apps for Android, iOS and

Blackberry 10. App development tools for HTML targeting mobile and HTML targeting desktop lag

behind.

CP Tools that are the most widely used to develop for different platforms are Marmalade, Air and

Unity 3D.

Table 5: Usage intensity of CP Tool supported platforms

iOs

And

roid

Win

dow

s Ph

one

Win

dow

s 8

Bla

ckbe

rry

7 an

d lo

we

r

Bla

ckbe

rry

10

Sym

bian

Bad

a

Ubu

ntu

Fire

fox

OS

Flas

h /

Flas

h Li

te

Tize

n

LG p

ropr

ieta

ry

Htm

l (ta

rget

ing

mob

ile)

Htm

l (ta

rget

ing

desk

top)

Win

dow

s (d

eskt

op)

OS

X (

desk

top)

Chro

me

Java

ME

Qt

Face

book

Mic

roso

ft X

Box

Sony

Pla

ysta

tion

Smar

t TV

s

AIR 65% 80% 7% 19% 41% 30% 20% 31% 15% 7% 4% 11%

Construct2 11% 67% 11% 22% 89% 11% 33% 22% 22%

Corona SDK 85% 90%

JQuery Mobile 67% 81% 17% 10% 25% 35% 10% 4% 46% 33% 8% 6% 4% 8% 4%

Marmalade 53% 66% 29% 79% 11% 21% 5% 11%

Phone Gap 80% 87% 22% 11% 24% 24% 11% 3% 22% 12%

Sencha Touch 75% 83% 0% 8% 25% 0% 58% 25% 17% 8%

Titanium 92% 92% 8% 8% 8%

Unity 3D 84% 94% 13% 13% 13% 3% 3% 0% 6% 13% 31% 22% 3% 6% 6% 9% 3% 3%

Xamarin 74% 95% 11% 16% 5% 16% 5%

<25% 25-50% 50-75% >75% not supported


Single User License

25


25

Marmalade (95%) and Unity 3D (94%) have the highest satisfaction for their platform support. The

vast majority of their users are satisfied with the possibilities to cross-develop for multiple OS.

They predominantly target Android, iOS and Blackberry. Sencha Touch receives also very good

user satisfaction ratings for their platform coverage, but results should be handled with care

because of low ratings.

JQuery Mobile, as a toolkit for the creation of web apps, is ranked number three for platform

support satisfaction.

Titanium has the lowest satisfaction for platform support. 36% of Titanium users are not satisfied

with their support. De facto, Titanium is a CP Tool that is being used to only develop apps for

Android and iOS, although Appcelerator offers support for a multitude of operating systems: iOS,

Android, Blackberry, Tizen, as well hybrid and HTML5.

Figure 8: Platform support benchmarking: user satisfaction with platform support

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Titanium

Construct2

AIR

Corona SDK

Phone Gap

Xamarin

JQuery Mobile

Unity 3D

Marmalade

Sencha Touch

Satisfied Unsatisfied


Single User License

26


26


Single User License

27


27

10. APP PROJECT DURATION AND TIME SAVINGS COMPARISON

CP Tools are designed to reduce app development time for multi- platform publishing.

The necessary development time is closely related to the size or complexity of an app project. A

comparison of the CP Tools based on the app development time is therefore difficult, but

development time can be used very well as an indicator for the complexity of the app which the

developer tries to build with the help of a CP Tool.

If the average development time for an app is only counted in days or weeks (as in the case of

Construct 2, JQuery or Titanium), it is a safe guess to say that these tools are mainly being used to

develop simple apps.

Unity 3D and Marmalade are the tools that show the longest average project durations.

Figure 9: Average app development time

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Unity 3D

Marmalade

Xamarin

Corona SDK

AIR

Phone Gap

Titanium

Sencha Touch

JQuery Mobile

Construct2

Days Weeks Months


Single User License

28


28

Against the background of the normal project length is the time saving, made possible by the use

of the tool, an important indicator of the quality of the tools.

The highest time savings for app development can be realized with Unity 3D, Corona SDK and

Xamarin. For all three tools, more than 50% of its users were able to develop more than 50%

faster.

JQuery Mobile, Marmalade and Titanium offer the smallest time saving for their users. The

realized time savings of these tools are below average. In case of Marmalade, the high complexity

of the tool (see above) might offset efficiency gains. For JQuery, the short project durations might

not give enough room for time savings.

Figure 10: Realized time-savings with CP Tools

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Titanium

Marmalade

JQuery Mobile

Sencha Touch

Phone Gap

AIR

Xamarin

Corona SDK

Unity 3D

Construct2

50%+ faster 30%-50% faster up to 30% faster 0% faster slower


Single User License

29


29

11. USER SATISFACTION WITH TECNOLOGY FEATURES

Availability of cloud API services (e.g. operator billing, in-app advertisement, app analytics or shop

systems), accessibility of device hardware features (e.g. microphone, camera, speaker) and pre-

installed applications (e.g. calendar, address book) help to manage the app lifecycle, ease the

development process and/or allow the creation of sophisticated apps in terms of functionalities as

well as look-and-feel.

CP Tools differ significantly in how intensively developers are using these features, how critical

they rate this service and how satisfied they are with the technology features.

11.1. CLOUD API SERVICES

Across all users of CP Tools, the availability of cloud API services is rated as of rather low

importance. Across all CP Tools the availability of cloud API services has been rated “critical” by

only 33% of users. Not surprisingly, the usage intensity of API services is moderate. Overall, less

than one third of the users say that they use the services “intensely” or “often”.

In the case of Corona and Sencha Touch, 50% of users regard the availability of cloud API services

as “critical”. 34% of Corona clients use these services frequently, which is the highest share of

intense users across all tools. At the same time, service satisfaction among Corona users is below

average.

AIR and Xamarin clients state the highest satisfaction with cloud API services. In the case of

Xamarin the usage intensity was higher than in the case of AIR. 30% of Xamarin users and 18% of

AIR users make extensive use of the services.

Titanium has a rather low share of users that are “satisfied” (30%) or “very satisfied” (10%) with

the Appcelerator cloud API services. In contrast 20% of users have been “unsatisfied”, which is the

largest share of non-satisfied users in the benchmarking regarding cloud API services.


Single User License

30


30

Figure 11: Cloud API services: service satisfaction, usage and importance

0% 20% 40% 60% 80% 100%

Construct2

Sencha Touch

Corona SDK

Titanium

Phone Gap

JQuery Mobile

Unity 3D

Marmalade

Xamarin

AIR

Very satisfied Satisfied Neutral

Unsatisfied Very unsatisfied

Cloud apis

0% 20% 40%

Intense or frequent usage

0% 50% 100%

Importance rating "critical"


Single User License

31


31

11.2. ACCESSIBIL ITY OF DEVICE HARDWARE FEATURES

Across all CP Tools in the benchmarking, 60% of users regard the accessibility of device hardware

features as “critical”. In particular Titanium users (85%), Unity 3D users (73%) and Phone Gap

users (72%) rate this feature as very important.

Besides Unity users, clients of other CP Tools with an emphasis on games (Corona SDK and

Marmalade) find this option very important.

Xamarin users are the most satisfied with the service of their tool: 69% are satisfied or very

satisfied with the accessibility of device hardware features.

Titanium and JQuery Mobile users are the least satisfied with the accessibility of device hardware

features.

Figure 12: Accessibility of device hardware features: service satisfaction, usage and importance

0% 20% 40% 60% 80% 100%

Titanium

JQuery Mobile

Phone Gap

Corona SDK

Xamarin

Unity 3D

Sencha Touch

AIR

Construct2

Marmalade



HW

0% 50% 100%


0% 50% 100%



Single User License

32


32

11.3. ACCESSIBILITY OF PRE-INSTALLED APPLICATIONS

Across all CP Tools the accessibility of pre-installed applications is rated by 40% of users as

important. Titanium clients rate the importance of accessibility of pre-installed applications the

highest (77%) and Marmalade clients the lowest (11%).

Phone Gap and Xamarin clients use this feature the most. 37% of Phone Gap users and 35% of

Xamarin users state that they use this feature “frequently” or “intensely”, followed by Titanium

users (28%). The lowest usage is amongst Corona SDK clients.

Xamarin users are the most satisfied with the service. Titanium users are the least satisfied.

Figure 13: Accessibility of pre-installed applications: service satisfaction, usage and importance

The results for all 3 benchmarked technology features indicate that Titanium users are the most

demanding developers and see technology features most critically.

On the other hand, Xamarin users tend to rate their tool support the most positively.

0% 20% 40% 60% 80% 100%

Sencha Touch

Corona SDK

Unity 3D

Titanium

Phone Gap

JQuery Mobile

Marmalade

Construct2

AIR

Xamarin



Pre-inst apps

0% 20% 40%


0% 50% 100%



Single User License

33


33

12. APP QUALITY BENCHMARKING

An important indicator for the performance of the different CP Tools is the quality of the apps that

has been developed with the help of a CP Tool. The quality of an app depends on the graphical

standard, the usability, the performance, the revenue potential and how secure the app is against

threats from viruses, data theft etc.

To evaluate the strengths and weaknesses of cross-platform apps, CP Tools have been compared

with native apps for the 5 app quality criteria.

Across all these factors, results for the different CP Tools are positive, but the differences between

the tools are significant.

12.1. DESIGN FEATURE COMPARISON

Design features of CP Tools are e.g. pre-installed design templates, functionality to edit and import

existing graphical resources and the support of animations.

In terms of “design features” AIR and Xamarin apps are rated best by far: 92% of AIR users and

89% of Xamarin users describe their cross-platform apps as good as or even better than native

apps in terms of design.

In contrast, the majority of Titanium users (57%) describe the design of their apps worse than

native apps. 22% even say that the scope of design for Titanium apps is “much lower” compared

to apps created with native SDK.


Single User License

34


34

Figure 14: App quality rating by CP Tool users: design features

12.2. USABILITY COMPARISON AGAINST NATIVE APPS

The usability of an app determines how effectively and efficiently users can accomplish the tasks

for which the app was designed.

Apps built with CP Tools have to make compromises, as the features and gestures that drive the

usability must work on multiple platforms and devices.

Nevertheless, CP Tools are being rated positively by users for the usability quality of their apps.

Xamarin is leading in usability as well: 89% of Xamarin users rate the usability of their apps up to

par with apps created with a native SDK. 81% of Unity 3D users rate their apps as having native-

like quality.

The lower end again is taken by Titanium and Phone Gap: 59% of Titanium users and 45% of

Phone Gap users rank their apps behind native apps, with 10% (Phone Gap) and 9% (Titanium) of

users rating the usability of their apps “much lower” than native.

-60% -40% -20% 0% 20% 40% 60% 80% 100%

Titanium

Phone Gap

JQuery Mobile

Sencha Touch

Marmalade

Corona SDK

Unity 3D

Construct2

Xamarin

AIR

Same or higher Lower Much lower


Single User License

35


35

Figure 15: App quality rating by CP Tool users: usability

12.3. PERFORMANCE COMPARISON AGAINST NATIVE APPS

The app performance describes the responsiveness, reliability and availability of an app.

Across all CP Tools in the benchmark, performance is the major weakness of cross-platform apps.

However, the results differ widely between CP Tools.

Again, Marmalade, Xamarin and Unity 3D apps are rated best. Especially Marmalade users (88%)

do not view their apps worse than apps created with an OS-specific native SDK.

Also, Titanium and Phone Gap apps are ranked on the lower end. 35% of Titanium users and 30%

of Phone Gap users describe the performance of their apps even “much lower” than native apps.

-60% -40% -20% 0% 20% 40% 60% 80% 100%

Titanium

Phone Gap

JQuery Mobile

Sencha Touch

Marmalade

Corona SDK

AIR

Unity 3D

Construct2

Xamarin



Single User License

36


36

Figure 16: App quality rating by CP Tool users: app performance

-80% -60% -40% -20% 0% 20% 40% 60% 80% 100%

Titanium

Phone Gap

Construct2

JQuery Mobile

Sencha Touch

AIR

Corona SDK

Unity 3D

Xamarin

Marmalade



Single User License

37


37

12.4. APP SECURITY COMPARISON

Mobile app security has become increasingly important in mobile computing. It is of particular

concern as it relates to the security of personal information now stored on smartphones with the

help of apps. Indeed, apps collect and compile an increasing amount of sensitive information, to

which access must be controlled to protect the privacy of the user and the intellectual property of

the company.

CP Tools can support app security by e.g. complying with international standards, providing

security guidelines or tutorials, or by making it easy to implement encryption technologies.

App security is rated highest for Xamarin and Construct2.

Phone Gap and Titanium apps are again on the lower end of the list. Especially for Titanium the

user rating is low: 35% complain about “much lower” app security compared to native apps.

Figure 17: App quality rating by CP Tool users: app security

-60% -40% -20% 0% 20% 40% 60% 80% 100%

Titanium

Phone Gap

JQuery Mobile

Corona SDK

AIR

Unity 3D

Sencha Touch

Marmalade

Construct2

Xamarin



Single User License

38


38


Single User License

39


39

12.5. APP REVENUE COMPARISON

App revenue which can be generated with apps that are being built with the help of CP Tools is

mainly impacted by increasing the reach/downloads of the app with easy multi-platform

publishing. In addition, CP Tools can support revenue generation by providing easy access to in-

app purchase, in-app advertisement as well as analytical tools that work on multiple platforms.

In terms of generated app revenues, all tools are being rated positively compared to native app

development.

AIR users rate their apps highest, followed by Unity 3D and Marmalade users.

On the lower end again are Titanium and Phone Gap apps.

Figure 18: App quality rating by CP Tool users: generated app revenues

-40% -20% 0% 20% 40% 60% 80% 100%

Construct2

Titanium

Phone Gap

Corona SDK

JQuery Mobile

Sencha Touch

Xamarin

Marmalade

Unity 3D

AIR



Single User License

40


40

13. COST-PERFORMANCE RATIO COMPARISON

Cost-performance ratio is an overall indicator of how efficient app developers rate the CP Tool-

supported development and publishing process of an app.

Overall, the cost-performance ratio is very high: on average 69% or more of the users rate the

cost-performance ratio “okay” or “good value”.

Construct2 and Marmalade cost efficiency rating is the highest with more than 50% of their users

saying that the CP Tool has a good value.

Titanium users make the highest share of unsatisfied users: 19% rate the cost-performance ratio

below average.

Figure 19: CP Tool user rating: cost-performance ratio

All CP Tools included in the benchmarking show strengths and weaknesses. In order to select the

best CP Tool for an app project or even for building an app portfolio, the pros and cons must be

understood. Listening to what users of these CP Tools are saying is a valuable input for the

decision.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Sencha Touch

Titanium

Phone Gap

Xamarin

Corona SDK

JQuery Mobile

Unity 3D

AIR

Marmalade

Construct2

Good value Okay Average Costly Poor value


Single User License

41


41

In general, the benchmarking of 10 of the most frequently used CP Tools shows that users are

satisfied with their tools.


Single User License

42


42

14. APPENDIX

ABOUT THE AUTHORS

The authors of this report have been following the app market and CP Tools for many years. The first report on multi-

platform app publishing tools was published in 2010. Since then, two other reports which cover the market for CP

Tools have been published.

OUR ANALYST TEAM

Ralf-Gordon Jahns

Ralf is the research director of research2guidance.

He has worked for more than 19 years in the

telecom and media industry. Prior to

research2guidance he worked as a partner for

Capgemini Telecom Media & Networks. Ralf is a

frequent keynote speaker on mobile industry

events, publisher of a multitude of mobile market

reports and executive consultant of more than

30 clients in the telecom and media industry.

Joachim Thiele-Schlesier

Joachim is a research analyst at research2guidance.

He is monitoring the CP Tool market since 2011 and

has been responsible for publishing 3 reports on this

topic. Joachim also engaged with projects on

converging telecommunication networks and

regulatory issues. He holds a degree in economics and

specializes on infrastructure and network economics.

Prior to research2guidance he worked for

Markedskraft ASA as analyst and for the Fraunhofer

Institute for Open Communication Systems and

Center for Network Industries.


Single User License

43


43


Single User License

44


44

LIST OF FIGURES AND TABLES

Figure 1: Geographical overview of cross-platform tool users ........................................................................................... 8

Figure 2: Background of cross-platform tool users ............................................................................................................. 9

Figure 3: CP Tool users by company size .......................................................................................................................... 14

Figure 4: CP Tool usage intensity by share of apps with CP Tool ...................................................................................... 16

Figure 5: Complexity benchmarking: time-to-familiarize ................................................................................................. 20

Figure 6: Complexity benchmarking: user rating complexity ........................................................................................... 21

Figure 7: Support services quality rating .......................................................................................................................... 23

Figure 8: Platform support benchmarking: user satisfaction with platform support ....................................................... 25

Figure 9: Average app development time ......................................................................................................................... 27

Figure 10: Realized time-savings with CP Tools ................................................................................................................ 28

Figure 11: Cloud API services: service satisfaction, usage and importance ...................................................................... 30

Figure 12: Accessibility of device hardware features: service satisfaction, usage and importance ................................. 31

Figure 13: Accessibility of pre-installed applications: service satisfaction, usage and importance .................................. 32

Figure 14: App quality rating by CP Tool users: design features ....................................................................................... 34

Figure 15: App quality rating by CP Tool users: usability .................................................................................................. 35

Figure 16: App quality rating by CP Tool users: app performance.................................................................................... 36

Figure 17: App quality rating by CP Tool users: app security ............................................................................................ 37

Figure 18: App quality rating by CP Tool users: generated app revenues ........................................................................ 39

Figure 19: CP Tool user rating: cost-performance ratio .................................................................................................... 40

Table 1: Benchmarked CP Tools and sample sizes ............................................................................................................ 10

Table 2: CP Tool users by industry focus ........................................................................................................................... 17

Table 3: CP Tool users by app category focus ................................................................................................................... 18

Table 4: Support services usage ........................................................................................................................................ 22

Table 5: Usage intensity of CP Tool supported platforms ................................................................................................. 24


Single User License

45


45

CONTACT RESEARCH2GUIDANCE

A one-on-one expert call with a Smartphone App Market analyst for

A personal 15-30 min. consultation

Questions and answers about our research

Matching your needs with our research services

Service and press inquiries

Support for articles

Background information on reports

Custom made services and bespoke projects

Direct web-shop access to our reports and PPTs

Browse our research products

Download immediately

Easy payment (VISA; Master Card; PayPal)

Contact our Research Director Ralf-Gordon Jahns

email: [email protected]

phone: +49 30 609893366

Visit www.research2guidance.com/shop



tool benchmarking 2013 - research2guidance€¦ · in terms of “design features”, air and...

Documents