p3 report mobile benchmark uk 2014

The first ever P3 Mobile Benchmark UK ended in a clear ranking of the country’s operators. The “best in test” seal was awarded to EE, thanks to the best data performance and a good voice service.Three ranked second with the best voice performance in the test, while O2 and Vodafone followed almost head-to-head.The P3 Mobile Benchmark used high-tech measure-ment equipment to test all four UK mobile networks in big cities, smaller cities and on connecting routes. The test cases were designed to push the networks as hard as possible, and were all based on proven P3 measurement methodologies.The results show, the UK networks provide much better services in larger metropolitan areas than in smaller cities and on connecting routes. Compared with other

leading operators in Europe and worldwide, all UK networks have significant potential to further improve their service quality and performance.The newest mobile network technology has just begun to show what customers can expect in the future.

The P3 Mobile Benchmark UK

EE emerged the clear winner and took the “best in test” accolade for 2014. Three ranked second overall and had the best voice performance. O2 and Vodafone followed almost level-pegging with a slightly better score for O2.

11/2014Mobile Benchmark UK

2

Thus, the UK is a natural area of focus for P3 communications, the international leader in mobile benchmarking. The P3 Mobile Benchmark UK is designed to provide an independent overview of the current network situation in the UK; it highlights the leading-edge innovations in the country’s mobile networks, but also reveals potential weak spots in mobile service quality as perceived by customers.

In Germany, Austria and Switzerland, the benchmark is now widely accepted as an objective and thorough test of network performance, drawing on hundreds of thousands of voice and data measurements. It has become a way both to improve market transparency and for the mobile operators themselves to measure and compare network quality. Today, the P3 benchmark provides an important incentive for the operators to continue investing in improving their infrastructure, services and performance. It also gives them a perfect platform to show how revenue from customers is used to deliver and improve service and quality.

The network operators are in the process of deploying 4G technology, also known as LTE, across large portions of their networks; LTE-

capable smartphones are available everywhere and are dominating handset sales. For these reasons, the P3 benchmark was conducted using modern 4G/ LTE devices.As technically-minded people know, voice is currently a particular challenge when smartphones use LTE. At this point neither the networks nor the end user devices allow telephony over LTE; for each phone call devices must therefore switch to legacy voice network standards like UMTS or GSM. Mobile experts refer to this as Circuit Switched Fallback (CSFB). In order to use CSFB, the device first needs to detect a UMTS or GSM network and then switch to it. Due to this procedure, call setup takes longer and the possibility of failure increases. The selected test methodology focused specifically on this issue.The benchmark results were based on several weeks of intensive testing. Two specially equipped measurement cars took tens of

thousands of test samples while travelling up and down the country, covering major metropolitan areas, smaller cities and connecting routes to provide the most representative data possible. The selected areas for the tests accounted for roughly 13 million people or 20 per cent of the UK population.To ensure fair competition between the operators, each was informed several months in advance about test methodologies, the main focuses of the test cases and the target areas. It was up to each operator to decide how best to prepare for the test. At the same time, the P3 benchmark was designed to be strictly independent and impartial. The final say for the test design, the routes and the score weighting was ours. And, needless to say, the operators had absolutely no influence over the final results. The scoring for the P3 Mobile Benchmark UK was based on the proven scoring systems used in other P3 benchmarks worldwide.P3 communications has well over a decade of experience in network testing of this type. In 2013 alone, P3 compiled about 40,000 measurement hours in 42 countries across five continents, with our test vehicles covering more than 500,000 miles. P3 works with clients around the world including network operators, device manufacturers and regulatory authorities.

Introduction

Background

3

P3 has been running similar annual public benchmarks in Germany since 2002 and in Austria and Switzerland since 2009. This year, it also launched an Australian benchmark.

The UK is one of the largest and most interesting mobile communications markets in Europe since it is moving very quickly at the forefront of trialling and deploying some of the newest LTE technologies.

Timeline 2014

P3 fi nally de-P3 fi nally de-P3 fi nally de-fi ned all test fi ned all test fi ned all test parameters.parameters.parameters.The decision was The decision was The decision was made to publish made to publish made to publish the benchmark the benchmark the benchmark results inresults inresults inNovember 2014.November 2014.November 2014.

Testi ng conclu-Testi ng conclu-Testi ng conclu-ded. P3 began ded. P3 began ded. P3 began post-processing post-processing post-processing on the results.on the results.on the results.The P3 Mobile The P3 Mobile The P3 Mobile Benchmark UK Benchmark UK Benchmark UK was published.was published.was published.

EE, O2, Three EE, O2, Three EE, O2, Three EE, O2, Three EE, O2, Three EE, O2, Three and Vodafone and Vodafone and Vodafone were noti fi ed were noti fi ed were noti fi ed of the planned of the planned of the planned benchmark by benchmark by benchmark by P3, including P3, including P3, including preliminary preliminary preliminary informati on informati on informati on around scope, around scope, around scope, and were invited and were invited and were invited to provide feed-to provide feed-to provide feed-back.back.back.

Testi ng began. Testi ng began. Testi ng began. All operators All operators All operators were noti fi ed were noti fi ed were noti fi ed and, in turn, and, in turn, and, in turn, were asked to were asked to were asked to provide noti ce provide noti ce provide noti ce of any outages of any outages of any outages which might which might which might impact results.impact results.impact results.

Start of scoping Start of scoping Start of scoping and planning and planning and planning work.work.work.

JunJunJun SepSepSep NovNovNovJulJulJul OctOctOct

About P3 communicati ons P3 communicati ons is a leading internati onal consulti ng, engineering and testi ng services company. P3 communicati ons is part of theP3 group, which has over 2,500 employees worldwide and posted a turnover of more than €270 million in 2013. The company provides a broad portf olio of independent technical and management consulti ng services including network planning, engineering, end-to-end opti misati on, security, QoS and QoE testi ng, internati onal benchmarking, device testi ng and acceptance services. P3 communicati ons’ clients include network operators, equipment vendors, device manufacturers, public safety organisati ons and regulatory authoriti es around the world; the company’s own experts combine strong technical know-how with many years of management experience.

4

The basis for the benchmark was the proven P3 measurement methodology, used by the company for similar testing around the world. The methodology ensures fair comparability of results be-tween all operators under tests, and yields reliable outcomes.

SmartphonesEvery year P3 reviews and decides which measurement device will be used for the upcoming tests. The selected smartphone must fulfil a detailed set of requirements. It is very important that it is widely used in the target markets, to ensure the benchmark results are representative of typical end-user experience; for this reason, P3 only considers market-leading devices. The test smartphones must also allow the installation of all necessary measurement software, making an Android operating system mandatory. Special local requirements are reflected by using local versions of the devices and, if available, operator specific firmware.The selected smartphone for the 2014 UK measurements was the Samsung Galaxy S4 LTE Cat 4. This device supports download speeds up to 150Mbit/s.

EE, O2, Three and Vodafone themselves provided the SIM cards for testing and ensured that any fair use policies – which are sometimes used to control data download speeds for customers with very high data usage – would not interfere with the test results. However, the P3 test staff also anonymously purchased SIM cards from normal retail outlets for a control comparison, ensuring that the SIM cards used for the test exactly matched those commercially available to consumers.

Measurement SystemEach measurement car was equipped with P3’s custom-built proprietary high end measurement systems; one voice and one data test smartphone per operator; and a set of external antennas, mounted in a roof top box, to ensure proper test conditions. All smartphones were set up with 12dB attenuation in the antenna feeders to reflect a balance of real-world conditions, such as users sometimes moving into buildings.

Voice MeasurementsAll voice test calls were mobile to mobile, and were set up from each car to the other. Voice measurements were taken while the cars were moving in cities, waiting at hot spots, or transferring between cities.

Speech quality was evaluated using the Perceptual Objective Listening Quality Analysis wideband algorithm (POLQA). POLQA-Wideband has become an important measurement standard for next-generation networks, as operators deploy high-definition voice services and offer compatible handsets. It delivers an advanced level of benchmarking accuracy and adds significant new capabilities for wideband and super-wideband (HD) voice signals, along with support for the most recent voice coding and VoIP transmission technologies.All test cases were executed in LTE preferred mode, which forced the networks to employ CSFB while the devices were in 4G areas. Voice calls in 3G- or 2G-only coverage areas were operated as usual. To simulate typical user behaviour, voice measurements were conducted while the test devices were handling data traffic at the same time – in this case, by receiving emails.Key performance metrics included call setup times, speech quality, call success ratios and sustainable connections.The difference between the last two metrics is worth explaining. Call success ratio looks only at the proportion of calls that are successfully set up and do not cut off. The sustainable connection measurement, on the other hand, looks at the proportion of calls where speech quality does not drop below a certain minimum level more than once or are even silent. In user

Methodology

5

Roof top box

Data MeasurementsFor data measurements, a variety of different data-specific test cases were used. Tests included HTTP downloads and uploads as well as live web browsing, static web browsing using a defined standard web page, and YouTube video streaming both in standard and high resolution (2.7MB video at 360p SD, 11.9MB video at 720p HD).The webpages were selected from amongst the most popular sites in the UK. For this year’s test, the following pages were selected:

▪ Google ▪ YouTube ▪ BBC ▪ Ebay ▪ Amazon

A public video was used for YouTube streaming in both resolutions. Finally, as a stress test, as much data as possible was pushed over the networks within a 10 second time window.The key metrics for data measurements were success ratios and session times, as well as up- and download speeds. One particularly

important metric from a customer point of view was the basic data rate: the minimum speed for the top 90% of all samples. For YouTube, the video start-up delay and the percentage of playback samples without interruption were important parts of the evaluation.

Logistics and routesThe testing was conducted in major metropolitan areas, smaller cities and on connecting routes. The test vehicles drove the same overall route around the country in parallel, but avoided staying in the same exact areas at the same time in order to prevent the risk of one car distorting the measurements of the other. They also took different routes within cities to maximize the area covered. Both vehicles conducted not just mobile drive tests but also stationary measurements at ‘areas of interest’ such as railway stations, airports, recreational areas and densely populated residential areas. These types of high-traffic and

frequently crowded areas present a challenge for mobile network operators, which have to provision and allocate sufficient capacity for all local users.

terms, unsustainable calls happen when there is so much noise on the line, or you have so many problems hearing the other person that you decide to give up and try again.

In total, the test vehicles covered more than 3,500 miles. Across all services combined on all networks, they collected more than 160,000 data samples and conducted roughly 16,000 voice calls with more than

125,000 speech samples. In total

more than 380 Gigabytes of data was transferred.

EdinburghGlasgow

East KilbridePaisley

S�rling

London

Leeds

Stafford

Evesham

StroudMarlow

Birmingham

Manchester

Penrith

Burnley

Oldham

Preston

▪ Wikipedia ▪ LinkedIn ▪ Wordpress ▪ The Telegraph ▪ Gumtree

Big CitiesSmaller Cities

LondonMarlowWitneyCirencesterStroudWorcesterDroitwich SpaBromsgroveBirminghamRedditchEveshamStratford-upon-AvonRoyal Leamington SpaNuneatonTamworthStaffordCongletonMacclesfieldWilmslow

ManchesterChorleyPrestonBurnleyBuryOldhamLeedsHarrogatePenrithEdinburghDunfemlineStirlingFalkirkLivingstonGlasgowPaisleyEast Kilbride

6

Waypoints on Testing Route

The results of the P3 Mobile Benchmark UK are represented by means of a scoring, that is used to map the technical performance metrics to a common scale and to aggregate the achievements in the different test categories to one final count.

As technology moves forward and customer demands change over time the scoring criteria are carefully adapted each year founding on P3 communications’ worldwide and long term experience in the rating

of mobile radio networks. Years ago, data had almost no importance at all, while voice and SMS texting was key to end user perceptions. Today, data dominates, while voice is perceived as something of a commodity. This might change again once Voice over LTE (VoLTE)

technology is implemented, with new challenges and opportunities for voice services.This way, P3 makes sure that the score always reflects the technical performance of the networks

compared to what is possible with state-of-the-art technology worldwide.The different performance aspects of the various test disciplines contribute with different weights to the final score.The weighting was designed to reflect the basic needs of mobile customers and, in general, put more emphasis on reliability and stability than on the peak performance of the networks. In other words, establishing a successful call in the first place was more important than a minor advantage in speech quality; by the same token, reliable data sessions with adequate speeds were weighted more highly than high peak data rates.

Results

Results: Voice TelephonyThe voice score was separated into two major sections. The first section contained the results of the metropolitan areas (big cities). The other section included smaller cities and connecting routes (Small Cities

and Transfers). The percentages in the table indicate how much of the maximum score in each section were achieved by each operator.Looking at the combined voice results achieved in cities and on

transfer routes, Three came out with the best voice score in the test. EE ranked in second place not far behind; third place went to Vodafone, closely followed by O2.

Operator

Voice Total max. 190 140 153 84 94Big Cities 125 73% 75% 32% 42%

Small Cities/Transfers 65 76% 91% 68% 65%

7

The score reflects the degree of fulfilment regarding the maximum achievable performance, while the best possible total score is 500.

Voice in big citiesIn cities EE and Three were on par in terms of voice scoring. While EE provided a slightly better call success ratio, Three impressed with very short call set up times and the best speech quality.The Vodafone network delivered good speech quality but could improve on call success ratio and call set up time. The challenges for O2 were call stability and call set up times as well as speech quality, which suffered from missing HD voice features. All networks performed well in terms of call sustainability.

Operator EE Three O2 Vodafone

Voice (Big Cities)

Call Success Ratio (%) 97.1 96.8 91.1 92.4

Call Setup Time (s) 7.9 5.5 8.1 8.2

Speech Quality (MOS-LQO) 3.3 3.6 2.6 3.0

Call Sustainability (%) 99.0 98.1 99.0 98.6

8


Voice (Small Cities and Transfers)

Call Success Ratio (%) 94.3 97.4 93.5 90.9

Call Setup Time (s) 7.0 4.5 8.0 7.3

Speech Quality (MOS-LQO) 3.3 3.6 2.5 3.0

Call Sustainability (%) 98.7 99.5 98.5 99.0

Voice in smaller cities and on transfer routes Three provided the best voice service in the tested smaller cities and on the transfer routes, leading in all Key Performance Indicators (KPIs). Test winner EE provided a stable voice network, followed by O2 and Vodafone.Call success ratio on transfer routes was an area of interest for almost all operators but Three, while call sustain-ability was managed well.

Operator

Data Total max. 310 226 163 199 186

Big Cities 200 81% 50% 74% 72%

Web Browsing 70.0 79% 75% 90% 88%

File Download 40.0 71% 32% 58% 60%

File Upload 40.0 83% 47% 64% 66%

YouTube 50.0 90% 31% 72% 63%

Small Cities 40 45% 45% 45% 32%

Web Browsing 14.0 47% 67% 67% 47%

File Download 8.0 39% 33% 30% 23%

File Upload 8.0 49% 37% 39% 20%

YouTube 10.0 43% 29% 30% 28%

Transfers 70 66% 65% 47% 41%

Web Browsing 24.5 70% 78% 65% 54%

File Download 14.0 70% 62% 27% 28%

File Upload 14.0 82% 68% 56% 50%

YouTube 17.5 45% 45% 29% 28%

The data score was separated into three major sections. The first section contained the results of the metropolitan areas (big cities), while the other sections included separately smaller cities (Small Cities) and connecting routes (Transfers). The percentages in the

table indicate how many of the maximum points in each section were achieved by each operator.

Overall EE achieved the highest data score. O2 ranked second, with Vodafone not far behind. Three could not show comparable results

to those for its voice performance and consequently ended up in fourth place.LTE has already begun to show its potential, with top data rates of more than 40Mbit/s measured in metropolitan areas.

Results: Data

9


DATA SERVICES (Big Cities)

Webpage download (Live/Static)

Success Ratio (% / %) 97.0 / 99.0 97.5 / 98.8 99.0 / 99.7 99.0 / 99.3

Avg. Session Time (s / s) 4.1 / 1.1 4.5 / 2.7 3.5 / 1.2 3.8 / 1.4

File download (3MB)

Success Ratio / Avg. Session Time (% / s) 99.7 / 2.9 98.8 / 9.9 99.4 / 3.9 99.3 / 3.4

90% faster than (kbit/s) 5271 1160 3640 4478

File upload (1MB)



File download (10 Seconds)

Avg. throughput (kbit/s) 17361 6355 13232 13973


File upload (10 Seconds)



Youtube SD

Success Ratio / Start Time (% / s) 99.9 / 1.6 95.7 / 2.2 98.8 / 1.3 99.5 / 1.7

Video playouts without interruptions (%) 99.6 93.9 97.6 99.3

Youtube HD



Data in metropolitan areasAll networks showed good data stability in most of the tested services, with different strengths and weaknesses.Vodafone and O2 delivered the best browsing results, while EE showed the best YouTube performance with impressive success ratios.EE achieved the highest data speeds in upload and download, followed by Vodafone, O2 and Three.

10


DATA SERVICES (Small Cities)


Success Ratio (% / %) 92.3 / 97.1 96.9 / 98.5 96.6 / 97.7 94.7 / 88.5


File download (3MB)



File upload (1MB)

Success Ratio / Avg. Session Time (%/s) 99.3 / 7.5 99.3 / 8.6 98.9 / 7.2 94.8 / 11.8








Youtube SD



Youtube HD



Data in small citiesThe results in the tested small cities showed where all operators could find a lot of potential to improve their data services. Smaller cities suffered from lower success ratios, longer session times and slower data speeds.In small cities EE, Three and O2 performed on-par, followed by Vodafone.

11


DATA SERVICES (Transfers)


Success Ratio (% / %) 90.9 / 95.2 94.9 / 97.9 90.8 / 85.8 87.8 / 81.4


File download (3MB)



File upload (1MB)









Youtube SD



Youtube HD



Data on transfer routesThe results on transfer routes showed even more improvement potential than those in smaller cities. Biggest challenges were the success ratios und the very slow throughputs. All services were suffering from those performances.EE and Three achieved the best scores followed by O2 and Vodafone.

12

Results LondonThe major metropolitan area of the UK is worth having a detailed view at in terms of network performances.If the London area were to be scored separately, with the same voice and data scoring used for big cities, the results would be as follows:

It is very interesting to see Vodafone, which performed in London much better in voice. EE achieved a better data score, while O2 and Three delivered voice and data scores in London at the same level as in the total Big City category.

Operator

TOTAL max. 325 255 187 185 200

Telephony max. 125 88 88 42 57London ONLY 125 71% 70% 33% 46%

Data Services max. 200 167 99 143 143London ONLY 200 83% 50% 71% 72%

13

The overall scores combined results from all of the categories previously detailed, and resulted in a clear winner for the first P3 Mobile Benchmark UK. EE achieved the highest overall score with a significant lead over second-ranked Three. O2 and Vodafone followed almost on par with each other, with O2 having a very slight edge.

Overall Scores

11/2014Mobile Benchmark UK

Operator

TOTAL max. 500 366 316 283 280

Telephony max. 190 140 153 84 94Big Cities 125 73% 75% 32% 42%

Small Cities/Transfers 65 76% 91% 68% 65%

Data Services max. 310 226 163 199 186Big Cities 200 81% 50% 74% 72%

Small Cities 40 45% 45% 45% 32%

Transfers 70 66% 65% 47% 41%

14

MAX.500

366

316283 280

VOIC

E

VOIC

E

VOIC

E

VOIC

EDA

TA

DATA

DATA

DATA

It revealed some very well per-forming services and some very well served areas, but also plenty of potential for improvement in all UK networks.

Comparing the UK results with other P3 mobile benchmarks in Europe and Australia makes it clear that much better performances are achievable. But the good news for

all UK customers is that the newest technology potentially provides for much more capability than is currently implemented; i.e. there is a lot more to expect.

Conclusions . & Highlights

EE achieved the “best in test” seal thanks to a dominant data performance combined with a reliable and well managed voice network. But there is potential for improvement in web browsing and data performance outside urban areas. For EE to defend its number 1 position, some extra attention to voice services could give it a valuable score boost for next year. 11/2014

Mobile Benchmark UK

O2 delivered well in data with the second best result, but has a lot of scope to improve its scores in voice. It was not far behind EE on data, however, with some improvements to suc-cess rates and throughput, the data story next year could be different. Higher voice success rates and better speech quality could further strengthen O2’s position.

Three finished in second place, with the best voice performance in the test. Had it not trailed the field in data, though, it could have put a lot more pressure on EE. Once its data chal-lenges are solved, we can expect much more from Three.

Vodafone ended up in third place for both voice and data. But O2 only pulled ahead in the total scoring via a slight advantage in data performance; it is a small gap to close for Voda-fone to draw level with O2, and a little more ground to cover to threaten Three. With good data results and an increased stability in voice, Vodafone could catch up in next year’s P3 Mobile Benchmark UK.

15

The P3 Mobile Benchmark was specifically designed to push the limits of the most advanced mobile network technology currently in service.

International Benchmark

16

While the best UK operator achieved a score of 366, the top performers in Germany, Austria and Switzerland scored in a range above 440; and even a score above 470 was achie-

ved in the last years. The lowest va-lues in those countries were above 325, which would still be enough for the second place in UK.In other words, compared with

other leading operators in Europe and worldwide, all UK networks have significant potential to further improve their service quality and performance.

The FutureThe P3 Mobile Benchmark 2014 is intended to be the first in a regular series of such reports.

The mobile industry in the UK moves very quickly and is at the forefront of trialling and deploying some of the newest LTE technologies, such as VoLTE and LTE-Advanced. Right now, each of the operators has a number of major cutting-edge initiatives under development to boost their networks in different ways.

We expect to see significant improvements by the time the next testing cycle begins.

P3 communications GmbH, Postfach 37 01 83, 52035 Aachen/Germanyphone +49 241 9437 400, [email protected]

Headquarters:Aachen (GER), Belgrade (SRB), Morristown (NJ, USA), Sydney (AUS)

All rights reserved. The information and statements provided in this document are for informational purposes only. Any measurements, recommendations, opinions or findings provided are only true and accurate for the conditions encountered at the time of any survey and P3 communications accepts no liability for any errors or omissions. In no event will P3 communications be liable for any consequential, indirect or incidental damages whatsoever, including lost profits, business, contracts, revenue or anticipated savings arising out of the use of any P3 communications products or documentation (or from errors or deficiencies therein), reliance on the data or measurements produced or information contained within.

p3 report mobile benchmark uk 2014

Documents

uk mobile networks

mobile operators

p3 benchmark

p3 mobile benchmark

mobile benchmark uk2thus

uk networks

countrys mobile networks

mobile service quality