Documente Academic
Documente Profesional
Documente Cultură
After conducting mobile network tests in the Netherlands in 2015 and 2016,
the network testing professionals of P3 communications and connect have Groningen
Leeuwarden
examined the Dutch mobile networks for the third time. As all four operators Win
inschoten
Drachten
showed excellent results in the previous tests, the race is all the more Sneek Gieten
Joure Heerenveen
exciting in 2017. How did the four candidates perform?
THE 2017
Emm
men
Alkmaar
P3 CONNECT
Amste
terd
rdam
am
Epe
Naarden
Naard
Hilversum
Leide
den Apeldoorn
Barneveld Ensch
chede
Waddinxveen
The Hagu
gue Utrecht
Woerden
Arnhem
MOBILE
Gouda Elst
Roerdam
Nijm
Nijmegen
Gorinchem
Breda
BENCHMARK
Bergen
Bergen op Zoom
Zoo Tilburg
ilb Eindhoven
Venlo
Roermon
Ro e d
IN THE NETHERLANDS
Gelee
Geleen
Valken
nburg
MOBILE BENCHMARK
A NUTSHELL
P3 connect Mobile Benchmark in the Nether-
lands, the Dutch operators were able to main-
tain their impressive results in 2017. However,
we observed some interesting changes too.
P3‘s network benchmarks are widely
accepted for setting industry standards as
well as being highly objective. The carefully
designed methodology scheduled four drive-
test cars to visit 17 large cities and 19 smaller T-Mobile
Vodafone KPN Tele2
towns in the Netherlands as well as the con-
necting roads. The areas in which we tested max. 1000 Points
2
MOBILE BENCHMARK
3
MOBILE BENCHMARK
VOICE
40%
Call Setup Time
DUTCH NETWORKS
Speech Quality
4
MOBILE BENCHMARK
VOICE
and good speech quality.
How do the Dutch networks
fulfill these expectations?
pline is T-Mobile – although this ope- Call Success Rao (%) 99.6 99.8 99.7 99.8 99.5 98.7 98.8 98.5
4.5 2.5 5.2 2.5
rator is the only one in the Netherlands Call Setup Time (s) 3.4 3.4 5.8 3.9
Speech Quality (MOS-LQO) 3.5 3.6 3.5 3.6 3.6 3.3 3.0 3.5
that has not yet introduced VoLTE.
Roads
Vodafone follows closely behind, while Call Success Rao (%) 99.2 98.8 98.1 99.4 99.0 97.6 98.0 97.1
Tele 2 loses points due to legacy natio- Call Setup Time (s) 3.8 3.7 5.0 2.6 6.2 4.1 5.5 2.5
nal roaming. KPN offers VoLTE, but not Speech Quality (MOS-LQO) 3.4 3.5 3.5 3.5 3.5 3.2 2.9 3.5
yet on the S7 test phones. If the eva-
luation had considered only legacy calls, The voice evaluation in this benchmark includes the results of both legacy and VoLTE calls in
those networks where we could actually measure VoLTE. However, in order to allow additional
KPN definitely would have ranked better. in-depth comparisons we have additionally specified the results of both connection types.
5
MOBILE BENCHMARK
DATA
growing. So, all operators face the challenge
to keep data rates and reaction times at a high
level. Which Dutch operator manages to best
meet the growing demand?
Good news for Dutch customers: All 600 of 1000 Points
mobile networks operating in the Nether-
lands show a very high degree of stability DATA
and performance in this benchmark.
Whether our test teams drove through
large cities, through smaller towns or on 96%
of 360 P.
the connecting roads – the success ratios 92%
observed in each of these scenarios were
altogether remarkably high. Cies 95%
In order to check the reliability and per- Drivetest 93%
formance of data connectivity in the Dutch
networks, one Samsung Galaxy S7 per 95%
operator was installed in our test cars. The
of 120 P.
phones constantly performed a suite of 90%
tests including web-page downloads, file Towns 95%
downloads and uploads as well as Youtube Drivetest 90%
video playbacks. For most of these checks,
success ratios and average session times
were logged. For the downloads and uploads 96%
of 120 P.
we also recorded the average throughputs. 94%
In order to get an idea of the typical perfor-
Roads 96%
mance, we determined the minimum data
Drivetest
rates that are available in 90 per cent of the 95%
cases. To also reflect peak date rates, we
additionally recorded the speeds that were T-Mobile Vodafone KPN Tele2
surpassed in 10 per cent of the cases.
Data in Ci es - Drivetest T-Mobile Vodafone KPN Tele2
P3‘s approach for testing Youtube play-
Web-Pa c)
back recognizes that this popular video
service has recently introduced adaptive bit Success Ra o (%/%) 99.9/99.8 99.7/99.9 99.6/100.0 99.7/99.9
rates. This decision of the streaming provider Avg. Session Time (s/s) 2.8/1.0 3.1/1.2 2.8/1.0 2.7/0.9
aims at a better user experience, surren- File Download (3 MB)
dering pixel resolution in favour of stable Success Ra o/Avg. Session Time (%/s) 100.0/0.8 100.0/1.1 100.0/0.9 100.0/1.0
playback. As a consequence, besides 10% faster than (kbit/s) 68926 53571 74074 72072
success ratios, start times and the absence 90% faster than (kbit/s) 21101 12512 16995 14889
of interruptions, the average value of the
File Upload (1 MB)
received video resolution became another
Success Ra o/Avg. Session Time (%/s) 99.4/0.7 99.7/1.2 100.0/0.8 99.6/1.8
important performance indicator.
10% faster than (kbit/s) 24242 20445 26059 24096
T-Mobile also leads the field in data 90% faster than (kbit/s) 7422 3558 6457 2138
Although all four operators reach very File Download (10 Seconds)
high scores in the cities, T-Mobile takes Success Ra o (%) 100.0 99.9 99.9 99.8
a distinctive lead over its competitors. Avg. Throughput (kbit/s) 77719 49053 63943 62041
Here, KPN reaches the second rank with a 10% faster than (kbit/s) 143182 85243 119424 95366
gap of only five points to T-Mobile. Tele2 90% faster than (kbit/s) 30068 16348 21292 18869
comes in third in the cities. However, slightly
File Upload (10 Seconds)
weaker values particularly in the file upload
Success Ra o (%) 100.0 99.9 100.0 99.9
category lead to clear distance to the
results of the two candidates on the higher Avg. Throughput (kbit/s) 28117 20358 23410 16456
ranks. With a very close gap of only one 10% faster than (kbit/s) 43440 41435 42439 39457
point behind Tele2, Vodafone comes in last 90% faster than (kbit/s) 9213 4137 6825 2279
in the cities. This position originates in the Youtube Video
loss of a couple of points especially in the web Success Ra o/Start Time (%/s) 99.9/1.4 99.9/1.6 99.9/1.6 99.9/1.4
browsing and file upload categories. Again, Video playouts without int ons (%) 100.0 100.0 100.0 100.0
we must emphasize that this ranking order >>
Average Video Resolu on (p) 704 703 701 686
6
MOBILE BENCHMARK
7
MOBILE BENCHMARK
TESTING METHODOLOGY
The P3 connect Mobile Benchmark acquired partly in 4G preferred to 3G
in the Netherlands took place from preferred mode and partly in 4G
February 27 to March 6, 2017. All preferred to 4G preferred mode.
samples were collected between 8am As a consequence, the phones in
and 10pm. The network tests covered most cases needed to switch (“fall
larger cities, smaller towns and connec- back”) to 2G or 3G when they were
ting roads. The combination of test areas logged in to the LTE (so called “cir-
had been carefully selected to provide cuit-switched fall back“ or CSFB).
a significant series of test results cove- In order to account for typical
ring the Dutch population. The areas smartphone usage scenarios during
chosen for the 2017 test account for the voice tests, background data
more than 5.1 million people, or about traffic was generated in a controlled
30 per cent of the total population. way through random injections of
P3 conducted the tests with four small amounts of HTTP traffic. The
drive-test cars, equipped with arrays voice test scores account for 40 per
Three boxes were mounted into the back and into
of twelve Samsung Galaxy S7 Cat 9 cent of the total benchmark results. the side windows of each measurement car in
smartphones each. Eight phones were order to support twelve smartphones per car.
used for the measurement of voice Data testing
services, and four additional S7 cap- Data performance was measured
tured the values for the data services. using four smartphones in each car
– one per operator. The radio access
Voice testing technology was set to LTE preferred
Two smartphones per operator in mode in order to reflect the customer
each car were used for the voice experience. The web tests accessed
tests, setting up test calls from one ten web pages according to the widely
car to another. The audio quality of recognized Alexa ranking. In addition,
the transmitted speech samples was the artificial (static) “Kepler” test web
evaluated using the HD-voice capable page as specified by ETSI (European
and ITU (International Telecommunica- Telecommunications Standards Insti-
tion Union) standardized so-called tute) for testing purposes was used.
POLQA wideband algorithm (Perceptu- In order to test the data service
al Objective Listening Quality Assess- performance, files of 3MB and 1MB
ment). All Dutch network operators for download and upload respectively
offer 4G capable subscriptions. were transferred from or to a test ser-
To take the high share of LTE into ver located on the Internet. In addition,
account, speech samples were the peak data performance was >>
DATA VOICE
60% 40%
Web Page Download Call Setup Time
Each box housed four smartphones allowing the
simultaneous testing of four mobile operators.
Success Rao
Call Success Rao
Youtube Quality
8
MOBILE BENCHMARK
DATA VOICE
ROADS
CITIES
9
Numbers in % of achievable maximum score
MOBILE BENCHMARK
CONCLUSION
All four Dutch operators score well over 900
out of 1000 points. Overall winner T-Mobile
even surpasses the 950 points threshold,
achieving the rare grade „outstanding“.
2015
2016
The results of the P3 connect mobile
2017 97 96 88 92 93 88 94 93 77 90 92
benchmark in the Netherlands had already
reached a very high level last year. In 2017, Operator T-Mobile Vodafone KPN Tele2
again, all four Dutch operators show a re-
markably high level of performance and The total score development over the last three years shows the very high level on which Dutch mobile
stability. In a year-on-year comparison, Tele2 networks operate. Although there have been some shifts, the overall trend shows a constant improvement.
was able to improve in all tested categories.
Vodafone takes over the second rank from T-Mobile
KPN, which mainly results from Vodafone‘s Vodafone KPN Tele2
distinct improvement in the voice category. max. 1000 Points
1
As in 2016, the third largest Dutch
2
With a considerable improvement
3
In a tight competitive environment,
4
The Netherlands’ smallest opera-
operator also wins the 2017 in its voice performance, Vodafone the largest Dutch operator has to tor scores last in this benchmark.
benchmark with a lead in the voice is able to outrank the market leader give way to the smaller operators, However, Tele2’s results are still
and data categories and the overall KPN in this year’s benchmark. In that rank on second and third posi- very competitive and deserve the
grade “outstanding“. The fact that the data category, this operator tion in terms of subscriber numbers. overall grade „very good“. The
T-Mobile achieves the best result still achieves a third place. However, However, KPN is beaten by Voda- loss of points in the voice category
in the voice category without offe- the differences are small, and es- fone with a gap of only one point. can mainly be explained by the
ring VoLTE shows that legacy voice pecially on the roads and to some But the Dutch market leader still disadvantageous effects of natio-
services can also offer perfect extent in smaller towns, Vodafone achieves very good results both in nal roaming for legacy voice ser-
calls, when they are handled well. holds up well with its competitors. the voice and data categories. vices on the T-Mobile network.
BEST IN TEST
T-MOBILE
4/2017
MOBILE BENCHMARK
THE NETHERLANDS
www.connect-testlab.com
10
MOBILE BENCHMARK
Objective testing will be essential to the ongoing evolution of the Dutch networks as to emerging tech nologies
like autonomous driving and smart cities. P3 communications is preparing for these future challenges.
OUTLOOK
As in the years before, all four through to completely new
candidates scored really well in mobile communications techno-
this year‘s mobile network test in logies like the upcoming super-
the Netherlands. This, however, fast 5G networks. However, 5G‘s
cannot be taken for granted. approach of assigning different
Upcoming improvements like network capabilities to different
the further extension of 4G use cases will also require an P3 communications is
coverage, the ongoing network extended approach to network constantly monitoring
the technological
evolution with technologies like testing.This will involve concepts development.
“4G Advanced“ and the conti- like advanced analytics and
nuing rollout of Voice over LTE big-data technology.
(VoLTE) may change this picture.
P3 communications is well Extensions to the test routines
prepared to accompany the fu- Readers interested in taking
ture developments of the Dutch part in our continuous efforts to
telecommunications market. In evaluate the performance of
this context, we are already pre- mobile networks can already do CROWDSOURCED NETWORK RATING
paring to include new technolo- so today – by downloading the P3 communications is increasingly focussing on aspects
like the retainability of voice services, the integrity of data
gies and enhancements menti- “U get“ wireless performance ra- services and “operational excellence“. An important instru-
oned above in next year‘s testing. ting app. See details on the right. ment for this approach is the “U get“ app that is available
Furthermore, P3 is also preparing This crowdsourcing approach under uget-app.com or via the adjoint QR code. The An-
for future challenges starting will give us valuable additional droid app checks and visualises current network performance.
Join the global community of users who understand their per-
with new communications appli- insights into the user experience sonal wireless performance, while contributing to the world’s
cations like autonomous driving and operational performance of most comprehensive picture of mobile customer experience.
and smart cities (see below), mobile networks in the near future.
The future of transport is quickly coming upon us – one kilometre Today, 54 per cent of the world’s population lives in urban areas,
loop at a time. With each new generation of car models, the auto- a proportion that is expected to increase to 66 per cent by 2050.
motive industry comes closer to its vision of highly automated Mobile communications will be an essential component to delive-
driving cars. However, to ensure that driverless cars can maintain ring on smart city promises. To enable smart cities to thrive and
connectivity and thus optimal performance and safety, we need host successful businesses in the digital era, their technological
to ensure that the technological infrastructure can manage the infrastructure must be capable of managing the increased demand
increasing demand that machines will place upon it. Therefore, on network usage. Therefore, it will be a future focus point of P3
autonomous driving scenarios play an important role in P3 com- communications to determine whether cities become truly smart
munications‘ concepts for the evolution of mobile network testing. by taking an even closer look at their advances in connectivity.
11