|

WHITE PAPER

Testing Wireless Voice Quality: Capturing the Customer Experience

December 2006

Testing Wireless Voice Quality

Contents
Executive Summary.......................................................................................................................... 3 Voice Quality The Customer Experience.......................................................................................... 4 Increased Network Complexity...................................................................................................... 4 Rising Customer Expectations....................................................................................................... 5 Put Your Company Ahead of the Curve with Ditech Networks......................................................... 6 VQA.................................................................................................................................... 6 Experience Intelligenc (EXi).................................................................................................... 6 Overview of Voice Quality Testing..................................................................................................... 7 Standards Governing Voice Quality Testing.................................................................................... 7 ITU P.800............................................................................................................................... 7 ITU P.862............................................................................................................................... 8 ITU G.107.............................................................................................................................. 9 Current Industry Tools for Assessing Voice Quality.........................................................................10 Intrusive Tests for Characterizing the Network.............................................................................. 10 Ditech Networks EXi Solution: Managing the Customers Live Voice Quality Experience....................... 11 Exposing External Impairments.................................................................................................... 11 Noise and Speech Level.......................................................................................................... 12 Echo and Delay...................................................................................................................... 12 Delivering Meaningful Customer Experience Data......................................................................... 12 A Future-Proof Solution.......................................................................................................... 14 Value for Multiple Quality Initiatives........................................................................................ 14 Control the Voice Quality Experience............................................................................................... 15

Copyright 2007 Ditech Networks

2/16

Testing Wireless Voice Quality

Executive Summary
A customers experience of voice quality extends beyond the physical network, as environmental issues and user equipment can have a profound impact on how a call sounds. The users environment and equipment impact the voice quality for a subscriber even in a perfect RF environment because they directly affect the speech signal input. Recognizing this issue is important because customers are becoming less tolerant of voice problems as they become more sophisticated wireless users. Existing industry voice quality tests focus on the network, primarily radio frequency (RF), and not on the speech signal. The following quotation from the documentation of a leading wireless voice quality test tool illustrates this bias: The Speech Quality Index (SQI) is a patented Ericsson algorithm which is built upon the fact that almost all speech distortions in a mobile network are due to problems with the radio transmission.1 RF- and codec-focused tests are not unimportant, but they provide a severely limited assessment of the overall customer experience. In addition to focusing only on network component performance, existing voice quality tests are intrusive, requiring the injection of sampled speech signals and measurement of the post-processed signal. This methodology provides no insight into the number of customer calls impaired by the users environment or sub-par equipment. Ditech Networks Experience Intelligence (EXi) experience intelligence solution provides the industrys first and only live monitoring of environmental and user-equipment-related impairments on every call, all the time. Ditechs EXi solution uses the ITU-Ts G.107 standard to provide Transmission Rating (R) Factor and Mean Opinion Score (MOS) values on the combined measured impact of background noise, echo, and audio level impairments on every call. EXi provides an unprecedented visibility into the customers real-life call experience.

1. Speech Quality Index in CDMA2000, published by Ericsson.

Copyright 2007 Ditech Networks

/16

Testing Wireless Voice Quality

Voice Quality The Customer Experience

The following trends are impacting the customers voice quality experience: 1. Network complexity is introducing additional voice quality impairments. 2. Customer expectations of voice quality from wireless phones are rising, so mere annoyances in years past are now creating churn. 3. As mobile become more commonplace, more calls originate from and end in noisy or acoustic-echo prone situations, adding voice quality issues. Figure 1 outlines the issues involved in managing the customers experience of voice quality:

Ability to Manage and Mitigate

Phone

SIP Phone

LOW

Environment

Other Networks

LIMITED Phones Accessories

Network

COMPREHENSIVE

Figure 1 : : Impairments Experienced by the Customer

Increased Network Complexity

The world of voice communications is becoming increasingly complex. Despite industry consolidation, device and IP-telephony proliferation are making it more difficult for carriers to engineer high voice quality. These trends introduce impairments beyond the physical boundaries of the network, but directly impacting the customer experience of the network. Carriers test handsets before they are offered to customers. In the early days, carriers could exhaustively test the small number of handsets in their offerings. But today, the number of devices on the market is increasing and device release cycles are shortening, making it difficult for carriers to dedicate the resources to extensively test each new model. Testing is further complicated by the intense competitive pressures to quickly introduce new models and tailor handset offerings to multiple market segments. The inability to tightly control voice quality in a wide range of handsets from multiple vendors has resulted in greater incidences of acoustic echo, inappropriate audio levels, and noise introduction.

Copyright 2007 Ditech Networks

4/16

Testing Wireless Voice Quality

If controlling voice quality among multiple handset offerings is difficult, controlling quality once aftermarket accessories are added to a handset is impossible. The market is awash with device accessories, which consumers purchase off the shelf and often personalize. The millions of possible device interaction permutations cannot be tested economically. One aspect of a call that a carrier can control is its own network, and the industry has developed a robust methodology for network characterization. In network testing, two attributes are primary used to define voice quality: Radio frequency (RF) Assesses the wireless link performance using metrics such as frame erasure rate (FER), bit error rate (BER), and carrier-to-interference ratio (C/I). Network-wide quality Assesses network delay and codec selection. However, this network control ends at the network edge, where traffic is delivered to other carriers for routing within their networks. Unfortunately, these interconnections often introduce acoustic and hybrid echoes, noise, varying audio levels, and delay, which add to poor voice quality.

Rising Customer Expectations

In addition to the mounting complexity in simply maintaining the status quo in voice quality, there is evidence from a leading research survey that consumer expectations of voice quality are increasing as the wireless industry matures. This has resulted in a marked reduction in tolerance for voice quality issues such as noise, echo, and level mismatch. The survey went further to identify that this reduction in tolerance is directly related to churn. In effect, service quality issues that did not cause a customer to consider switching services last year are now considered unacceptable and cause churn. Figure 2 shows the voice quality impairments discussed above and where they originate within the overall call path.

SIP Phone PSTN BSC MSC

Environment
Acoustic Echo Ambient Noise

Acoustic Echo Audio Level

Device

RF

FER BER C/I

Network Wide
Codec Delay

Other Networks
Acoustic Echo Noise Audio Level Hybrid Echo

Figure 2 : : Voice Quality Impairment Origins in the Network

Copyright 2007 Ditech Networks

5/16

Testing Wireless Voice Quality

Put Your Company Ahead of the Curve with Ditech Networks

What can be done to mitigate the problems that come from outside a network, but directly impact customer experience and cause churn? With the following products, Ditech Networks offers the most comprehensive solution for managing and improving the customer experience beyond the reach of a wireless carriers network boundaries: VQA The most comprehensive suite of voice improvement capabilities Superior Voice Quality Improvement - Advanced Noise Cancellation (ANC) Cancels noise without distorting the speakers voice Effective even in extremely low SNR environments - Automatic Level Control (ALC) Removes level mismatches caused by both the network and the device Prevents clipped speech and distortion - Enhanced Voice Intelligibility (EVI) Improves speech clarity, especially for calls that use low bit-rate codecs in noisy environments Speech-based signal processing without amplification or distortion Dramatic statistical word recognition improvement Advanced Echo Control - Industry-best Hybrid Echo Cancellation (HEC) 192 ms tail length, the highest in industry - Bidirectional Acoustic Echo Control (AEC) Cancels echo a full 3 dB louder than the competition DSP-enabled nonlinear processing to address unique acoustic echo challenges EXi The industrys only real-time customer experience metrics that include external impairments 24/7 measurement of every call Provides visibility to impairment issues outside of the carriers network Measures noise, echo, delay, and audio level impairments Delivers ITU-T G.107-based R Factor and MOS values for every call

Copyright 2007 Ditech Networks

6/16

Testing Wireless Voice Quality

Overview of Voice Quality Testing

How do wireless carriers test for voice quality impairments and remedy them? This section discusses the capabilities and limitations of existing standards and the industry tests associated with voice quality testing.

Standards Governing Voice Quality Testing

The most common standards that assess the impact of various elements on voice quality are the International Telecommunication Unions Telecommunication Standardization Sector (ITU-T) Recommendations P.800, P.862, and G.107. These standards were designed to address specific voice quality issues, as outlined in Table 1.
Table 1 : : Focus of ITU-T Standards for Voice Quality Testing

Standard

Summary

Environment

Device

RF

Network

Other Network

ITU-T P.800 ITU-T P.862 ITU-T G.107 ITU-T P.800

Subjective Voice Quality Test Objective Voice Quality Test PESQ E-Model for Voice Quality Analysis All

Level All

All All

Codec Codec All

Noise Level All

ITU-T Recommendation P.800 identifies a subjective standard that attempts to measure listening quality and listening effort using Absolute Category Rating (ACR), Degradation Category Rating (DCR), and Comparison Category Rating (CCR) procedures. These procedures require a panel of listeners to assign voice calls a score based on specific criteria. These scores are then aggregated into Mean Opinion Scores (MOS) that are quantified from 1 (worst) to 5 (best). ITU-T P.800 is limited in its ability to provide accurate results when assessing the performance of a voice enhancement device. Dynastat, a leader in the subjective testing industry stated the following in an ITU-T submission: Standard rating scale methodologies, e.g., ACR, DCR, and CCR, can give confounding and even spurious results when used to evaluate the effects of [voice enhancement devices].2 The paper went on to postulate that the inability for people to reliably rate quality is that individuals cannot weigh the value of the two variables in voice enhancement of modification to the original signal and noise reduction level in a repeatable manner. Therefore, ITU-T P.800 is of limited use in assessing the impact of noise in the network since it cannot be relied upon to provide reliable results on the change in customer experience created by introducing noise reduction technology. ITU-T P.800 is also impractical due to the high cost of performing subjective testing for network monitoring purposes.

2. From ITU-T Submission AH-00.

Copyright 2007 Ditech Networks

7/16

Testing Wireless Voice Quality

ITU-T P.862 ITU-T P.862s Perceptual Evaluation of Speech Quality (PESQ) procedure is designed to be analogous to the ITU-T P.800 procedure, providing an objective method for predicting a subscribers experience of voice quality from a signal that passes through a communication system. PESQ uses a defined series of phrases to produce a score that can be directly mapped to the MOS value. However, PESQ is only accurate for a limited range of circumstances, none of which impact the speech signal itself from noise, echo, or level issues. Additionally, PESQ only measures during active speech no weight is given to the listening experience in pauses or silence. The three tables below are from the ITU-T P.862 Recommendation and identify the capabilities and limitations of PESQ. PESQ has demonstrated acceptable accuracy for the applications listed in Table 2.
Table 2 : : Factors for Which PESQ Has Acceptable Accuracy (ITU-T P.862)

Codec evaluation Codec selection Live network testing using digital or analog connection to the network Testing of emulated and prototype networks
Note: When environmental noise is present, the quality can be measured by passing PESQ the clean original without noise, and the degraded signal with noise.

PESQ is known to provide inaccurate predictions when used in conjunction with the variables in Table 3, or is otherwise not intended to be used with these variables.
Table 3 : : Factors for Which PESQ Has Inaccuracy (ITU-T P.862)

Test Factors

Listening levels (see note below) Loudness loss Effect of delay in conversational tests Talker echo Sidetone
Coding Technologies

Replacement of continuous sections of speech making up more than 25% of active speech by silence (extreme temporal clipping)
Applications

In-service non-intrusive measurements device Two-way communications performance

Note: PESQ assumes a standard listening level of 79 dB SPL and compensates for non-optimum signal levels in the input files. The subjective effect of deviation from optimum listening level is therefore not taken into account.

Copyright 2007 Ditech Networks

/16

Testing Wireless Voice Quality

PESQ has not been validated for measuring the test factors and applications listed in Table 4.
Table 4 : :Factors for Which PESQ Is Not a Valid Test (ITU-T P.862)

Test Factors

Music as input to a codec Listener echo Effects/artifacts from operation of echo cancellers Effects/artifacts from noise reduction algorithms Coding technologies CELP and hybrid codecs <4 kbps MPEG4 HVXC
Applications

Acoustic terminal/handset testing, e.g. using HATS

Note 1: PESQ appears to be more sensitive than subjects to front-end temporal clipping, especially in the case of missing words, which may not be perceived by subjects. Conversely, PESQ may be less sensitive than subjects to regular, short time clipping (replacement of short sections of speech by silence). In both of these cases there may be reduced correlation between PESQ and subjective MOS. Note 2: There is some evidence to suggest that PESQ is able to account for amplitude clipping, but only four conditions are known to have been included (in two 50-condition experiments) in the validation database described in clause 7.

ITU-T G.107 This Recommendation, known as the E-model, was developed as a planning tool to help network designers estimate voice quality based on certain impairment factors assigned to each piece of equipment in a transmission chain. The resulting score, the R Factor, can be related to a MOS value and used for transmission planning purposes. ITU-T G.107 has evolved beyond the original purpose of planning to become a network management tool as well. This is because G.107 has the power to capture many impairments and provide an overall quality score for live calls. For this reason, G.107 is widely used in the VoIP world for network voice quality monitoring. ITU-T G.107 has two primary advantages: It provides multiple impairment measurements on live calls It is positioned for future use as the world migrates to VoIP Ditech Networks leverages ITU-T G.107 to provide wireless carriers with visibility to the customer experience impacting impairments for echo, noise, and speech level.

Copyright 2007 Ditech Networks

/16

Testing Wireless Voice Quality

Current Industry Tools for Assessing Voice Quality

Existing testing tools are based primarily on the P.862 PESQ standard and on the Spirent (formerly SwissQual) SQUADLQ proprietary algorithm. These tools include intrusive tests, using recorded files, to measure voice quality within the deterministic network environment. These tests have two primary limitations for measuring aspects of the voice quality customer experience outside of the network component: 1. The tests rely on speech files to characterize the network performance rather than capture actual call metrics. 2. The algorithms have a limited ability to accurately assess non-network-based impairments to customer experience even for recorded speech files due to the limitations of the PESQ and SQUAD tools in modeling speech signal impairments (as was identified in the previous discussion of ITU P.862). Table 5 below identifies the capabilities of various toolsets with respect to the overall customer experience.
Table 5 : : Ability of Various Tools to Measure Overall Customer Experience

Tool

Algorithm

Live Calls

Environment

Device

RF

Network

Other Network

TEMs TEMs SwissQual Invex3G Invex3G QVoice Agilent GL

PESQ SQI SQUAD-LQ PESQ SQUAD-LQ PESQ PESQ PESQ

No Yes No No No No No No Audio Level Audio Level

All All All All All All All All

Codec Codec Codec Codec Codec Codec Codec Codec

Intrusive Tests for Characterizing the Network

The objective of intrusive tests is to characterize the network while minimizing introduced impairments. Intrusive testing is a very effective tool for this because the network component of the voice quality experience is a controlled environment that will provide consistent output for consistent input. Therefore, under a given network load and with a defined RF environment the network will deliver the same quality level every time. The same is not true for impacts to the speech signal that occurs on actual customer calls. A noisy background can cause degradation to the customer experience even in a high-quality RF environment. The same is true of acoustic echo or improper speech level coming in from another network or a poor device. These issues cannot be identified using recorded files, but only through an assessment of actual calls.

Copyright 2007 Ditech Networks

10/16

Testing Wireless Voice Quality

Ditech Networks EXi Solution: Managing the Customers Live Voice Quality Experience
Ditechs EXi solution delivers new visibility into impairments that occur during real calls, accurately identifying where and when impairments occur, and quantifying the impairments impact on the subscribers experience. Ditechs EXi provides the following features: Measurement of live customer experience Visibility to impairments outside of the carriers network R Factor and MOS value per call based on measured noise, echo, and level impairments Table 6 lists EXis current and future capabilities and features.

SIP Phone BSC PSTN

Environment
Acoustic Echo Ambient Noise

Acoustic Echo Audio Level

Device

FER BER C/I

RF

SNRI NPLR WAEPL EPL MOS R-Value

VED

MSC

Network Wide
Codec Delay

Other Networks
Acoustic Echo Noise Audio Level Hybrid Echo

Table 6 : : Ditech Networks EXi Capabilities

Tool

Algorithm

Live Calls

Environment

Device

RF

VED

Network

Other Network

Ditech EXi

ITU-T G.107

Yes

All

All

Future

All

Future

All

Accurate Production Environment Measurements Ditechs EXi provides 24-hour monitoring on a per-call basis. Live monitoring delivers accurate measurements of subscribers actual experiences using a production network.

Exposing External Impairments

With Ditechs EXi technology, for the first time carriers have visibility into impairments that originate outside of their network. Ditechs solution monitors for acoustic echo, noise, audio level, and hybrid echo pinpointing the impairments and removing them. Ditechs EXi also helps the carrier identify problems caused by other networks, further aiding in troubleshooting. EXi measures the following voice quality impairments: Noise level Speech level Hybrid echo delay Hybrid echo return loss Acoustic echo delay
Copyright 2007 Ditech Networks 11/16

Testing Wireless Voice Quality

Noise and Speech Level An increasing number of calls originate from noisy urban environments, and background noise can be a major degradation factor to the listening quality of a call. Ditechs EXi measures average per-call speech level, noise level, and signal-tonoise ratio (SNR) in both directions on every call. Echo and Delay The migration of voice networks from circuit to packet technologies is resulting in greater and more variable latency. The majority of legacy hybrid echo cancellers are unable to handle this added delay, leaving callers unprotected from annoying hybrid echo. In addition, variability in terminal equipment (handsets, headsets, and handsfree kits) continues to expand without necessarily providing adequate acoustic isolation, exposing more users to acoustic echo. Ditechs EXi provides comprehensive statistics about echo delay and return loss for both hybrid (linear) echo and acoustic (non-linear) echo. It monitors from 0 to 400 ms in both directions, ensuring complete and accurate measurement coverage. For a statistical understanding of how echo and delay combine to affect customer satisfaction, Ditechs EXi also provides Echo Objection Rates per ITU-T G.131.

Delivering Meaningful Customer Experience Data

Ditechs EXi solution delivers a unified MOS score of the actual subscriber voice quality experience, measuring all calls, all the time, for multiple quality impairments. The solution also complements existing test equipment for easy deployment into an existing carrier environment. Per-call listening and conversational R Factors and MOS values are continuously and non-intrusively computed based on the measured voice quality impairments (speech, noise, and echo), as well as codec type. Ditechs objective scores follow the ITU-T G.107 E-Model standard using an industry-unique approach of DSP-based measurements of the live voice signal. Ditech has partnered with Telchemy, a leading provider of performance monitoring technology, and employs their VQmon computational model to provide R Factors and MOS values for the following: Listening Quality (LQ) This score includes noise and speech level. It does not include impairments that affect conversation, such as delay. Conversational Quality (CQ) This score includes the impairments measure for the LQ score and adds echo and delay, which affect conversational quality.

Copyright 2007 Ditech Networks

12/16

Testing Wireless Voice Quality

The impact of the impairments is reported in any of the following mechanisms: Transmission Rating (R) Factors o Listening Quality (R-LQ) o Conversational Quality (R-CQ) Mean Opinion Scores (MOS) o Listening Quality (MOS-LQ) o Conversational Quality (MOS-CQ) Weighted acoustic echo path loss Echo Objection Rate per ITU-T G.131 Ditech EXi data can be aggregated from the local, regional, and national levels to gain visibility into the customer experience at any point in the network. Results are easily integrated into existing management and executive dashboards for at-a-glance status reporting. Below is an example of how EXi data can be graphically presented to show the level of voice quality customers are experiencing.

MOS Conversational Quality 18,000 16,000 16,100 15,207 14,563 14,000 14,750 14,332 12,000 12,328 10,000 10,432 8,923 8,000 6,000 4,000 2,000 2,447 1,254 1 2 3 MOS Score 4 5

Call Count

Rin Rout

Figure 3 : : Sample EXi Data

Copyright 2007 Ditech Networks

1/16

Testing Wireless Voice Quality

A Future-Proof Solution Ditech EXi uses a Telchemy scoring algorithm that delivers an R Factor and MOS values per call. The Telchemy algorithm is widely adopted in IP networks, with more than two million licenses issued. This ensures that as wireless networks evolve to VoIP, the quality measurements initiatives put in place today will continue to be of value in the future. Value for Multiple Quality Initiatives Ditech EXi is easily integrated with other tools from multiple vendors to improve voice quality management and expand visibility into call quality and the origin of impairments. MOS data can be correlated with other network data, such as cell identifier and Electronic Serial Number (ESN). Integrating these multiple data points provides wireless carriers the ability to recognize and identify network trouble points, receive automatic alerts when a problem arises, and provide advanced reporting for network planning and customer Service Level Agreements (SLAs).
Partners Ditech

Figure 4 : : Managing Network-wide Voice Quality

Copyright 2007 Ditech Networks

14/16

Testing Wireless Voice Quality

Control the Voice Quality Experience

Ditech EXi provides the ability to manage the complete end-to-end voice quality experience, so that wireless carriers can use their existing PESQ, SQUAD-LQ, or SQI tools for RF-related network quality measurements, and EXi for monitoring their noise, echo, and level impairments (Figure 5).

PESQ
Phone SIP Phone

SQUAD-LQ Fair

Ditech EXi Good

Environment

Other Networks

Poor

Poor Phones Accessories

Fair

Good

Network

Good

Good

Fair

Figure 5 : : Tools for Controlling the Customer Experience

Ditechs EXi and VQA solutions provide carriers with visibility into and mitigation of impairments that, until now, have been outside of a carriers control. Used in conjunction with industry-standard voice quality network tests, wireless carriers now have tools to identify and mitigate all aspects of voice quality impairments impacting the customer experience.

Copyright 2007 Ditech Networks

15/16

Testing Wireless Voice Quality

About Ditech Networks

Ditech Networks supplies voice processing equipment for telecommunication networks around the world. Ditech Networks technology solutions include voice, media processing, SIP, and security delivered on carrier-grade scalable platforms to enhance the delivery of communications services over mobile, Voice over IP, and wireline networks. Ditech Networks customers are premier network operators including Verizon, AT&T, Orascom Telecom, and others that collectively serve more than 150 million subscribers.

Corporate Headquarters
US Sales West Ditech Networks 825 East Middlefield Rd. Mountain View, CA 94043 USA 1-800-234-0884 (Toll Free) 1-650-623-1300 (Direct) 1-650-564-9599 (Fax) NA-Sales@ditechnetworks.com US Sales Office East and Central Ditech Networks 8360 Greensboro Dr. McLean, VA 22102 USA 1-800-234-0884 (Toll Free) 1-650-623-1300 (Direct) 1-650-564-9599 (Fax) NA-Sales@ditechnetworks.com Canada Sales Office Ditech Networks 2275 Lakeshore Blvd West, Suite 500 Toronto, ON M8V 3Y3 Canada 1-416-255-7776 (Phone) NA-Sales@ditechnetworks.com Mexico Sales Office Ditech Networks Torcuato Tasso 245-6o. Piso Col. Polanco Mxico, D.F. C.P. 11570 Mxico +52 (55) 5254-5422 (Main) +52 (55) 5350-8679 (Sales) Mexico-Sales@ditechnetworks.com South-East Asia Sales Office Ditech Networks Lippo Plaza 3rd Floor Jl. Jend. Sudirman Kav. 25 Jakarta 12920 Indonesia +62 (21) 5291 3780 (Phone) +62 (21) 522 1977 (Fax) SE-Asia-Sales@ditechnetworks.com South Asia Sales Office Ditech Networks No. 8/11, Sarvapriya Vihar New Delhi 110016 India +91 9810372555 (Phone) South-Asia-Sales@ditechnetworks.com China Sales Office Ditech Networks Unit 3010, No. 500 Xiangyang South Road Xuhui District Shanghai 200031 PRC +86 (21) 5456 0305 (Phone) China-Sales@ditechnetworks.com Brazil Sales Office Ditech Networks Alameda Araguaia, 933 Alphaville Barueri, SP 06455-000 Brazil +55 (11) 4208-6266 (Main) +55 (21) 3521-5543 (Sales) Brazil-Sales@ditechnetworks.com South-America-Sales@ditechnetworks.com Middle East/Africa Sales Office Ditech Networks 21 El-Fawakeh St., 3rd Floor Dokki, Giza 12311 Egypt +20 2 336-5100 (Phone) +20 2 761-8964 (Fax) MEA-Sales@ditechnetworks.com Spain Sales Office Ditech Networks Torres Quevedo, 1 (P.T.M.) 28760 Tres Cantos Madrid Spain +34 (91) 803 74 44 (Main) +34 (91) 829 26 90 (Sales) Spain-Sales@ditechnetworks.com Europe-Sales@ditechnetworks.com

Copyright 2007 Ditech Networks. All rights reserved. VQA and Experience Intelligence are trademarks of Ditech Networks. All other brands are the property of their respective owners. Specifications may change without notice. This document was last revised 03/07.

16/16