Transition from KPI to KQI

-- Huawei Service Quality Improvement Solution

HUAWEI TECHNOLOGIES CO., LTD.

Content
What is Service Quality Improvement Case Study

Technology Trend
User Email Video/Voice Messaging MMS Device Video telephony

Cafe

Working

Office

Watching TV

Subway

Learning

Home

Driving

Multimedia Apps

Social Networking

Network IP Transformation

SMS

PTT Location Based Services IMS

Mobile TV

User culture drives Technology Trend

HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 3

End-user Experience

CamGSMs Challenges in 3G Era

To understand end user Demand
-Understand end user interest & service distributions

Maintaining network

Behavior Performance
Total Value of Ownership

Performance during rapid growth

-Maintain VIP urban 3G network During 300 NodeB launch

Complexity
Increasing network Complexity after new services introduction
-Maintain 2G&3G networks simultaneously

Visibility Services
Defend and grow top line Services
-Support new service to launch & grow

Visibility of user experience

on offered services
-KPI User Experience (QoE), visibility to manage user experience, E2E, All-IP etc

HUAWEI TECHNOLOGIES CO., LTD.

Huawei Confidential

Page 4

What is Service Quality Improvement about?

Service Quality Improvement
Is about End-To-End Visualization of real end-user experience Improve application service quality Root cause analysis of service quality issues

Network KPI
RRC Setup Success Rate Call Drop Rate Call Setup Success rate

User Experience
No service Poor voice quality Slow to web

Customer Hotline

HUAWEI TECHNOLOGIES CO., LTD.

Huawei Confidential

Page 5

From KPI to Service Quality Improvement

As is To be

Good KPI, but Poor User Experience

CTOs Experience: Full signal bar, but difficult to call out Poor voice quality, One-way audio Connecting to internet, but cant browse internet

Build SQI System, Improve End-User Experience

Level 1 QoE: Quality of Experience, ranking from 1-5; Level 2 KQI: Key Quality Index, measure end user experiences; Level 3 Measurement KQI: Measurable, Manageable, Assessable, Improvable KQIs;

Cell KPI Value RAN Access Success >99% Rate Call Drop Rate <0.3% Cell Average KPI is Good

QoE WEB page access satisfaction

KQI WEB page access success ratio WEB page access delay

Measurement KQI E2E access success ratio DT access success rate E2E page access delay DT page Access delay

Sample DT KPI Value Coverage >99% Voice Quality (MOS) 3.6 Sample DT KPI is Good

Good KPI Good Service Quality

Huawei Confidential Page 6

HUAWEI TECHNOLOGIES CO., LTD.

From Network layer to User layer

As is To be

Indicator Care ---Monitoring Network KPI

Call Setup Success Ratio-NCR[%]
0 00 .001
99.20 99.25 99.30 99.30

VIP Care ---Quick Response to User Complaint

At same time:
Cause types: Normal Access failure Handover failure Call drop Web refresh delay Other exceptions Call record Call information Subscriber Access/release CS service PS service Handover

..
10:00 am 18:00 pm

Several Key users worse experience will not indicate from KPI, then complaint exists
HUAWEI TECHNOLOGIES CO., LTD.

00 .99 00 .89 00 .79 00 .69

Week 39 Week 40 Week 41 Week 42
Call Set-up Success Rate

KPI Normal

10:00 am

10:30 am

11:00 am

11:10 am

Signaling

Huawei Confidential

Page 7

Huawei SQI Capability - Tools & Platform

Accessibility Service Quality KQI System
E2E access completion rate E2E access duration

Retainability
Web offline ratio Speech Quality

Integrity
-One-way Audio -Echo -Time delay -Chopping -Noise -Crosstalk

User perception of the entire network

VIP&VAP user perception

Monitoring (tool) Tool Service

M2000 (traffic/log) PRS U2520 Nastar MR

Assessment & Delimiting & Identification (Service)

Nastar

E2E Active/passive joint testing

Assessment tool set

Equipment capability

End-to-End Network
BTS Abis

Resource-based refined tracking

User-based refined tracking

IP
BSC A UMG MSC MSC UMG A BSC Abis BTS

HUAWEI TECHNOLOGIES CO., LTD.

Huawei Confidential

Page 8

Huawei SQI Capability User Experience Research Team

Special Experts Dr. Dong Biography
The User Experience Principle at Huawei. Ph.D from Purdue University in the USA. More than 15 years of research and Design experience in top companies, such as IBM and eBay. He is now in charge of the user research and consumer programs in the Department of User Centered Design. Ph.D. degree in psychology from the Zhejiang University, Zhejiang, China. More than 6-year research experiences on human perception and mental processing. Expert of QoE modeling- establishing quantitative relation between KQIs and QoE.

Dr. Wan

Dr. Cui

Ph.D in Cognitive Psychology from the Zhejiang University, China. Senior user researcher at UCD center, Huawei R&D. Expert of QoE research User Experience Design and Human-Computer Interaction.

From Technologies to Research in End User Experience & Human Perception

HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 9

Tina

Zhang Tingting received her master of Arts (Sociology) in Nanjing University, accumulated wealth of experience in user research. She organized and participated in lots of large sample social research and focused on quantitative analysis.

Q1: How to Establish SQI System?

Example
Project Kick-off Tool & Service Setup KQI Data Collection KQI Analysis & Optimization KQI Monitoring & QoE Survey QoE KQI Modeling

Project End

Step 5 KQI Monitoring

1.Precision KQIs for use of each specific service 2.Time Span: 2~4 weeks, 7*24 hours 3.Sample Size About 20,000 4.Sample Structure User groups of different user behavior, and from different regions.
5

Step5 QoE Survey

1.Method Phone survey 2.Questionnaire Satisfaction of the quality of Service, including QoE and sub-QoE. 3.Performer Call center (Trained by QoE Experts) 4.Sample Size: 900~1,000 users from those monitored 5.Potential Risk: Might annoy the end users, how to control?
5

Objective
Data from KQIs monitoring

KQI monitored

Mapping KQI to QoE Step6 QoE-KQI Modeling

HUAWEI TECHNOLOGIES CO., LTD.

Huawei Confidential

Method

Regression analysis+ QoE = f(KQI1, KQI2, KQI3 ) Page 10

Subjective
QoE score from phone survey

QoE surveyed

Q2: How to Measure QoE?

Mapping Category KQI name KQI value QoE Weight score QOE score 4.5 4~4.5 3.5~4 3~3.5 <3 QoE grade Excellent Very good Good Fair Poor
E2E access completion rate (Passive Test) Accessibility Front page access delay

Example
User perception Very satisfied Satisfied Neutral Very unsatisfied Extremely unsatisfied

Integrity

HUAWEI TECHNOLOGIES CO., LTD.

Huawei Confidential

%51

1 2 3

spbk46 < spbk821~46 spbk652~821

Web page refresh delay

%52

Web page refresh success ratio

%01

Retainability

Web browsing data transfer cut-off ratio

%04 %01

5 1 2 3 4 5 1 2 3 4 5 5 1 2 3 3 3 3 4 5 1 2 3 5
4 4 4 4 4 4 4 4

spbk215~652 spbk215~652 spbk215~652 spbk215~652

spbk215 %07 < %08 ~ %07 %59 ~ %08 %99~%59 %99 %2 > %2 ~ %1 %1 ~ %5.0 %5.0 ~ %1.0 %5.0 ~ %1.0 %1.0 %1.0 spbk46 < spbk821~46 spbk652~821 spbk652~821 spbk652~821 spbk652~821 spbk215~652 spbk215 %07 < %08 ~ %07 %59 ~ %08 %99
%99~%59 %99~%59 %99~%59 %99~%59

QoE
3* 4 * 40% + 3*10% + 4*10% + 3*25% + 4*15%

3.65 (Good)

Initialized with default weight value & benchmark for KQIs , yet need to get agreement with CamGSM first.

Page 11

Content
What is Service Quality Assurance Case Study

Data Service Quality Improvement China Mobile

Customer Needs
Measurable Establish WAP/MMS/FTP/Stream data service measurement system, which is tailored for China Mobile Hangzhou; Visible Quality monitoring of WAP/MMS/Stream data services; Manageable Establish methodology to identify key factors and improve KQI; Improvement 3G Quality Improve 3G network quality. (E.g. WEB front page access delay/Unstable MMS downloading speed/Frequent 3G complaints etc)
Hangzhou, China Mobile

Solutions
Establishment - China Mobile Hangzhou SQI system through partnership
Analysis & Optimization
- Data Service Quality Analysis - User complaints Analysis - 2G/3G Hotspot Analysis - User Behavior Analysis - P2P Service Analysis & Restriction - PS Core Network Parameter Audit & Resource Evaluation

Delivery Period: Aug.2010 Dec.2010 Organization: An End-to-End project team including: Wireless / PS / SQI PDT team / Nastar Tool Team Tools: - Nastar
- Wireless tool set - Probe /CDS - U2520

HUAWEI TECHNOLOGIES CO., LTD.

Huawei Confidential

Page 13

Data Service Quality Improvement China Mobile

TOP regions WEB Delay comparison
Front Page Access Delay Before & After (s) Refresh delay Page Refresh Delay Before & After (s)

Connection access delay

BF BF AF BF AF BF AF BF AF

AF

BF

AF

BF

AF

BF

AF

TOP regions WEB access failure comparison

TOP Cell Access Failures Before & After

Access failure rate

Values: - Established customized SQI solution for Hangzhou, China Mobile. - New O&M methodology for long-term benefits; - WEB Data Service Quality was remarkably Improved; - Joint Benefits Analysis is undertaken.

BF

AF

BF

AF

BF

AF

BF

AF

HUAWEI TECHNOLOGIES CO., LTD.

Huawei Confidential

Page 14

Thank you
www.huawei.com