Voice quality assessment

SW/HW migration and feature

activation
- Non-regression test plan -
Anca Beca
NE/ Voice Quality Epert
Alcatel-!ucent "omania
Alcatel-Lucent File Date Edition Page
B11 VQ measurement 19/3/09 1.0 1
Table of Contents
1 SCOPE..............................................................................................................................................................................
2 NEEDED EQ!P"EN#.................................................................................................................................................
2.1 PREREQUISITES...........................................................................................................................................................
2.2 GENERAL OVERVIEW ON QVOICE...............................................................................................................................
2.3 SPEEC SA!PLE CLASSI"ICATION TRESOL#S.........................................................................................................
3 P$OPOSED #ES#S........................................................................................................................................................
% E&PEC#ED $EPO$#ED $ESL#S............................................................................................................................
Alcatel-Lucent File Date Edition Page
B11 VQ measurement 19/3/09 1.0 2
1 Scope
#$e scope of t$is document is to define t$e non-regression test cases% needed equipment and
time sc$edule for voice quality assessment at t$e introduction of ne& soft&are release% ne&
optional features or $ard&are en$ancements t$at mig$t impact t$e voice quality'
(t$er tests% t$at mig$t )e required to asses t$e improvement in voice quality% do not ma*e t$e
scope of t$is document'
#$e tests s$all perform +S to +S and +S to ,S-N calls for E."% ."% H"% A+"-." and A+"-H"' ,f
one or more of t$is codecs is not availa)le in t$e net&or*% t$e correspondent tests &ill )e
removed'
.or a fair comparison on field of t$e codec performance )efore and after t$e c$anges in t$e
net&or* /e0 feature activation1% t$e tests $ave to )e done as muc$ as possi)le in t$e same radio
conditions and at t$e same net&or* load' .or t$is% static tests are recommended to )e
performed' #$e static tests $ave to )e performed )ot$ in good and )ad radio conditions to
evaluate t$e )est and &orst case scenario'
As t$e voice quality in t$e BSS is affected not only )y t$e radio conditions )ut also )y $andovers
it is recommended to perform as &ell mo)ility tests' .or t$e mo)ility tests% t$e same route
s$ould )e drove% in t$e same conditions and in t$e same direction% for t$e measurements )efore
and after feature activation'
Alcatel-Lucent File Date Edition Page
B11 VQ measurement 19/3/09 1.0 3
2 Needed equipment
2.1 Prerequisites
,n order to carry on &it$ t$e measurements t$e follo&ing points s$ould )e ta*en into
consideration
#$e QVoice tool /from AS2(+1 s$ould )e used% equipped &it$ at least 3 test mo)iles &it$
forcing functions /speec$ version% )and1'
QV+ /for mo)ile to mo)ile calls1 and QVS /for mo)ile to ,S-N calls1 are required
QV4 for post-processing can )e remotely
#$e frequency )and $as to )e t$e same during t$e tests'
Availa)le ,S-N line at t$e +S2 is required
Significant num)er of samples is necessary for relia)le statistical distri)utions /5677
samples/mo)ility test case and 377 samples/static test case1'
#$e net&or* s$ould allo& ena)ling E."% H"% A+" 8 ."% A+" 8 H"
#$e A+" parameters s$ould )e t$e A!9 optimi:ed0
2odec su)set0 53'3% 57'3% ;'< and <';6
#$res$olds0 #5=<dB% #3=;dB% #>=57dB% Hyst=3dB
.or t$e stationary tests% at least t&o points &it$ good radio conditions and t&o &it$ )ad radio
conditions $ave to )e considered' .or t$is% 3 different cells &it$ normal load $ave to )e c$osen'
#$e parameter settings in t$e cells c$osen for t$ese tests $ave to )e t$e ones used in t$e most
part of t$e net&or*'
#$e route for t$e mo)ility tests $as to )e defined /same route for )efore/after tests1 including
good and )ad radio conditions% &it$ reduced num)er of $andovers /avoid ping-pong1' #$e route
$as to de drove in t$e same direction &it$ ma' 67*m/$'
-rive test car needed'
N4( access needed% for t$e monitoring of t$e net&or* ?4,s in t$e tested area% during t$e tests% to
see any a)normal )e$avior of t$e net&or* t$at mig$t impact t$e results of t$e tests'
#$e reference tests and t$e tests after feature activation or s&/$& migration $ave to )e
performed in t$e same traffic conditions% same time of t$e day /load conditions1 and eactly on
t$e same route /for t$e mo)ility tests1'
2.2 General overview on Qvoice
#$e QVoice consists of t&o modules /QV+ @ QVS1' #$e QV+ /QVoice +o)ile unit1 is t$e mo)ile
measurement tool installed in t$e car and QVS /QVoice Server unit1 is t$e stationary one eisting
on t$e ,S-N side'
#$e principle of operation is t$e follo&ing0 t$e AQV+A /or AQVSA according to t$e direction%
9!/-!% &e &ant to measure1 sends an audio pattern in a c$osen language /Englis$% .renc$%
Berman or 2antonese1' #$e AQVSA /or AQV+A1 receives t$e pattern% compares it to t$e reference
and calculates a note% under t$e 4ESQ metric' 6 seconds speec$ samples in different languages
are transmitted in )ot$ directions'
After t$e measurements are terminated% log files /C'Q+-1 from )ot$ modules are imported in t$e
QVoice 4resentation station in order to perform analysis and produces reports and maps'
.or +S-+S measurements% t$e QV+ is )ot$ t$e transmitter and t$e receiver of t$e audio pattern%
as t$e ,S-N/4S#N net&or* is not involved in t$e call' #$e call is esta)lis$ )et&een t&o mo)iles
located at t$e same QV+'
Alcatel-Lucent File Date Edition Page
B11 VQ measurement 19/3/09 1.0 $
.igure 50 -ifferent modules of QVoice tool for +S-,S-N measurement
#$e 4rinciple of all end-to-end Voice Quality tools is t$e follo&ing0
5' A reference speec$ sample ,N'
3' A degraded speec$ sample (9#'
>' #$e comparison )et&een t$em giving a score'
"%&ure 2' Pr%n(%)*es +, V+%(e -ua*%t. t++*s
2.3 Speech sample classification thresholds
,n t$e follo&ing ta)le t$e ranges of t$e 4ESQ and corresponding classification are presented'
Speech
Quality
PESQ
Speech Sample
From To
Ecellent 6 >'6
Excellent
Sample.wav
Bood >'6 3';
Good Sample.wav
.air 3'; 3'5
Fair Sample.wav
4oor 3'5 5'6
Poor Sample.wav
Bad 5'6 5
Bad Sample.wav
Alcatel-Lucent File Date Edition Page
B11 VQ measurement 19/3/09 1.0 /
#a'le 1( S)eec* +a,)le cla++i-ication
Alcatel-Lucent File Date Edition Page
B11 VQ measurement 19/3/09 1.0 0
3 Proposed tests
#$e measurement test cases s$ould )e t$e ones in t$e follo&ing ta)le' Eac$ of t$e tests
proposed )ello& s$ould )e made )efore and after t$e s&/$& migration or feature activation'
Also% t$e results of t$e tests $ave to )e sent )y t$e measurement team after eac$ day% so t$at
&e can find in time some pro)lems t$at mig$t appear and correct t$em'
Time Tas
5 day Setting up equipment /QV+ D QVS1% testing equipment% validating
proposed drive route% administrative issues
5 day
/377 samples E
37 min1
(~ 4h with 4
mobiles)
Static measurements in t&o selected cells in good radio conditions
/"!ev F= -G7dBm% "Qual 710
+o)ile to ,S-N0 ." 8 ,S-N /377 samples/cells1
E." 8 ,S-N /377 samples/cell1
H" 8 ,S-N /377 samples/cell1
A+"-." 8 ,S-N /377 samples/cell1
A+"-H" 8 ,S-N /377 samples/cell1
+o)ile to mo)ile0 ." 8 ." /377 samples/cell1
E." 8 E." /377 samples/cell1
H" 8 H" /377 samples/cell1
A+"-." 8 A+"-." /377 samples/cell1
A+"-H" 8 A+"-H" /377 samples/cell1
5 day
(~ 4h with 4
mobiles)
Static measurements in t&o selected cells in )ad radio conditions
/-H7 I= "!ev I= -577dBm% "Qual 8 no constraints10
+o)ile to ,S-N0 ." 8 ,S-N /377 samples/cell1
E." 8 ,S-N /377 samples/cell1
H" 8 ,S-N /377 samples/cell1
A+"-." 8 ,S-N /377 samples/cell1
A+"-H" 8 ,S-N /377 samples/cell1
+o)ile to mo)ile0 ." 8 ." /377 samples/cell1
E." 8 E." /377 samples/cell1
H" 8 H" /377 samples/cell1
A+"-." 8 A+"-." /377 samples/cell1
A+"-H" 8 A+"-H" /377 samples/cell1
3'6 days
(1.5 days with 4
mobiles)
+o)ility test measurements in t$e selected area0
+o)ile to ,S-N0 ." 8 ,S-N /5677 samples1
E." 8 ,S-N /5677 samples1
H" 8 ,S-N /5677 samples1
A+"-." 8 ,S-N /5677 samples1
A+"-H" 8 ,S-N /5677 samples1
Alcatel-Lucent File Date Edition Page
B11 VQ measurement 19/3/09 1.0 1
+o)ile to +o)ile0 ." 8 ." /5677 samples1
E." 8 E." /5677 samples1
H" 8 H" /5677 samples1
A+"-." 8 A+"-." /5677 samples1
A+"-H" 8 A+"-H" /5677 samples1
> days 4ost processing and report creation
Note 1! 377 samples are measured in a)out 37 minutes and 5677 samples are measured in a)out
3 $ours'
Note 2! #$e duration of t$e tests is estimated in t$e supposition t$at t&o mo)iles are used' ,n
t$is case% for t$e +S-,S-N test cases% on eac$ mo)ile is forced a different codec% and t&o test
cases are performed at t$e same time' .or t$e +S-+S test cases% )ot$ mo)iles are used to
perform one test' ,f more mo)iles are used /< or G1 t$e duration of t$e tests decreases'
Here are summari:ed t$e foreseen duration% resources for all t$e tests descri)ed a)ove'
2odec #ype of test Human resources #ools -uration
." 8 ,S-N
Static good radio conditions
5car and driver D
5 person
AS2(+
QVoice
3C37 min
Static )ad radio conditions 3C37 min
+o)ility 3$
." - ."
Static good radio conditions 3C37 min
Static )ad radio conditions 3C37 min
+o)ility 3$
." codec All test cases G$ @ <7 min
E." 8 ,S-N
Static good radio conditions
5car and driver D
5 person
AS2(+
QVoice
37 min
Static )ad radio conditions 37 min
+o)ility 3$
E." - E."
Static good radio conditions 37 min
Static )ad radio conditions 37 min
+o)ility 3$
E." codec All test cases G$ @ <7 min
H" 8 ,S-N
Static good radio conditions
5car and driver D
5 person
AS2(+
QVoice
37 min
Static )ad radio conditions 37 min
+o)ility 3$
H" - H"
Static good radio conditions 37 min
Static )ad radio conditions 37 min
+o)ility 3$
H" codec All test cases G$ @ <7 min
A+"-." 8 ,S-N
Static good radio conditions
5car and driver D
5 person
AS2(+
QVoice
37 min
Static )ad radio conditions 37 min
+o)ility 3$
A+"-." - A+"-."
Static good radio conditions 37 min
Static )ad radio conditions 37 min
+o)ility 3$
A+"-." codec All test cases G$ @ <7 min
A+"-H" 8 ,S-N
Static good radio conditions
5car and driver D
5 person
AS2(+
QVoice
37 min
Static )ad radio conditions 37 min
+o)ility 3$
A+"-H" - A+"-H"
Static good radio conditions 37 min
Static )ad radio conditions 37 min
+o)ility 3$
A+"-H" codec All test cases G$ @ <7 min
All codecs All test cases E6 days
Alcatel-Lucent File Date Edition Page
B11 VQ measurement 19/3/09 1.0 2
Note! #$e a)ove ta)le doesnJt ta*e into account t$e possi)ility of performing more +S-,S-N tests
at a time using more mo)iles' #$e ta)le is intended to )e $elpful in t$e assessment of t$e tests
duration in case t$at t$e net&or* doesnJt support some of t$e codecs and some proposed tests
$ave to )e removed'
#$e estimation of t$e total test duration% ta*ing into account t$e a)ove considerations0
A(t%3%t.
#urat%+n
#eta%*s
#est preparation - 5 day
"eference tests
." codec 5 day
E." codec 5 day
H" codec 5 day
A+"-." codec 5 day
A+"-H" codec 5 day
"eference tests All codecs 6 days
.ollo&-up tests
." codec 5 day
E." codec 5 day
H" codec 5 day
A+"-." codec 5 day
A+"-H" codec 5 day
.ollo&-up tests All codecs 6 days
4ost processing and
report creation
- > days
#est trial - 3 &ee*s
#$e follo&ing ?4,s is recommended to )e monitored during t$e period t$at t$e tests are
performed and in t$e designated test area /for mo)ility tests1 or in t$e designated test cells /for
static tests1'
CS Summary "# monitorin$
K S-22H Assign 2ongestion K H( ,nc ,ntra .ail
K S-22H Assign .ail K H( ,nc ,ntra 2ong
K S-22H -rop K H( ,nc ,ntra Success
S-22H #raffic Erl #otal K H( (ut ,ntra "(2
"#2H #raffic Erl #otal K H( (ut ,ntra -rop
K "#2H Assign 2ongestion K H( (ut ,ntra Success
K "#2H Assign Success K H( ,nc ,nter .ail
K2all Setup Suc K H( ,nc ,nter 2ong
K "#2H fail "adio K H( ,nc ,nter Success
K 2all -rop K H( (ut ,nter "(2
K "#2H -rop K H( (ut ,nter -rop
K H( (ut ,nter Success
KH( Better 2ell
Alcatel-Lucent File Date Edition Page
B11 VQ measurement 19/3/09 1.0 4
% E&pected reported results
After t$e measurements campaign of eac$ day% t$e results s$ould )e given )y t$e su)contractor
in Ecel format /or 2SV1 containing t$e follo&ing fields0
#$e file containing t$e 4ESQ/4A2E results0
C:\Work\Issues\
oice!"ualit#\Exemple\$e%uested &ormat\EF$!Pace!C'(!)o*!+,(-+--..xls
#$e file containing t$e H( information0
/:\work\Issues\
oice!"ualit#\Exemple\$e%uested &ormat\EF$!01!)o*!C'(!)o*!+,(-+--..xls
#$e file containing t$e "!ev/ "Qual information0
C:\Work\Issues\
oice!"ualit#\Exemple\$e%uested &ormat\EF$!$x"!$x2!C'(!)o*!+,(-+--..xls

#$e final report s$all include t$e comparison of Voice Quality per codec and in general% )efore
and after soft&are migration or feature activation in static and drive test conditions'
#$e comparison &ill include
average voice quality /&it$ and &it$out $andovers1
sample distri)ution /according to sample classification1%
comparison on map /"!ev% "Qual% Voice Quality distri)ution1
#$e outcome of t$e test &ill reveal if t$e activation of t$e feature )roug$t any c$anges in t$e
Voice Quality'
EN' #F '#C()ENT
Alcatel-Lucent File Date Edition Page
B11 VQ measurement 19/3/09 1.0 10