Documente Academic
Documente Profesional
Documente Cultură
SwissQual License AG Allmendweg 8 CH-4528 Zuchwil Switzerland t +41 32 686 65 65 f +41 32 686 65 66 e info@swissqual.com www.swissqual.com
SwissQual has made every effort to ensure that eventual instructions contained in the document are adequate and free of errors and omissions. SwissQual will, if necessary, explain issues which may not be covered by the documents. SwissQuals liability for any errors in the documents is limited to the correction of errors and the aforementioned advisory services. Copyright 2000 - 2011 SwissQual AG. All rights reserved. No part of this publication may be copied, distributed, transmitted, transcribed, stored in a retrieval system, or translated into any human or computer language without the prior written permission of SwissQual AG. Confidential materials. All information in this document is regarded as commercial valuable, protected and privileged intellectual property, and is provided under the terms of existing Non-Disclosure Agreements or as commercial-in-confidence material. When you refer to a SwissQual technology or product, you must acknowledge the respective text or logo trademark somewhere in your text. SwissQual, Seven.Five, SQuad, QualiPoc, NetQual, VQuad, Diversity as well as the following logos are registered trademarks of SwissQual AG.
Diversity Explorer, Diversity Ranger, Diversity Unattended, NiNA+, NiNA, NQAgent, NQComm, NQDI, NQTM, NQView, NQWeb, QPControl, QPView, QualiPoc Freerider, QualiPoc iQ, QualiPoc Mobile, QualiPoc Static, QualiWatch-M, QualiWatch-S, SystemInspector, TestManager, VMon, VQuad-HD are trademarks of SwissQual AG. SwissQual acknowledges the following trademarks for company names and products: Adobe, Adobe Acrobat, and Adobe Postscript are trademarks of Adobe Systems Incorporated. Apple is a trademark of Apple Computer, Inc. DIMENSION, LATITUDE, and OPTIPLEX are registered trademarks of Dell Inc. ELEKTROBIT is a registered trademark of Elektrobit Group Plc. Google is a registered trademark of Google Inc. Intel, Intel Itanium, Intel Pentium, and Intel Xeon are trademarks or registered trademarks of Intel Corporation. INTERNET EXPLORER, SMARTPHONE, TABLET are registered trademarks of Microsoft Corporation. Java is a U.S. trademark of Sun Microsystems, Inc. Linux is a registered trademark of Linus Torvalds. Microsoft, Microsoft Windows, Microsoft Windows NT, and Windows Vista are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries U.S. NOKIA is a registered trademark of Nokia Corporation. Oracle is a registered US trademark of Oracle Corporation, Redwood City, California. SAMSUNG is a registered trademark of Samsung Corporation. SIERRA WIRELESS is a registered trademark of Sierra Wireless, Inc. TRIMBLE is a registered trademark of Trimble Navigation Limited. U-BLOX is a registered trademark of u-blox Holding AG. UNIX is a registered trademark of The Open Group.
Contents
1 Introduction .......................................................................................................................................... 1 Video Streaming Service ....................................................................................................................... 1 2 Configuring a Video over IP Streaming Test .................................................................................... 3 Overview ................................................................................................................................................ 3 QuickTime and RealPlayer Settings ...................................................................................................... 3 Configuring QuickTime ..................................................................................................................... 4 Configuring RealPlayer .................................................................................................................... 6 Creating a Streaming Test ..................................................................................................................... 8 Configuring a Video No Reference Test .......................................................................................... 9 Configuring a Video Full Reference Test............................................................................................. 13 Configuration - Timings ....................................................................................................................... 17 3 Viewing Test Results in NQDI .......................................................................................................... 18 Stream and Session Results ............................................................................................................... 19 Video and Audio Content Results ........................................................................................................ 22 4 Video Full Reference Streams .......................................................................................................... 23 Available Video Full Reference Streams ............................................................................................. 23 Overview of VQuad Results ................................................................................................................ 24 QuickTime Player Results .............................................................................................................. 24 RealPlayer Results ......................................................................................................................... 27 5 Streaming From YouTube ................................................................................................................. 30 Source Signals ..................................................................................................................................... 30 YouTube Test Emulating PC ............................................................................................................... 30 YouTube Test Emulating a Mobile Phone ........................................................................................... 32 Uploading Video Material on YouTube ................................................................................................ 33 Referring to YouTube Video Clips in TestManager ....................................................................... 34
Figures
Figure 1-1 Overview of Video Streaming test .................................................................................................... 1 Figure 2-1 Video Streaming - Layer model........................................................................................................ 3 Figure 2-2 QuickTime Audio settings ................................................................................................................ 4 Figure 2-3 QuickTime Streaming settings ......................................................................................................... 5 Figure 2-4 QuickTime Advanced settings .......................................................................................................... 5 Figure 2-5 RealPlayer Content / clip cache settings ......................................................................................... 6 Figure 2-6 RealPlayer Hardware settings / Video Card compatibility ............................................................... 6 Figure 2-7 RealPlayer Hardware settings / Sound Card compatibility .............................................................. 7 Figure 2-8 RealPlayer AutoUpdate setting ........................................................................................................ 7
Contents | CONFIDENTIAL MATERIALS ii
Figure 2-9 RealPlayer Message Center Preferences ........................................................................................ 8 Figure 2-10 Video Streaming Job definition ................................................................................................... 8 Figure 2-11 Select "Video No Reference" Streaming test type ......................................................................... 9 Figure 2-12 Configuration of a Video No Reference test ................................................................................ 10 Figure 2-13 Overview of timings in a Video No Reference test....................................................................... 12 Figure 2-14 Image resolution options for a video streaming test .................................................................... 12 Figure 2-15 Key-word settings for a dynamic navigation ................................................................................ 13 Figure 2-16 Test selection of a Video Full Reference test .............................................................................. 14 Figure 2-17 Settings on the General tab of a Video Full Reference test ......................................................... 14 Figure 2-18 Schematic timing overview Full Reference ............................................................................... 16 Figure 2-19 Schematic timing overview Full Reference for high resolution / frame-rate video sequences . 16 Figure 2-20 Timings required for a Video Streaming test................................................................................ 17 Figure 3-1 Importing a measurement file into NQDI ........................................................................................ 18 Figure 3-2 Data selection in NQDI ................................................................................................................... 19 Figure 3-3 Session, Service and Test Layer in NQDI ..................................................................................... 19 Figure 3-4 Session and Stream information in NQDI ...................................................................................... 20 Figure 3-5 Video Status trace providing video protocol information ................................................................ 20 Figure 3-6 Protocol information in NQDI ......................................................................................................... 21 Figure 3-7 Media information in NQDI ............................................................................................................. 21 Figure 3-8 Frame information in the media info tab-sheet .............................................................................. 22 Figure 3-9 Data analysis in NQDI .................................................................................................................... 22 Figure 5-1 Selection of Flash-Player in YouTube tests emulating PC ............................................................ 30 Figure 5-2 YouTube Full Reference tests emulating PC ................................................................................. 31 Figure 5-3 YouTube test emulating a mobile phone ....................................................................................... 32
Tables
Table 2-1 Description of the settings on the General tab for a Video No Reference streaming test .............. 10 Table 2-2 Description of the settings on the General tab for a Video Full Reference streaming test ............. 14 Table 4-1 QuickTime player results from the john clip in QCIF resolution for different coding settings ....... 24 Table 4-2 QuickTime player results from the laugh clip in QCIF resolution for different coding settings ..... 25 Table 4-3 QuickTime player results from the solothurn clip in QCIF resolution for different coding settings 25 Table 4-4 QuickTime player results from the john clip in QVGA resolution for different coding settings ..... 26 Table 4-5 QuickTime player results from the laugh clip in QVGA resolution for different coding settings ... 26 Table 4-6 QuickTime player results from the solothurn clip in QVGA resolution for different coding settings ......................................................................................................................................................................... 27 Table 4-7 RealPlayer results from the john clip in QCIF resolution for different coding settings .................. 27 Table 4-8 RealPlayer results from the laugh clip in QCIF resolution for different coding settings................ 27 Table 4-9 RealPlayer results from the solothurn clip in QCIF resolution for different coding settings ......... 28 Table 4-10 RealPlayer results from the john clip in QVGA resolution for different coding settings .............. 28 Table 4-11 RealPlayer results from the laugh clip in QVGA resolution for different coding settings ............ 29
Contents | CONFIDENTIAL MATERIALS iii
Table 4-12 RealPlayer results from the solothurn clip in QVGA resolution for different coding settings ...... 29
iv
Introduction
The Diversity Streaming Service Test Manual describes how the SwissQual Diversity software evaluates the quality of video streaming services as well as how to configure streaming service tests, that is, Diversity Streaming PC tests. Note: As of Diversity 10.6, SwissQual renamed the former Job-Type Streaming PC to Video over IP that matches to the extended capabilities of this set of tests. Diversity 10.6 enables for first time the video streaming from YouTube using a FlashPlayer client. This emulates a PC user interface. Streaming from YouTube using a mobile phone ID allows the use of RealPlayer and QuickTime as in a phone as user interface.
Objective video streaming tests require a streaming server, a video client, and an objective algorithm for video quality assessment. SwissQual uses a Darwin Streaming Server to host the and stream the video clips for a test and the QuickTime and RealPlayer software on the client computer to play the streams. The client uses the RTSP or HTTP protocol to send a streaming connection request to the Darwin server. After the connection is established, the client starts the video capture and stores the output video sequence on the hard drive.
Chapter 1 | Introduction CONFIDENTIAL MATERIALS 1
An additional package enables the video streaming from YouTube either emulating a mobile phone or a PC as user interface. In a full reference approach, the VQuad algorithm evaluates the difference between the output video sequence and the corresponding source video reference sequence on a psycho-visual scale to, predict the video quality. In a no reference approach, the VMon algorithm only evaluates the output video sequence.
This chapter describes how to configure Video No Reference and Video Full Reference Streaming PC tests in Diversity TestManager. These tests are designed to evaluate a subscribers experience with a video streaming service.
Overview
To set up and to maintain a video stream for a specific amount of time, for example, 2 minutes, a Streaming PC test requires a data connection. During the streaming process, the test performs a series of visual quality measurements and analyzes the transmission success of the incoming stream. These measurements provide a signal quality profile for the duration of the streaming activity. The test uses a mobile phone as a dial-up device and a software video player, that is QuickTime or RealPlayer, on a computer to play back the video stream. This configuration enables full access to the entire IP layer as well as all of the stream and the media layer parameters. The SwissQual VQuad algorithm objectively evaluates the quality of a full reference video capture whereas the VMon algorithm evaluates the quality of a no reference video capture. The tests capture the video from the display of a mobile phone. Streaming PC streaming tests use the following layers to provide a set of corresponding results: Data Session layer: Contains the dial-up and the IP layers Service layer: Contains the media stream Test layer: Contains the content-based quality analysis DialUp Connect VQ Data Session Streaming Test VQ VQ VQ Connect VQ Streaming Test VQ VQ VQ
A Streaming PC test analyzes the data session for dial-up times, throughput on IP protocol information, and the success state of the dial up. Note: You can also use a permanent IP connection during a test instead of a dial-up device. The quality evaluation of the streaming service is based on connection and buffering times as well as the success state. A content analysis within the stream provides detailed information about the perceived quality and the main signal degradations. Before you can create a Streaming PC test, you need to verify the QuickTime and RealPlayer settings.
Configuring QuickTime
This section describes how to configure QuickTime for use with Diversity.
In the Sound Out area, specify the following values: In the Rate list, click 48 kHz. In the Size list, click 16 bit. In the Channels list, click Mono.
3. On the Streaming tab, click Automatic in the Streaming Speed list and clear the following check boxes: Override speed for downloads Enable Instant-On
4. On the Advanced tab, click Automatic in the Transport Setup list, and then specify the following options: Clear the RTSP Proxy Server check box. Drag the Download Cache slider to 0. In the Video area, click Safe mode (GDI only).
Note: DirectX strongly affects the appearance of the video image, which in turn has a negative impact on the measurement results. Clear the Enable encoding using legacy codecs and Install QuickTime icon in system tray check boxes.
Configuring RealPlayer
This section describes how to configure RealPlayer for use with Diversity.
4. Specify the video card compatibility. a b In the Category tree, click Hardware. In the Video Card Compatibility area, drag the slider to Most reliable, and then click OK.
5. Specify the playback sound quality. a b c d In the Category tree, click Hardware. In the Sound Card Compatibility area, click Settings. Clear the Disable Direct Sound check box. Drag the Playback Quality slider to Best quality, and then click OK.
6. Disable the automatic update feature. a b In the Category tree, expand Automatic Services, and then click AutoUpdate. Clear the Automatically download and install important updates check box.
a b
In the Category tree, expand Automatic Services, and then click Message Center. In the Message Center window, clear the check boxes, and then click OK.
5. On the Video over IP tests tab and the Timings tab, specify the settings for the job. Note: For more information, see the "Configuring a Video No Reference Test" and "Configuring a Video Full Reference Test" sections on pages 9 and 13 respectively.
You can use the settings on the Video over IP tests tab to define one of the following test types: Video No Reference: Video test for unknown content, such as public streaming servers
Important: This test does not evaluate the audio signal. Video Full Reference: Video test that requires a known reference video signal. You can only use this test to evaluate video signals from a streaming server that you control.
Important: In case you have enabled he YouTube test option you can select up to four further tests YouTube Full Reference (emulate PC): Video test that requires a known reference video signal pre-stored on YouTube. This test emulates a user at a PC approaching the YouTube site and watches the video using FlashPlayer. YouTube No Reference (emulate PC): Video test that streams a video signal from YouTube at a given URL. This test emulates a user at a PC approaching the YouTube site and watches the video using FlashPlayer. YouTube Full Reference (emulate Phone): Video test that requires a known reference video signal pre-stored on YouTube. This test emulates a user using a mobile phone approaching the YouTube site and identifies itself as a mobile phone. The stream will be enabled from a special YouTube site tailored for mobile phones and allowing RealPlayer or QuickTime as viewers. YouTube No Reference (emulate Phone): Video test that streams a video signal from YouTube at a given URL. This test emulates a user using a mobile phone approaching the YouTube site and identifies itself as a mobile phone. The stream will be enabled from a special YouTube site tailored for mobile phones and allowing RealPlayer or QuickTime as viewers.
Important: Use the Video No Reference test type for unknown content, such as public streaming servers. Use the Video Full Reference test type when you evaluate a known reference video stream that originates on a server that you control. Although Video No Reference tests can use video streams in VGA and SDTV resolution, the recording and evaluation of the video is limited by Diversity processing power restrictions. Diversity records and calculates session statistics, session-related KPIs as well as the IP throughput and Media Info of each measurement. Note: VMon does not evaluate each frame in the spatial domain during image analysis and assumes that the skipped frames contain similar image artifacts as the evaluated frames.
General Tab
The following table describes the settings that you can configure on the General tab for a Video No Reference streaming test.
Table 2-1 Description of the settings on the General tab for a Video No Reference streaming test
Setting
Stream dur.
Description
Specifies the duration in seconds of the streaming test. If the test loses the stream during this time, the test status is set to DROPPED, otherwise the status is set to COMPLETED. Specifies the length of the sample in seconds to capture for the quality analysis. For best results, enter a value in the range of 6 to 15 seconds. The VMon algorithm uses the capture time to calculate a set of integrative quality values, for example, visual quality and blockiness, from the video capture sample. VMon seamlessly analyzes each portion of the complete capture. For an overview of the timing options for a video streaming test, see Figure 2-13 on page 12.
Capture time
Specifies the time after the test becomes aborted in case no connection to the video server can be established. The streaming session is set FAILED / Connection TimeOut in this case. Specifies the time after the test becomes aborted in case no video packages are received anymore. The streaming session is set DROPPED / TimeOut in this case. Specifies which video player to use for the test, that is, QuickTime or RealPlayer. Important: Only QuickTime 7.1 and higher supports H.264 video streams.
URL
10
Setting
Image Resolution
Description
Select Auto if you want the video stream to play at the native resolution, which is defined in the session description of the stream. If the player cannot interpret this description, the video plays at a resolution of 176x144 pixels, that is, QCIF. If you want to force the player to use a specific resolution, click a resolution in the list, for example, QCIF(176x144) or QVGA(320x240).
Figure 2-14 on page 12 for the available image resolution options. Note: If the forced size does not match the native image resolution of the stream, the internal re-scaling functions of the player are applied to achieve the forced size, which can result in artifacts. Stream This option is only for Video Full Reference tests.
11
Setting
Save A/V file
Description
Specifies the file saving options for a video stream during a measurement. Diversity saves the videos as *.AVI files. The AVI files contain uncompressed video in RGB format at the original frame rate and resolution. Since the size of these files can be quite large, saving and copying the files can require a significant amount of CPU load and time. Note: For best results in Diversity Q, click None.
This option is only for Video Full Reference tests. The captured video becomes compresses in three quality levels. It saves storage capacity but may lead to compression artefacts by re-viewing the recording. The compression only affects the storage after evaluation. The evaluation is made on the un-compressed video. Allows you to manually specify a dynamic link configuration by keywords to be spotted in websites and following the link behind.
Advanced
Video evaluation
Video evaluation
Report Vmon
Figure 2-13 Overview of timings in a Video No Reference test
Report VMon
Advanced Tab
In the Dynamic URL box, enter a series of keywords for dynamic link navigation. Add each new keyword to a new line. Note: You can use an asterisk (*) as a wildcard to replace an undefined number of characters in a keyword. The test searches the Web site that you specified in the URL box on the General tab, see Table 2-1 on page 10, for the first keyword in the list. If the keyword is found, the test opens page that contains the keyword and searches for the second keyword, and so on. The test only searches for the first occurrence of a keyword on a page.
Important: TestManager cannot retrieve a list of keywords from an HTTP page. Instead, you need to use external HTTP source interpreter to recognize the keywords, which you can then copy into the Dynamic URL box. Keywords are often hidden behind images.
13
A Video Full Reference test requires a known reference video stream that is on a server, which you control. For a description on how to set up the server, see the Installation - RedHat Mediaserver.pdf document. For information on the available video streams, see the "Video Full Reference Streams" chapter on page 23. Note: A blue sequence separates each content clip in the stream so that you can recognize the start and the end of a clip.
Figure 2-17 Settings on the General tab of a Video Full Reference test Table 2-2 Description of the settings on the General tab for a Video Full Reference streaming test
Setting
Stream duration
Description
Specifies the duration in seconds for the streaming test. If the test loses the stream during this time, the test status is set to DROPPED, otherwise the status is set to COMPLETED. Automatically displays the duration in seconds for the streaming sequence. The capture time is included in the video stream. The capture starts with a blue synchronization sequence and ends with the detection of the next blue marker. These sequences are between 05 and 1 seconds in length and use a sequence marker in the bottom lines to identify the next streaming sequence. Specifies the time after the test becomes aborted in case no connection to the video server can be established. The streaming session is set FAILED / Connection TimeOut in this case.
14
Capture time
Connection timeout
Setting
Stream Lost timeout
Description
Specifies the time after the test becomes aborted in case no video packages are received anymore. The streaming session is set DROPPED / TimeOut in this case. Specifies which video player to use for the test, that is, QuickTime or RealPlayer. Important: Only QuickTime 7.1 and higher supports H.264 video streams.
Player
Enter the URL that points to the server as well as the reference video sample that you want to stream. For a Video Full Reference test, the resolution of the captured video must be the same as the reference video stream, that is, select Auto. However, if the player cannot determine the resolution from the session description in the video stream, you need to force the player to use the correct resolution, that is, click the resolution in the list. Important: If you do not select the same resolution as the reference video, the quality analysis of the measurement yields incorrect results.
Select the reference video sample that you want to use for the test. Specifies the file saving options for a video stream during a measurement. Diversity saves the videos as *.AVI files. The AVI files contain uncompressed video in RGB format at the original frame rate and resolution. Since the size of these files can be quite large, saving and copying the files can require a significant amount of CPU load and time. Note: For best results in Diversity Q, click None.
Evaluates the voice signal in a video stream with SQuad-LQ in wideband mode. The captured video becomes compresses in three quality levels. It saves storage capacity but may lead to compression artefacts by re-viewing the recording. The compression only affects the storage after evaluation. The evaluation is made on the un-compressed video. Allows you to manually specify a dynamic link configuration by keywords to be spotted in websites and following the link behind.
Advanced
The concept of the full reference video evaluation is slightly different from No Reference using VMon. Due to the comparison to the reference signal there is synchronization between the two signals required. For that reason the video sequence is interlaced with short blue sequences for synch-up. In addition, the evaluation of the video sequence requires a complete capturing and cant be done in real-time as for VMon in the Video No Reference test. Since, the full reference video evaluation is very resource consuming; the capturing of the following video stops so long the previous video is evaluated.
15
Capture Time
Capture Time
Blue synch
Report VQuad
Report VQuad
Fast evaluation (low resolution, low frame rate, high performance CPU)
Figure 2-18 Schematic timing overview Full Reference
The computation time depends on the CPU performance but also on image resolution and frame rate (the more frames and the bigger the frames are, the higher is the computation effort). The computation can take between a few seconds until several 10s for QVGA or higher and frame rates > 20fps.
Capture Time
Blue synch
VQuad evaluation
Time consuming evaluation (high resolution, high frame rate, low performance CPU)
Report VQuad
Figure 2-19 Schematic timing overview Full Reference for high resolution / frame-rate video sequences
16
Configuration - Timings
Figure 2-20 shows the required timings for the video streaming test. The influence of the individual timers can be seen in the graphics below the settings.
17
This chapter describes how to view the results of a video streaming test in the NQDI post-processing software.
5. Select the data that you want to analyze. a b Click the Data Selection tab and click the Refresh icon on the Selection Criterion toolbar.
In the Test List of Current Section area, double-click a test block to view the stream and session results.
18
The figure also shows the related radio and transport layer information for a radio connection or, more specifically, HSDPA over UMTS 2100. On the Session info tab of the Analysis tab, you can analyze the results of the streaming PC test session.
19
The left panel of the Session info tab contains general information about the data session as well as the session status, that is, FAILED, DROPPED, or COMPLETED. The main panel of the tab contains the status information of the streaming service as well as other stream-related information such as defined player, URL, used protocol, image resolution, Duration, and time to first picture. The Video Status Trace tab provides an overview of protocol-related negotiations and player information, that is, buffering status.
On the Protocol info tab of the Analysis tab, you can view the IP log and a graphical representation of the IP traffic.
20
The Media info tab of the Analysis tab contains partially decoded MPEG4 part 2, H.263, and H.264 elementary streams. Note: This media information, including the video codec in NQFI, is only decoded for RTSP streams.
In Figure 3-7 and Figure 3-8, each bar of the top graph represents an individual frame, each colour represents a frame type, and the height of each bar represents the frame size in bytes. See an enlarged view in. The lower graph in Figure 3-7 is a copy of the IP-throughput graph and is included for direct comparison.
For a description and interpretation of the individual results, see the following documents: SQuad-LQ: Manual - SQuad Voice Test Result Description.pdf VMon and VQuad: Manual - VMon and VQuad Results Description.pdf
In a Video Full Reference test, the audio signal component of the AV signal is evaluated by SQuad-LQ with respect to the clean reference audio signal. In a Video No Reference test, the audio signal is not analysed regarding quality since the audio signal cannot be evaluated with respect to a pre-defined audio signal.
22
This chapter describes the pre-encoded video streams that are available for Video Full Reference tests. The available streams consist of 3 different video sequences that have been encoded with different settings. Each stream is 5 minutes long and contains multiple repetitions of an 8 second long video clip, which are separated by one second blue sequences.
Set 2: 72kbps MPEG4/H.264 video, 12.5 fps, 24kbps AAC audio o o 5min_*_QCIF_12.5fps_72k-mpg4_24k-aac_08.3gp 5min_*_QCIF_12.5fps_72k-h264_24k-aac_08.3gp
Set 3: 112kbps MPEG4/H.264 video, 12.5 fps, 24kbps AAC audio o o o o 5min_*_QCIF_12.5fps_112k-mpg4_24k-aac_08.3gp 5min_*_QCIF_12.5fps_112k-h264_24k-aac_08.3gp 5min_*_QVGA_12.5fps_112k-mpg4_24k-aac_08.3gp 5min_*_QVGA_12.5fps_112k-h264_24k-aac_08.3gp
Set 4: 172kbps MPEG4/H.264 video, 25 fps, 32kbps AAC audio o o o o 5min_*_QCIF_25fps_172k-mpg4_32k-aac_08.3gp 5min_*_QCIF_25fps_172k-h264_32k-aac_08.3gp 5min_*_QVGA_25fps_172k-mpg4_32k-aac_08.3gp 5min_*_QVGA_25fps_172k-h264_32k-aac_08.3gp
Set 5: 320kbps MPEG4/H.264 video, 25 fps, 48kbps AAC audio o o 5min_*_QVGA_25fps_320k-mpg4_48k-aac_08.3gp 5min_*_QVGA_25fps_320k-h264_48k-aac_08.3gp
Set 6: 700kbps H.264 video, 25 fps, 64kbps AAC audio o o 5min_*_QVGA_25fps_700k-h264_64k-aac_08.3gp 5min_*_VGA_25fps_700k-h264_64k-aac_08.3gp
o o
5min_*_QVGA_25fps_1200k-h264_128k-aac_08.3gp 5min_*_VGA_25fps_1200k-h264_128k-aac_08.3gp
Set 8: 3100kbps/5500kbps H.264/MPEG2 video, 25 fps, 128kbps AAC audio o o o 5min_*_VGA_25fps_3100k-h264_128k-aac_08.3gp 5min_*_SD_25fps_3100k-h264_128k-aac_08.3gp 5min_*_SD_25fps_5500k-h264_128k-aac_08.3gp
The tables in this section present VQuad results for lossless transmission. These values represent the expectation that one has for the different individual results in perfect transmission conditions. The tests were conducted on a Diversity system and involved streaming each video clip 10 to 20 times through a LAN connection. The tables also contain the 90th percentile values of the VQuad and SQuad results for each clip as well as the 10th percentile values of the individual degradations that were calculated by VQuad. The MOS values have an uncertainty of approximately +/- 0.05 MOS, while the other values have an uncertainty of approximately +/- 5 %. Important: To obtain comparable results, you need to use the same QuickTime player and RealPlayer settings as described in "QuickTime and RealPlayer Settings" on page 3.
Encoding Settings
[%]
25 29 27 17 12 13 7 11 6
[%]
9 3 3 3 2 3 2 0 0
MOS
40kbps MPEG4 video, 8.333 fps, 8kbps AMR 40kbps MPEG4 video, 12.5 fps, 8kbps AMR 40kbps H.264 video, 12.5 fps, 8kbps AMR 72kbps MPEG4 video, 12.5 fps, 24kbps AAC 72kbps H.264 video, 12.5 fps, 24kbps AAC 112kbps MPEG4 video, 12.5 fps, 24kbps AAC 112kbps H.264 video, 12.5 fps, 24kbps AAC 172kbps MPEG4 video, 25 fps, 32kbps AAC 172kbps H.264 video, 25 fps, 32kbps AAC
24
Encoding Settings
[%]
24 28 33 20 17 13 10 12 7
[%]
13 5 5 5 4 5 5 0 0
MOS
40kbps MPEG4 video, 8.333 fps, 8kbps AMR 40kbps MPEG4 video, 12.5 fps, 8kbps AMR 40kbps H.264 video, 12.5 fps, 8kbps AMR 72kbps MPEG4 video, 12.5 fps, 24kbps AAC 72kbps H.264 video, 12.5 fps, 24kbps AAC 112kbps MPEG4 video, 12.5 fps, 24kbps AAC 112kbps H.264 video, 12.5 fps, 24kbps AAC 172kbps MPEG4 video, 25 fps, 32kbps AAC 172kbps H.264 video, 25 fps, 32kbps AAC
Encoding Settings
[%]
28 32 33 25 19 18 10 18 10
[%]
15 6 6 6 6 6 6 0 0
MOS
40kbps MPEG4 video, 8.333 fps, 8kbps AMR 40kbps MPEG4 video, 12.5 fps, 8kbps AMR 40kbps H.264 video, 12.5 fps, 8kbps AMR 72kbps MPEG4 video, 12.5 fps, 24kbps AAC 72kbps H.264 video, 12.5 fps, 24kbps AAC 112kbps MPEG4 video, 12.5 fps, 24kbps AAC 112kbps H.264 video, 12.5 fps, 24kbps AAC 172kbps MPEG4 video, 25 fps, 32kbps AAC 172kbps H.264 video, 25 fps, 32kbps AAC
25
Encoding Settings
[%]
28 21 27 17 15 7 3 1
[%]
3 2 0 0 0 0 0 0
MOS
112kbps MPEG4 video, 12.5 fps, 24kbps AAC 112kbps H.264 video, 12.5 fps, 24kbps AAC 172kbps MPEG4 video, 25 fps, 32kbps AAC 172kbps H.264 video, 25 fps, 32kbps AAC 320kbps MPEG4 video, 25 fps, 48kbps AAC 320kbps H.264 video, 25 fps, 48kbps AAC 700kbps H.264 video, 25 fps, 64kbps AAC 1200kbps H.264 video, 25 fps, 128kbps AAC
Encoding Settings
[%]
40 34 37 29 20 13 3 2
[%]
5 5 0 0 0 0 0 0
MOS
112kbps MPEG4 video, 12.5 fps, 24kbps AAC 112kbps H.264 video, 12.5 fps, 24kbps AAC 172kbps MPEG4 video, 25 fps, 32kbps AAC 172kbps H.264 video, 25 fps, 32kbps AAC 320kbps MPEG4 video, 25 fps, 48kbps AAC 320kbps H.264 video, 25 fps, 48kbps AAC 700kbps H.264 video, 25 fps, 64kbps AAC 1200kbps H.264 video, 25 fps, 128kbps AAC
26
Encoding Settings
[%]
42 30 41 32 27 16 8 5
[%]
6 6 0 0 0 1 1 2
MOS
112kbps MPEG4 video, 12.5 fps, 24kbps AAC 112kbps H.264 video, 12.5 fps, 24kbps AAC 172kbps MPEG4 video, 25 fps, 32kbps AAC 172kbps H.264 video, 25 fps, 32kbps AAC 320kbps MPEG4 video, 25 fps, 48kbps AAC 320kbps H.264 video, 25 fps, 48kbps AAC 700kbps H.264 video, 25 fps, 64kbps AAC 1200kbps H.264 video, 25 fps, 128kbps AAC
RealPlayer Results
Video clip john, Resolution QCIF, Player RealPlayer
Table 4-7 RealPlayer results from the john clip in QCIF resolution for different coding settings
Encoding Settings
[%]
29 33 18 11 9
[%]
8 3 3 2 0
MOS
40kbps MPEG4 video, 8.333 fps, 8kbps AMR 40kbps MPEG4 video, 12.5 fps, 8kbps AMR 72kbps MPEG4 video, 12.5 fps, 24kbps AAC 112kbps MPEG4 video, 12.5 fps, 24kbps AAC 172kbps MPEG4 video, 25 fps, 32kbps AAC
Encoding Settings
[%]
29
[%]
12
MOS
2.36
27 CONFIDENTIAL MATERIALS
Encoding Settings
[%]
34 21 14 12
[%]
4 5 5 0
MOS
40kbps MPEG4 video, 12.5 fps, 8kbps AMR 72kbps MPEG4 video, 12.5 fps, 24kbps AAC 112kbps MPEG4 video, 12.5 fps, 24kbps AAC 172kbps MPEG4 video, 25 fps, 32kbps AAC
Encoding Settings
[%]
31 37 26 17 16
[%]
14 6 6 6 0
MOS
40kbps MPEG4 video, 8.333 fps, 8kbps AMR 40kbps MPEG4 video, 12.5 fps, 8kbps AMR 72kbps MPEG4 video, 12.5 fps, 24kbps AAC 112kbps MPEG4 video, 12.5 fps, 24kbps AAC 172kbps MPEG4 video, 25 fps, 32kbps AAC
Encoding Settings
[%]
29 28 14
[%]
2 0 0
MOS
112kbps MPEG4 video, 12.5 fps, 24kbps AAC 172kbps MPEG4 video, 25 fps, 32kbps AAC 320kbps MPEG4 video, 25 fps, 48kbps AAC
28
Encoding Settings
[%]
40 36 18
[%]
4 0 0
MOS
112kbps MPEG4 video, 12.5 fps, 24kbps AAC 172kbps MPEG4 video, 25 fps, 32kbps AAC 320kbps MPEG4 video, 25 fps, 48kbps AAC
Encoding Settings
[%]
39 37 22
[%]
6 0 0
MOS
112kbps MPEG4 video, 12.5 fps, 24kbps AAC 172kbps MPEG4 video, 25 fps, 32kbps AAC 320kbps MPEG4 video, 25 fps, 48kbps AAC
29
As of Diversity Release 10.6, a YouTube streaming option is also available. This option is an extension of the regular Diversity video streaming test; however, unlike the regular test, the video streams from the YouTube.com server and not from a video server that you control, for example, a Helix or Darwin server. The design and definition of a YouTube test is almost identical to the regular video streaming approach since the technical process of streaming, capturing and evaluation is essentially the same for both tests. One difference between the tests is that the YouTube video streams are not on a single server or restricted to a single URL. YouTube provides uploaded video content in different resolutions and bitrates for different user clients and qualities. When you upload a video clip to the site, YouTube transcodes the clip automatically, an action that becomes part of the processing chain for SwissQual to measure. The dynamic nature of the device and user dependent quality of the uploaded clips results in Diversity YouTube tests that are close to the real user experience. The YouTube test is an application of the SwissQual IP video streaming concept in that the KPIs and the results that VQuad and VMon produce are identical to the regular IP streaming tests.
Source Signals
SwissQual provides a set of video clips on YouTube that have the resolutions and qualities of video content that people typically access through mobile phones. More specifically, the contents of the clips are the same as the clips that Diversity uses for the regular Video over IP tests on Darwin and Helix media servers, that is, John, Laugh and Solothurn. These clips contain low motion head and shoulder, median motion, and high motion content. Each of the contents is available with an aspect ratio of 4:3 in three of the natively supported YouTube resolutions, that is, QQVGA (160x120), QVGA (320x240), and ~HVGA (480x360). SwissQual provides high quality video clips without visible encoding distortions for upload to the YouTube Web site. YouTube performs the actual transcoding according to the available bandwidth.
YouTube No Reference: Uses the same settings as the regular No Reference video tests except for the URL that uses HTTP to link to the video stream on YouTube, that is, http://www.youtube.com/watch?v=0PaGUwhwK-0. YouTube Full Reference: Uses the same settings as the regular Full Reference test except for the absence of a default stream selection. Unlike the regular test where the stream is stored on a media server that can be directly selected and assessed, YouTube processes and hosts the stream for the test. Note: You need to enter the complete HTTP address in the URL box. If you want the address to appear in the list of available URLs, you need to manually add the address to the SQTM reference file list. For more information, see the "Referring to YouTube Video Clips in " section on page 34.
Important: Flash Player does not report the native image resolution of the clip, which can lead to incorrect results if the 'Auto' is the image size. In the event of a un-reported resolution, Auto forces the player into QVGA resolution that might not be the correct resolution. As a result, you need to manually force the resolution whenever possible to ensure the correct native image resolution of the clip. For the Full-Reference test, this information should be available due to the uploaded clip. For the No Reference test, you need to determine this manually. For demonstration purposes only, video examples are stored in a SwissQual test account on YouTube. These examples are stored in semi-private mode and cannot be found by searching YouTube. http://www.youtube.com/watch?v=7Sibwe0HRYI http://www.youtube.com/watch?v=0PaGUwhwK-0 http://www.youtube.com/watch?v=kxqvGy0-d_c http://www.youtube.com/watch?v=5TeNIDCBd-U http://www.youtube.com/watch?v=V-A_UfaC2gg http://www.youtube.com/watch?v=UmBNNZXUNqA http://www.youtube.com/watch?v=MPPuEZSW7oM http://www.youtube.com/watch?v=5HbMtM1CGVA http://www.youtube.com/watch?v=teNsDytzN1M // John QQVGA // John QVGA // John HVGA // Laugh QQVGA // Laugh QVGA // Laugh HVGA // Solothurn QQVGA // Solothurn QVGA // Solothurn HVGA
SwissQual recommends that you create your own YouTube account for testing where you can upload the reference video clips from SwissQual. This approach lets you have complete control over the account, including the privacy settings and the number of parallel approaches to the links in the account. Important: These links are only for YouTube tests that emulate PCs with a Flash Player and not for
Chapter 5 | Streaming From YouTube CONFIDENTIAL MATERIALS 31
The settings for this test are the same as for Diversity tests that emulate a PC, except for the video players and the URLs. The following YouTube URLs are only for demonstration purposes of the mobile phone emulation mode. These video clip examples are saved in a SwissQual test account in semi-private mode and cannot be found by searching on the YouTube site.
rtsp://v8.cache1.c.youtube.com/CjYLENy73wIaLQmCRQftwZso7RMYJCAkFEIJbXYtZ29vZ2xlSARSBXdhdGNoYJTs7Mr2jKfjTAw=/0/0/ 0/video.3gp // John QQVGA rtsp://v5.cache2.c.youtube.com/CjYLENy73wIaLQntK3AIU4b20BMYJCAkFEIJbXYtZ29vZ2xlSARSBXdhdGNoYLnqoJDxy5vOTAw=/0/0 /0/video.3gp // John QVGA rtsp://v2.cache2.c.youtube.com/CjYLENy73wIaLQnld4EwII035RMYJCAkFEIJbXYtZ29vZ2xlSARSBXdhdGNoYLismYriqanjTAw=/0/0/0/v ideo.3gp // Laugh QQVGA rtsp://v2.cache6.c.youtube.com/CjYLENy73wIaLQkI2oL2UT_gVxMYJCAkFEIJbXYtZ29vZ2xlSARSBXdhdGNoYJu9s23rJvOTAw=/0/0/0/video.3gp // Laugh QVGA rtsp://v7.cache3.c.youtube.com/CjYLENy73wIaLQmD7paUEe7zMBMYJCAkFEIJbXYtZ29vZ2xlSARSBXdhdGNoYNqbhZ7nqKfjTAw=/0/ 0/0/video.3gp // Solothurn QQVGA rtsp://v7.cache7.c.youtube.com/CjYLENy73wIaLQlQGULNtMx25BMYJCAkFEIJbXYtZ29vZ2xlSARSBXdhdGNoYPrukd3RgJvOTAw=/0/ 0/0/video.3gp // Solothurn QVGA
Note: The maximum supported resolution of the YouTube mobile site is QVGA. Do not use higher resolutions for the mobile phone emulation mode.
32
33
<!-- YouTube PC emulation SwissQual, YouTube API player in Flash --> <Sample> <DisplayName>YouTube PC emulation, john qqvga 25fps</DisplayName> <FileName>http://www.youtube.com/watch?v=7Sibwe0HRYI</FileName> <Duration>303088</Duration> <Framerate>25</Framerate> <BitRate>0</BitRate> <BitRateClass>Low</BitRateClass> <Format>QQVGA</Format> <Quality>High</Quality> <Complexity>Low</Complexity> <ContainsAudio>true</ContainsAudio> <VideoCodec>H264</VideoCodec> <AudioCodec>AAC_128k</AudioCodec> </Sample> c Replace the DisplayName and FileName values in bold with the following relevant information: DisplayName: Name of the video stream to display in the TestManager selection list. Depending on the test type, the name must include the keywords "YouTube PC" or "YouTube Phone". The keyword check is not case sensitive. FileName: Enter the complete URL, including the protocol, of the video stream on the YouTube server. This URL is unique for each stream and YouTube account that you create.
Important: Keep a copy of the Reference.xml file as this file is overwritten when you update TestManager. After the update, you can replace the new file with your own.
34