1、 International Telecommunication Union ITU-T P.913TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (01/2014) SERIES P: TERMINALS AND SUBJECTIVE AND OBJECTIVE ASSESSMENT METHODS Audiovisual quality in multimedia services Methods for the subjective assessment of video quality, audio quality and audiovi
2、sual quality of Internet video and distribution quality television in any environment Recommendation ITU-T P.913 ITU-T P-SERIES RECOMMENDATIONS TERMINALS AND SUBJECTIVE AND OBJECTIVE ASSESSMENT METHODS Vocabulary and effects of transmission parameters on customer opinion of transmission quality Seri
3、es P.10 Voice terminal characteristics Series P.30 P.300 Reference systems Series P.40 Objective measuring apparatus Series P.50 P.500 Objective electro-acoustical measurements Series P.60 Measurements related to speech loudness Series P.70 Methods for objective and subjective assessment of speech q
4、uality Series P.80 P.800 Audiovisual quality in multimedia services Series P.900Transmission performance and QoS aspects of IP end-points Series P.1000 Communications involving vehicles Series P.1100 Models and tools for quality assessment of streamed media Series P.1200 Telemeeting assessment Serie
5、s P.1300 Statistical analysis, evaluation and reporting guidelines of quality measurements Series P.1400 Methods for objective and subjective assessment of quality of services other than voice services Series P.1500 For further details, please refer to the list of ITU-T Recommendations. Rec. ITU-T P
6、.913 (01/2014) i Recommendation ITU-T P.913 Methods for the subjective assessment of video quality, audio quality and audiovisual quality of Internet video and distribution quality television in any environment Summary Recommendation ITU-T P.913 describes non-interactive subjective assessment method
7、s for evaluating the one-way overall video quality, audio quality and/or audiovisual quality for applications such as Internet video and distribution quality video. These methods can be used for several different purposes including, but not limited to, comparing the quality of multiple devices, comp
8、aring the performance of a device in multiple environments, and subjective assessment where the quality impact of the device and the audiovisual material is confounded. History Edition Recommendation Approval Study Group Unique ID*1.0 ITU-T P.913 2014-01-13 9 11.1002/1000/12106 _ *To access the Reco
9、mmendation, type the URL http:/handle.itu.int/ in the address field of your web browser, followed by the Recommendations unique ID. For example, http:/handle.itu.int/11.1002/1000/11830-en. ii Rec. ITU-T P.913 (01/2014) FOREWORD The International Telecommunication Union (ITU) is the United Nations sp
10、ecialized agency in the field of telecommunications, information and communication technologies (ICTs). The ITU Telecommunication Standardization Sector (ITU-T) is a permanent organ of ITU. ITU-T is responsible for studying technical, operating and tariff questions and issuing Recommendations on the
11、m with a view to standardizing telecommunications on a worldwide basis. The World Telecommunication Standardization Assembly (WTSA), which meets every four years, establishes the topics for study by the ITU-T study groups which, in turn, produce Recommendations on these topics. The approval of ITU-T
12、 Recommendations is covered by the procedure laid down in WTSA Resolution 1. In some areas of information technology which fall within ITU-Ts purview, the necessary standards are prepared on a collaborative basis with ISO and IEC. NOTE In this Recommendation, the expression “Administration“ is used
13、for conciseness to indicate both a telecommunication administration and a recognized operating agency. Compliance with this Recommendation is voluntary. However, the Recommendation may contain certain mandatory provisions (to ensure, e.g., interoperability or applicability) and compliance with the R
14、ecommendation is achieved when all of these mandatory provisions are met. The words “shall“ or some other obligatory language such as “must“ and the negative equivalents are used to express requirements. The use of such words does not suggest that compliance with the Recommendation is required of an
15、y party. INTELLECTUAL PROPERTY RIGHTS ITU draws attention to the possibility that the practice or implementation of this Recommendation may involve the use of a claimed Intellectual Property Right. ITU takes no position concerning the evidence, validity or applicability of claimed Intellectual Prope
16、rty Rights, whether asserted by ITU members or others outside of the Recommendation development process. As of the date of approval of this Recommendation, ITU had not received notice of intellectual property, protected by patents, which may be required to implement this Recommendation. However, imp
17、lementers are cautioned that this may not represent the latest information and are therefore strongly urged to consult the TSB patent database at http:/www.itu.int/ITU-T/ipr/. ITU 2014 All rights reserved. No part of this publication may be reproduced, by any means whatsoever, without the prior writ
18、ten permission of ITU. Rec. ITU-T P.913 (01/2014) iii Table of Contents Page 1 Scope 1 1.1 Limitations 1 2 References. 1 3 Definitions 2 3.1 Terms defined elsewhere 2 3.2 Terms defined in this Recommendation . 2 4 Abbreviations and acronyms 3 5 Conventions 4 6 Source stimuli . 4 6.1 Source signals r
19、ecordings . 4 6.2 Video considerations 4 6.3 Audio considerations 4 6.4 Audiovisual considerations 5 6.5 Duration of stimuli . 5 6.6 Number of source stimuli . 5 7 Test methods, rating scales and allowed changes . 6 7.1 List of methods . 6 7.2 Acceptable changes to the methods 9 7.3 Discouraged but
20、acceptable changes to the methods . 9 8 Environment . 10 8.1 Controlled environment 10 8.2 Public environment . 10 8.3 Viewing distance 11 8.4 Documentation of environment 11 9 Subjects . 11 10 Experiment design 12 10.1 Size of the experiment and subject fatigue . 12 10.2 Special considerations for
21、transmission error, rebuffering, and audiovisual synchronization impairments 12 10.3 Special considerations for longer stalling events . 12 11 Experiment implementation . 13 11.1 Informed consent 13 11.2 Optional pre-screening of subjects . 13 11.3 Post-screening of subjects 14 11.4 Instructions and
22、 training . 14 11.5 Experiment sessions and breaks . 15 11.6 Voting . 17 iv Rec. ITU-T P.913 (01/2014) Page 11.7 Questionnaires and interviews 18 12 Data analysis . 19 12.1 Documenting the experiment . 19 12.2 Calculate MOS or DMOS 19 12.3 Evaluating objective metrics 19 Annex A Method for post-expe
23、riment screening of subjects using Pearsons linear correlation . 20 A.1 Screen by PVS 20 A.2 Screen by PVS and HRC 21 Appendix I Informed consent form: example . 22 Appendix II Example instructions 24 Bibliography. 25 Rec. ITU-T P.913 (01/2014) v Introduction ITU-T P.910, ITU-T P.911 and ITU-R BT.50
24、0-13 have been successfully used for many years to perform video quality and audiovisual quality subjective assessments. These Recommendations were initially designed around the paradigm of a fixed video service that transmits video over a reliable link to an immobile cathode ray tube (CRT) televisi
25、on located in a quiet and non-distracting environment, such as a living room or office. These Recommendations have been updated and expanded as technology shifted and they have proven to be valuable and useful for the displays and questions addressed in their original scopes. However, the initial pr
26、emise of these Recommendations does not include the new paradigms of Internet video and distribution quality television. One new paradigm of video watching is an on-demand video service transmitted over an unreliable link to a variety of mobile and immobile devices located in a distracting environme
27、nt, using LCDs and other flat-screen displays. This new paradigm impacts key characteristics of the subjective test, such as the viewing environment, the listening environment, and the questions to be answered. Users of Internet video and distribution quality television are moving from one device to
28、 another and from one environment to another throughout the day, perhaps even observing the same video using multiple devices. For example, someone might start watching a sporting event on their computer using IPTV, move to an over-the-air broadcast in their living room when the IPTV connection disp
29、lays a re-buffering event, and then switch to a mobile Internet device (MID) or even a smart phone when leaving the house. Thus, subjective quality assessments into Internet video and distribution quality television ask unique questions that are not considered in the existing Recommendations. These
30、questions may require situation-specific modifications to the subjective scale (e.g., presentation of additional information defining what “good“ means in this context). Consider the pristine viewing environment defined by ITU-R BT.500-13, with its exact lighting conditions and non-distracting walls
31、. The intention is to remove the impact of the viewing environment and listening environment from the experiment. For some subjective audiovisual quality experiments, this is not appropriate. First, consider an experiment that investigates the quality of service observed by video-conferencing users
32、in an office with fluorescent lights and the steady hum of a computer. Second, consider an experiment that analyses a communications device for emergency personnel. A highly distracting background may be a critical element of the experiment design (e.g., to simulate video watched inside a moving fir
33、e truck with sirens blaring). The impact of environment is an integral part of these experiments. These questions and environments cannot be accommodated with the existing subjective assessment Recommendations. Modifying these Recommendations would reduce the value of the intended experiments and pa
34、radigms addressed therein. The main differences in this Recommendation when compared to existing ITU subjective assessment Recommendations are: 1) inclusion of multiple testing environment options (e.g., pristine laboratory environment, simulated office within a laboratory, public environment); 2) f
35、lexibility for the user to modify the subjective scale (e.g., modified words, added information); 3) applicability for interaction effects that confound the data (e.g., evaluating a device that can only accept compressed material, impact of mobility on quality perception); 4) mandatory reporting req
36、uirement (e.g., choices made where this Recommendation allows for flexibility, experimental variables that cannot be separated due to the experiment design); and 5) inclusion of multiple display technologies (e.g., flat screen, 2D, 3D). Rec. ITU-T P.913 (01/2014) 1 Recommendation ITU-T P.913 Methods
37、 for the subjective assessment of video quality, audio quality and audiovisual quality of Internet video and distribution quality television in any environment 1 Scope The devices and usage scenarios of interest herein are Internet video and distribution quality television. The focus is on the quali
38、ty perceived by the end user. This Recommendation describes methods to be used for subjective assessment of the audiovisual quality of Internet video and distribution quality. This may include assessment of visual quality only, audio quality only and/or the overall audiovisual quality. This Recommen
39、dation may be used to compare audiovisual device performance in multiple environments, and to compare the quality impact of multiple audiovisual devices. It is appropriate for subjective assessment of devices where the quality impact of the device and the material is confounded. It is appropriate fo
40、r a wide variety of display technologies, including flat screen, 2D, 3D, multi-view and autostereoscopic. 1.1 Limitations This Recommendation does not address the specialized needs of broadcasters and contribution quality television. This Recommendation is not intended to be used in the evaluation o
41、f audio-only stimuli alone, but rather audiovisual subjective assessments that may or may not include audio-only sessions. Caution should be taken when examining adaptive streaming impairments, due to the slow variations in quality within one stimulus over a long period of time. 2 References The fol
42、lowing ITU-T Recommendations and other references contain provisions which, through reference in this text, constitute provisions of this Recommendation. At the time of publication, the editions indicated were valid. All Recommendations and other references are subject to revision; users of this Rec
43、ommendation are therefore encouraged to investigate the possibility of applying the most recent edition of the Recommendations and other references listed below. A list of the currently valid ITU-T Recommendations is regularly published. The reference to a document within this Recommendation does no
44、t give it, as a stand-alone document, the status of a Recommendation. ITU-T J.144 Recommendation ITU-T J.144 (2004), Objective perceptual video quality measurement techniques for digital cable television in the presence of a full reference. ITU-T J.340 Recommendation ITU-T J.340 (2010), Reference al
45、gorithm for computing peak signal to noise ratio of a processed video sequence with compensation for constant spatial shifts, constant temporal shift, and constant luminance gain and offset. ITU-T P.78 Recommendation ITU-T P.78 (1996), Subjective testing method for determination of loudness ratings
46、in accordance with Recommendation P.76. ITU-T P.800 Recommendation ITU-T P.800 (1996), Methods for subjective determination of transmission quality. ITU-T P.800.2 Recommendation ITU-T P.800.2 (2013), Mean opinion score interpretation and reporting. 2 Rec. ITU-T P.913 (01/2014) ITU-T P.910 Recommenda
47、tion ITU-T P.910 (2008), Subjective video quality assessment methods for multimedia applications. ITU-T P.911 Recommendation ITU-T P.911 (1998), Subjective audiovisual quality assessment methods for multimedia applications. ITU-T P.1401 Recommendation ITU-T P.1401 (2012), Methods, metrics and proced
48、ures for statistical evaluation, qualification and comparison of objective quality prediction models. ITU-T X.1244 Recommendation ITU-T X.1244 (2008), Overall aspects of countering spam in IP-based multimedia applications. ITU-R BS.1534-1 Recommendation ITU-R BS.1534-1 (2003), Method for the subject
49、ive assessment of intermediate quality levels of coding systems. ITU-R BT.500-13 Recommendation ITU-R BT.500-13 (2012), Methodology for the subjective assessment of the quality of television pictures. ITU-R BT.1788 Recommendation ITU-R BT.1788 (2007), Methodology for the subjective assessment of video quality in multimedia applications. 3 Definitions 3.1 Terms defined elsewhere This Recommendation uses the following terms defined elsewhere: 3.1.1 modality ITU-T X.1244: In general usage, this term refers t
