1、 TIA TELECOMMUNICATIONS SYSTEMS BULLETIN Project 25 Vocoder Evaluation Mean Opinion Score Test TSB-102.BABE May 2007 TELECOMMUNICATIONS INDUSTRY ASSOCIATION The Telecommunications Industry Association represents the communications sector of Copyright Telecommunications Industry Association Provided
2、by IHS under license with EIANot for ResaleNo reproduction or networking permitted without license from IHS-,-,-NOTICE TIA Engineering Standards and Publications are designed to serve the public interest through eliminating misunderstandings between manufacturers and purchasers, facilitating interch
3、angeability and improvement of products, and assisting the purchaser in selecting and obtaining with minimum delay the proper product for their particular need. The existence of such Standards and Publications shall not in any respect preclude any member or non-member of TIA from manufacturing or se
4、lling products not conforming to such Standards and Publications. Neither shall the existence of such Standards and Publications preclude their voluntary use by Non-TIA members, either domestically or internationally. Standards and Publications are adopted by TIA in accordance with the American Nati
5、onal Standards Institute (ANSI) patent policy. By such action, TIA does not assume any liability to any patent owner, nor does it assume any obligation whatever to parties adopting the Standard or Publication. This Standard does not purport to address all safety problems associated with its use or a
6、ll applicable regulatory requirements. It is the responsibility of the user of this Standard to establish appropriate safety and health practices and to determine the applicability of regulatory limitations before its use. (From Standards Proposal No. 3-0201-A, formulated under the cognizance of the
7、 TIA, TR-8 Mobile and Personal Private Radio Standards TR-8.4 Subcommittee on Vocoders ). Published by TELECOMMUNICATIONS INDUSTRY ASSOCIATION Standards and Technology Department 2500 Wilson Boulevard Arlington, VA 22201 U.S.A. PRICE: Please refer to current Catalog of TIA TELECOMMUNICATIONS INDUSTR
8、Y ASSOCIATION STANDARDS AND ENGINEERING PUBLICATIONS or call IHS USA and Canada (1-800-525-7052 ) International (303-790-0600) or search online at http:/www.tiaonline.org/standards/catalog/ All rights reserved Printed in U.S.A. Copyright Telecommunications Industry Association Provided by IHS under
9、license with EIANot for ResaleNo reproduction or networking permitted without license from IHS-,-,-NOTICE OF COPYRIGHT This document is copyrighted by the TIA. Reproduction of these documents either in hard copy or soft copy (including posting on the web) is prohibited without copyright permission.
10、For copyright permission to reproduce portions of this document, please contact TIA Standards Department or go to the TIA website (www.tiaonline.org) for details on how to request permission. Details are located at: http:/www.tiaonline.org/standards/catalog/info.cfm#copyright OR Telecommunications I
11、ndustry Association Standards (b) there is no assurance that the Document will be approved by any Committee of TIA or any other body in its present or any other form; (c) the Document may be amended, modified or changed in the standards development or any editing process. The use or practice of cont
12、ents of this Document may involve the use of intellectual property rights (“IPR”), including pending or issued patents, or copyrights, owned by one or more parties. TIA makes no search or investigation for IPR. When IPR consisting of patents and published pending patent applications are claimed and
13、called to TIAs attention, a statement from the holder thereof is requested, all in accordance with the Manual. TIA takes no position with reference to, and disclaims any obligation to investigate or inquire into, the scope or validity of any claims of IPR. TIA will neither be a party to discussions
14、of any licensing terms or conditions, which are instead left to the parties involved, nor will TIA opine or judge whether proposed licensing terms or conditions are reasonable or non-discriminatory. TIA does not warrant or represent that procedures or practices suggested or provided in the Manual ha
15、ve been complied with as respects the Document or its contents. If the Document contains one or more Normative References to a document published by another organization (“other SSO”) engaged in the formulation, development or publication of standards (whether designated as a standard, specification
16、, recommendation or otherwise), whether such reference consists of mandatory, alternate or optional elements (as defined in the TIA Engineering Manual, 4thedition) then (i) TIA disclaims any duty or obligation to search or investigate the records of any other SSO for IPR or letters of assurance rela
17、ting to any such Normative Reference; (ii) TIAs policy of encouragement of voluntary disclosure (see Engineering Manual Section 6.5.1) of Essential Patent(s) and published pending patent applications shall apply; and (iii) Information as to claims of IPR in the records or publications of the other S
18、SO shall not constitute identification to TIA of a claim of Essential Patent(s) or published pending patent applications. TIA does not enforce or monitor compliance with the contents of the Document. TIA does not certify, inspect, test or otherwise investigate products, designs or services or any cl
19、aims of compliance with the contents of the Document. ALL WARRANTIES, EXPRESS OR IMPLIED, ARE DISCLAIMED, INCLUDING WITHOUT LIMITATION, ANY AND ALL WARRANTIES CONCERNING THE ACCURACY OF THE CONTENTS, ITS FITNESS OR APPROPRIATENESS FOR A PARTICULAR PURPOSE OR USE, ITS MERCHANTABILITY AND ITS NONINFRI
20、NGEMENT OF ANY THIRD PARTYS INTELLECTUAL PROPERTY RIGHTS. TIA EXPRESSLY DISCLAIMS ANY AND ALL RESPONSIBILITIES FOR THE ACCURACY OF THE CONTENTS AND MAKES NO REPRESENTATIONS OR WARRANTIES REGARDING THE CONTENTS COMPLIANCE WITH ANY APPLICABLE STATUTE, RULE OR REGULATION, OR THE SAFETY OR HEALTH EFFECT
21、S OF THE CONTENTS OR ANY PRODUCT OR SERVICE REFERRED TO IN THE DOCUMENT OR PRODUCED OR RENDERED TO COMPLY WITH THE CONTENTS. Copyright Telecommunications Industry Association Provided by IHS under license with EIANot for ResaleNo reproduction or networking permitted without license from IHS-,-,-TIA
22、SHALL NOT BE LIABLE FOR ANY AND ALL DAMAGES, DIRECT OR INDIRECT, ARISING FROM OR RELATING TO ANY USE OF THE CONTENTS CONTAINED HEREIN, INCLUDING WITHOUT LIMITATION ANY AND ALL INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES (INCLUDING DAMAGES FOR LOSS OF BUSINESS, LOSS OF PROFITS, LITIGATION,
23、 OR THE LIKE), WHETHER BASED UPON BREACH OF CONTRACT, BREACH OF WARRANTY, TORT (INCLUDING NEGLIGENCE), PRODUCT LIABILITY OR OTHERWISE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. THE FOREGOING NEGATION OF DAMAGES IS A FUNDAMENTAL ELEMENT OF THE USE OF THE CONTENTS HEREOF, AND THESE CONTENTS
24、WOULD NOT BE PUBLISHED BY TIA WITHOUT SUCH LIMITATIONS. Copyright Telecommunications Industry Association Provided by IHS under license with EIANot for ResaleNo reproduction or networking permitted without license from IHS-,-,-TSB-102.BABE ii Foreword The foreword is not a normative part of this doc
25、ument. The Project 25 vocoder standard was selected in 1992, and subsequently standardized by TIA as TIA-102.BABA. Since that time, various improvements have been developed in the industry, and the original suite of Project 25 standards has diversified into Phase 1 and Phase 2. During the Project 25
26、 Steering Committee meeting of June 13, 2003, it unanimously agreed that the results of a formal and independent Mean Opinion Score test were necessary for formal consideration of the use of a rate vocoder in Project 25 Phase 2. The Project 25 Steering Committee encouraged that TIA and the TIA / APC
27、O Project 25 Interface Committee (APIC) begin to schedule independent testing of a Project 25 compatible rate vocoder for comparison to the baseline Phase 1 vocoder. This testing should be conducted at an independent facility. Prior to making these tests, the participants would like to clearly under
28、stand the nature of any rate vocoder to be tested, or any enhanced vocoders that are claimed to provide significant improvements in voice quality. Furthermore, any vocoder tested should include any bit stealing proposed for encryption sync or other purposes which reduces the overall vocoder data rat
29、e. Copyright Telecommunications Industry Association Provided by IHS under license with EIANot for ResaleNo reproduction or networking permitted without license from IHS-,-,-TSB-102.BABE iii Table of Contents 1. Introduction1 1.1 Scope.1 1.2 Overview.2 1.3 Revision History .4 2. References6 3. Defin
30、itions .7 4. Test Elements.8 4.1 Vocoder Descriptions .8 4.2 Radio Channel Model .9 4.3 Acoustic Background Noise .12 5. Speech Data Bases .14 5.1 Speech Data Base Requirements 14 5.2 Speech Data Base Reference Conditions15 5.3 Speech Data Base for MOS Tests.15 5.4 Speech Data Base Levels15 5.5 Sign
31、al and Acoustic Noise Levels 16 6. Production of Digital Vocoder Recorded Files18 6.1 Software Test Elements 18 6.2 Test Signal Preparation.18 7. Subjective Evaluation of Speech Quality 20 7.1 Stimulus File Preparation20 7.1.1 Experiment Randomization .20 7.2 MOS Evaluation Laboratory.21 7.3 Transmi
32、ssion Quality Testing.21 7.3.1 Conducting the MOS Listening Test .23 7.4 Reporting of Results .25 8. MOS Result Analysis.27 8.1 ANOVA Calculation.27 8.2 Newman-Keuls Analysis 31 9. Disclosure of MOS Test Results35 9.1 Experiment 1 Results 35 9.1.1 Clean Condition Results 36 9.1.2 Fading 1 mph 1% BER
33、 Results .38 9.1.3 Fading 5 mph 1% BER Results .39 9.1.4 Fading 5 mph 3% BER Results .41 9.1.5 Fading 60 mph 3% BER Results .43 9.1.6 Fading 60 mph 5% BER Results .45 9.2 Experiment 2 Results 47 9.2.1 Clean Condition Results 48 9.2.2 Car Noise at 10 dB SNR 50 9.2.3 Babble Noise at 15 dB SNR.52 9.2.4
34、 Street Noise at 10 dB SNR 54 Copyright Telecommunications Industry Association Provided by IHS under license with EIANot for ResaleNo reproduction or networking permitted without license from IHS-,-,-TSB-102.BABE iv 10. Application of Results .56 Annex A. Tables .57 Annex B. Listener Scores .93 Cop
35、yright Telecommunications Industry Association Provided by IHS under license with EIANot for ResaleNo reproduction or networking permitted without license from IHS-,-,-TSB-102.BABE 1 1. Introduction This document describes the evaluation procedure to be employed in the assessment of various digital
36、voice coding technology enhancements for Project 25. The choice of a digital voice coder is important since it determines the speech quality performance of a communication system. The evaluation procedure is designed to give a fair comparison of the vocoder enhancements. This is accomplished through
37、 the examination of the vocoder performance in a subjective listening test in which the relative performance among the enhancements is measured in a quantitative way. The purpose of the evaluation is to enable the recommendation of any enhancements for Project 25. The purpose of the subjective liste
38、ning test is to evaluate different vocoders under a variety of operating conditions. The operating conditions were chosen to be representative of those expected to be experienced in a land mobile radio environment. Only a limited number of operating conditions are tested. To test all possible operat
39、ing conditions would lead to a test that would be too unwieldy to conduct. Section 4 describes the vocoders that are the subject of this test, and also the channel coding, and channel operating conditions. Section 4 also discusses any background noise for the test conditions. Section 5 discusses the
40、 speech data base that was chosen to be used in the vocoder testing and comparison. Section 6 contains the details of the vocoder testing as well as the speech data base used for the testing. Section 7 discusses the design of the listening tests themselves. Section 8 discusses the overall evaluation
41、 procedure of the vocoders. Section 9 describes the disclosure of the MOS test results. Lastly, an annex is included which contains additional detail. 1.1 Scope This document specifies the procedures to be employed to test the different enhancements of TIA-102.BABA compatible speech codecs. The orig
42、inal baseline speech codec from 1992 is the Improved Multi-Band Excitation system described in TIA-102.BABA, Project 25 Vocoder Description. The Improved Multi-Band Excitation speech codec is used to digitally encode the speech signal and provide forward error control for transmission at a data rate
43、 of 7200 b/s. The baseline speech codec is defined as the reference codec for the Project 25 MOS Conformance Test, reference 2. The testing of the codecs is based upon subjective tests which employ listening tests to judge overall speech quality. This document takes the approach of comparing the bas
44、eline speech codec with enhanced implementations that are now available. The baseline speech codec is based upon the Digital Voice Systems Incorporated implementation of the Improved Multi-Band Excitation algorithm on the VC-20 Project 25 hardware card, or the equivalent software version. The softwa
45、re version is of primary interest in this test. The comparison is done in two experiments. The first experiment compares the vocoders with various channel conditions as might occur on a land mobile radio channel. The second experiment compares the vocoders with speech added to background noise as mi
46、ght be anticipated in public safety environments. Copyright Telecommunications Industry Association Provided by IHS under license with EIANot for ResaleNo reproduction or networking permitted without license from IHS-,-,-TSB-102.BABE 2 1.2 Overview The way that the implementation of the speech coder
47、 is tested is by comparing it with a standardized baseline implementation. The test works by comparing the tested vocoders with the baseline, to determine if listeners perceive a statistically significant difference. There are five vocoders in the test. These vocoders are listed below, and they are
48、also described in section 4. 1. Baseline full rate Improved Multi-Band Excitation vocoder, equivalent to the vocoder selected by Project 25 in 1992. This is described as a Full Rate vocoder. It operates at a gross bit rate of 7200 b/s, a net bit rate of 4400 b/s, with 2800 b/s of parity checks for c
49、hannel error correction. This does not include any enhancements that might have been developed since 1992. 2. Enhanced full rate vocoder, interoperable with the baseline full rate vocoder. This vocoder operates at a gross bit rate of 7200 b/s, a net bit rate of 4400 b/s, with 2800 b/s of parity checks for channel error correction. It includes non
