1、_ SAE Technical Standards Board Rules provide that: “This report is published by SAE to advance the state of technical and engineering sciences. The use of this report is entirely voluntary, and its applicability and suitability for any particular use, including any patent infringement arising there
2、from, is the sole responsibility of the user.” SAE reviews each technical report at least every five years at which time it may be revised, reaffirmed, stabilized, or cancelled. SAE invites your written comments and suggestions. Copyright 2016 SAE International All rights reserved. No part of this p
3、ublication may be reproduced, stored in a retrieval system or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of SAE. TO PLACE A DOCUMENT ORDER: Tel: 877-606-7323 (inside USA and Canada) Tel: +1 724-776-497
4、0 (outside USA) Fax: 724-776-0790 Email: CustomerServicesae.org SAE WEB ADDRESS: http:/www.sae.org SAE values your input. To provide feedback on this Technical Report, please visit http:/www.sae.org/technical/standards/AIR5909 AEROSPACE INFORMATION REPORT AIR5909 Issued 2016-02 Prognostic Metrics fo
5、r Engine Health Management Systems RATIONALE With the increasing application of prognostic technologies within the propulsion system community, there is a need for standardized metrics that can be used by developers and end users alike for assessing the performance of specific software algorithms fo
6、r estimating and predicting remaining useful life. The focus of this document is to introduce a variety of metrics that can be used for this purpose. FOREWORD Engine Health Management (EHM) prognostic technologies are becoming more common and are being tied directly to maintaining engine reliability
7、 and performance. The implementation of these prognostic technologies is in turn supporting the more automated and integrated logistics concepts that are being fielded for many commercial and defense systems. This document provides an overview and introduction to the metrics that can be applied for
8、assessing the capability of prognostic technologies for gas turbine engines. This will include a discussion of the required definitions, necessary data sources and a comprehensive set of metrics to assess the performance and effectiveness of prognostic methods. SAE INTERNATIONAL AIR5909 Page 2 of 26
9、 TABLE OF CONTENTS 1. SCOPE 3 1.1 Purpose . 3 2. REFERENCES 3 2.1 Applicable Documents 3 2.1.1 SAE Publications . 3 2.2 External Publications . 3 2.3 Definitions/Acronyms/Abbreviations . 4 3. PROGNOSTIC METRICS FOR EHM SYSTEMS . 6 3.1 Engine Prognostics . 6 3.2 Required Information and Data for Appl
10、ying Prognostic Metrics 9 3.3 Prognostic Metrics . 9 3.3.1 Introduction of Select Prognostic Metrics 11 3.3.2 Metrics Applicable to Condition-based Prognostics 12 4. EXAMPLE APPLICATION OF PROGNOSTIC METRICS 14 4.1 Description of Simulated Aircraft Engine EGT Margin Forecasting Problem . 15 4.2 Desc
11、ription of Applied Prognostic Methods 16 4.2.1 Method #1: Fleet Average EGT Margin Forecast . 16 4.2.2 Method #2: Third Order Least Squares Fit . 16 4.3 Applied Prognostic Metrics and Results . 17 5. USER CONSIDERATIONS FOR SELECTING AND APPLYING PROGNOSTIC METRICS 22 6. NOTES 22 6.1 Revision Indica
12、tor 22 APPENDIX A COMPREHENSIVE LIST OF PROGNOSTIC METRICS . 23 FIGURE 1 RUL VERSUS TIME PLOT . 8 FIGURE 2 ILLUSTRATION OF TRAJECTORY PREDICTION AND RUL CALCULATION . 8 FIGURE 3 PROGNOSTIC METRICS CLASSIFICATION 10 FIGURE 4 RUL VERSUS TIME PLOT INCLUDING UNCERTAINTY DISTRIBUTIONS OF ESTIMATED RUL 13
13、 FIGURE 5 ILLUSTRATION OF PREDICTION UNCERTAINTY 13 FIGURE 6 ILLUSTRATION OF EGT MARGIN FORECASTING PROBLEM. 15 FIGURE 7 ILLUSTRATION OF EGT MARGIN FORECASTING PROBLEM. 17 FIGURE 8 EGT MARGIN VERSUS FLIGHT NUMBER TEST CASES (TWO INDIVIDUAL ENGINES PLUS THE FLEET AVERAGE ENGINE) 18 FIGURE 9 ESTIMATED
14、 REMAINING USEFUL LIFE (RUL) VERSUS HORIZON PRIOR TO EOL . 19 FIGURE 10 ENGINE A PROGNOSTIC HORIZON (PH) RESULTS 20 FIGURE 11 ENGINE B PROGNOSTIC HORIZON (PH) RESULTS 20 FIGURE 12 FLEET AVERAGE PROGNOSTIC HORIZON (PH) RESULTS . 20 TABLE 1 AVERAGE BIAS, MAE, MAPE AND SAMPLE STANDARD DEVIATION RESULTS
15、 18 TABLE 2 ESTIMATED REMAINING USEFUL LIFE (RUL) . 19 TABLE 3 - Accuracy Results 21 TABLE 4 RELATIVE ACCURACY (RA) AND CUMULATIVE RELATIVE ACCURACY (CRA) RESULTS . 21 SAE INTERNATIONAL AIR5909 Page 3 of 26 1. SCOPE This SAE Aerospace Information Report (AIR) presents metrics for assessing the perfo
16、rmance of prognostic algorithms applied for Engine Health Management (EHM) functions. The emphasis is entirely on prognostics and as such is intended to provide an extension and complement to such documents as AIR5871, which offers information and guidance on general prognostic approaches relevant t
17、o gas turbines, and AIR4985 which offers general metrics for evaluating diagnostic systems and their impact on engine health management activities. 1.1 Purpose The purpose of this AIR is to present metrics that can be applied to assess the performance of prognostic methods. Various metrics are prese
18、nted and discussed. Additionally, the application of select metrics is illustrated through a given example. 2. REFERENCES These references contain useful information that may have been used in this report or may be beneficial in understanding the subject matter and its application. 2.1 Applicable Do
19、cuments The following publications form a part of this document to the extent specified herein. The latest issue of SAE publications shall apply. The applicable issue of other publications shall be the issue in effect on the date of the purchase order. In the event of conflict between the text of th
20、is document and references cited herein, the text of this document takes precedence. Nothing in this document, however, supersedes applicable laws and regulations unless a finding of Equivalent Level of Safety or a specific exemption has been obtained from the governing regulatory authority. 2.1.1 S
21、AE Publications Available from SAE World Headquarters, 400 Commonwealth Drive, Warrendale, PA 15096-0001, Tel: 877-606-7323 (inside USA and Canada) or 724-776-4970 (outside USA), www.sae.org AIR4985 A Methodology for Quantifying the Performance of an Engine Monitoring System AIR5871 Prognostics for
22、Gas Turbine Engines ARP5783 Health and Usage Monitoring Metrics Monitoring the Monitor 2.2 External Publications Byington, C., Roemer, M., and Watson, M., “Prognostic Enhancements to Diagnostic Systems (PEDS) Applied to Shipboard Power Generation Systems,” Proceedings of the ASME/IGTI Turbo Expo 200
23、4-Power for Land, Sea, and Air, June 1417, 2004, Austria, Paper Number: GT2004-54135. Engel, S. J., Gilmartin, B. J., Bongort, K., and Hess, A, “Prognostics, The Real Issues Involved with Predicting Life Remaining,“ IEEE 0-7803-5846-5/00, 2000. Kacprzynski, G. J., Hess, A. J., and Begin, M., “Metric
24、s and Development Tools for Prognostic Algorithms,” 2004 IEEE Aerospace Conference, March 6-13, 2004, Big Sky, MT. Koutsoukos, X., Biswas, G., Mylaraswamy, D. A., Hadden, G. D., Mack, D., and Hamilton, D., “Benchmarking the Vehicle Integrated Prognostic Reasoner,” 2010 Annual Conference of the Progn
25、ostics and Health Management Society, October 10-16, 2010, Portland, OR. Frederick, D. K., DeCastro, J. A., and Litt, J. S., “Users Guide for the Commercial Modular Aero-Propulsion System Simulation (C-MAPSS),” NASA/TM-2007-215026, October 1, 2007. SAE INTERNATIONAL AIR5909 Page 4 of 26 Patrick, R.,
26、 Smith, M. J., Byington, C. S., Vachtsevanos, G. J., Tom, K., and Ly, C., “Integrated Software Platform for Fleet Data Analysis, Enhanced Diagnostics, and Safe Transition to Prognostics for Helicopter Component CBM,” Annual Conference of the Prognostics and Health Management Society (PHM10), October
27、 13-16th, 2010, Portland, OR. Roemer, M. J., Dzakowic, J., Orsage, R. F., Byington, C. S., and Vachtsevanos, G., “Validation and Verification of Prognostic and Health Management Technologies,” 2005 IEEE Aerospace Conference, March 5-12, 2005, Big Sky, MT. Roemer, M., Byington, C., Kacprzynski, G., a
28、nd Vachtsevanos, G., “An Overview of Selected Prognostic Technologies with Application to Engine Health Management,” GT2006-90677, Proceedings of ASME Turbo Expo 2006: Power for Land, Sea and Air, Barcelona, Spain, May 8-11, 2006. Roemer, M. J. and Byington, C.S., “Prognostics and Health Management
29、Software for Gas Turbine Engine Bearings,” ASME Turbo Expo 2007: Power for Land, Sea, and Air, Paper #GT2007-27984, Montreal, Canada, May 14-17, 2007. Saxena, A., Celaya, J., Balaban, E., Goebel, K., Saha, B., Saha, S., and Schwabacher, M., “Metrics for Evaluating Performance of Prognostic Technique
30、s,” IEEE International Conference on Prognostics and Health Management, October 6-9, 2008, Denver, CO. Saxena, A., Celaya, J., Balaban, E., Saha, B., Saha, S., and Goebel, K., “Evaluating Algorithm Performance Metrics Tailored for Prognostics,” 2009 IEEE Aerospace Conference, March 7-14, 2009, Big S
31、ky, MT. Saxena, A., Celaya, J., Balaban, E., Saha, B., Saha, S., and Goebel, K., “On Applying the Prognostic Performance Metrics,” Annual Conference of the Prognostics and Health Management Society (PHM09), September 27 - October 1, 2009, San Diego, CA. Saxena, A., Celaya, J., Balaban, E., Saha, B.,
32、 Saha, S., and Goebel, K., “Evaluating Prognostics Performance for Algorithms Incorporating Uncertainty Estimates,” Aerospace Conference, 2010 IEEE, March 6-13 2010, Big Sky, MT. Saxena, A., Celaya, J., Saha, B., Saha, S., and Goebel, K., “Metrics for Offline Evaluation of Prognostic Performance,” I
33、nternational Journal of Prognostics and Health Management, vol. 1, no. 1, p. 20, 2010. Sen-Gupta, J., Trinquier, C., Medjaher, K., and Zerhouni, N., “Continuous Validation of the PHM Function in Aircraft Industry,” 2015 Prognostics and System Health Management Conference, October 21-23, 2015, Beijin
34、g, China. Uckun, S., Goebel, K., and Lucas, P. J. F, “Standardizing Research Methods for Prognostics,” IEEE International Conference on Prognostics and Health Management, October 6-9, 2008, Denver, CO. Wheeler, K. R., Kurtoglu, T., and Poll, S. D., “A Survey of Health Management User Objectives in A
35、erospace Systems Related to Diagnostic and Prognostic Metrics,” ASME 2009 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, August 30September 2, 2009, San Diego, CA. 2.3 Definitions/Acronyms/Abbreviations Acronyms ARMA Auto-Regressive Mo
36、ving Average B Average Bias C-MAPSS Commercial Modular Aero-Propulsion System Simulation CRA Cumulative Relative Accuracy EHM Engine Health Management EGT Exhaust Gas Temperature EoL End-of-Life SAE INTERNATIONAL AIR5909 Page 5 of 26 EoLP Predicted End-of-Life H Horizon LLP Life Limited Part MAE Mea
37、n Absolute Error MAPE Mean Absolute Percentage Error PH Prognostic Horizon RA Relative Accuracy RUL Remaining Useful Life S Sample Standard Deviation UUT Unit Under Test Definitions t time tFtime index when fault of interest initiates tDtime index when fault is detected by diagnostic system tPtime i
38、ndex when prognostic prediction begins tEoLactual time index when UUT reaches its end-of-life or failure threshold tEoLPpredicted time index when UUT will reach its end-of-life or failure threshold i index representing time index tiithe first time index when predictions satisfy the -criterion for a
39、given -bounds k index representing the kthUUT “ number of time indices where predictions are made between tP and tEoLP L number of UUTs p set of all time indices when predictions are made irkactualactual RUL at time i for the kthUUT irkpredictedpredicted RUL at time i for the kthUUT w weight factor
40、k(i) error, or difference between calculated and actual end-of-life for the kthUUT at time tikI prediction of variable of interest for the kthUUT SAE INTERNATIONAL AIR5909 Page 6 of 26 jik|I prediction of variable of interest for the kthUUT at time tigiven data up through the time tj ik) trajectory
41、of future predictions of kI defined as P|iEoLiiiikkkIII ,|1,| accuracy bounds on prediction +maximum allowable positive error -maximum allowable negative error minimum desired probability threshold time window modifier such that PEoLPtttt OO xM non-parameterized probability distribution for any vari
42、able x DDS |predictedRUL probability mass of the prediction residing within the defined -bounds 3. PROGNOSTIC METRICS FOR EHM SYSTEMS Engine health management (EHM) systems play a critical role in assisting operators in managing the reliability, availability, and affordability of their gas turbine e
43、ngine assets, which may also provide an intrinsic enhancement to safety. The functionality provided by an EHM system includes the capability to evaluate the current health of the system (diagnostics) as well as the emerging approach of predicting or forecasting the future health of the system (progn
44、ostics). The emphasis and need for effective prognostic methods continues to grow. The desire for condition-based maintenance, optimal scheduling and performance of maintenance, and fleet management are all technology pulls. Operators are facing prognostic technology design choices, and with these c
45、hoices, they require an effective means to compare the relative performance and benefits of different prognostic methods. The purpose of this document is to set forth a series of metrics that can be applied to quantify the performance of EHM prognostic algorithms. 3.1 Engine Prognostics A prognostic
46、 entails the prediction in time of the health state of a system relative to some threshold. Prognostics is the ability to predict the future condition of a component and/or system of components. For the purposes of gas turbine engine prognostics, this definition is often further described in terms o
47、f life (usage) consumption of a component or the rate at which a faulted component continues to degrade. Generally speaking, prognostics can be delineated into two different categories: Usage-based prognostics and Health-based prognostics. These are defined as follows Patrick, et al, 2010: Usage-bas
48、ed Prognostics: Combines usage monitoring (load/stress tracking) with a wear or life model to estimate the rate at which a component degrades, accumulates damage or “consumes” its design life. The prognosis estimates the remaining design life at any given time. It can track wear or degradation in pa
49、rts and can also offer support for replacement logistics. It can reduce risk of conventional time-based or cycle-based approaches when usage is intense. Usage-based prognostics does not take into account unanticipated faults, and is most applicable for components that degrade as intended by design. It does require an estimate of future syste
