1、Lessons Learned Entry: 0664Lesson Info:a71 Lesson Number: 0664a71 Lesson Date: 1999-12-01a71 Submitting Organization: HQa71 Submitted by: Eric RaynorSubject: SOHO Mission Interruption Joint NASA/ESA Investigation Board Description of Driving Event: This is a depiction of the SOlar Heliospheric Obser
2、vatory (SOHO) spacecraftContact with the SOlar Heliospheric Observatory (SOHO) spacecraft was lost in the early morning hours of June 25, 1998, Eastern Daylight Time (EDT), during a planned period of calibrations, maneuvers, and spacecraft reconfigurations. Prior to this the SOHO operations team had
3、 concluded two years of extremely successful science operations. A joint European Space Agency (ESA)/National Aeronautics and Space Administration (NASA) engineering team has been planning and executing recovery efforts since loss of contact with some success to date.ESA and NASA management establis
4、hed the SOHO Mission Interruption Joint Investigation Board to determine the actual or probable cause(s) of the SOHO spacecraft mishap.Lesson(s) Learned: The Board has concluded that there were no anomalies on-board the SOHO spacecraft but that a number of ground errors led to the major loss of atti
5、tude experienced by the spacecraft.The Board finds that the loss of the SOHO spacecraft was a direct result of operational errors, a failure to adequately monitor spacecraft status, and an erroneous decision which disabled part of the on-board autonomous failure detection. Further, following the occ
6、urrence of the emergency situation, the Board finds that insufficient time was taken by the operations team to fully assess the spacecraft Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-status prior to initiating recovery operations. The Board disco
7、vered that a number of factors contributed to the circumstances that allowed the direct causes to occur.FACTORS DIRECTLY CONTRIBUTING TO THE LOSSThe Board has concluded that there were no anomalies on-board the SOHO spacecraft but that a number of ground errors led to the major loss of attitude expe
8、rienced by the spacecraft.The first two errors were contained in predefined command sequences executed from the Ground System, while the last error was a decision to send a command to the spacecraft in response to unexpected telemetry. The sequence of these errors and their relationship to the trigg
9、ered-ESRs are graphically depicted in Diagram 1.refer to D descriptionD Diagram 1. Failure Event Tree - Top Level (Click image for a larger view) This series of events was preceded by a routine calibration of the spacecrafts three roll gyros. As stated earlier, the gyros are not required during most
10、 of the mission. They are used for thruster-based activities such as momentum management, ISA, and ESR.Since the gyro calibration in the compressed timeline is immediately followed by the execution of momentum management, the previously employed procedure to despin the gyros at the end of the gyro c
11、alibration and to re-enable the on-board software gyro control function was not required. Following gyro calibration, Gyro A Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-was specifically deactivated (despun) in order to conserve its life, while Gy
12、ros B and C remained fully active. Due to an omission in the modified predefined command sequence actually used, the onboard software function that activates the gyro needed by ESR was not enabled. This omission resulted in the removal of the functionality of the normal safe mode and ultimately caus
13、ed the catastrophic sequence of events.Following the momentum management maneuver, Gyro B, which is used for fault detection, was erroneously left in its high gain setting, resulting in an indicated roll rate of 20 times greater than actual. The incorrect gain was due to an error in another predefin
14、ed command sequence; this error resulted in an on-board fault detection output that triggered an ESR. This ESR, the 5th since launch, occurred at 7:16 PM EDT (23:16 UT), June 24, 1998.During ESR-5, the control Gyro A was not active because of the first error referenced above; however, there is no ev
15、idence or belief that any anomalous spacecraft behavior had occurred. As per design, the ESR event resulted in a reconfiguration of the gyros. Gyro A replaced Gyro C as the roll gyro used for the ESR thruster-based control mode, while Gyro B remained configured as the fault detection gyro. The error
16、 in Gyro Bs gain was discovered and corrected, but the Gyro A despun status was not identified.After transitioning to the ISA mode as part of the normal ESR recovery sequence, the attitude control system began integrating the gyro drift rate bias associated with the still despun Gyro A. After 15 min
17、utes, this resulted in roll thruster firings intended to null the apparent (but non-existent) roll attitude error. In less than one minute, the roll rate was sufficiently high to trigger the Gyro B based fault detection once again, resulting in ESR-6 at 10:35 PM EDT (02:35 UT), June 24, 1998.Althoug
18、h the spacecraft remained Sun-pointing within nominal limits and was therefore in a power-positive and thermally-safe attitude, the state of the spacecraft was precarious at this point in time. It had an anomalous roll rate and was depending on a deactivated gyro for roll control in both ESR and ISA
19、 modes. The personnel on the ground were not aware of either of these facts at that time. Gyro C was correctly configured to the ACU since the reconfiguration at ESR-5. Gyro B was active and on-line for fault detection, and it was correctly measuring the anomalous roll rate. A rapid decision was mad
20、e that Gyro B was faulty because its output disagreed with the rate indicated by Gyro A. This decision led to the commanding off of Gyro B.During ESR-6 recovery, Ground Operations commanded the spacecraft to ISA mode. In ISA, the attitude control system resumed firing roll thrusters in an attempt to
21、 null the attitude error associated with the electrical rate bias term of the despun Gyro A. Gyro B and the associated fault detection were now inactive. The increasing roll rate eventually resulted in pitch and yaw Sun-pointing errors that exceeded a prescribed limit of five degrees, resulting in E
22、SR-7 at 12:38 AM EDT (04:38 UT), June 25, 1998. Due to the gyroscopic cross-coupling torques caused by pitch and yaw thruster firings, and the absence of true roll rate indications, the ESR controller was no longer stable, and the spacecraft attitude diverged. The incorrect diagnosis of a Gyro B fau
23、lt and the subsequent ground response to this diagnosis ultimately resulted in loss of attitude control, subsequent loss of telemetry, and loss of power and thermal control. Loss of telemetry occurred at 12:43:56 AM EDT (04:43:56 UT), June 25, 1998. It can not be determined whether this loss was a c
24、onsequence of insufficient power or a loss of communication link caused by spacecraft attitude.At any time during the over five hour emergency situation, the verification of the spinning status of Gyro A Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-
25、,-would have precluded the mishap.For further information, a more detailed timeline summarizing the sequence of events leading up to the mishap is provided in Appendix C. A detailed fault tree analysis supporting the above scenario and the Boards conclusions has been performed by MMS and is on file.
26、FACTORS INDIRECTLY CONTRIBUTING TO THE FAILURE:A. Ground Procedures (failure to control change; failure to perform risk analysis of a modified procedure set; Failure to comunicate change).B. Procedure Implementation (failure to properly respect autonomous Safe Mode triggers; Failure to follow the op
27、erations script; failure to evaluate primary and ancillary data; failure to question telemetry discrepancies)C. Management Structure and Process (Failure to recognize risk caused by operations team overload; failure to recognize shortcomings in implementation of ESA/NASA agreements; emphasis on scie
28、nce return at expense of spacecraft safety; Over-reliance of flight operations teams on ESA and MMS representatives; dilution of observatory engineering support)D. Ground Systems (Failure to resolve a critical deficiency report in a timely manner; failure to validate the planned sequence of events i
29、n advance)Recommendation(s): The Board strongly recommends that the two Agencies proceed immediately with a comprehensive review of SOHO operations addressing issues in the ground procedures, procedure implementation, management structure and process, and ground systems. This review process should b
30、e completed and process improvements initiated prior to the resumption of SOHO normal operations. The recommendations are described below in detail.A. GROUND PROCEDURES 1. An ESA and NASA review of the process for SOHO operational procedure change should be implemented forthwith. The review should c
31、ritically assess the process from beginning to end. The review should include matters such as who can initiate a change, who agrees it should be made, how is the modification process monitored, how is it validated, how is it introduced into operations, how it is signed-off, how are the users of the
32、procedure informed of the change, and how are users trained on the new version.2. All SOHO procedures modified since launch should be identified and subjected to a thorough review of the changes made and their verification and validation status. The review should be led by ESA and supported by MMS,
33、NASA, and the FOT. This review should be completed before the resumption of routine operations. If there is any doubt about a procedure, it should be re-validated.B. PROCEDURE IMPLEMENTATION 1. NASA should perform an immediate comprehensive audit of all ISTP on-going flight operations activities to
34、assess conformance to contractual requirements addressing areas such as leadership, configuration management, roles/responsibilities, anomaly handling, and Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-general procedure implementation and validatio
35、n. The activities pertinent to SOHO should include ESA in the review.2. ESA and NASA should review the decision authority for real-time divergence from agreed upon ground or spacecraft procedures. In the event that a problem is encountered during any procedure execution, the decision authority for t
36、he subsequent action must be clearly defined. Spacecraft safety must never be compromised.3. It is critical that NASA should review the relevance of selected metrics to determine adequacy for contractor performance evaluation.C. MANAGEMENT STRUCTURE AND PROCESS 1. ESA and NASA should review the allo
37、cation of their responsibilities for the operation of the SOHO mission, as defined in the STSP MOU, Program Plan, and SOHO Mission Management Plan. Assessment should be made as to how these responsibilities have been undertaken by both parties. Changes should be proposed and implemented where approp
38、riate. In particular, ESAs responsibilities for the technical integrity and safety of the spacecraft and science instrumentation and NASAs responsibilities for Ground Segment infrastructure and the conduct of the mission in accordance with the approved Flight Operations Plan should receive critical
39、focus. This review should be preceded by internal reviews in both Agencies and completed prior to return to normal operations. It is anticipated that the review should result in an improved Mission Management Plan.2. ESA and NASA should re-assess staffing to ensure it is commensurate with the comple
40、xity and criticality of the SOHO mission and consistent with the updated Mission Management Plan. The staffing should be strengthened as required. Surge capability should exist to support non-routine and contingency operations.3. NASA should perform a risk-based analysis of operations plans to deter
41、mine the level of insight/oversight appropriate for joint, cooperative, and PI missions with special attention to accountability.D. GROUND SYSTEMS 1. All operational timelines should be planned and validated well before implementation with proper attention to risk assessment and contingency planning
42、.2. The operations scripts (the string of procedures used by the FOT) for other than routine science operations should be put under configuration control and any change formally approved by ESA and NASA. Each time such a script is changed, the whole script should be validated.3. Flight operations sh
43、ould verify response to spacecraft configuration changes and critical commands to ensure proper execution. Flight operations should verify configuration prior to initiating critical spacecraft activity to ensure it is consistent with planned events.4. ESA/ESOC should lead an independent assessment o
44、f the capabilities of the NASA SOHO simulator and provide recommendations for suggested maintenance and enhancements.5. The FOT should review the current database to ensure that all critical parameters are flagged as out-of-limits (and preserved) if they violate values as defined in the ESA provided
45、 satellite users manual. If possible, automatic monitoring should be extended to all telemetry.6. An ESA and NASA board should review all outstanding Ground System Problem Reports and the plans to close them.7. ESA and NASA flight operations personnel should be conversant with both the ESA and NASA
46、systems to the maximum extent possible to form a more synergistic, integrated team.Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-Evidence of Recurrence Control Effectiveness: N/ADocuments Related to Lesson: N/AMission Directorate(s): a71 ScienceAdd
47、itional Key Phrase(s): a71 Administration/Organizationa71 Communication Systemsa71 Computer-Aided Design/Manufacturing/Engineeringa71 External Relationsa71 Flight Operationsa71 Flight Equipmenta71 Ground Operationsa71 Ground Equipmenta71 Hardwarea71 Human Factorsa71 Mishap Reportinga71 Parts Materia
48、ls & Processesa71 Risk Management/Assessmenta71 Safety & Mission Assurancea71 SpacecraftMishap Report References: SOHO Mission Interruption Joint NASA/ESA Investigation BoardAdditional Info: Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-Approval Info: a71 Approval Date: 2000-03-06a71 Approval Name: Eric Raynora71 Approval Organization: QSa71 Approval Phone Number: 202-358-4738Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-
