1、Public Lessons Learned Entry: 6776 Lesson Info: Lesson Number: 6776 Lesson Date: 2012-09-25 Submitting Organization: JPL Submitted by: David Oberhettinger Subject: Triggering an Office Fire Alarm Shut Down an Adjacent Data Center Abstract: A sprinkler head failure in office spaces triggered a power
2、shutdown in an adjacent data center during non-office hours. Assure that Center security and facilities organizations are prepared to respond appropriately to emergencies. Assess the need for major data centers to be equipped with an isolated fire alarm/suppression system that will not be triggered
3、by alarms originating in the buildings non-critical spaces. Description of Driving Event: On August 26, 2011, a fire suppression sprinkler head leaked and then failed in Building 600 at the NASA/Caltech Jet Propulsion Laboratory (JPL), inundating several office cubicles. The initial leak was reporte
4、d earlier in the day to the JPL Office of Protective Services (OPS), OPS contacted the building owner, and a maintenance crew was responding when the sprinkler head triggered the full water release. The impact of the water damage was minor. However, the affected office space shares the fire alarm sy
5、stem with an adjacent data center that, among other activities, supported Mars Science Laboratory (MSL) Assembly, Test, and Launch Operations (ATLO). There was no water release in the data center, but it was believed that the fire alarm system was programmed to shut down power to the data center unl
6、ess the shutdown was manually countermanded within ten minutes. The data center was not staffed because August 26 was an RDO*, the security guard arrived after the ten-minute period had elapsed, and the data center suffered a power outage. Building 600 is one of the few JPL buildings that is leased
7、from a private owner. Although the procedure for placing the data center electrical system in bypass is posted in the electrical room of the data center, subsequent investigation found that the shutdown was immediate: there is no ten-minute delay in triggering the power shutdown after the fire alarm
8、 is triggered. JPL was misinformed, and such a delay is not compatible with the fire code. To prevent a recurrence of this incident, JPL installed a double-interlock, preaction, fire suppression system to replace the existing system in the Building 600 data center. Preaction systems are suitable for
9、 water sensitive environments like data centers because (until a valve is electrically activated) they contain no water in the sprinkler piping so a sprinkler head failure does not release water. The double-interlock feature prevents flow of water until both a sprinkler head and a separate detector
10、are triggered simultaneously. (Triggering only one of the two devices will produce an alarm, but no water flow into the system.) Because this fire suppression system is separate and isolated from the system in the Building 600 office space, an alarm in the office space will not trigger an electrical
11、 shutdown in the data center. *Most JPL personnel work a schedule that allows them an additional day off work every other Friday, and August 26, 2011 was such a Regular Day Off (RDO). References: 1. “B600 Power Failure,“ Incident Surprise Anomaly Report No. 49896, NASA/Caltech Jet Propulsion Laborat
12、ory, August 30, 2011. 2. “DNS Server Outage Caused Loss of Uplink In MSTB During ORT2,“ Incident Surprise Anomaly Report No. 49699, NASA/Caltech Jet Propulsion Laboratory, August 11, 2011. Lesson(s) Learned: Electrical outages in data centers and other critical facilities that are triggered in error
13、 may have a major impact on critical NASA operations. Two weeks prior to the Building 600 incident, a server outage (Reference (2), caused loss of telemetry flow to an MSL testbed during operational readiness testing. The Building 600 incident might have had even more serious consequences had it occ
14、urred in an MSL command and telemetry processing facility because the MSL spacecraft was due to launch in another three months. Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-Especially water-sensitive facilities like major data centers and museums
15、are known to require special (i.e., double-interlock, preaction) fire suppression systems. Although JPL is continuing to install double-lock isolated systems in major data centers where appropriate (e.g., the new data center in Building 230), they are not suitable for all data processing facilities.
16、 (Small rooms with one or two servers may not warrant special measures, and a gas suppression system (e.g., halon) is often preferable for data centers.) Because protective systems may be triggered during night or weekend hours when cognizant personnel are not present, potential first responders (e.
17、g., security guards) must be familiar with the required steps to be taken. Recommendation(s): 1. Assure that the NASA Center security and facilities organizations are familiar with the configuration and features of fire suppression systems in any leased facilities. For special purpose facilities tha
18、t may operate without trained personnel present (e.g., data centers, certain hardware test facilities, and mission operations centers), ensure that: o Cognizant personnel for critical facilities identify any special measures that need to be taken by first responders during emergency incidents. o Sec
19、urity guards are fully trained and aware of their responsibilities as first responders to automated electrical shutdown events. Assure that clear instructions are posted for any special emergency procedures. The instructions should also provide contact information so the guards can promptly inform t
20、he responsible person(s) for the facility, and inform Center Facilities about the incident. 2. For major data centers, and for other major facilities having a potentially mission-critical sensitivity to power loss, assess the need for an isolated fire alarm/suppression system that will not be trigge
21、red by an alarm originating in the buildings non-critical spaces. Evidence of Recurrence Control Effectiveness: The JPL Office of the Chief Information Officer submitted FSR 139013 to JPL Facilities requesting them to provide a double lock, preaction, fire suppression replacement for the B600 data c
22、enter, and the system has been installed. JPL has referenced this lesson learned as additional rationale and guidance supporting Paragraph 5.19 (“Management Practices: Crisis Response”) in the Jet Propulsion Laboratory standard “Flight Project Practices, Rev. 9,” JPL DocID 58032, August 21, 2012. Do
23、cuments Related to Lesson: N/A Mission Directorate(s): Aeronautics Research Exploration Systems Science Space Operations Additional Key Phrase(s): Safety and Mission Assurance.Advanced planning of safety systems Additional Categories.Facilities Additional Categories.Fire Protection Provided by IHSNo
24、t for ResaleNo reproduction or networking permitted without license from IHS-,-,- Additional Categories.Security Additional Info: Project: Mars Science Laboratory Approval Info: Approval Date: 2012-11-28 Approval Name: mbell Approval Organization: HQ Provided by IHSNot for ResaleNo reproduction or networking permitted without license from IHS-,-,-
