1、 1 NRSC Bulletin No. 2017-001 DS3 Non-Simplex Outages March 2017 Background In the December 2015 quarterly meeting of the Alliance for Telecommunications Industry Solutions (ATIS) Network Reliability Steering Committee (NRSC), the Federal Communications Commission (FCC) identified that the number of
2、 outages per month had risen dramatically. As a result of this observation, the NRSC established the Outage Increase Task Force (TF). Based on this TFs findings, the team was renamed to the DS3 Non-Simplex TF with the acceptance of NRSC Issue 33. During 2016, the NRSC DS3 TF gathered Network Outage
3、Reporting System (NORS) data from all of the NRSC members for the period between March 2015-October 2016 and conducted extensive analysis to attempt to determine if there were any underlying actionable causes to the rise in reportable outages. This Bulletin will describe the analysis and the resulti
4、ng recommendations; however, other than increased vigilance, no actionable tasks were identified. Methodology The FCC presentation to the NRSC indicated that “Hardware Failures”, “Power Failures”, and “Cable Damage” were the causes that had most increased and thus were the first focus of the analysi
5、s. The NRSC member companies initially provided March 2015-March 2016 NORS data to an independent third party for analysis. The analysis examined: null Root Cause however, a single service provider was identified to have contributed a large portion of the increase within the NRSC-only data. Further
6、analysis indicated that the identified service provider had coded the “Equipment Failed” field as “Customer DS3” in a large number of its NORS reports. An initial report detailing these findings was provided to the FCC in August 2016. At that meeting, the FCC indicated that the DS3 outages were cont
7、inuing to rise at a higher rate than had historically been seen. The NRSC continued its investigation focusing on DS3 Outages. Subsequently, the NRSC member companies provided additional NORS data for the period of March-October 2016 to the independent third party for analysis. Analysis of this data
8、 followed a similar pattern to the initial data; however, the Best Practices used were also examined. Findings While no actionable characteristic of the NORS reports were identified, there are several NRSC observations: null Technological advances have resulted in a higher number of DS3 equivalent c
9、ircuits on a single facility which has resulted in a higher number of outages becoming reportable as soon as the 30 minute threshold is met. 2 null Not all service providers include the name of the vendor on the report; however, among those that do, two vendors were observed more frequently than oth
10、ers. null The states with the highest proportion of outages generally were on the East Coast of the United States, Texas, and California (i.e., states with large populations). null One NRSC company reported a higher portion of the DS3 Non-Simplex Outage Report during this time period. Recommendation
11、s Based on the NRSCs analysis, it is recommended that service providers revisit ATIS-0100045, NRSC Bulletin No. 2009-2, DS3 Outage March 2009 and ATIS-0100038, NRSC Technical Report Analysis of Large DS3 FCC Reportable Outages. Additionally, the NRSC recommends service providers review the following
12、 Best Practices and assess the applicability of implementation to their operations: Best Practices Related to Power Failures Number Description 9-7-0496 Network Operators and Property Managers should consider storing their portable generators at critical sites that are not otherwise equipped with st
13、ationary generators. 9-7-0650 Network Operators, Service Providers, and Property Managers should place strong emphasis on human activities related to the operation of power systems (e.g., maintenance procedures, alarm system operation, response procedures, and training) for operations personnel. 9-8
14、-0799 Cell Site & Remote Location Power Backup: Service Providers, Network Operators, and Property Managers should periodically evaluate the need for and feasibility of providing back up power at cell sites and remote locations, taking into consideration the criticality of the site or location, as w
15、ell as local zoning laws, statutes, and contractual obligations. 9-9-0548 Post Mortem Review: Network Operators and Service Providers should have an internal post mortem process to complete root cause analysis of major network events with follow-up implementation of corrective and preventive actions
16、 to minimize the probability of recurrence. Network Operators and Service Providers should engage Equipment Suppliers and other involved parties, as appropriate, to assist in the analysis and implementation of corrective measures. 9-9-0819 For the deployment of Residential Internet Access Service, N
17、etwork Operators should provide backup power for broadband network equipment when economically and technically practical. Best Practices Related to Hardware Failures Number Description 9-5-0511 Network Operators and Service Providers should provide training for their operations personnel on network-
18、level troubleshooting. 9-7-0588 Network Operators, Service Providers, and Equipment Suppliers should provide awareness training that stresses the services impact of network failure, the risks of various levels of threatening conditions, and the roles components play in the overall architecture. Trai
19、ning should be provided for personnel involved in the direct operation, maintenance, provisioning, security, and support of network elements. 3 Number Description 9-7-5080 Network Operators should identify and track critical network equipment, location of spares, and sources of spares to ensure the
20、long term continuity and availability of communication service. 9-7-5262 Network Operators, Service Providers, and Equipment Suppliers should evaluate the vulnerability of storage locations in an effort to protect critical spares. 9-9-5237 Network Operators, Service Providers, Public Safety, and Equ
21、ipment Suppliers should verify the integrity of system spares and replenish spares, as appropriate, as part of a disaster response and at the conclusion of a disaster response at a facility. Best Practices Related to Cable Damage Number Description 9-7-0434 Employee Training: Network Operators, Serv
22、ice Providers, Equipment Suppliers, and Property Managers should provide appropriate training and periodic refresher courses for their employees. 9-7-0741 Network Operators and Service Providers should review, and adopt as appropriate, best practices aimed at reducing damage to underground facilitie
23、s that are maintained by the Common Ground Alliance (). 9-8-0731 Network Operators and Service Providers should provide physical diversity on critical inter-office and wireless backhaul routes when justified by a risk or value analysis. 9-8-0755 Network Operators, Service Providers, and Property Man
24、agers should document and communicate their installation and maintenance guidelines (e.g., Maintenance Operation Protocol MOP) and the expectation of compliance by all involved parties. 9-9-0548 Post Mortem Review: Network Operators and Service Providers should have an internal post mortem process t
25、o complete root cause analysis of major network events with follow-up implementation of corrective and preventive actions to minimize the probability of recurrence. Network Operators and Service Providers should engage Equipment Suppliers and other involved parties, as appropriate, to assist in the analysis and implementation of corrective measures.
