Editor’s note: Richard Deason is a 20-plus year veteran in telecommunications and is currently with Lightwave Consulting Group in Charlotte.When the lights flickered for a third time, Harry sensed something was wrong so he dialed the number for the bank’s network management center on his cell phone.

Nothing. Not even the “bonk-bonk-bonk” tones of an overloaded cellular network.

He tried his home phone. No dial tone.

He attempted to access the bank’s network management system across his home DSL connection. He noticed the modem lights were solid red.

Harry activated his television with the remote control and turned to the local news channel. He then heard the unthinkable.

Two nearly simultaneous events were wreaking havoc on the city’s telecommunications networks. A drunk driver had leveled two telephone poles and severed a major fiber optic cable artery into the telephone company’s main central office. An unrelated power problem at a nearby substation sent a surge into the central office, thereby causing the failure of the fiber optic terminal equipment that normally re-routed traffic around cable breaks. These two events had also impacted the main telecom carrier hotel where multiple competitive telecom providers interconnected with each other and to the local telephone company’s network.

A cascading effect was occurring. The telephone company’s switch failed when thousands of call attempts from frantic callers were attempted at nearly the same time. Since this switch served as the tandem for all other central office switches and cellular switches, they also soon ceased to process due to the volume of calls. Data traffic across the Internet and private networks also backed up. Routers could no longer handle the traffic loads; therefore, they shut down. Automatic teller machines and traffic signals started failing.

Harry grabbed his keys and headed to his car. He knew the eight-mile drive to the bank’s network management center would be longer than normal but it might give him a chance to gather his thoughts. He would have much to explain to his superiors at the bank.

Fact or Fiction

Could this fictional story take place in Charlotte, Greenville, or Raleigh? Absolutely.

For example, 1988 was a rough year for central offices. In early May, the Illinois Bell central office in Hinsdale, IL was completely destroyed by fire. Telephone services were mostly restored by the end of May. In our local area, the main Southern Bell central office in Charlotte failed on May 22, 1988, which left thousands without long distance service, emergency 911 services, or access to automatic teller machines. Services were restored the next day.

More recently, on May 2, 2004, active DS-3 cards were stolen from a Verizon
co-location facility at 240 E. 38th Street in Manhattan. Internet access to a number of corporate customers was disrupted for hours. On Nov. 24, burglars temporarily stole more than 100 circuit boards from the Verizon central office in White Plains, NY and disrupted phone service in Westchester County for seven hours. The FBI is now investigating.

Are these rare events? From October 2001 to September 2002, there were 123 outages reported to the Federal Communications Commission (FCC) by the major telecommunications service providers. Each outage lasted more than 30 minutes with a minimum of 90,000 blocked calls. Due to security concerns, the FCC recently decided to discontinue the publication of network outage statistics.

Check and Recheck

Many companies, pre- and post-9/11, implemented disaster recovery plans for their data centers and other mission-critical sites. How many of them evaluated the risks of losing communications links into these locations and implemented safeguards? Probably not enough.

Some telecommunications contingency plans are obvious while others are more complex. Simple procedures include locating backup sites far enough away from primary sites so that communications failures have minimal impact. Extreme diligence is required to map out each critical circuit serving these sites to find common points of failure, and maintaining these circuit routing reports is even more painstaking since telecommunications companies often permanently or temporarily re-route circuits. Also, companies should not be lulled into a false sense of security by believing that circuits provisioned across Synchronous Optical Networks (SONET) or other automatic restoration systems are completely immune from failures. Equipment fails. Humans make errors. Software gets “buggy”.

How protected are your critical communications links?

Lightwave: www.lightwavegroup.com