Report on Outage on the Night of Nov 14, 2022
Direct Communications operations and engineering staff have completed a post-outage review of the unscheduled outage that began intermittently around 9:30pm on November 14 2022 and ended around 2:00am.
In the spirit of full transparency with our customers here in Eagle Mountain, since we understand many customers are frustrated with seemingly compounding issues over the past week, we are posting as much information as we have been able to aggregate. Most providers would say this is too much information, but we are not most providers. We only service Eagle Mountain, we live and work here too, and we do care greatly about the quality of our service and our customers’ experience. We are sincerely trying our best to constantly improve all our systems and processes.
Last night’s outage was not related to the few smaller short individual subdivision outages that we have been seeing over the course of the last week. These smaller outages have occurred in single subdivisions randomly and have been due to a software glitch in a recent software update to some central network distribution electronics cards, which have caused some individual neighborhood fiber ports to reboot automatically. Although this issue appears to be resolved, we are still working with the manufacturer to confirm everything is back to normal functioning.
Last night’s outage was due to an IP failure at a data center in downtown Salt Lake City, where one of our upstream providers connects to the national Internet backbone. Unfortunately, this was out of our control, and even more frustrating, was a veiled failure at the national provider, (meaning nobody could see it was failing at that particular point) so we as wholesale customers were not made aware of the outage until much later, so part of our network was still trying to pass traffic through that non-functioning data center.
We do have multiple redundant fiber routes and multiple upstream providers to carry traffic from Eagle Mountain to Salt Lake City, but this was apparently a strange situation where the traffic didn’t know a road was closed, which caused congestion throughout the network. We wish we could have handled this more efficiently, but even after 17 years of serving Eagle Mountain, the fact is every new system outage is unique in it’s own way, and we have learned from this one too, like we have from any past outages.
We appreciate your patience and your continued support. It is a privilege to be the provider of such an important public utility as fiber Internet and to be part of your lives here in Eagle Mountain. As Thanksgiving approaches we do want to say how grateful we are for all our customers.
Sincerely,
Your Friends at Direct Communications