We experienced several related network disruptions Friday afternoon. Beginning with reports at noon of DNS outages from Sutardja Dai Hall, there initially appeared to be problems with that building’s router failing to forward packets correctly. Resetting the links to the DNS service switches initially appeared to solve the issue, but additional connectivity problems between Cory and S-D were discovered later.
While following error messages generated by the primary 10Gb line card in the Cory Hall router, some wired subnets in Cory were temporarily isolated from the rest of the network and the Internet. This outage from 4:20-4:30pm occurred when the suspect line card failed to resume forwarding traffic after running diagnostics. Network staff were immediately aware of that disruption and quickly reestablished basic connectivity on a different port.
While the symptoms have been mitigated, the root cause of the outage has not been fully diagnosed and network is running without full redundancy. Network personnel are planning followup work and are actively monitoring this problem in the event that it reoccurs.
[Read more…] about Network routing & DNS disruptions