Tuesday afternoon and evening, November 19, we had a disruption of DNS resolution for the eecs.berkeley.edu domain on the campus nameservers. This meant that many systems outside of the EECS network were temporarily unable to resolve *.eecs.berkeley.edu hostnames — so people may have been temporarily unable to reach EECS websites or deliver email to some @eecs.berkeley.edu addresses.
The problem started at approximately 3:15pm, and was ultimately fixed at 10:30pm, with the help of campus DNS staff and technicians from InfoBlox, the campus DNS vendor. It appears that a change to our eecs.berkeley.edu zone file was not tolerated well by campus InfoBlox servers, and the zone transfers had been silently failing; cached records began timing out in the afternoon, so DNS queries from outside of EECS then began to fail. Fortunately, we had already begun working with campus DNS staff, to investigate why certain new zone updates weren’t propagating to campus servers.
Machines using EECS nameservers, like most that are on EECS networks, should have been fine during this time.
We are currently working to re-do some of our most recent DNS updates, and they should be in place by Wednesday morning, Nov 20. We are also looking into tuning our system monitoring to alert on potential zone transfer issues, while we work with campus to help monitor from their side.
Resolved as of 2024-11-19 10:30pm