Staff are investigating intermittent disturbances in network traffic at or around the EECS border. We’ll provide updates here as we learn more.
Final Update 11/4 12:09:
This issue was caused by a DDoS or DDoS-style traffic which was overloading our firewall, which is near the end of its life. Using a number of active and passive mitigation techniques, we believe the impact of this traffic has been fully mitigated and there should be no further issues; the firewall has been stable since Friday morning. We are also accelerating the project already in the works to replace this firewall.
Update 10/26:
This issue affects any traffic which traverses the EECS border firewalls:
* EECS networks (wired, EECS-Secure, EECS-PSK) to/from campus/Internet
* EECS research networks (Millennium) to/from EECS networks
The issue relates to new session creation; if you are able to establish a tcp/udp session (e.g. SSH/RDP) across the border, the session should remain stable as long as it does not reach an idle timeout. The difficulty is in actually establishing the session.
Update 10/26 11:40: We have confirmed that we are hitting a hardware limit on our border firewall. As load is naturally lower on the weekend, compounded with the campus power shutdown, the issue ceased approx. 12:00AM 10/26. We are attempting to reduce load on the firewall, but may need to wait until Monday for load to increase (and the problem to reappear) to gather additional data.
Update 10/26 13:30: We believe we have identified and mitigated the root cause of this issue. Staff continue to monitor the situation through next week to ensure continued stability. We expect to provide a further update next week which will include greater detail.
Final Update 11/4 12:09:
This issue was caused by a DDoS or DDoS-style traffic which was overloading our firewall, which is near the end of its life. Using a number of active and passive mitigation techniques, we believe the impact of this traffic has been fully mitigated and there should be no further issues; the firewall has been stable since Friday morning. We are also accelerating the project already in the works to replace this firewall.