Today between 1206 and 1212, EECS networks were isolated from the outside world due to an unanticipated routing protocol behavior during a reconfiguration related to our firewall replacement project. This has been corrected and there will be no further issues.
Wired Networking
Wired network service
Intermittent Network Trouble at EECS Border
Staff are investigating intermittent disturbances in network traffic at or around the EECS border. We’ll provide updates here as we learn more.
Final Update 11/4 12:09:
This issue was caused by a DDoS or DDoS-style traffic which was overloading our firewall, which is near the end of its life. Using a number of active and passive mitigation techniques, we believe the impact of this traffic has been fully mitigated and there should be no further issues; the firewall has been stable since Friday morning. We are also accelerating the project already in the works to replace this firewall.
Potential Campus Power Outage
We have been alerted to possible PG&E power outages due to fire danger, and that campus power may be shutdown at some point over the next 2-3 days. If we receive notice of imminent shutdown, we will work to safely shutdown our NetApp fileserver and other critical infrastructure systems; this may occur on very short notice.
We will endeavor to update this News item as we receive any updates.
See also https://berkeley.statusdashboard.com/
We recommend you proceed with shutting down any non-essential machines in advance, e.g. shutting down workstations at the end of the work day.
[Read more…] about Potential Campus Power OutageScheduled Network Maintenance: Labor Day Weekend 8/31 – 9/2, 3-4AM
We will be performing firmware upgrades on the EECS core routers at 3AM on the mornings of Saturday 8/31, Sunday 9/1, and Monday 9/2 (Labor Day weekend). Each outage is expected to last under 20 minutes; however a full hour has been scheduled in case of unanticipated issues. Specific times and expected impact for each maintenance window is below.
These upgrades were completed successfully as of Monday 9/2 at 4AM.
[Read more…] about Scheduled Network Maintenance: Labor Day Weekend 8/31 – 9/2, 3-4AMBWRC Network Down
As of 10:38am, the network connection to BWRC (Berkeley Wireless Research Center) is down. Network staff are investigating, and we’ll post updates here as they become available.
[Read more…] about BWRC Network DownCory 1F Network Outage 11/18-19
A switch appears to have failed around 8PM on Sunday, causing wired and wireless network outages on the first floor of Cory Hall. We will investigate and replace the switch ASAP on Monday.
[Read more…] about Cory 1F Network Outage 11/18-19
BWRC Uplink Issues, 8/23/18
Following this morning’s IST network maintenance we are experiencing issues passing traffic in and out of BWRC. We may have identified the root cause and are working with the campus NOC to resolve.
The impact specifically seems to be traffic between BWRC and EECS. Local traffic within BWRC is not affected. General traffic to/from the Internet does not appear affected. We are unsure about traffic between BWRC and the rest of campus outside EECS.
We will be performing an experimental configuration change tomorrow morning on the related EECS routers to try and alleviate the issue until the true cause can be addressed with IST. An update will be posted here before or around noon.
[Read more…] about BWRC Uplink Issues, 8/23/18
HMMB BiD Center Wired Network Outage 7/13/18 – Ongoing
The EECS wired network in the HMMB BiD center (169.229.63.0/25) is currently down due to a link issue. According to our router, the link has been down since Friday 7/13 at 22:47.
This part of the network is a little unusual, in that it is served by an EECS router but an IST switch and IST cabling for the last-mile. We have confirmed there seems to be no issue on our end and are working with IST networks toward a resolution.
Wireless networks in HMMB are not affected, including EECS-Secure and EECS-PSK.
Update 12:03 PM: We are in contact with IST Network Operations and they are aware of the severity of the issue. Due to heavy workload, they may not be able to address the issue until tomorrow, but they will look at it as soon as possible.
[Read more…] about HMMB BiD Center Wired Network Outage 7/13/18 – Ongoing
IT and Electrical Maintenance Shutdown, Sat. 6/16/18
On Saturday, June 16th, the campus high voltage team will be cutting power to **Soda, Jacobs, Etcheverry, and North Gate Halls** to complete critical power grid maintenance. The electrical shutdown is currently scheduled for **8AM to 1PM** and these buildings will be **closed** during this time to all but critical staff related to the maintenance projects.
We are also taking this time to perform safety tests in our server and network rooms, as well as replacing the failed battery backup system in our primary server room. As a result, **the outage window for most IT services** (other than basic network connectivity) **is 11PM 6/15 – 9PM 6/16**.
Anybody with computers and electronics in Soda and Jacobs is **STRONGLY RECOMMENDED** to power off their noncritical systems when leaving the office on Friday, June 15. Critical systems **MUST** be powered off by 6AM on Saturday the 16th. This includes machines hosted in the 340 Soda server room, and a separate message will be sent to those system administrators next week.
As most of EECS’ IT infrastructure is hosted in Soda Hall, this will have some impacts which are detailed below. This will affect all EECS locations which are not undergoing electrical maintenance: **Cory, Sutardja Dai and Blum Halls, the Calvin Lab, and BWRC** as well as remote users.
* **Network infrastructure** (wired and wireless connections, including DHCP and DNS services) **will be offline for a brief period between 6:00 and 6:30 AM** in order to perform annual Emergency Power Off and Fire Alarm tests in our network core. Basic networking (including DHCP/DNS) will be restored by 6:30 and should remain up for the remainder of the day for all locations outside of Soda and Jacobs Hall.
* The IRIS website and portions of the department website including personal homepages, as well as other IRIS services that depend on file storage, will be shut down **starting at 11PM Fri. 6/15**
* Storage services, including home and project file storage, will be shut down at **12 AM Sat. 6/16**
* All services are scheduled to be **restored by 9PM**, but may be online earlier if maintenance is completed ahead of schedule.
As the IRIS website will be unavailable during this maintenance, https://status.eecs.berkeley.edu will be available for in-the-moment status updates.
Please contact help@eecs.berkeley.edu with any questions or concerns.
[Read more…] about IT and Electrical Maintenance Shutdown, Sat. 6/16/18
Network slowness, dropped packets
Staff are investigating widespread reports of network slowness and dropped packets.
Network traffic solely within EECS networks appears to be fine, but network traffic traveling past the EECS border to campus or beyond is incurring significant packet loss. It is unclear at this time whether the problem exists on the EECS side or the campus side.
We will update this news item with more information as it becomes available.
[Read more…] about Network slowness, dropped packets