Today between 1206 and 1212, EECS networks were isolated from the outside world due to an unanticipated routing protocol behavior during a reconfiguration related to our firewall replacement project. This has been corrected and there will be no further issues.
Intermittent Network Trouble at EECS Border
Staff are investigating intermittent disturbances in network traffic at or around the EECS border. We’ll provide updates here as we learn more.
Final Update 11/4 12:09:
This issue was caused by a DDoS or DDoS-style traffic which was overloading our firewall, which is near the end of its life. Using a number of active and passive mitigation techniques, we believe the impact of this traffic has been fully mitigated and there should be no further issues; the firewall has been stable since Friday morning. We are also accelerating the project already in the works to replace this firewall.
Oct 26 Planned Power Outage
We’ve been notified of another impending PG&E power outage scheduled to begin Saturday evening, October 26; campus Co-Generated power should take over, similar to the last outage earlier this month (https://iris.eecs.berkeley.edu/news/resolved-incidents/potential-campus-power-outage/)
We are planning to keep essential services up and running, including networking, fileservice, web services (people.eecs and group project sites), DHCP, DNS, login.eecs, WINTERM, and lists.eecs (maillists). Some redundant systems may be powered down (e.g. cronus will stay up for DNS/NIS, but rhea may be shut down).
We encourage everyone to shut down all non-essential systems at the end of day today (Friday). Any reduction in power consumption will make it easier to keep other essential services running.
[Read more…] about Oct 26 Planned Power Outagegateway.eecs downtime – Tuesday October 22, 2019 07:15-07:45am
The departmental authenticating outgoing smtp server gateway.eecs will be down for a needed reboot to activate changes delivered by some RHEL patches.
Though the scheduled downtime is 7:15am to 7:45am, we expect actual downtime to be less.
[Read more…] about gateway.eecs downtime – Tuesday October 22, 2019 07:15-07:45amcronus and rhea downtime – Tuesday October 22, 2019 06:45-07:15
cronus one of the departmental DNS, NIS and time servers will be down for a needed reboot to activate changes delivered by some RHEL patches from 6:45-7:00am.
rhea the other departmental DNS, NIS and time server will be down for a needed reboot to activate changes delivered by some RHEL patches from 7:00-7:15 am.
Though the total scheduled downtime is 6:45am to 7:15am, we expect actual downtime to be less.
[Read more…] about cronus and rhea downtime – Tuesday October 22, 2019 06:45-07:15dhcp server downtime – Tuesday October22, 2019 06:30-07:00am
The departmental dhcp server will be down for a needed reboot to activate changes delivered by some RHEL patches.
Though the scheduled downtime is 6:30am to 7:00am, we expect actual downtime to be less.
[Read more…] about dhcp server downtime – Tuesday October22, 2019 06:30-07:00amlogin.eecs downtime – Tuesday October 22, 2019 06:30-07:00am
login.EECS.Berkeley.EDU will be down for patching and needed reboot to activate changes delivered by some RHEL patches.
Though the scheduled downtime is 6:30am to 7am, we expect actual downtime to be less.
[Read more…] about login.eecs downtime – Tuesday October 22, 2019 06:30-07:00amNetApp archive backups not working
When fileservice was restored after the PG&E outage last Friday, the part of the service that handles backups became non-functional. Staff are working to restore backup service as soon as possible.
Snapshots remain functional and available, providing easy access to file versions from the last 7 days. Older archive backups, from before October 9, also remain available. The broken archiving process means that backups after Oct 9 will not be available.
We make nightly copies of current fileserver data to our Disaster Recovery (DR) site at the San Diego Supercomputing Center (SDSC). This mirror of current data remains operational, and is not affected by the archive process failure.
Potential Campus Power Outage
We have been alerted to possible PG&E power outages due to fire danger, and that campus power may be shutdown at some point over the next 2-3 days. If we receive notice of imminent shutdown, we will work to safely shutdown our NetApp fileserver and other critical infrastructure systems; this may occur on very short notice.
We will endeavor to update this News item as we receive any updates.
See also https://berkeley.statusdashboard.com/
We recommend you proceed with shutting down any non-essential machines in advance, e.g. shutting down workstations at the end of the work day.
[Read more…] about Potential Campus Power OutageOctober 2019 Windows Server Patching
Microsoft will release their patches for the month on Tuesday, October 8th, 2019.
PRINT, WINTERM and WDS will automatically patch and reboot Wednesday, October 9th, at 3:00 am.
WINTERM may be unavailable till 10:00 am Wednesday, October 9th, depending on additional software updates.