Instructional & Research Information Systems
| Lars Rohrbach
Microsoft will release their patches for the month on Tuesday, November 12th, 2019.
PRINT, WINTERM and WDS will automatically patch and reboot Wednesday, November 13th, at 3:00 am.
WINTERM may be unavailable till 10:00 am Wednesday, November 13th, depending on additional software updates.
Resolved as of 2019-11-13 07:00
| Derek Calderon
Today between 1206 and 1212, EECS networks were isolated from the outside world due to an unanticipated routing protocol behavior during a reconfiguration related to our firewall replacement project. This has been corrected and there will be no further issues.
| Lars Rohrbach
Staff are investigating intermittent disturbances in network traffic at or around the EECS border. We’ll provide updates here as we learn more.
Final Update 11/4 12:09:
This issue was caused by a DDoS or DDoS-style traffic which was overloading our firewall, which is near the end of its life. Using a number of active and passive mitigation techniques, we believe the impact of this traffic has been fully mitigated and there should be no further issues; the firewall has been stable since Friday morning. We are also accelerating the project already in the works to replace this firewall.
| Lars Rohrbach
We’ve been notified of another impending PG&E power outage scheduled to begin Saturday evening, October 26; campus Co-Generated power should take over, similar to the last outage earlier this month (https://iris.eecs.berkeley.edu/news/resolved-incidents/potential-campus-power-outage/)
We are planning to keep essential services up and running, including networking, fileservice, web services (people.eecs and group project sites), DHCP, DNS, login.eecs, WINTERM, and lists.eecs (maillists). Some redundant systems may be powered down (e.g. cronus will stay up for DNS/NIS, but rhea may be shut down).
We encourage everyone to shut down all non-essential systems at the end of day today (Friday). Any reduction in power consumption will make it easier to keep other essential services running.
[Read more…] about Oct 26 Planned Power Outage| Mark Kraitchman
The departmental authenticating outgoing smtp server gateway.eecs will be down for a needed reboot to activate changes delivered by some RHEL patches.
Though the scheduled downtime is 7:15am to 7:45am, we expect actual downtime to be less.
[Read more…] about gateway.eecs downtime – Tuesday October 22, 2019 07:15-07:45am| Mark Kraitchman
cronus one of the departmental DNS, NIS and time servers will be down for a needed reboot to activate changes delivered by some RHEL patches from 6:45-7:00am.
rhea the other departmental DNS, NIS and time server will be down for a needed reboot to activate changes delivered by some RHEL patches from 7:00-7:15 am.
Though the total scheduled downtime is 6:45am to 7:15am, we expect actual downtime to be less.
[Read more…] about cronus and rhea downtime – Tuesday October 22, 2019 06:45-07:15| Mark Kraitchman
The departmental dhcp server will be down for a needed reboot to activate changes delivered by some RHEL patches.
Though the scheduled downtime is 6:30am to 7:00am, we expect actual downtime to be less.
[Read more…] about dhcp server downtime – Tuesday October22, 2019 06:30-07:00am| Mark Kraitchman
login.EECS.Berkeley.EDU will be down for patching and needed reboot to activate changes delivered by some RHEL patches.
Though the scheduled downtime is 6:30am to 7am, we expect actual downtime to be less.
[Read more…] about login.eecs downtime – Tuesday October 22, 2019 06:30-07:00am| Lars Rohrbach
When fileservice was restored after the PG&E outage last Friday, the part of the service that handles backups became non-functional. Staff are working to restore backup service as soon as possible.
Snapshots remain functional and available, providing easy access to file versions from the last 7 days. Older archive backups, from before October 9, also remain available. The broken archiving process means that backups after Oct 9 will not be available.
We make nightly copies of current fileserver data to our Disaster Recovery (DR) site at the San Diego Supercomputing Center (SDSC). This mirror of current data remains operational, and is not affected by the archive process failure.
[2019-10-22 09:31 | pathma ]
This has now been fixed. The archival gap apart from the PG&E shutdown is from Oct 11th evening to Oct 21st. The cause of the problem was a netapp OEM’d product, “snapprotect”, which is no longer supported since end of August, but was fully functional and was planned to be replaced by an alternative solution in the weeks after. After the power outage, the snapprotect “permanent” license failed to be accepted and a netapp website to correct this also failed to work. Netapp didn’t want to fix their website, so I accelerated the alternative solution to completion, and that is now working as of my testing this morning.
Resolved as of 2019-10-22 09:31:00
| Lars Rohrbach
We have been alerted to possible PG&E power outages due to fire danger, and that campus power may be shutdown at some point over the next 2-3 days. If we receive notice of imminent shutdown, we will work to safely shutdown our NetApp fileserver and other critical infrastructure systems; this may occur on very short notice.
We will endeavor to update this News item as we receive any updates.
See also https://berkeley.statusdashboard.com/
We recommend you proceed with shutting down any non-essential machines in advance, e.g. shutting down workstations at the end of the work day.
[Read more…] about Potential Campus Power Outage