We are investigating reports that Windows/CIFS shares from home.eecs and project.eecs became unavailable around 3pm today.
[Read more…] about CIFS Shares From home & project UnresponsiveProject Storage
Department Project storage
Firewall Migrations for IRIS NetApps and Instructional Servers, 3/21 6AM
At 6AM on Monday, March 21, we will be migrating the following networks to our new border firewall:
169.229.3.96/27 (Instructional Support Group servers)
169.229.3.128/26 (IRIS NetApp file servers, including home and project shares)
There will be a brief interruption to these networks as they are migrated. This will affect all home, project, and other file storage provided by the IRIS NetApps. After the migration, we encourage system administrators to test that connections to the NetApps are behaving as expected. Some file service clients may need to be rebooted or have their nfs service restarted.
Please contact us at help@eecs.berkeley.edu with any questions or concerns.
[Read more…] about Firewall Migrations for IRIS NetApps and Instructional Servers, 3/21 6AMChange of Some Fileservice IP Addresses March 15
Our current NetApp fileservice architecture has multiple hostnames pointing to the same IP address. As a performance measure, we will be introducing some new IP addresses for some of the hostnames, very early on Tuesday, March 15. If you have any custom firewall rules for the current IP addresses, you may need to modify them to include the new ones.
This change should not create any interruption in service. Any active mounts of the former IP addresses will continue to work.
[Read more…] about Change of Some Fileservice IP Addresses March 15Dept NetApp Filer Maintenance, 3pm January 3, 2022
We will be performing an OS upgrade on our departmental NetApp filers (upgrading from OnTAP version 9.7 to 9.8), which will require a service outage from 3pm to 9pm on Monday, January 3, 2022. All NFS and CIFS access to the filer (home and project storage) will be down during that time. Services (e.g. webservice) that depend on this storage will also be down or unresponsive.
While we are scheduling a 6 hour maintenance window, we will attempt to minimize the service outage if possible.
When maintenance is completed, NFS clients should automatically resume normal operation. CIFS clients (such as Windows and MacOS) may need to remount filesystems, or simply be rebooted.
[Read more…] about Dept NetApp Filer Maintenance, 3pm January 3, 2022Soda Hall Power Shutdown Tuesday 8/18 8am-8pm
On Tuesday, August 18th, Soda Hall building power will be shutdown from approximately 8am to 8pm for required facilities work. There will also be sporadic brief outages to various IRIS services between 9am and 2pm due to network maintenance.
Our primary datacenter and network core will receive backup generator support and we generally expect all services to remain online throughout the day. However, we are taking the opportunity to perform a number of network maintenance items which would have otherwise necessitated multiple outages later in the year.
As a result there will be multiple brief interruptions to various services including NetApp file servers, DHCP and DNS service, and the login.eecs and winterm terminal servers. We expect each individual interruption will last for less than five minutes, and this work will be performed between 9AM – 2PM. This post will be updated when all network maintenance is complete.
After network services are fully restored, sysadmins may need to reboot their systems, restart/remount nfs file systems, etc. to get fully back online.
[Read more…] about Soda Hall Power Shutdown Tuesday 8/18 8am-8pmNetApp archive backups not working
When fileservice was restored after the PG&E outage last Friday, the part of the service that handles backups became non-functional. Staff are working to restore backup service as soon as possible.
Snapshots remain functional and available, providing easy access to file versions from the last 7 days. Older archive backups, from before October 9, also remain available. The broken archiving process means that backups after Oct 9 will not be available.
We make nightly copies of current fileserver data to our Disaster Recovery (DR) site at the San Diego Supercomputing Center (SDSC). This mirror of current data remains operational, and is not affected by the archive process failure.
Potential Campus Power Outage
We have been alerted to possible PG&E power outages due to fire danger, and that campus power may be shutdown at some point over the next 2-3 days. If we receive notice of imminent shutdown, we will work to safely shutdown our NetApp fileserver and other critical infrastructure systems; this may occur on very short notice.
We will endeavor to update this News item as we receive any updates.
See also https://berkeley.statusdashboard.com/
We recommend you proceed with shutting down any non-essential machines in advance, e.g. shutting down workstations at the end of the work day.
[Read more…] about Potential Campus Power OutageNetApp fileserver maintenance – Phase III: Mon, July 22, 6-8pm
On Monday evening, July 22, we will be taking the next step in our fileserver migration: moving network interfaces from the old system to the new system. This will result in brief interruptions of access to home directories and project shares.
Our expectation is that each move of a network interface should only take a minute or so, and that most NFS and CIFS clients will likely recover just fine without a reboot. Any ongoing file copies or moves will be interrupted and terminated; any unsaved file editing should be closed before the maintenance period.
For any systems experiencing issues after maintenance is completed, a reboot would be recommended.
[Read more…] about NetApp fileserver maintenance – Phase III: Mon, July 22, 6-8pmNetApp fileserver maintenance – Phase II: Wed, June 19, 12-2pm
As part of our NetApp fileserver refresh cycle, we have a NetApp technician scheduled to be on site June 19, to assist with installation of the new hardware, and transfer of storage racks from the old NetApp system to the new one. Work is planned to take place between noon and 2pm.
Our expectation is that this work will not be disruptive to existing fileservice, but it involves a good amount of work around the existing hardware, so there is potential for unexpected problems.
NetApp fileserver downtime in preparation for upgrade – Phase I: Monday, June 10, 6-8pm
NetApp filers are due for their 4 year refresh cycle. Work on this has commenced, with new NetApp hardware on site. New filers are going to be added and data migration will occur some weeks down the road. While most of the work will be non-disruptive, some elements will be disruptive.
The first of this will happen on 10th June Monday, and the scheduled downtime will be from 6-8pm, to incorporate some network changes. All NFS and CIFS access to the filer (home and project storage) will be down during that time.
[Read more…] about NetApp fileserver downtime in preparation for upgrade – Phase I: Monday, June 10, 6-8pm