IRIS

Instructional & Research Information Systems

EECS

Project Storage

Department Project storage

NetApp archive backups not working

|

When fileservice was restored after the PG&E outage last Friday, the part of the service that handles backups became non-functional. Staff are working to restore backup service as soon as possible.

Snapshots remain functional and available, providing easy access to file versions from the last 7 days. Older archive backups, from before October 9, also remain available. The broken archiving process means that backups after Oct 9 will not be available.

We make nightly copies of current fileserver data to our Disaster Recovery (DR) site at the San Diego Supercomputing Center (SDSC). This mirror of current data remains operational, and is not affected by the archive process failure.

UPDATE

[2019-10-22 09:31 | pathma ]

This has now been fixed. The archival gap apart from the PG&E shutdown is from Oct 11th evening to Oct 21st. The cause of the problem was a netapp OEM’d product, “snapprotect”, which is no longer supported since end of August, but was fully functional and was planned to be replaced by an alternative solution in the weeks after. After the power outage, the snapprotect “permanent” license failed to be accepted and a netapp website to correct this also failed to work. Netapp didn’t want to fix their website, so I accelerated the alternative solution to completion, and that is now working as of my testing this morning.

Resolved as of 2019-10-22 09:31:00

Potential Campus Power Outage

|

We have been alerted to possible PG&E power outages due to fire danger, and that campus power may be shutdown at some point over the next 2-3 days. If we receive notice of imminent shutdown, we will work to safely shutdown our NetApp fileserver and other critical infrastructure systems; this may occur on very short notice.

We will endeavor to update this News item as we receive any updates.

See also https://berkeley.statusdashboard.com/

We recommend you proceed with shutting down any non-essential machines in advance, e.g. shutting down workstations at the end of the work day.

[Read more…] about Potential Campus Power Outage

NetApp fileserver maintenance – Phase III: Mon, July 22, 6-8pm

|

On Monday evening, July 22, we will be taking the next step in our fileserver migration: moving network interfaces from the old system to the new system. This will result in brief interruptions of access to home directories and project shares.

Our expectation is that each move of a network interface should only take a minute or so, and that most NFS and CIFS clients will likely recover just fine without a reboot. Any ongoing file copies or moves will be interrupted and terminated; any unsaved file editing should be closed before the maintenance period.

For any systems experiencing issues after maintenance is completed, a reboot would be recommended.

[Read more…] about NetApp fileserver maintenance – Phase III: Mon, July 22, 6-8pm

NetApp fileserver maintenance – Phase II: Wed, June 19, 12-2pm

|

As part of our NetApp fileserver refresh cycle, we have a NetApp technician scheduled to be on site June 19, to assist with installation of the new hardware, and transfer of storage racks from the old NetApp system to the new one. Work is planned to take place between noon and 2pm.

Our expectation is that this work will not be disruptive to existing fileservice, but it involves a good amount of work around the existing hardware, so there is potential for unexpected problems.

NetApp fileserver downtime in preparation for upgrade – Phase I: Monday, June 10, 6-8pm

|

NetApp filers are due for their 4 year refresh cycle. Work on this has commenced, with new NetApp hardware on site. New filers are going to be added and data migration will occur some weeks down the road. While most of the work will be non-disruptive, some elements will be disruptive.

The first of this will happen on 10th June Monday, and the scheduled downtime will be from 6-8pm, to incorporate some network changes. All NFS and CIFS access to the filer (home and project storage) will be down during that time.

[Read more…] about NetApp fileserver downtime in preparation for upgrade – Phase I: Monday, June 10, 6-8pm

Power Failure in Soda Machine room 11:40am Dec 20

|

Contractors attempting to repair the UPS in our primary machine room tripped a breaker which disabled most IRIS services. Service has been mostly restored as of around 2:05pm.

The lists.eecs.berkeley.edu mailing list server is still being rebooted but should be back shortly.

Resolved as of 2017-12-20 14:07:00