When fileservice was restored after the PG&E outage last Friday, the part of the service that handles backups became non-functional. Staff are working to restore backup service as soon as possible.
Snapshots remain functional and available, providing easy access to file versions from the last 7 days. Older archive backups, from before October 9, also remain available. The broken archiving process means that backups after Oct 9 will not be available.
We make nightly copies of current fileserver data to our Disaster Recovery (DR) site at the San Diego Supercomputing Center (SDSC). This mirror of current data remains operational, and is not affected by the archive process failure.
UPDATE
[2019-10-22 09:31 | pathma ]
This has now been fixed. The archival gap apart from the PG&E shutdown is from Oct 11th evening to Oct 21st. The cause of the problem was a netapp OEM’d product, “snapprotect”, which is no longer supported since end of August, but was fully functional and was planned to be replaced by an alternative solution in the weeks after. After the power outage, the snapprotect “permanent” license failed to be accepted and a netapp website to correct this also failed to work. Netapp didn’t want to fix their website, so I accelerated the alternative solution to completion, and that is now working as of my testing this morning.
Resolved as of 2019-10-22 09:31:00