UPDATE
[2016-03-31 18:36:11 | Lars Rohrbach]
Webservice on www.eecs is down, while we continue to investigate issues of NFS access to the fileservers.
UPDATE
[2016-03-31 19:31:52 | Lars Rohrbach]
Normal webservice on the department webserver has resumed as of 7pm.
The NFS access problem was the result of a corrupted DNS cache on the NetApp fileservers. After flushing the caches, NFS mounts were once again functional.
As of 7am, we are seeing further NFS access denied issues on the webserver. We will work to re-establish webservice, and escalate the issue with the vendor.
As of 9:08am, most webservice resumed as normal. But some NFS mounts were still being denied until shortly after 9:30am. We are still working with NetApp to fully resolve this issue.
UPDATE
[2016-04-02 08:40:39 | Lars Rohrbach]
Department webservice was stabilized yesterday, mid-afternoon. Homepages and virtual web hosts on the department webserver are being displayed as expected.
While the majority of webpages were working properly as of about 9:30am Friday, throughout the day we continued to experience occasional denied NFS access, causing some webpages to not load while others were fine. By 3pm, all was well.
I would like to apologize for the inconvenience this may have caused. We understand that downtime for department homepages impacts classes, research, and those on the job circuit, or seeking internships or funding.
The NFS issue stems from the very large number of NFS requests from the department webserver, coupled with an occasional error on the fileserver in getting the wrong hostname via reverse-DNS added to its netgroup cache. The problem seems to have been introduced Thursday midday, when a minor update was applied to the NetApps. With the help of NetApp technical support, we now have a workaround in place so that we can avoid having to flush the cache every few minutes. Our case with NetApp remains open, as technicians work on a permanent fix.
Resolved as of 2016-04-01 22:00:00