After the planned work on the network switches this morning, the department website remains down, as well as the department maillist server lists.eecs and jabber server jabber.eecs.
[Read more…] about www.eecs, lists.eecs, jabber.eecs down this morning
Department Webservice
Department Webservice
Follow-up SAN updates
Some of the issues with yesterday’s upgrade could not be resolved yesterday, so we’ll be addressing them today. We expect the services to stay up, but there could be minor delays/slowdowns or short outages.
[Read more…] about Follow-up SAN updates
Major Upgrades on June 6, 2009
On Saturday, June 6th, we will be doing major upgrades to some of our
backend systems. This work will start at 10 am. We expect that some of
services will start coming back up in about 3 hours, but due to the
complexity and extensiveness of the upgrades, the downtime may extend well
beyond the 3 hour mark. These upgrades are imperative for the continuing
reliability of the infrastructure services. We apologize for the
inconvenience.
The following services will be affected:
* Home directories
* Project space
* UNIX SWW
* Departmental/IDSG web servers
* IMAP (due to the additional work, IMAP will be down for an additional 2 hours)
* argus/login (UNIX login servers)
* Department websites www.eecs / www.cs and all hosted project sites
* Department mailing lists hosted on lists.eecs
* buffy and all ACG applications
* wikis hosted on wiki.eecs
* Department’s jabber server, jabber.eecs
[Read more…] about Major Upgrades on June 6, 2009
Department website, mailing lists, jabber offline
The department’s web server (www.eecs & www.cs), mailing list server (lists.eecs), and jabber server (jabber.eecs) all run on a Xen virtual cluster. After last night’s network failure was resolved around 2:30am, the Xen cluster failed to recover gracefully. These services had to be manually restarted this morning. Web and mailing list services were restored around 9:20am, and jabber was restored around 9:45am.
Staff are investigating why this occurred and how to prevent it from happening again.
Staff are investigating why this occurred and how to prevent it from happening again.
Resolved as of 2009-05-22 09:30:00
Brief Downtime for Multiple IRIS Services
The EECS Department website, FTP server, Windows Terminal Server (winterm), the IRIS website, Jabber, and Sympa (mailing list) services will all experience brief periods of unavailability beginning at 7:00am on Wednesday, February 25, 2009, possibly lasting until 8:00am.
The servers hosting these services will be changing IP addresses during this time as their current subnet (128.32.139.0/24) is being repurposed. DNS will be updated at 7:00am as well, so you shouldn’t need to make any changes to client applications unless you’ve been accessing any of these services directly by their IP rather than by their DNS name.
[Read more…] about Brief Downtime for Multiple IRIS Services
Multiple Services Offline for About Eight Hours
The EECS website, FTP server, Jabber server and Sympa mailing-list server were offline for about eight hours on Sunday, Dec 28, 2008 beginning at 2:48 p.m.
The virtualization cluster that hosts these services experienced a failure when the network switch that supplies its management network was rebooted during the [scheduled network maintenance](https://iris.eecs.berkeley.edu/news/2246-eecs-network-maintenance-sunday-dec). We didn’t notice the failure until this evening, but were able to restore service quickly once we realized what was wrong.
I apologize for the length of the downtime and am planning on working to make the cluster more robust and improve our notification and monitoring so that any future issues will be noticed more quickly.
The virtualization cluster that hosts these services experienced a failure when the network switch that supplies its management network was rebooted during the [scheduled network maintenance](https://iris.eecs.berkeley.edu/news/2246-eecs-network-maintenance-sunday-dec). We didn’t notice the failure until this evening, but were able to restore service quickly once we realized what was wrong.
I apologize for the length of the downtime and am planning on working to make the cluster more robust and improve our notification and monitoring so that any future issues will be noticed more quickly.
Resolved as of 2008-12-28 22:32:00
EECS Website, FTP, Sympa and Jabber Offline Briefly for Maintenance
The EECS website, EECS FTP server, Sympa mailing-lists and Jabber (IM) service will all experience brief outages on Thursday, Dec 18, 2008 beginning at 10:00p.m. due to system maintenance.
These services won’t all be offline simultaneously, but each will experience a brief (less than 5 minutes) outage while the server hosting it is rebooted to apply operating system patches.
[Read more…] about EECS Website, FTP, Sympa and Jabber Offline Briefly for Maintenance
Brief Interruption of Some IRIS Services
The department website, Jabber and Sympa services were briefly interrupted this morning due to an issue with the cluster that provides them.
Each service was unavailable for a few minutes between 10:00 a.m. and 10:30 a.m. as the server cluster they run on was restarted. Restarting the cluster was necessary to alleviate a deadlock in the clustered file system used by the services.
My apologies for any inconvenience this has caused, I’m looking into how to avoid this situation in the future and bring more stability to the cluster.
Each service was unavailable for a few minutes between 10:00 a.m. and 10:30 a.m. as the server cluster they run on was restarted. Restarting the cluster was necessary to alleviate a deadlock in the clustered file system used by the services.
My apologies for any inconvenience this has caused, I’m looking into how to avoid this situation in the future and bring more stability to the cluster.
Resolved as of 2008-10-28 10:30:00
Unexpected Outage of Dept. Website, FTP, and Mailing Lists
Due to some necessary maintenance to some SAN hardware, the department’s website, FTP server, and mailing lists experienced brief outages this evening beginning around 6:24 p.m.
The website and FTP server were only offline for about six minutes, but mail sent to lists handled by lists.eecs.berkeley.edu (Sympa) during the outage may have been delayed up to 45 minutes. I apologize for any inconveniences this may have caused.
The server hosting the department web and FTP services along with the one hosting the database used by Sympa went offline when they unintentionally lost their connection to the SAN during maintenance. The maintenance was last-minute and shouldn’t have caused any disruption, but it didn’t quite go as planned.
Hardware has arrived to allow us to make redundant connections to the SAN for these services which will make them more resilient in future situations like this one. We plan to have the hardware installed and in use soon.
The website and FTP server were only offline for about six minutes, but mail sent to lists handled by lists.eecs.berkeley.edu (Sympa) during the outage may have been delayed up to 45 minutes. I apologize for any inconveniences this may have caused.
The server hosting the department web and FTP services along with the one hosting the database used by Sympa went offline when they unintentionally lost their connection to the SAN during maintenance. The maintenance was last-minute and shouldn’t have caused any disruption, but it didn’t quite go as planned.
Hardware has arrived to allow us to make redundant connections to the SAN for these services which will make them more resilient in future situations like this one. We plan to have the hardware installed and in use soon.
Resolved as of 2008-10-08 19:20:00
EECS Website, FTP, Jabber, and Sympa Offline for Urgent SAN Maintenance
Due to some urgent maintenance to the SAN, the EECS Website, EECS FTP server, Jabber, and Sympa (mailing lists) services will be offline from 5:00 p.m. to 6:00 p.m. on Tuesday, September 23, 2008.
The servers hosting the aforementioned services lack redundant connections to the SAN which prevents them from remaining in service during the maintenance. We realize this is an unfortunate limitation and are procuring new hardware which will add the redundancy necessary to handle similar situations more gracefully in the future.
Our apologies for the short-notice and inconvenience.
[Read more…] about EECS Website, FTP, Jabber, and Sympa Offline for Urgent SAN Maintenance