The department’s mailing list server ran out of disk space late Friday, April 26, around 4:30pm. A single very large message to one list caused what should have been a temporary issue. I removed some unnecessary files on Friday and thought that everything was working normally, but this morning noticed messages were not being distributed.

The server was restarted and has since caught up with all outstanding messages. Because of this you may notice messages sent over the last 3 days just showing up in your mailbox today, but no mail has been lost.

An investigation is underway as to why our old monitoring software did not alert us to this problem sooner. A new monitoring solution is being deployed in IRIS and this server will be moved to this new monitoring solution as soon as possible.

---

UPDATE

[2013-04-29 15:54:20 | Robert McNicholas]

IRIS is deploying a new monitoring system. Since our old system did not properly alert us to the weekend’s mailing list issue, we have migrated lists.eecs to this new server today and tested the basic functionality. We plan to add more comprehensive testing of end-to-end delivery of messages in the future, but for now basic system monitoring is in place and working.

Resolved as of 2013-04-29 10:30:00