We were hit today with a massive failure of events that caused a major network outage this afternoon. I sincerely regret the inconvenience that I know you all felt. Most services were back up by about 7PM this evening.
Here are the details for those who want specifics:
It appears that our core router had a serious failure. We attempted to switch to the back-up management module. However, that too failed. No amount of poking and prodding would bring either of them back. We then attempted to put in our hot spare. However, that unit began smoking! At this point we did not dare install anything in the chassis and we decided that we needed to replace the entire unit. This is no easy task as it is large, very heavy and has many fiber and copper cables connected to it. Once we swapped the spare chassis and all the cards and cables, a seemingly new problem arose, our firewall cluster started misbehaving immediately followed by a failure of one of the Gigabit Fiber I/O cards(the last one is what I now believe led to the whole business). We managed to stabilize the firewall and swap the last blade and slowly, networks services have come back. Many thanks to all of those who helped! It truly was a team effort. Whew, this has been a day that I will not wish to live over again.
Thank you for your patience,
Fred