Beginning at 7pm these services will be operating in a degraded mode while preparations are made. During this time less computing power will be available and the web servers might be slightly slower than normal. At 8pm all servers will all be shut down while the data is copied over. This should take approximately 20-30 minutes, but we are scheduling 1 hour in case of unexpected problems.
This maintenance will allow us to retire some buggy SAN management software which has been the cause of most of our recent storage outages.
UPDATE
[2010-03-12 04:54:07 | Robert McNicholas]
Unexpected problems arose during the storage migration which caused the project to take far longer than expected. The downtime began at 00:30, and the maintenance was canceled at 4:30am.
Copying the data went was much slower than expected. Additionally, post-migration filesystem checks were taking extraordinarily long. The migration was canceled at 4:30am when it was determined the filesystem checks would not complete by 6am.
This maintenance will have to be rescheduled. I am investigating some patches for GFS2 that are available which should greatly speed up the filesystem checking.
Resolved as of 2010-03-12 04:44:00