The problem is that the vast majority of our 5GHz wireless radios are currently using the same wi-fi channel, causing a large amount of co-channel interference which is greatly degrading wireless performance.
Last night we made changes to the configured bandwidth of the 5GHz channels in an attempt to increase general reliability. Unfortunately we did not notice until well after the fact that the wireless controller’s dynamic channel assignment (DCA) algorithm was malfunctioning; this resulted in all or most radios defaulting to channel 36 and being unable to change.
Correction of the issue will require a full outage of the wireless network, which is why with user feedback we have decided to wait until after the business day to make any changes.
UPDATE
[2017-10-20 15:01:52 | Derek Calderon]
We managed to (ab)use the high availability clustering function of our wireless controllers to un-stick the DCA algorithm, and the 5GHz radios are now properly selecting their wi-fi channels. The majority of APs are now operating correctly and the remainder will correct themselves as DCA continues to run.
We still plan to make some minor changes this evening to clean things up, however we believe the issue is largely resolved at this time.
Resolved as of 2017-10-20 13:40:00