The wired network in Cory Hall was experiencing intermittent slowness today due to a router issue. We narrowed the issue down to a specific uplink, which was temporarily disabled to improve network performance. This issue occurred on the same device as yesterday’s LACP problem, and we are beginning to suspect a buggy config or device. In the coming weeks we will analyze our configurations, perform some deferred software upgrades and additional testing on this router, and open a bug ticket with the vendor if necessary.
All traffic has been diverted away from this router and we believe the issue to be resolved at this time. If you’re continuing to notice slowness or any other network issues, please email firstname.lastname@example.org.
Soda Hall will be powered down between 0600 and 1900 on Saturday, May 30th in order to complete the electrical interconnect between Soda and Jacobs Halls. During this time the building will be completely shut down and entrance to the building will not be allowed.
We have arranged for reserve power to be provided to the network and server rooms in 288, 290, and 420A Soda, such that department and research infrastructure and services will remain online and available to users outside Soda Hall.
We recommend powering off all equipment such as desktop computers ahead of time to prevent any issues caused by unexpected power loss. Additionally, any sysadmins hosting equipment in machine rooms other than 288, 290, and 420A will need to power their machines down in advance of the outage.
Should you need access to the building during the outage (for example, you are a sysadmin who would like to take advantage of the downtime to perform deferred maintenance), please email Miles Bianchi (email@example.com) for consideration by the Safety Committee. This access will only be granted for special cases.
An attempt at adding a new switched link in the network core resulted in a broadcast storm which crippled the EECS network between approximately 5 and 6 PM this evening. The maintenance was rolled back and the network was allowed to stabilize. We will reattempt this maintenance in the future after we have analyzed the data gathered to determine what went wrong. [Read more…] about Intermittent Network Outages 5/4/15
We are pleased to announce a project to fully upgrade the EECS Wi-Fi infrastructure. The community has borne with us through years of wireless issues, including poor coverage, frequent disconnects, inability to connect, and others. After working through a sizable backlog of technical debt, we’re finally ready to get to work on a solution.
Summary of main points:
* We will be upgrading equipment as well as redesigning our coverage plan
* Adding 802.11ac support and increasing density in the 5GHz band
* Phase I: Survey – April through May
* Phase II: Install – June through September (possibly October, contingent on unforeseen delays) [Update: please see timeline post below for live details]
* Engineers will need access to all spaces (offices, labs, classrooms, etc.) during both phases of the project (we cannot ensure good coverage for any area that is not surveyed properly!)
Beginning this week, engineers will be performing detailed surveys of Soda, Cory, and Sutardja Dai Halls in order to fully understand each space and its uses. Our current deployment was designed during a much earlier time in Wi-Fi’s history and was never intended to cope with the number of wireless devices in use today or the way in which we use them; it also includes some oddities such as access points being mounted in hallways where they have trouble reaching into offices and labs. This is being taken into account and access points will be moved into the areas where they will be used, which will require our engineers to fully explore all classrooms, labs, offices, and conference rooms. In addition to reevaluating where and how we place access points, we will be increasing density (more APs) and upgrading to newer and more powerful models which support 802.11ac.
This project is broken up into two main phases. As mentioned above, the survey phase is beginning this week, and will consist of detailed walkthroughs of all wireless locations. The second phase, which will begin after the current semester is concluded, will involve contractors running cables and installing the new access points.
The current wireless deployment will remain online until the new APs are fully installed, at which point we will switch over.
To ensure that our new design provides complete coverage, our engineers will need access to all spaces within EECS, including individual offices. We will do our best to work around busy people and spaces, but if possible please try to take a moment for us when we come around. Remember: we cannot ensure good coverage for any area that is not surveyed properly.
Tonight, March 12th at 6:30, there will be a scheduled maintenance in Cory Hall to add a redundant 10-gigabit link between the building’s main router and the access network. A less-than-one-minute outage is expected while the cutover is performed, which will affect both the wired and wireless networks in Cory Hall. This maintenance will increase the resiliency of the Cory Hall network in the event of a device or link failure. [Read more…] about Scheduled Network Maintenance, Cory Hall, Tonight 3/12
Approximately 4AM this evening, we lost contact with the access switch in the Nanolab on the 6th floor of Sutardja Dai Hall. We presume the network to be down at this time, and network staff will begin diagnosis as soon as they arrive onsite. This should affect users in the 6th floor area of the Nanolab only, as the rest of SDH uses separate switches. [Read more…] about Network Outage, 6th Floor SDH Nanolab
On Wednesday evening at 6PM, we will be performing maintenance on the EECS firewall in order to to reinsert the primary device (which failed during the January power maintenance) into the production cluster. Per the vendor, no interruption in service is expected; however, in the event of unforeseen complications Internet connectivity may be lost to the department for some time. [Read more…] about EECS Firewall Maintenance, Wed. 2/11
There will be another emergency scheduled maintenance tonight at 7PM. This window is scheduled to close at 10PM but the network will be available earlier if all goes well. The wireless network in all locations except BWRC will be unavailable during reconfiguration. Wired networking also will be briefly (less than ten minutes) unavailable for a routing change. We believe these steps will take us closer to fully eliminating the current traffic issues.
We will be performing emergency network maintenance tonight at 7:30-10. The network will not be completely unavailable during this time, but the wireless network as well as the wired network in some buildings may periodically go offline. It is our hope that we can use this time to gain insight into the ongoing network issues and help with a resolution before the semester begins.
We are currently experiencing worse-than-normal issues with the wireless network including inability to connect, random drops, increased latency, and reduced throughput. We believe we have an idea of the cause and will be working to clear things up over the next couple of days.