Partial network shutdown scheduled for the evening of March 11th

On Monday evening, March 11th, we will be putting the new network switch in Brothers College into production as the first step in upgrading our campus network infrastructure. We ran into problems during our previous attempt in January and switched back to the old equipment while we did some additional testing to work around the issue. This is the largest change we've made to the network infrastructure in a decade. We expect that it will take a few hours to swap out the switch and reconnect everything. Our intention is to make sure that nearly all public-facing services continue to operate, with only a few exceptions, and barring unforeseen problems. We will be posting updates to the CNS Facebook and Twitter feeds during the work.

The following buildings will have no network connectivity (wired or wireless) on the evening of Monday 11th: Brothers College, Hall of Science, Davies, Faulkner, Gilbert, Hannon, Lewis, Sitterly, Smith and Sycamore.

During the day on Monday, we will take steps to host Internet-facing services (Moodle, Library, faculty/staff GroupWise email, the web site, etc.) on servers in Pepin and the Learning Center so that they remain operational while we work. At 6:30 PM, there will be a very brief internet outage as we re-route our internet connection from Brothers to Learning Center. A second, brief internet outage will occur at the end of the evening when we switch the internet connection back to Brothers.

At 7pm, we are going to start to disconnect equipment and the buildings connected to Brothers College. At that point, the buildings listed above will be off the network for the evening.

Once Brothers is back up and running, we will be carefully monitoring for problems that might be caused by the change in configuration. We will announce a schedule for replacing the switch in the Learning Center, which will give us the full benefit of the network upgrade in the near future. Given the scope of this change to the backbone of the network, we may need to schedule additional work in order to make configuration changes should unexpected complications arise.

We will be updating this entry, adding additional details.

Just a little background . . .

We ran into an unexpected problem during the January attempt, which lead to additional disruptions to some services. Around 2am, we made a decision to put the old switch back into use, so that services were largely restored to normal by 5am. However, the old switch is actually sitting next to the new one - we don't have to physically remove it from the rack and install the new one this time (that alone took two hours last time). This time, it's simply a matter of unhooking the old switch, plugging it in, and testing. While we did find the cause of the major problem that came up in January, and we've done additional testing, it is still possible that unexpected issues will come up - we can't completely duplicate the production network (equipment, connections, configuration, network traffic, and so forth). Once we're done, we still to have to schedule the Learning Center switch replacement. We'd originally hoped to do BC over January and LC during this coming week. If this upgrade goes well, we will be looking at the calender to try to find the least disruptive time possible to do that. If things are working well, we will be more conservative about the LC replacement schedule. If the BC switch is working well, but some of the configuration changes needed are causing problems between BC and LC (which would be very disruptive), we might adopt a more aggressive schedule for replacing the Learning Center switch if we believe it will solve the issue.

What we're doing

We're replacing our 10-year-old Cisco Catalyst 6500 swiches in Brothers College and the Learning Center with new data center switches from Brocade. Nearly all of the network traffic on campus passes through one or both. Once it's complete, we'll not only have new, faster switches, but also a faster data connection between the BC and LC (10 gigabits per second of traffic, instead of a pair of 1 gigabit per second connections). In addition, the computer systems room in Pepin, where we have some important systems, will no longer pass traffic through a smaller switch in Pepin, but instead pass network traffic directly to either LC or BC. We're also going to patch the Pepin switches to the rest of the network in a way that will allow that west side of campus to remain connected to the internet and more campus systems if Learning Center loses power and emergency power. Doing this ahead of the Learning Center upgrade will also make that upgrade substantially less disruptive.