Disaster at Amherst
Earlier this year Amherst College, another top-five US liberal arts college like Carleton, experienced a total failure of its networking infrastructure, leaving the entire campus without usable internet service for five full days. The disruption and loss of work was enormous.
You may wonder: Could that happen here at Carleton?
Yes, of course it could, or something like it! Technology is vastly complicated, which leads to unpredictable behaviors. Nonetheless, increasing reliability is a high priority for ITS, which includes reducing complexity when appropriate and actively learning from situations like Amherst’s.
What Carleton has done
ITS has built backups into our network and related infrastructure. We have multiple physical paths to our upstream internet providers and built-in redundancy to almost everything in the datacenter. Despite all this prep, though, Murphy’s Law applies here the same as anywhere else. You may recall the outage from last September, when an equipment failure in the datacenter brought most of our internal systems down on the second day of classes. Six hours then was bad enough. Can you imagine being down for multiple days?
Knowing how disruptive these outages can be, ITS continues to invest in external “cloud” services like Gmail, because we recognize the need for those services to remain accessible even when there’s a problem with our internet connection or datacenter. Currently, though, logging in to Google with your Carleton account still depends on our local login system, and therefore also on our internal network.
We’ve changed that. On March 13, ITS deployed a backup authentication system in Google’s cloud infrastructure that can take over for Carleton’s normal login page. That means that you’ll still be able to log into Gmail, Google Drive, Dropbox, and dozens of other cloud-hosted applications even if our campus internet connection is down or there’s a problem with our datacenter.
It doesn’t make sense for us to invest in cloud services like Gmail if they require on-campus systems in order to work. Moving our authentication infrastructure into the cloud ensures that those systems remain accessible even if we’re dealing with a local disaster, and makes it easier for us to communicate with our clients during and after an outage.
– Dave Flynn, Director of Systems and Infrastructure for ITS
If nothing else, the failure of Amherst’s network simply serves as a wake-up call that reinforces ITS’s already significant commitment to continuous re-evaluation and improvement, and motivates us to think of new and better ways to structure our network, so as to avoid that kind of disaster.