Oregon Employment Division Servers and Phones Crash: 10/04/2009

• Length of outage: Officially, 10 hours. In reality, 24+ hours

• Number of people affected: 165,000 Unemployment recipients, plus OED staffers.

I happened to be one of those affected by this outage, because at the time, I was drawing unemployment!

The original article in the Oregonian the following Monday spun the story to make it sound as if it was the extra load of new people applying for benefits that crashed the system. Even in this later, edited version, you don't find the truth until well down the page:

Original news story HERE.

Here's where the truth comes out:

Problems started Sunday when a computer server crashed while state workers were doing maintenance on the state's computer network. The 60 percent of unemployed who usually file online for their weekly checks turned to the telephone to file their claims on the state's interactive voice response system. At the same time, the group looking for emergency extensions also were swamping the phone lines.

So, they don't explicitly say it was an upgrade, but the system was down when I tried to use it early on Sunday morning, indicating that they had been working on the system during the overnight shift. This smells suspiciously like an upgrade was being applied. Pinkston's Law!

Also, it is an interesting example of the cascading failure effect; when people could not file online, they moved to the phones to file on Monday (so much for the 10-hour outage -- the system was still down Monday morning). The phone system is not sized to handle all of the traffic that the online system handles, so it crashed, too.