8 Years since the Eastern Seaboard Blackout - Has it Been that Long?

8 Years since the Eastern Seaboard Blackout - Has it Been that Long?
The Eastern Seaboard power blackout that occurred in 2003 (started at 4:10 on Aug 14, 2003, with the recovery varying by region) was a milestone in many of our lives. Not only was it full of personal consequences - I can remember my wife calling me in a panic as I was driving home, but it had some severe business and societal impacts, and changed how we view service interruptions in IT. The blackout forced many businesses to seriously consider what an interruption in basic services could cost the organization, and also to consider how to do business without various services. In short, we now do Disaster Recovery Planning (DRP) and Business Continuity Planning (BCP) a lot more, and a lot more rigorously than we did pre-2003. The blackout also forced us as a society to consider just how critical our "Critical Infrastructure" is, and how long it had been since it was last looked at closely (post WWII in a lot of cases). It also forced us to look at security in a whole new light - the electrical grid had been built on a "we trust our neighbours" model, which was one of the root problems that made the 2003 event so wide-spread. Most utilities are now a lot more self-contained, or at least aware of the "good fences make good neighbours" design approach these days. We're a lot more aware now of just how complex our utility infrastructure is now, we've seen first hand what happens when the power goes off, and how complex it was to get the power back on after a widespread hit. While NERC (North American Electric Reliability Council) has been around since 1968, the power outage was one of the catalysts in re-formulating it as The North American Electric Reliability Corporation, and re-writing the Critical Infrastructure Protection (NERC CIP) regulations in 2006. Above all, to me the 2003 blackout illustrates just how short our memory is. We had a power hit that affected New York City in 1977 (which I remember), and a much larger Northeast area event back in 1965 (I was 3 then, so before my time). I guess as a society we're a lot like my cat - bad things need to take place a few times at least before it sinks in. Hopefully, now that we've got critical infrastructure standards and particularly security written into regulations and law, it'll stick. Also, now that we've got some momentum in BCP and DR planning, the private sector will follow along. We'd love to hear your comments, either from your experiences during any of the larger power problems, or how they've affected your organization. =============== Rob VandenBrink Metafore	Rob VandenBrink 578 Posts ISC Handler Aug 17th 2011
Thread locked Subscribe	Aug 17th 2011 1 decade ago
I spent 8 hours sitting in a R.A.C.E.S. radio room doing backup communications. There were 3 of us total. Thank goodness we just did rough outage assessments from our QTH. Amateur Radio is often looked at as 'old technology' but usually works when other stuff doesn't. Not sure what happens if we have a good solar flare. 73	Anonymous
Quote	Aug 17th 2011 1 decade ago
I remember it well. I was at my last job and Blaster was knocking down systems over all across the company but management said we could not patch the servers because people needed to do their work. Which Blaster was interrupting, of course. When we went on UPS, everything went dark except for the data center so we began applying the Blaster patch and rebooting servers. It was odd how the power loss was distributed. My small suburb never had a power loss except for about fifteen minutes in the beginning. But every town on our every border was black. Everyone was coming to our town to fill up with gas. My home is in a lower part of the area so we had full water pressure for the three days. But the people just a few streets east of us, and higher, lost water pressure after one day. I guess this was one time when it was good to be downhill.	Anonymous
Quote	Aug 17th 2011 1 decade ago
Oh yeah, I forgot to mention I was in Akron, OH at the time, the home of FirstEnergy, supposedly the company that kicked it all off by not trimming the trees around their power lines. Or maybe not: http://www.schneier.com/essay-002.html "The Blaster worm affected more than a million computers running Windows during the days after Aug. 11. The computers controlling power generation and delivery were insulated from the Internet, and they were unaffected by Blaster. But critical to the blackout were a series of alarm failures at FirstEnergy, a power company in Ohio. " Maybe this event is where the Stuxnet authors got their inspiration.	Anonymous
Quote	Aug 17th 2011 1 decade ago

The Eastern Seaboard power blackout that occurred in 2003 (started at 4:10 on Aug 14, 2003, with the recovery varying by region) was a milestone in many of our lives. Not only was it full of personal consequences - I can remember my wife calling me in a panic as I was driving home, but it had some severe business and societal impacts, and changed how we view service interruptions in IT.

The blackout forced many businesses to seriously consider what an interruption in basic services could cost the organization, and also to consider how to do business without various services. In short, we now do Disaster Recovery Planning (DRP) and Business Continuity Planning (BCP) a lot more, and a lot more rigorously than we did pre-2003.

The blackout also forced us as a society to consider just how critical our "Critical Infrastructure" is, and how long it had been since it was last looked at closely (post WWII in a lot of cases). It also forced us to look at security in a whole new light - the electrical grid had been built on a "we trust our neighbours" model, which was one of the root problems that made the 2003 event so wide-spread. Most utilities are now a lot more self-contained, or at least aware of the "good fences make good neighbours" design approach these days.

We're a lot more aware now of just how complex our utility infrastructure is now, we've seen first hand what happens when the power goes off, and how complex it was to get the power back on after a widespread hit.

While NERC (North American Electric Reliability Council) has been around since 1968, the power outage was one of the catalysts in re-formulating it as The North American Electric Reliability Corporation, and re-writing the Critical Infrastructure Protection (NERC CIP) regulations in 2006.

Above all, to me the 2003 blackout illustrates just how short our memory is. We had a power hit that affected New York City in 1977 (which I remember), and a much larger Northeast area event back in 1965 (I was 3 then, so before my time). I guess as a society we're a lot like my cat - bad things need to take place a few times at least before it sinks in. Hopefully, now that we've got critical infrastructure standards and particularly security written into regulations and law, it'll stick. Also, now that we've got some momentum in BCP and DR planning, the private sector will follow along.

We'd love to hear your comments, either from your experiences during any of the larger power problems, or how they've affected your organization.

===============

Rob VandenBrink
Metafore

Rob VandenBrink

578 Posts
ISC Handler

Aug 17th 2011

I spent 8 hours sitting in a R.A.C.E.S. radio room doing backup communications. There were 3 of us total. Thank goodness we just did rough outage assessments from our QTH. Amateur Radio is often looked at as 'old technology' but usually works when other stuff doesn't. Not sure what happens if we have a good solar flare. 73

Anonymous

I remember it well. I was at my last job and Blaster was knocking down systems over all across the company but management said we could not patch the servers because people needed to do their work. Which Blaster was interrupting, of course.

When we went on UPS, everything went dark except for the data center so we began applying the Blaster patch and rebooting servers. :-)

It was odd how the power loss was distributed. My small suburb never had a power loss except for about fifteen minutes in the beginning. But every town on our every border was black. Everyone was coming to our town to fill up with gas. My home is in a lower part of the area so we had full water pressure for the three days. But the people just a few streets east of us, and higher, lost water pressure after one day. I guess this was one time when it was good to be downhill. :-)

Anonymous

Oh yeah, I forgot to mention I was in Akron, OH at the time, the home of FirstEnergy, supposedly the company that kicked it all off by not trimming the trees around their power lines.

Or maybe not: http://www.schneier.com/essay-002.html

"The Blaster worm affected more than a million computers running Windows during the days after Aug. 11. The computers controlling power generation and delivery were insulated from the Internet, and they were unaffected by Blaster. But critical to the blackout were a series of alarm failures at FirstEnergy, a power company in Ohio. "

Maybe this event is where the Stuxnet authors got their inspiration.

Anonymous