Where did we
go?!?
At 9:38pm PST on Wednesday February
23, projectcoolmedia went off the net. 55 hours and 34
minutes later, we are back online and I'm composing this
message to you. It is Saturday morning, far too early to
be out of bed.
You didn't see anything about this in
the news. We're not sexy enough. We don't have enough
dollars and we weren't hit by hackers. I did try to get
some of the press interested, but to no avail. So let me
tell you what happened, from my perspective.
I do a fairly good job of keeping
track of our site from home. About 20 minutes after we
went down I noticed it. It looked like we may have had a
power failure so I drove into the office to investigate.
Power was still on but our router didn't have a
connection to the rest of the world.
A call to our ISP, Network
Architects, revealed that our DSL connection had
dropped. We were off the air. They were on the phone
with our DSL provisioner, Covad,
trying to find out what was wrong. Things were about to
get strange.
Covad promotes itself as the leader in
business DSL connectivity. It says so right there on the
front page of their
website, embedded as text in a graphic image. Here,
first hand, is our experience with Covad over the next
55 hours and also why you couldn't find us on the net.
The first thing we were told was that
no one could be sent to look at the problem except
between the hours of 8am and 5pm. Huh?!? I don't know
about you, but the net's a 24 hour operation. 8-5? Oh,
and no one would be able to do that until Monday, the
28th.
At this point it's time for the
wonderful people at Network Archtects to start beating
their heads against the wall. At this point I also learn
another scary fact: They've been beating their heads
against a wall since last Friday when another of their
customers went off the air with Covad.
So no service that night. Someone
should be able to take a look at it at 8am the next day.
(It's only a 9 hour a day internet, remember.) We'd only
be off the net for half a day.
Well...
The next story I heard was 'it seems
like the software upgrade they recently did may not have
been completely tested/compatible with the hardware and
they are unable to reprovision DSL lines. Engineers are
working on it.'
Then came 'It's a bad card, a
replacement has been ordered and you should be up by the
end of the day.'
The next day, Friday, the story became
'the part didn't come in last night. It should be here
today then it will be replaced at 2am Saturday morning.
Repairs like this are only done in the early morning
maintenence window.'
Well at 4:50am this morning, Saturday,
I got a phone call from my ISP. 'Can you cycle the power
on your router?' Ok, I climb out of bed and into the car
and come into the office. I spend the next 47 minutes on
the phone talking to my ISP who is talking to Covad. I'm
guessing that endusers talking to covad is a no-no for
some reason. Finally, we have a working connection. Days
late and far too many dollars short.
I'd like to thank Network Architects,
our ISP, for all the wall banging they did with their
heads throughout this ordeal. I'd also like to Heather
Champ, Phil Glatz and Jeffrey Zeldman for helping me
spread the word via grass roots net methods on why we
were down. I'd also like to thank the others who passed
on the letter I wrote explaining the situation. They are
too great a number to list.
Finally, this bit of follow up. Covad
must house it's equipement in PacBell's offices since
DSL depends on phone line wiring. I received the
following in a message from a high integrity mailing
list I am on just shortly after we came back up:
