It was early 1995, and EarthLink was in one of its earliest growth periods. We'd just moved from our first 800 square foot office in Los Angeles to a 3,000 foot space down the hall in the same building. Even then, we quickly filled the new office, and people were sitting at desks in the hallway. We were adding subscribers at a rate faster than we could handle, sometimes growing 10% a week, and our systems were straining under the demand. We were barely managing to keep up.
Then, on February 14th, 1995, our main subscriber database — the file that held the usernames and passwords — got totally corrupted, and suddenly, we could no longer authenticate our users when they dialed in. (It was modems in those days, remember.) EarthLink was completely down. Fortunately, we had a back-up tape at the office that was only a few days old. Our engineers went to load the tape and restore our system. We would be back on in an hour, still a horrific outage, but survivable. Then, we discovered that the back-up tape was damaged and totally unreadable. We were dead in the water. If we couldn’t figure out how to get the data off that tape, EarthLink would be out of business, and I would be back to running a coffee house.
I put the word out to everyone I knew for a data recovery expert. An EarthLink customer referred me to a firm in San Diego, and within minutes, the damaged tape was speeding south in a car. We fielded calls from angry customers while waiting anxiously for news. Many hours later, we received word that although some of the data was was lost, most of the information on the tape had been taken off bit-by-bit and reconstructed. We were saved! The recovered file was loaded onto a hard drive and driven (there was no Gobbler back then) back to EarthLink's offices in LA. Our engineers stayed up all night and got us back online.
At EarthLink, February 14th, 1995 became known as the Valentines Day Massacre. We survived, and although our customers were rightfully angry with us, I think they also sensed that this was the early days of the Internet. Stuff didn't "just work" the way it does now. The next day, our modem banks were once again full of happily connected EarthLink customers. And I was back to figuring out out to hire and expand faster to keep up with demand.
Like many start-ups, there times in those early days at EarthLink when we were up against seemingly insurmountable obstacles. But we knew we had something that people wanted, so we kept going, forging ahead despite it all. Being successful, especially in a start-up, is all about perseverance.
For a future post: How we blew up our building's power transformer and ran EarthLink for two months on a generator in the parking lot.