30 hours of panic; or how to survive a DDoS attack
As you may have noticed, Smokingpipes.com has been singularly unresponsive during the past twenty-four or so hours. At 1pm yesterday, we were hit with a DDoS attack (Distributed Denial of Service). Since then, with a nice stretch of working between 3am and 8am this morning (yeah, hardly a peak site traffic time), our servers have been slammed by millions of bogus requests from thousands of IPs. We've come up with a temporary solution, by moving Smokingpipes.com to another server and just serving up a flat html page from there, which redirects to the full site's web server on our old servers, which handles legitimate traffic normally.
We've also discovered that we are one of at least four major premium tobacco retailers (with the other three being major cigar retailers) to be hit with a DDoS attack in the last 24 hours. This appears to be directed against purveyors of tasty, high quality smokables. These are targeted attacks. Still, in the Wild West that is the internet, the good code slingers are winning this particular round over the bad ones with our stop gap measure; we shall see how it progresses over the next few hours.
So, what is a DDoS attack and what have we done to make the site available?
|Normal: Happy users visit Smokingpipes.com and see a bevy of beautiful briars|
In our wildly simplified diagram, this is how things are normally. Happy pipe smokers go to Smokingpipes.com, read about or purchase or drool over pictures of pipes, pipe tobacco and cigars. Your computer asks our web server (by way of various servers in between) and our server nicely responds by serving up lots of fun pictures, images, tons of wonderful information, all stored on either our database server or our assets server.
|A DDoS Attack: Lots of Zombies trying to get in the door makes it impossible for the regular user to access the site.|
When someone initiates a DDoS attack against us, they've used thousands of slaved computers (think of them as zombies, perhaps) to remotely make http requests to our servers, specifically to the Smokingpipes.com domain. Our servers, though they are shiny and fast, are utterly unable to serve up the information fast enough and end up getting completely bogged down trying to contend with all of the bogus traffic. The thousands of computers are innocent bystanders too-- more than likely they were infected with a trojan that causes them to make these requests at the behest of the master (evil!) computer, much like zombies at the behest of some wicked puppet master.
Keep in mind that this traffic doesn't do anything to us except just ask us questions. We've not been hacked, nothing has been compromised, everything is safe. All is normal, except that thousands of extra computers are asking our servers for information and we just can serve it up fast enough.
|Our Solution: behind the splash page (bouncer) life is normal, but the bouncer is there to keep out the bogus traffic.|
Part of the problem is that our regular servers, sort of like our store staff at Low Country Pipe & Cigar, like to greet folks with lots of fun stuff, show them what's new, and point out interesting odds and ends. That first page people reach is filled with dynamic content, pictures and other things that, in the normal running of things aren't taxing at all for our servers, but multiply that normal load times fifty or a hundred and things slow to a crawl or stop altogether. So, it's sort of like Ron, Kelly, Vince and Jennifer in the store all trying to show pipes to a thousand customers simultaneously, most of which really just want to stand in the middle of the store and not really do anything. Obviously, as good as our store staff is, they're going to grind to a halt in a hurry if they had to contend with this.
So, what do you do if you have this problem in real life? You hire a bouncer. Our digital bouncer lets in anyone who asks nicely, but doesn't try to help anyone or be particularly nice about his greeting. It makes it much easier for him to keep up with the multitudes. He then lets in anyone who asks nicely, and inside the store, the customer experience (and our poor, harried store staff in the metaphor) return to normal. The digital bouncer, our splash page, does this by serving up the simplest code possible (a bunch of explanatory text) and letting in those who click the link to enter Smokingpipes.com. We'll leave the bouncer out front until we're confident the throng of zombies has passed and just normal good folks are trying to get to the site again.
So, we very much appreciate your patience and kind words as you've waited for us to return. Hopefully, we'll be able to drop the splash page in the next few days and return everything completely to normal. Brian and Ted will cover the phones until 10pm tonight to accommodate extra call volume. I, however, having been at this for almost all of the past twenty four hours, will go take a shower and get some sleep!