How to Detect a Denial of Service (DoS) Attack

Next: Who are the players in this drama?

Previous: What is a Denial of Service Attack

Back to Table of Contents

Here is what your website may look like after a successful Denial of Service Attack:
Service Unavailable

At this point, your web server has given up on trying to service new requests. Using software, the hacker has simulated a very large number of people (or connections) to your website, your website is no longer able to handle all these connections, and you and your customers see an error message when you visit your website.

Note that you can also get this error message if you make a mistake when deploying your website, or if your webserver implements memory limits and your site uses too much memory (by displaying big data sets, or by doing big calculations, or ...), or if your web application crashes (you wrapped all your page events in try/catch blocks, right?). However, if you are certain that you have not recently changed your website in any way, and you are sure that your website programming is sound, then read on.

If you've setup your website to handle 10,000 simultaneous visitors, the hacker can bring your website down by simulating 100,000 simultaneous visitors.

If you've setup your website to handle 100,000 simultaneous visitors, the hacker can bring your website down by simulating 1,000,000 simultaneous visitors.

Regardless of how much hardware and performance tuning you have done, the hacker can still simulate more users than your website can handle.

Later in this article, I will show you how you can thwart these DoS attempts using various techniques.  If your website is hosted by a 3rd party ISP, you may not be able to use all of these techniques yourself, and you may need your ISP's assistance.  If your ISP refuses to help you (as mine did), I will show you how you can thwart these attacks by moving to Microsoft Windows Azure

You can detect the onset of a Denial of Service Attack in the following ways:

1.) Check your website periodically to see if it is running more sluggishly than usual.  Also, watch to see if you are sporadically receiving the "Service Unavailable" message shown above.  As the hacker starts ramping up his attack, you may notice these symptoms.


2.) Check your web stats.  If you don't currently have web stats software setup, consider using SmarterStats (my personal favorite) or the free Analog.   These packages provide many graphs and reports, and it is a good idea to familiarize yourself with them, and look at them often so that you know how the graphs look when your site is functioning normally. In this way, you can easily spot anomalies which indicate a DoS attack (not to mention a whole bunch of other problems, but that's another story).

Here is what a normal graph looks like (for my site, anyway). As you can see, I get a fairly consistent number of page views every weekday (approximately 30K).

Normal Site Usage

Now, here is a graph showing a DoS attack. Instead of seeing approximately 30K page views per day, I'm getting over 1 million page views per day.  Obviously, something is wrong.

Abnormal Site Usage

When I first saw this anomaly, I paid it little attention, thinking that my ISP might have been having some problem with my web log files. I made a big mistake here by not quickly addressing this issue.   Also, note that the hacker gradually ramped up his attack over time.  Later on, I will discuss how a hacker can set this up.

Here is a graph showing me which IP addresses used my site the most. This shows relatively normal conditions, although it looks like a few IP addresses may be using an inordinate amount of bandwidth. These are likely some kind of Search Engine Spiders, and I'll show you later how to find out more about these IP addresses.

Normal IP Address Report

Here is the same report when my site is under attack. Note that there are a couple of IP addresses here that are using an inordinate amount of bandwidth. As you find these IP addresses, you should keep a list, as I will be showing you how to contact the ISPs and website owners later on. Remember that these organizations are also victims of your attacker.

Abnormal IP Address Report

3.) If you have some kind of Remote Desktop Access (e.g. Remote Desktop Connection, VNC, PCAnywhere, etc.) you can log in to your web server, open a command prompt, and type "netstat -an" (without the quotes).   (If you find command line applications like netstat unwieldy, you might try using TCPView by legendary programmer Mark Russinovich.)

Here is what the netstat results should look like under normal circumstances:

Normal NetStat Results

The first column of the netstat results shows the protocol. For the purposes of our current discussion, we only care about the TCP protocol. However, if you are confronted with a more sophisticated attacker, you will want to learn more about other protocols, particularly UDP and ICMP.

The second column of the netstat results shows your computer's IP address. Note that your computer has several IP addresses. The IP address is followed by a colon and a port number. Different services use different port numbers. For example, your web server is likely using port 80, whereas Remote Desktop Connection uses port 3389.   For the purposes of this article, we only care about connections to port 80 (web server) or port 443 (web server with SSL).

The third column shows the IP address of a remote computer. For example, if a customer is looking at your website (on port 80), and the customer has IP address 24.200.167.248, and your webserver has IP address 10.114.242.92, then you will see the following line:

TCP     10.114.242.92:80    24.200.167.248:60734    ESTABLISHED

Note that the customer's machine is using port 60734. All you need to know for now is that well-defined services like your Web Server listen on well-defined ports like port 80. In contrast, clients who are connecting to your web server also need a port number to make this connection, but the port number is some random high number. This is why the "Foreign Address" column has such funky numbers after the colon, whereas the Local Address column has (mostly) well-known port numbers.

In the "State" column, you see a bunch of lines that end with "LISTENING", "ESTABLISHED", "TIME_WAIT", etc. These tell you about the current state of that connection (socket).

If a socket is "LISTENING", it a program is waiting for some remote computer to connect to it via the network.

If a socket is "ESTABLISHED", it means a client is connected to your machine (e.g. a customer is visiting your website).

If the socket says "TIME_WAIT", the socket may be setting up a connection, or it may be tearing down a connection. In any case, its waiting for something.

For this article, I'm mainly focusing on simple Denial of Service (DoS) attacks. When dealing with a more high-tech attacker, I urge you to investigate which services are "LISTENING" on your server (did the hacker set something up), do you see many, many TIME_WAIT's (could be a more sophisticated DoS attack), etc.

Now, here is what the netstat output looks like during a DoS attack.

abnormal NetStat

As you can see, 216.36.50.65 is making many, many connections to my webserver (thousands of connections). You need to be a little careful here. When someone looks at a page on your website, their web browser will open several connections to your webserver. So it is not uncommon to see a legitimate user with 8-20 connections to your web server. Also, many companies have a single public IP address for a large number of their employees use. So if many employees at that company are simultaneously visiting your website, you may see a whole bunch of legitimate connections from the same IP address. However, if you scroll through hundreds and hundreds of lines of your netstat results, and you are seeing the same IP address (in the third column) again and again, then this indicates a DoS attack. If you see such an IP address, write it down, as we'll later use it to track down and contact the owners of this other computer so that we can notify them that their machine has been compromised and is being used to attack your webserver.

 

4.) You can also check your log files for suspicious activity. Here is what a normal log file looks like:

date time s-ip cs-method cs-uri-stem cs-uri-query s-port c-ip cs-version cs(User-Agent)
2012-02-06 08:00:03 96.31.33.24 GET /Financial/Default.aspx action=browseByState&arg=TN 80 89.248.174.2 HTTP/1.0 Mozilla/4.0+(compatible;+MSIE+8.0;...)
2012-02-06 08:00:35 96.31.33.24 GET /financial/StyleSheet.css - 80 217.196.17.20 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+8.0;...)
2012-02-06 08:00:35 96.31.33.24 GET /financial/ - 80 217.196.17.20 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+8.0;+...)
2012-02-06 08:00:36 96.31.33.24 GET /Financial/WebResource.axd d=XlhVf... 80 217.196.17.20 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+8.0;+...)
2012-02-06 08:00:36 96.31.33.24 GET /Financial/WebResource.axd d=Z_gdL... 80 217.196.17.20 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+8.0;+...)
2012-02-06 08:01:06 96.31.33.24 GET /Bank/RoutingNumbers/State/StyleSheet.css - 80 217.196.17.20 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+8.0;+...)
2012-02-06 08:01:11 96.31.33.24 GET /Bank/RoutingNumbers/State/IL - 80 217.196.17.20 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+8.0;... )
2012-02-06 08:01:27 96.31.33.24 GET /Bank/Routing/Numbers/P - 80 66.249.67.153 HTTP/1.1 Mozilla/5.0+(compatible;+Googlebot/2.1;
++http://www.google.com/bot.html)
2012-02-06 08:01:36 96.31.33.24 GET /Financial/Default.aspx - 80 74.114.36.65 HTTP/1.1 Mozilla/4.0+...

IIS (and other webservers) allow you to configure the log files in many ways, so your log files may look different, but will contain the information I discuss below. The important fields to look at are c-ip (the ip address of the person or program that is visiting your site), cs-uri-stem (which tells you which page they are visiting, and cs(User-Agent), which tells you which program is being used to visit your website (e.g. IE, Firefox, Google Chrome, etc.) Note that in the table above, we see several different IP addresses for our clients (89.248.174.2, 217.196.17.20, 66.249.67.153, etc.), different webpages (/Financial/Default.aspx, /financial/StyleSheet.css, etc.) and different user agents (IE 8 and a Google "bot" )

Here is what the log file might look like during a Denial of Service attack:

date time s-ip cs-method cs-uri-stem cs-uri-query s-port c-ip cs-version cs(User-Agent)
2012-02-06 10:12:51 96.31.33.24 GET / - 80 69.163.239.247 HTTP/1.0 ApacheBench/2.3
2012-02-06 10:12:51 96.31.33.24 GET / - 80 69.163.239.247 HTTP/1.0 ApacheBench/2.3
2012-02-06 10:12:51 96.31.33.24 GET / - 80 69.163.239.247 HTTP/1.0 ApacheBench/2.3
2012-02-06 10:12:51 96.31.33.24 GET / - 80 69.163.239.247 HTTP/1.0 ApacheBench/2.3
2012-02-06 10:12:51 96.31.33.24 GET / - 80 69.163.239.247 HTTP/1.0 ApacheBench/2.3
2012-02-06 10:12:51 96.31.33.24 GET / - 80 69.163.239.247 HTTP/1.0 ApacheBench/2.3
2012-02-06 10:12:51 96.31.33.24 GET / - 80 69.163.239.247 HTTP/1.0 ApacheBench/2.3
2012-02-06 10:12:51 96.31.33.24 GET / - 80 69.163.239.247 HTTP/1.0 ApacheBench/2.3
2012-02-06 10:12:51 96.31.33.24 GET / - 80 69.163.239.247 HTTP/1.0 ApacheBench/2.3
2012-02-06 10:12:51 96.31.33.24 GET / - 80 69.163.239.247 HTTP/1.0 ApacheBench/2.3
2012-02-06 10:12:51 96.31.33.24 GET / - 80 69.163.239.247 HTTP/1.0 ApacheBench/2.3
2012-02-06 10:12:51 96.31.33.24 GET / - 80 69.163.239.247 HTTP/1.0 ApacheBench/2.3
2012-02-06 10:12:51 96.31.33.24 GET / - 80 69.163.239.247 HTTP/1.0 ApacheBench/2.3
2012-02-06 10:12:51 96.31.33.24 GET / - 80 69.163.239.247 HTTP/1.0 ApacheBench/2.3
2012-02-06 10:12:51 96.31.33.24 GET / - 80 69.163.239.247 HTTP/1.0 ApacheBench/2.3
2012-02-06 10:12:51 96.31.33.24 GET / - 80 69.163.239.247 HTTP/1.0 ApacheBench/2.3
2012-02-06 10:12:51 96.31.33.24 GET / - 80 69.163.239.247 HTTP/1.0 ApacheBench/2.3

Note that we are seeing the same IP address (69.163.239.247) again and again (for hundreds of lines), that this bot or user is accessing the same web page again and again (/ is the root page of my website), and that the program accessing our website is called ApacheBench. Each time a user visits one of your pages, you should expect to see about 1-10 lines in the log files for that user's visit, and you should see the user's web browser downloading an html page (on one line), and any art or links embedded in that page downloaded (each also on its own line). What you don't expect to see is the same item downloaded again and again and again by the same IP address hundreds of times. This is what a Denial of Service (DoS) looks like.

Also note, that the suspicious lines also show the use of ApacheBench. From apache.org, "ApacheBench is a tool for benchmarking your Apache Hypertext Transfer Protocol (HTTP) server". That is, if you were trying to find out how many customers your website could handle in a short period of time, you would run this tool from the command line (it's called "ab"), the tool would simulate many, many visitors coming to your site at the same time, and then you could watch for the "Service Unavailable" message shown at the top of this page, and you would know how many visitors could simultaneously access your website before it crashed.

Unfortunately, this helpful tool can also be used by a hacker to bring your website down. If the hacker runs this tool from several locations against your website (making it a Distributed Denial of Service Attack), he will be able to overwhelm your website with the phony traffic from ApacheBench.

At this point, you will want to keep a list of these IP addresses, and I will show you later how we can use them to stop and/or mitigate the attack, but first we need to understand who the players are in this fiasco. We may need to contact some of these people, so we need to be sure we understand who the victims are, and who are the culprits.

Next: Who are the players in this drama?

Previous: What is a Denial of Service Attack

Back to Table of Contents

Problems, Comments, Suggestions? Click here to contact Greg Thatcher

Please read my Disclaimer





Copyright (c) 2013 Thatcher Development Software, LLC. All rights reserved. No claim to original U.S. Gov't works.