Limiting brute-force attacks with IPTables

Update: a more comprehensive, tiered version of the procedure below is outlined in Escalating Consquences with ipTables. The material below is, however still applicable, and forms the basis for many more advanced implementations.

For the past few weeks, I have noticed that the scale of attempted intrusions, on ssh, pop3, and ftp ports has increased dramatically, reaching levels as high as 10000 attempts over the course of an hour.

The following chronicles a few observations about the attacks, and my approach towards a solution. It is worth noting that it would be preferable to use Amazon’s security groups for blocking as opposed to IPTables, however, the former does not support custom rules of any sort – it is limited to simply opening and closing ports.

Attack Description

The majority of these attacks have originated from the US, Germany, Spain, and China (in no particular order). Given the small scale of the sites running on this server, combined with a relatively ‘peaceful’ first few months, I did not expect quite the extent of attempts I have recently seen. Fortunately, all attempts so far have been futile.

Since this article is on IPTables, I will try to keep the deviations to a minimum, but a few points are worth a mention:

• The attempted logins on pop3 greatly resemble a dictionary attack
  • One IP tried common usernames (adm, apache, bin, calendar, cyrus, daemon, …, www) 9 times each followed by an alphabetical listing of names appear to have been tried (aaron, abdiel, abdullah, abel, abraham, abram, adam, adan, addison, aden, aditya, adolfo, adonis, adrian, …, zion), with 4 attempts for each, followed by a list of words and less common names, 1 time each.
  • Another tried the several usernames (Admin01, Admin02, Administrador, Administrateur, Administrator, Admins, backup, guest, mail, postmaster) 1020 times each
• SSH login attempts were typically at the rate of 2/second, with 500 attempts made per IP address per day.
• FTP login attempts were on a much smaller scale – only about 85 attempts on a single site’s FTP with some common username variations tried.

Needless to say, uncommon usernames and strong passwords would go a long way towards preventing a successful intrusion of this sort; as well as the use of key/certificate based authentication on SSH.

Solutions Considered

The preferable approach to defending against such an attack appears to be blocking the attack by use of a firewall, although, the implementation varies considerably.

I briefly looked into the following options, but decided against them:

• pam_shield:
  • Limitation: only works with PAM
• fail2ban, daemon_shield, BlockHosts
  • Limitation: All of the above scan log files and as such do not appear to be instant acting

The options above all add a rule to IPTables based on specified triggers. Of all of them, PAM_shield does look the most appealing (its trigger is a login failure), it is even possible to customize the IPTables rules that PAM_shield executes. However, on my setup, SSH doesn’t authenticate through PAM, and I don’t currently wish to change that.

Adding rules to IPTables and using the recent module was the method I decided on. It however, does have the limitation of not distinguishing between successful and unsuccessful login attempts.

Solution Implemented

Objective: if an IP address attempts to initiate a connection to ssh, pop3, imap, or ftp, more than 5 times in 60 seconds, or more than 15 times in 10 minutes, ban them from accessing all of the above ports for an hour (and extend the time if they continue to try – so that an hour without any attempts is necessary).

Note: most edits/commands listed below must be run as root (sudo), and some parts are specific to Amazon’s Linux AMI.

Background Information

Firstly, a couple of quick points about IPTables on Amazon’s Linux AMI:

• lsmod displays the recent module as xt_recent (instead of ipt_recent)
• The recent module creates files under the folder /proc/net/xt_recent (instead of /proc/net/ipt_recent)
• IPTables runs under the syslog facility user (instead of kern)

IPTables can be temporarily (until restart) modified using the command iptables, or modifications can be made permanent by saving the changes to /etc/sysconfig/iptables.

Keep in mind that incorrect IPTables rules may lock you out of your server – a good recommendation that I came across was to setup a cron task to flush (iptables –F) the IPTables rules every few minutes while testing.

The Rules

I added my rules directly to /etc/sysconfig/iptables (these are modifications, not a complete rule set):

:ATTACKED - [0:0]
:ATTK_CHECK - [0:0]

-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p tcp -m multiport --dports 21,22,110,143 -m recent --update --seconds 3600 --name BANNED --rsource -j DROP
-A INPUT -p tcp -m multiport --dports 21,22,110,143 -m state --state NEW -j ATTK_CHECK

#OTHER PRE-EXISTING RULES
#...

-A ATTACKED -m limit --limit 5/min -j LOG --log-prefix "IPTABLES (Rule ATTACKED): " --log-level 7
-A ATTACKED -m recent --set --name BANNED --rsource -j DROP
-A ATTK_CHECK -m recent --set --name ATTK –-rsource
-A ATTK_CHECK -m recent --update --seconds 600 --hitcount 16 --name ATTK --rsource -j ATTACKED
-A ATTK_CHECK -m recent --update --seconds 60 --hitcount 6 --name ATTK --rsource -j ATTACKED
-A ATTK_CHECK -j ACCEPT

After completing your edits, restart iptables:

service iptables restart

If you add the above rules using the iptables command:

• you can view the current rules using: iptables –S (or a more ‘explained’ version with iptables –L)
• you can save the rules to /etc/sysconfig/iptables using: iptables-save > /etc/sysconfig/iptables

You will need to specify the correct positions in order to have the first rules added to the top of the list.

Explanations

(Rules precede the explanation in each case)

:ATTACKED - [0:0]
:ATTK_CHECK - [0:0]

Create two new chains (ATTACKED and ATTK_CHECK) that will be referenced later.

-A INPUT -i lo -j ACCEPT

Accept all input on the loopback interface, so that local services (which might just make more requests than ‘permitted’ don’t get locked out); adding source (sport) and destination (dport) values of 127.0.0.1 might be desired, but seems redundant. Since rules are carried out in order, this ‘accept all’ rule must precede other rules that could drop the packets.

-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

Accept packets from connections that have already been established – this will save a bit of processing power, but perhaps, more importantly, will allow not close an already open connection if another connection (from the same source) exceeds the limit. One use is keeping an ssh session open for testing, and exceeding the limit by opening new ssh sessions. Without this rule, the original session will be locked out as soon at the rule is triggered, while with this, the successfully established sessions can continue, and only the one that exceeds the limit (and subsequent attempts) will be blocked.

-A INPUT -p tcp -m multiport --dports 21,22,110,143 -m recent --update --seconds 3600 --name BANNED --rsource -j DROP

Drop packets from IPs on the banned list (those that are recorded by the ATTACKED chain), to the selected ports, until 1 hour without any connections (using –update) have elapsed. We want this rule before other checks and accepts, so that banned IPs will be dealt with here.

-A INPUT -p tcp -m multiport --dports 21,22,110,143 -m state --state NEW -j ATTK_CHECK

For all new connections on the selected ports, jump to the ATTK_CHECK chain – we are going to check the new (i.e. syn set) packets against the recent connections list. If we have made it this far (not loopback and not banned), only then check the packet.

Chain: ATTACKED

All packets that enter this chain originate from IPs deemed ‘attackers’ (5+ connections/min; 15+ connections/10 min).

-A ATTACKED -m limit --limit 5/min -j LOG --log-prefix "IPTABLES (Rule ATTACKED): " --log-level 7

Log packets that have reached here (however, to prevent excess logging don’t exceed 5 log entries a minute), send to syslog with priority debug (log-level 7)

-A ATTACKED -m recent --set --name BANNED --rsource -j DROP

Drop all packets that have made it here and add IP to the banned list (so that further connection attempts are also dropped)

Chain: ATTK_CHECK

Check the packets that get here (new connections on specified ports) for repeated previous connections from the same IP.

-A ATTK_CHECK -m recent --set --name ATTK --rsource

Keep track of IPs (in file /proc/net/xt_recent/ATTK) to check for repeats

-A ATTK_CHECK -m recent --update --seconds 600 --hitcount 16 --name ATTK --rsource -j ATTACKED

If an IP has exceeded 16 checks in 10 minutes, treat it as an attack (goto the ATTACKED chain). This rule must precede the one below.

-A ATTK_CHECK -m recent --update --seconds 60 --hitcount 6 --name ATTK --rsource -j ATTACKED

If an IP has exceeded 5 checks in 1 minute, treat it as an attack

-A ATTK_CHECK -j ACCEPT

All packets we are not treating as an attack, we will accept

Setting up Logging

To setup the log for IPTables, add the following to /etc/syslog.conf:

user.debug          /var/log/iptables

and restart the syslog service:

service syslog restart

Note: Some other scripts/processes might also log as user.debug (for instance, Suhosin), so you could try a different log-level to see what works best for you.

Final Explanations and Results

A few final points:

• The use of recent with –update as opposed to –rcheck is so that continued attempts to access the server, will keep resetting the time.
• The use of DROP vs REJECT is seemingly frowned upon, but it doesn’t let the attacking machine know that the connection has been dropped – which will often slow down the attack somewhat, tie up the attacker’s resources a bit, and requires a slightly more sophisticated attack script to effectively overcome. If many people other than myself were legitimately logging onto the server, it might be better to use REJECT.

Following the implementation of these rules, I have seen a drop in the number of SSH and POP3 attempts, to about 10 per day. It appears that each IP tries once a day and once they are blocked, stop trying for that day.

Log Samples

For reference, sample log entries matching the ban and intrusion attempts are provided below:

/var/log/secure:

Jan 2 21:48:21 servername sshd[20071]: reverse mapping checking getaddrinfo for sub.example.com [aa.bb.cc.ddd] failed - POSSIBLE BREAK-IN ATTEMPT!
Jan 2 21:48:21 servername sshd[20072]: Received disconnect from aa.bb.cc.ddd: 11: Bye Bye

Attempts from the same IP were repeated at: 21:48:22 (twice), 21:48:23, 21:48:24 (total of 5 attempts logged)

/var/log/iptables:

Jan 2 21:48:24 servername klogd: [683891.966643] IPTABLES (Rule ATTACKED): IN=eth0 OUT= MAC=xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx SRC= aa.bb.cc.ddd DST=ee.fff.ggg.hhh LEN=60 TOS=0x00 PREC=0x00 TTL=46 DF PROTO=TCP SPT=35485 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0

This entry corresponds to a trigger of the ATTACKED chain, from the same IP above, on port 22 (SSH).