Fail2Ban Behind a Reverse Proxy: The Almost-Correct Way

Table of Contents

Fail2Ban is a wonderful tool for managing failed authentication or usage attempts for anything public facing. However, by default, it’s not without it’s drawbacks: Fail2Ban uses iptables to manage it’s bans, inserting a --reject-with icmp-port-unreachable rule for each banned host. The thing with this is that I use a fairly large amount of reverse-proxying on this network to handle things like TLS termination and just general upper-layer routing. Since it’s the proxy that’s accepting the client connections, the actual server host, even if its logging system understands what’s happening (say, with PROXY protocol) and logs the real client’s IP address, even if Fail2Ban puts that IP into the iptables rules, since that’s not the connecting IP, it means nothing. What I really need is some way for Fail2Ban to manage it’s ban list, effectively, remotely. Luckily, it’s not that hard to change it to do something like that, with a little fiddling.

Quick iptables Fundamentals

Just for a little background if you’re not aware, iptables is a utility for running packet filtering and NAT on Linux. It’s uh… how do I put this, it’s one of those tools that you will never remember how to use, and there will be a second screen available with either the man page, or some kind soul’s blog post explaining how to use it. But, when you need it, it’s indispensable.

Chains

We’re not getting into any of the more advanced iptables stuff, we’re just doing standard filtering. For that, you need to know that iptables is defined by executing a list of rules, called a chain. Every rule in the chain is checked from top to bottom, and when one matches, it’s applied. Each chain also has a name. The main one we care about right now is INPUT, which is checked on every packet a host receives. However, we can create other chains, and one action on a rule is to “jump” to another chain and start evaluating it.

Rules

Each rule basically has two main parts: the condition, and the action. The condition is further split into the source, and the destination. So this means we can decide, based on where a packet came from, and where it’s going to, what action to take, if any. An action is usually simple. For all we care about, a rule’s action is one of three things:

• Do nothing, and evaluate the next rule
• Reject or drop the packet, maybe with extra options for how
• Check the packet against another chain. If that chain didn’t do anything, then it comes back here and starts at the next rule.

Fail2Ban’s Actions

When Fail2Ban matches enough log lines to trigger a ban, it executes an action. Generally this is set globally, for all jails, though individual jails can change the action or parameters themselves. Each action is a script in action.d/ in the Fail2Ban configuration directory (/etc/fail2ban). These scripts define five lists of shell commands to execute:

• The start list, for when Fail2Ban starts up and initializes a jail
• The stop list, for when Fail2Ban is shutting down
• The check list, to see if the jail is active
• The ban list, to add an IP address to restrict
• The unban list, to remove an IP address after its ban time expired.

By default, Fail2Ban uses an action file called iptables-multiport, found on my system in action.d/iptables-multiport.conf. This has a pretty simple sequence of events:

1. When started, create an additional chain off the jail name. For example, the dovecot jail creates a chain named fail2ban-dovecot.
2. Add a rule to the INPUT chain that any packets matching the jail’s ports are to be passed through the chain it just created.
3. When banned, just add the IP address to the jail’s chain, by default specifying a REJECT action, rejecting with an ICMP Port Unreachable message.
4. When unbanned, delete the rule that matches that IP address.
5. When stopping, flush the chain (delete all its rules), and then delete the now empty chain. Finally, remove the rule from INPUT.

So naturally, when host 192.0.2.7 says “Hey here’s a connection from 203.0.11.45, the application knows that 203.0.11.45 is the client, and what it should log, but iptables isn’t seeing a connection from 203.0.11.45, it’s seeing a connection from 192.0.2.7 that’s passing it on. If you’ve ever done some proxying and see Fail2Ban complaining that a host is “already banned,” this is one cause. So the solution to this is to put the iptables rules on 192.0.2.7 instead, since that’s the one taking the actual connections. There are… a few ways to do this.

Multi-System Fail2Ban

There’s a number of actions that Fail2Ban can trigger, but most of them are localized to the local machine (plus maybe some reporting). To influence multiple hosts, you need to write your own actions. Some people have gone overkill, having Fail2Ban run the ban and do something like insert a row into a central SQL database, that other hosts check every minute or so to send ban or unban requests to their local Fail2Ban. We don’t need all that. All I need is some way to modify the iptables rules on a remote system using shell commands. Well, iptables is a shell command, meaning I need to find some way to send shell commands to a remote system. Any guesses? Yes, it’s SSH.

Fail2Ban + SSH

Really, it’s simple. My mail host has IMAP and POP proxied, meaning their bans need to be put on the proxy. Fail2Ban runs as root on this system, meaning I added root’s SSH key to the authorized_keys of the proxy host’s user with iptables access, so that one can SSH into the other. All I needed to do now was add the custom action file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

[INCLUDES]

before = iptables-common.conf

[Definition]

actionflush = ssh [email protected] '<iptables> -F mail-remote-f2b-<name>'

actionstart = ssh [email protected] '<iptables> -N mail-remote-f2b-<name>'
              ssh [email protected] '<iptables> -A mail-remote-f2b-<name> -j <returntype>'
              ssh [email protected] '<iptables> -I <chain> -p <protocol> -m multiport --dports <port> -j mail-remote-f2b-<name>'

actionstop = ssh [email protected] '<iptables> -D <chain> -p <protocol> -m multiport --dports <port> -j mail-remote-f2b-<name>'
             <actionflush>
             ssh [email protected] '<iptables> -X mail-remote-f2b-<name>'

actioncheck = ssh [email protected] "<iptables> -n -L <chain> | grep -q 'mail-remote-f2b-<name>[ \t]'"

actionban = ssh [email protected] '<iptables> -I mail-remote-f2b-<name> 1 -s <ip> -j <blocktype>'

actionunban = ssh [email protected] '<iptables> -D mail-remote-f2b-<name> -s <ip> -j <blocktype>'

[Init]

It’s actually pretty simple, I more-or-less copied iptables-multiport.conf and wrapped all the commands in a ssh [email protected] '...' so that it’ll start an SSH session, run the one provided command, dump it’s output to STDOUT, and then exit.

The one thing I didn’t really explain is the actionflush line, which is defines in iptables-common.conf. Because this also modifies the chains, I had to re-define it as well. The only place (that I know of) that it’s used is in the actionstop line, to clear a chain before it’s deleted.

After all that, you just need to tell a jail to use that action:

[dovecot]
enabled = true
port    = pop3,pop3s,imap,imaps,sieve
logpath = %(dovecot_log)s
backend = %(dovecot_backend)s
findtime = 21600
action  = proxy-iptables[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]

All I really added was the action line there. In this case, the action is proxy-iptables (which is what I called the file, proxy-iptables.conf), and everything after it in [ ] brackets are the parameters. The name is used to name the chain, which is taken from the name of this jail (dovecot), port is taken from the port list, which are symbolic port names from /etc/services, and protocol and chain are taken from the global config, and not overridden for this specific jail.

Note that most jails don’t define their own actions, and this is the global one:

iptables-multiport[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]

So all I had to do was just take this part from the top of the file, and drop it down.

This… took several tries, mostly just restarting Fail2Ban, checking the logs to see what error it gave this time, correct it, manually clear any rules on the proxy host, and try again. But at the end of the day, it’s working. And now, even with a reverse proxy in place, Fail2Ban is still effective.

Security Considerations

Because how my system is set up, I’m SSH’ing as root which is usually not recommended. Sure, it’s using SSH keys, but it’s using the keys of another host, meaning if you compromise root on one system then you get immediate root access over SSH to the other. To this extent, I might see about creating another user with no permissions except for iptables. Sure, that’s still risky, allowing iptables access like this is always risky, but that’s what needs to be done barring some much more complex setups.