This article has narration:
(Press the 'p' key to toggle playback)
Table of Contents
Fail2Ban is a wonderful tool for managing failed authentication or usage attempts for anything public facing.
However, by default, it’s not without it’s drawbacks: Fail2Ban uses
iptables to manage it’s bans, inserting a
--reject-with icmp-port-unreachable rule for each banned host.
The thing with this is that I use a fairly large amount of reverse-proxying on this network to handle things like TLS termination and just general upper-layer routing.
Since it’s the proxy that’s accepting the client connections, the actual server host, even if its logging system understands what’s happening (say, with PROXY protocol) and logs the real client’s IP address, even if Fail2Ban puts that IP into the
iptables rules, since that’s not the connecting IP, it means nothing.
What I really need is some way for Fail2Ban to manage it’s ban list, effectively, remotely.
Luckily, it’s not that hard to change it to do something like that, with a little fiddling.
Just for a little background if you’re not aware,
iptables is a utility for running packet filtering and NAT on Linux.
It’s uh… how do I put this, it’s one of those tools that you will never remember how to use, and there will be a second screen available with either the man page, or some kind soul’s blog post explaining how to use it.
But, when you need it, it’s indispensable.
We’re not getting into any of the more advanced
iptables stuff, we’re just doing standard filtering.
For that, you need to know that
iptables is defined by executing a list of rules, called a chain.
Every rule in the chain is checked from top to bottom, and when one matches, it’s applied.
Each chain also has a name.
The main one we care about right now is
INPUT, which is checked on every packet a host receives.
However, we can create other chains, and one action on a rule is to “jump” to another chain and start evaluating it.
Each rule basically has two main parts: the condition, and the action. The condition is further split into the source, and the destination. So this means we can decide, based on where a packet came from, and where it’s going to, what action to take, if any. An action is usually simple. For all we care about, a rule’s action is one of three things:
- Do nothing, and evaluate the next rule
- Reject or drop the packet, maybe with extra options for how
- Check the packet against another chain. If that chain didn’t do anything, then it comes back here and starts at the next rule.
When Fail2Ban matches enough log lines to trigger a ban, it executes an action.
Generally this is set globally, for all jails, though individual jails can change the action or parameters themselves.
Each action is a script in
action.d/ in the Fail2Ban configuration directory (
These scripts define five lists of shell commands to execute:
- The start list, for when Fail2Ban starts up and initializes a jail
- The stop list, for when Fail2Ban is shutting down
- The check list, to see if the jail is active
- The ban list, to add an IP address to restrict
- The unban list, to remove an IP address after its ban time expired.
By default, Fail2Ban uses an action file called
iptables-multiport, found on my system in
This has a pretty simple sequence of events:
- When started, create an additional chain off the jail name. For example, the
dovecotjail creates a chain named
- Add a rule to the
INPUTchain that any packets matching the jail’s ports are to be passed through the chain it just created.
- When banned, just add the IP address to the jail’s chain, by default specifying a
REJECTaction, rejecting with an ICMP Port Unreachable message.
- When unbanned, delete the rule that matches that IP address.
- When stopping, flush the chain (delete all its rules), and then delete the now empty chain. Finally, remove the rule from
So naturally, when host
192.0.2.7 says “Hey here’s a connection from
22.214.171.124, the application knows that
126.96.36.199 is the client, and what it should log, but
iptables isn’t seeing a connection from
188.8.131.52, it’s seeing a connection from
192.0.2.7 that’s passing it on.
If you’ve ever done some proxying and see Fail2Ban complaining that a host is “already banned,” this is one cause.
So the solution to this is to put the
iptables rules on
192.0.2.7 instead, since that’s the one taking the actual connections.
There are… a few ways to do this.
There’s a number of actions that Fail2Ban can trigger, but most of them are localized to the local machine (plus maybe some reporting).
To influence multiple hosts, you need to write your own actions.
Some people have gone overkill, having Fail2Ban run the ban and do something like insert a row into a central SQL database, that other hosts check every minute or so to send
unban requests to their local Fail2Ban.
We don’t need all that.
All I need is some way to modify the
iptables rules on a remote system using shell commands.
iptables is a shell command, meaning I need to find some way to send shell commands to a remote system.
Yes, it’s SSH.
Fail2Ban + SSH
Really, it’s simple.
My mail host has IMAP and POP proxied, meaning their bans need to be put on the proxy.
Fail2Ban runs as
root on this system, meaning I added
root’s SSH key to the
authorized_keys of the proxy host’s user with
iptables access, so that one can SSH into the other.
All I needed to do now was add the custom action file:
It’s actually pretty simple, I more-or-less copied
iptables-multiport.conf and wrapped all the commands in a
ssh [email protected] '...' so that it’ll start an SSH session, run the one provided command, dump it’s output to STDOUT, and then exit.
The one thing I didn’t really explain is the
actionflush line, which is defines in
Because this also modifies the chains, I had to re-define it as well.
The only place (that I know of) that it’s used is in the
actionstop line, to clear a chain before it’s deleted.
After all that, you just need to tell a jail to use that action:
[dovecot] enabled = true port = pop3,pop3s,imap,imaps,sieve logpath = %(dovecot_log)s backend = %(dovecot_backend)s findtime = 21600 action = proxy-iptables[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
All I really added was the
action line there.
In this case, the action is
proxy-iptables (which is what I called the file,
proxy-iptables.conf), and everything after it in
[ ] brackets are the parameters.
name is used to name the chain, which is taken from the name of this jail (
port is taken from the port list, which are symbolic port names from
chain are taken from the global config, and not overridden for this specific jail.
Note that most jails don’t define their own actions, and this is the global one:
iptables-multiport[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
So all I had to do was just take this part from the top of the file, and drop it down.
This… took several tries, mostly just restarting Fail2Ban, checking the logs to see what error it gave this time, correct it, manually clear any rules on the proxy host, and try again. But at the end of the day, it’s working. And now, even with a reverse proxy in place, Fail2Ban is still effective.
Because how my system is set up, I’m SSH’ing as root which is usually not recommended.
Sure, it’s using SSH keys, but it’s using the keys of another host, meaning if you compromise
root on one system then you get immediate
root access over SSH to the other.
To this extent, I might see about creating another user with no permissions except for
Sure, that’s still risky, allowing
iptables access like this is always risky, but that’s what needs to be done barring some much more complex setups.