AbuseIPDB Checking With Postfix

Table of Contents

So if you’ve not heard, there’s this website called AbuseIPDB, which, no affiliation, is a website where webmasters can submit reports of abusive IP addresses, and then query those reports, either manually, or using their REST API. And this is how I did exactly that, to help cut down some of the spam on my email server. Let’s get started.

So here’s the deal. Spam is just something that, if you manage a mail server, are going to have to accept that it exists. There’s no way around that. But what you don’t have to accept is the spam itself, there’s plenty of ways to block it if you wish. Now, I already have the basics in: SPF, DKIM, DMARC, a few checks to DNSBLs / RBLs, but there’s still one other line of defense that I can add: Just a general abuse check.

Note: if you want to do something similar yourself, I’ve put some more cleaned-up code up in a repository. More on that at the end.

The Plan

So the goal for this idea was as such: Every time an email is received by Postfix, I want it to query the AbuseIPDB API for the client’s IP address, and use that to judge if it should reject the message or not. Given the API call limits per day of AbuseIPDB, and the amount of mail I receive, this isn’t an issue whatsoever. And hand-rolling my own API client using Python’s Requests lib, that was an extremely simple process. So, I have some way of querying an IP address, and receiving it’s abuse confidence, which is a value from 0% to 100%, where 100% is “abusive” and 0% is “not abusive”. And I know that Postfix has a system in place for sending incoming emails through a series of checks. So, how do I marry the two?

smtpd Restrictions

A brief bit of background: How Postfix handles this. Postfix has a series of configuration directives, of the form smtpd_X_rescrictions, where X is something like client, recipient, helo, etc. At every major stage of the SMTP transaction, Postfix can run a sequence of checks to say if a particular client or message is allowed to progress, or be sent a denial message. We can use this to check, say, the connecting IP address against a lookup table, your HELO name’s validity with an MX or A record, if the IP is on a certain RBL, there’s a number of them. And each can give a result that’s, according to Postfix, OK, REJECT, or DUNNO. OK results are a “this is good, don’t bother processing any further”, say, have all servers on the local network have full relay access. REJECT results cause… a rejection, and DUNNO is kinda the middle ground — a DUNNO check behaves as if it didn’t exist: it doesn’t reject the message, but doesn’t give it the green light either, it just goes to the next step in the line.¹

Well, one of the items you can slot into these restrictions is called a policy service, which will send a summary of the transaction as it stands to a program of your choosing, and it’ll expect back a result to act on. And this is what we can use. When Postfix wants to use a policy service, it opens a socket to the program, and sends it a series of attribute=value lines, followed by a single blank line, to signify the end of it’s side of the conversation, and it’ll wait for a reply of the form action=X, where X is a valid filter action, followed by a blank line. One of the items that Postfix will send is client_address. And that’s what I need to check!

The Implementation

The bulk of the processing is a Python script:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157

#!/usr/bin/python3

import sys
import requests
import syslog

MIN_CONFIDENCE = 75
API_KEY = "XXXX"
HEADER_NAME = "AbuseIPDB-Confidence"
WHITELIST_VALUE = "-1"
WHITELIST_IMMEDIATE_OK = False
REJECT_ACTION = "521"
ADD_HEADER = True

# Called whenever the filter passes a message.
# Handles checking for header addition and the like.
# All calls should be of the form 'return msg_ok("header-value")'
def msg_ok(val, logmsg):
    syslog.syslog(logmsg)
    if ADD_HEADER:
        return f"action=prepend X-{HEADER_NAME}: {val}\n"
    return "action=dunno\n"

# Called when the filter sees a message as whitelisted.
# Normally, just an alias for msg_ok(), unless WHITELIST_IMMEDIATE_OK is True.
def msg_whitelist(val, logmsg):
    if not WHITELIST_IMMEDIATE_OK:
        return msg_ok(val, logmsg)
    syslog.syslog(logmsg)
    if ADD_HEADER:
        return f"action=prepend X-{HEADER_NAME}: {val}\n"
    return "action=ok\n"

# Called whenever the filter rejects a message.
# No log message specified, since the rejection notice will be logged
def msg_fail(val):
    return f"action={REJECT_ACTION} {val}\n"

# Called whenever a message can't be processed.
# Returns a DUNNO, and logs a warning.
def msg_error(logmsg):
    syslog.syslog(syslog.LOG_WARNING, logmsg)
    return "action=dunno\n"

# Called when the filter doesn't want to take any action, and has nothing
# Really to log about it either.
def msg_skip():
    return "action=dunno\n"

# Main check function
def check_ip(addr):
    urlenc_addr = addr.replace(
        ":", "%3A"
    )  # Replace all ':' in IPv6 addrs with '%3A' for url encoding

    syslog.syslog(f"Checking AbuseIPDB statistics for {addr}...")
    resp = requests.get(
        f"https://api.abuseipdb.com/api/v2/check?ipAddress={urlenc_addr}&maxAgeInDays=90",
        headers={"Key": API_KEY, "Accept": "application/json"},
    )  # Get results from AbIPDB API

    # Only process if we got a 200 OK response, else, warn, and abort.
    if resp.status_code != 200:
        return msg_error(
            f"Cannot get AbuseIPDB data for {addr}, responded with code {resp.status_code}. Checking skipped."
        )

    # All information held in the "data" key
    data = resp.json()["data"]

    # If AbuseIPDB has whitelisted the IP, then immediately return, using the special value
    # "-1" in the header to denote a whitelist.
    if data["isWhitelisted"]:
        return msg_whitelist(
            WHITELIST_VALUE,
            f"AbuseIPDB statistics for {addr}: Address is whitelisted, bypassing check.",
        )

    # Remember, the higher the percentage, the more likely it's spam.
    if data["abuseConfidenceScore"] >= MIN_CONFIDENCE:
        # Send rejection, including link to view results.
        return msg_fail(
            f"Your IP has been blocked, AbuseIPDB has {data['abuseConfidenceScore']}% confidence. See https://www.abuseipdb.com/check/{urlenc_addr} for details."
        )

    # IP address is good, so don't take action.
    # The PREPEND action here is used to add a header with the confidence score,
    # and PREPEND by itself doesn't specify a specific message delivery action,
    # so in effect there's an implicit DUNNO attached, meaning that other filters
    # will still evaluate the message and can cause a reject.
    # using OK here would instead cause all other filters to be skipped.
    return msg_ok(
        data["abuseConfidenceScore"],
        f"AbuseIPDB statistics for {addr}: confidence of abuse is {data['abuseConfidenceScore']}%.",
    )


# Logging init
syslog.openlog("abuseipdb-check", 0, syslog.LOG_MAIL)

# Setup for fallback: no client_address presented
ip = ""

# Postfix's dialog begins with a lot of key=value pairs,
# One per line, defining everything Postfix knows.
# A final blank line (meaning an ending sequence of \n\n)
# signals end-of-data.
#
# Since we're just filtering, all our response is, as seen above,
# is a single action key, then a blank line.

# Loop through each line one by one
while True:
    line = input()
    line = line.strip()
    if line == "":
        # Empty line, finish reading, and process.
        break
    try:
        # We do this weirdness to allow for attributes with an '=' in the value,
        # like VERP or BATV in your 'subject' line.
        attrib_parts = line.split("=")
        key = attrib_parts[0]
        value = '='.join(attrib_parts[1:])
    except:
        # Malformed? Ignore.
        # Some attributes like the sender address might have an '=' in their value,
        # but the ONLY one we care about will never, so it's safe to just take the
        # error and discard the line.
        syslog.syslog(
            syslog.LOG_WARNING,
            f"Encountered malformed line: '{line}' processing policy exchange. Ignoring.",
        )
        continue

    # Store address into 'ip' var.
    # Yes, this is the ONLY key we care about,
    # the rest get ignored.
    if key == "client_address":
        ip = value
        continue

# Failsafe: exit if address was never defined and print warning.
if ip == "":
    print(
        msg_error(
            "Never received 'client_address' attribute in policy filter exchange. Cannot check message."
        ),
        file=sys.stdout,
    )
    sys.exit(0)

# Run check. Results are immediately returned.
# Note: As much as you'd expect STDOUT to be the default
# print target, without explicitly specifying it, this doesn't work.
# At all. So for now, it stays.
print(check_ip(ip), file=sys.stdout)

To me it seemed pretty simple, but… okay that’s pretty long. But basically, here’s what happens:

Each invocation processes one connection, and therefore, one message. The only thing it cares about is a client_address line, so it only looks for that and no others. After seeing this, it’ll send that to the check function which will URL-encode the address and query AbueIPDB’s API with it (line 51). We ignore reports over 90 days old for this.

Now, if an HTTP response other than 200 was returned, we log an error, but pass the message through untouched. Otherwise, we run two checks:

1. If the isWhitelisted field from the API response is true, then AbuseIPDB has whitelisted this IP for some reason. We don’t need to continue checking confidence, and can immediately return. If the WHITELIST_IMMEDIATE_OK variable was set at the top, then also skip any other restrictions afterwards too by returning an OK response.
2. Otherwise, check the abuse confidence against the MIN_CONFIDENCE value. If it’s higher, then reject the message. The reject action is 521 and not REJECT, meaning that Postfix will return 521 as the code and not its default REJECT code, which is 554 unless specified in the config.²

Assuming neither checks caught the message, we’ll send a DUNNO back, so Postfix will continue evaluation. However, if ADD_HEADER is set, then instead of DUNNO, we use PREPEND, which will add a header to the email message, in this case, with the confidence score. A PREPEND action has an implicit DUNNO in there as well.

As a failsafe, if no client_address line is sent, then the entire thing is skipped completely.

Postfix Integration

Really there’s two parts: adding it to Postfix’s config, and then giving Postfix a way to manage the process. The first is simple, and is the check_policy_service restriction, which I added like this to smtpd_client_restrictions:

smtpd_client_restrictions = permit_mynetworks,
        permit_sasl_authenticated,
        reject_rbl_client zen.spamhaus.org,
        reject_rbl_client bl.spamcop.net,
        reject_rbl_client cbl.abuseat.org,
        check_policy_service { unix:private/abuse_chk, default_action=DUNNO }

This means that, if nothing else has issued an OK or REJECT-like action, then the check_policy_service will take effect, opening a UNIX socket at private/abuse_chk, and sending the policy server data there. If this doesn’t return correctly for some reason, then, by default, use the DUNNO action. Now, private/abuse_chk is a named socket that’s relative to Postfix’s chroot, which lives at /var/spool/postfix. So somehow, I need to get the socket file /var/spool/postfix/private/abuse_chk to exist, be readable and writable by Postfix, and have my program connected to the other end of it. Luckily, Postfix itself can do that for us, with the master table.

Postfix Master Config

Ever peeked at master.cf? This is a table of every process that Postfix starts, and some specifics about them. We just need to add two lines:

abuse_chk unix  -       n       n       -       0       spawn
        user=nobody argv=/etc/postfix/abuseipdb-check

This instructs Postfix to spawn a service, named abuse_chk, as a unix service type, which, since it’s privacy is then unspecified (-), it’s assumed to be yes, meaning it places a UNIX socket for communications at /var/spool/postfix/private/abuse_chk. This also means that only the local Postfix process can access it. After this, the first n means that it’s not unprivileged. Unprivileged processes run as the Postfix user, privileged run as root. The second n means that the process is not chrooted to the mail queue. The second - means not to periodically wake up the service, since… we don’t need it. That 0 means there are no limits on the number of running copies of this process. And finally, spawn is the actual command to execute, and below it, indented, are its arguments. In this case, we’re spawning the actual process as nobody (spawn is who’s running as root), and the process to spawn is /etc/postfix/abuseipdb-check, which is that script above.

Since this means that Postfix will automatically allocate the named socket as we require, all we need to do it instruct Postfix to use it, as shown above. The socket’s input is fed to the processes STDIN, and the process’s STDOUT is put back to the socket. The only oddity here is that the Python print() doesn’t work 100% as intended, you need to manually specify the output is sys.stdout, but besides that, it’s nothing special.

Running This Yourself

Disclaimer: Be very careful, messing with Postfix delivery rules and restrictions can cause Bad Things™ to happen. However if you want to give it a try, I’ve cleaned up the script a little and stuck it on my local code hosting, you can grab the code from here (or just stick to the releases). The stuff in there is no longer just one script, since I moved all the configuration out of global variables into an INI file, but that README should tell you everything you need to know. If you want to have some more in-depth information on what’s happening, then you can check the wiki tab which is where I’m documenting the extras that are at play here, but aren’t required for knowing how to get it up and running.

Because I want to keep that instance closed (for now), instead of creating an issue the usual way, you can genuinely just email it and it should create one. Or, alternatively, sign-up with a Google or GitHub account is turned on, but direct registration can only happen if you already have an email address on my domain, which, you don’t.