### Tek's Domain

#<NTA:NnT:SSrgS:H6.6-198:W200-90.72:CBWg>

# AbuseIPDB Checking With Postfix

Updated Dec 31, 2021

So if you’ve not heard, there’s this website called AbuseIPDB, which, no affiliation, is a website where webmasters can submit reports of abusive IP addresses, and then query those reports, either manually, or using their REST API. And this is how I did exactly that, to help cut down some of the spam on my email server. Let’s get started.

So here’s the deal. Spam is just something that, if you manage a mail server, are going to have to accept that it exists. There’s no way around that. But what you don’t have to accept is the spam itself, there’s plenty of ways to block it if you wish. Now, I already have the basics in: SPF, DKIM, DMARC, a few checks to DNSBLs / RBLs, but there’s still one other line of defense that I can add: Just a general abuse check.

Note: if you want to do something similar yourself, I’ve put some more cleaned-up code up in a repository. More on that at the end.

## The Plan

So the goal for this idea was as such: Every time an email is received by Postfix, I want it to query the AbuseIPDB API for the client’s IP address, and use that to judge if it should reject the message or not. Given the API call limits per day of AbuseIPDB, and the amount of mail I receive, this isn’t an issue whatsoever. And hand-rolling my own API client using Python’s Requests lib, that was an extremely simple process. So, I have some way of querying an IP address, and receiving it’s abuse confidence, which is a value from 0% to 100%, where 100% is “abusive” and 0% is “not abusive”. And I know that Postfix has a system in place for sending incoming emails through a series of checks. So, how do I marry the two?

## smtpd Restrictions

A brief bit of background: How Postfix handles this. Postfix has a series of configuration directives, of the form smtpd_X_rescrictions, where X is something like client, recipient, helo, etc. At every major stage of the SMTP transaction, Postfix can run a sequence of checks to say if a particular client or message is allowed to progress, or be sent a denial message. We can use this to check, say, the connecting IP address against a lookup table, your HELO name’s validity with an MX or A record, if the IP is on a certain RBL, there’s a number of them. And each can give a result that’s, according to Postfix, OK, REJECT, or DUNNO. OK results are a “this is good, don’t bother processing any further”, say, have all servers on the local network have full relay access. REJECT results cause… a rejection, and DUNNO is kinda the middle ground — a DUNNO check behaves as if it didn’t exist: it doesn’t reject the message, but doesn’t give it the green light either, it just goes to the next step in the line.1

Well, one of the items you can slot into these restrictions is called a policy service, which will send a summary of the transaction as it stands to a program of your choosing, and it’ll expect back a result to act on. And this is what we can use. When Postfix wants to use a policy service, it opens a socket to the program, and sends it a series of attribute=value lines, followed by a single blank line, to signify the end of it’s side of the conversation, and it’ll wait for a reply of the form action=X, where X is a valid filter action, followed by a blank line. One of the items that Postfix will send is client_address. And that’s what I need to check!

## The Implementation

The bulk of the processing is a Python script:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157  #!/usr/bin/python3 import sys import requests import syslog MIN_CONFIDENCE = 75 API_KEY = "XXXX" HEADER_NAME = "AbuseIPDB-Confidence" WHITELIST_VALUE = "-1" WHITELIST_IMMEDIATE_OK = False REJECT_ACTION = "521" ADD_HEADER = True # Called whenever the filter passes a message. # Handles checking for header addition and the like. # All calls should be of the form 'return msg_ok("header-value")' def msg_ok(val, logmsg): syslog.syslog(logmsg) if ADD_HEADER: return f"action=prepend X-{HEADER_NAME}: {val}\n" return "action=dunno\n" # Called when the filter sees a message as whitelisted. # Normally, just an alias for msg_ok(), unless WHITELIST_IMMEDIATE_OK is True. def msg_whitelist(val, logmsg): if not WHITELIST_IMMEDIATE_OK: return msg_ok(val, logmsg) syslog.syslog(logmsg) if ADD_HEADER: return f"action=prepend X-{HEADER_NAME}: {val}\n" return "action=ok\n" # Called whenever the filter rejects a message. # No log message specified, since the rejection notice will be logged def msg_fail(val): return f"action={REJECT_ACTION} {val}\n" # Called whenever a message can't be processed. # Returns a DUNNO, and logs a warning. def msg_error(logmsg): syslog.syslog(syslog.LOG_WARNING, logmsg) return "action=dunno\n" # Called when the filter doesn't want to take any action, and has nothing # Really to log about it either. def msg_skip(): return "action=dunno\n" # Main check function def check_ip(addr): urlenc_addr = addr.replace( ":", "%3A" ) # Replace all ':' in IPv6 addrs with '%3A' for url encoding syslog.syslog(f"Checking AbuseIPDB statistics for {addr}...") resp = requests.get( f"https://api.abuseipdb.com/api/v2/check?ipAddress={urlenc_addr}&maxAgeInDays=90", headers={"Key": API_KEY, "Accept": "application/json"}, ) # Get results from AbIPDB API # Only process if we got a 200 OK response, else, warn, and abort. if resp.status_code != 200: return msg_error( f"Cannot get AbuseIPDB data for {addr}, responded with code {resp.status_code}. Checking skipped." ) # All information held in the "data" key data = resp.json()["data"] # If AbuseIPDB has whitelisted the IP, then immediately return, using the special value # "-1" in the header to denote a whitelist. if data["isWhitelisted"]: return msg_whitelist( WHITELIST_VALUE, f"AbuseIPDB statistics for {addr}: Address is whitelisted, bypassing check.", ) # Remember, the higher the percentage, the more likely it's spam. if data["abuseConfidenceScore"] >= MIN_CONFIDENCE: # Send rejection, including link to view results. return msg_fail( f"Your IP has been blocked, AbuseIPDB has {data['abuseConfidenceScore']}% confidence. See https://www.abuseipdb.com/check/{urlenc_addr} for details." ) # IP address is good, so don't take action. # The PREPEND action here is used to add a header with the confidence score, # and PREPEND by itself doesn't specify a specific message delivery action, # so in effect there's an implicit DUNNO attached, meaning that other filters # will still evaluate the message and can cause a reject. # using OK here would instead cause all other filters to be skipped. return msg_ok( data["abuseConfidenceScore"], f"AbuseIPDB statistics for {addr}: confidence of abuse is {data['abuseConfidenceScore']}%.", ) # Logging init syslog.openlog("abuseipdb-check", 0, syslog.LOG_MAIL) # Setup for fallback: no client_address presented ip = "" # Postfix's dialog begins with a lot of key=value pairs, # One per line, defining everything Postfix knows. # A final blank line (meaning an ending sequence of \n\n) # signals end-of-data. # # Since we're just filtering, all our response is, as seen above, # is a single action key, then a blank line. # Loop through each line one by one while True: line = input() line = line.strip() if line == "": # Empty line, finish reading, and process. break try: # We do this weirdness to allow for attributes with an '=' in the value, # like VERP or BATV in your 'subject' line. attrib_parts = line.split("=") key = attrib_parts[0] value = '='.join(attrib_parts[1:]) except: # Malformed? Ignore. # Some attributes like the sender address might have an '=' in their value, # but the ONLY one we care about will never, so it's safe to just take the # error and discard the line. syslog.syslog( syslog.LOG_WARNING, f"Encountered malformed line: '{line}' processing policy exchange. Ignoring.", ) continue # Store address into 'ip' var. # Yes, this is the ONLY key we care about, # the rest get ignored. if key == "client_address": ip = value continue # Failsafe: exit if address was never defined and print warning. if ip == "": print( msg_error( "Never received 'client_address' attribute in policy filter exchange. Cannot check message." ), file=sys.stdout, ) sys.exit(0) # Run check. Results are immediately returned. # Note: As much as you'd expect STDOUT to be the default # print target, without explicitly specifying it, this doesn't work. # At all. So for now, it stays. print(check_ip(ip), file=sys.stdout) 

To me it seemed pretty simple, but… okay that’s pretty long. But basically, here’s what happens:

Each invocation processes one connection, and therefore, one message. The only thing it cares about is a client_address line, so it only looks for that and no others. After seeing this, it’ll send that to the check function which will URL-encode the address and query AbueIPDB’s API with it (line 51). We ignore reports over 90 days old for this.

Now, if an HTTP response other than 200 was returned, we log an error, but pass the message through untouched. Otherwise, we run two checks:

1. If the isWhitelisted field from the API response is true, then AbuseIPDB has whitelisted this IP for some reason. We don’t need to continue checking confidence, and can immediately return. If the WHITELIST_IMMEDIATE_OK variable was set at the top, then also skip any other restrictions afterwards too by returning an OK response.
2. Otherwise, check the abuse confidence against the MIN_CONFIDENCE value. If it’s higher, then reject the message. The reject action is 521 and not REJECT, meaning that Postfix will return 521 as the code and not its default REJECT code, which is 554 unless specified in the config.2

Assuming neither checks caught the message, we’ll send a DUNNO back, so Postfix will continue evaluation. However, if ADD_HEADER is set, then instead of DUNNO, we use PREPEND, which will add a header to the email message, in this case, with the confidence score. A PREPEND action has an implicit DUNNO in there as well.

As a failsafe, if no client_address line is sent, then the entire thing is skipped completely.

## Postfix Integration

Really there’s two parts: adding it to Postfix’s config, and then giving Postfix a way to manage the process. The first is simple, and is the check_policy_service restriction, which I added like this to smtpd_client_restrictions:

smtpd_client_restrictions = permit_mynetworks,
permit_sasl_authenticated,
reject_rbl_client zen.spamhaus.org,
reject_rbl_client bl.spamcop.net,
reject_rbl_client cbl.abuseat.org,
check_policy_service { unix:private/abuse_chk, default_action=DUNNO }


This means that, if nothing else has issued an OK or REJECT-like action, then the check_policy_service will take effect, opening a UNIX socket at private/abuse_chk, and sending the policy server data there. If this doesn’t return correctly for some reason, then, by default, use the DUNNO action. Now, private/abuse_chk is a named socket that’s relative to Postfix’s chroot, which lives at /var/spool/postfix. So somehow, I need to get the socket file /var/spool/postfix/private/abuse_chk to exist, be readable and writable by Postfix, and have my program connected to the other end of it. Luckily, Postfix itself can do that for us, with the master table.

### Postfix Master Config

Ever peeked at master.cf? This is a table of every process that Postfix starts, and some specifics about them. We just need to add two lines:

abuse_chk unix  -       n       n       -       0       spawn
user=nobody argv=/etc/postfix/abuseipdb-check


This instructs Postfix to spawn a service, named abuse_chk, as a unix service type, which, since it’s privacy is then unspecified (-), it’s assumed to be yes, meaning it places a UNIX socket for communications at /var/spool/postfix/private/abuse_chk. This also means that only the local Postfix process can access it. After this, the first n means that it’s not unprivileged. Unprivileged processes run as the Postfix user, privileged run as root. The second n means that the process is not chrooted to the mail queue. The second - means not to periodically wake up the service, since… we don’t need it. That 0 means there are no limits on the number of running copies of this process. And finally, spawn is the actual command to execute, and below it, indented, are its arguments. In this case, we’re spawning the actual process as nobody (spawn is who’s running as root), and the process to spawn is /etc/postfix/abuseipdb-check, which is that script above.

Since this means that Postfix will automatically allocate the named socket as we require, all we need to do it instruct Postfix to use it, as shown above. The socket’s input is fed to the processes STDIN, and the process’s STDOUT is put back to the socket. The only oddity here is that the Python print() doesn’t work 100% as intended, you need to manually specify the output is sys.stdout, but besides that, it’s nothing special.

## Running This Yourself

Disclaimer: Be very careful, messing with Postfix delivery rules and restrictions can cause Bad Things™ to happen. However if you want to give it a try, I’ve cleaned up the script a little and stuck it on my local code hosting, you can grab the code from here (or just stick to the releases). The stuff in there is no longer just one script, since I moved all the configuration out of global variables into an INI file, but that README should tell you everything you need to know. If you want to have some more in-depth information on what’s happening, then you can check the wiki tab which is where I’m documenting the extras that are at play here, but aren’t required for knowing how to get it up and running.

Because I want to keep that instance closed (for now), instead of creating an issue the usual way, you can genuinely just email it and it should create one. Or, alternatively, sign-up with a Google or GitHub account is turned on, but direct registration can only happen if you already have an email address on my domain, which, you don’t.

1. if you check the man page for access(5), you’d know there’s a few other results, like a specific error code, or to log a warning but let it continue. We’ll get to those in a second. ↩︎

2. 421 and 521 are special in Postfix: these indicate that you want it to disconnect immediately after giving the response, which is non-standard behavior. ↩︎