Spam reduction with greylisting

This article brought to you by LWN subscribers

Subscribers to LWN.net made this article — and everything that surrounds it — possible. If you appreciate our content, please buy a subscription and make the next set of articles possible.

A common body of opinion says that there is no good reason to run your own mail server. Organizations such as Google will, for not much money, allow you to run your entire corporate email as a hosted service. You can use the mail client of your choice, thanks to services like IMAP and SMTP AUTH decoupling the client experience from the back-end provider; there's also an excellent webmail interface for when you're out and about. Nevertheless, my experience is that, sooner or later, some shortcoming proves to be an inconvenience to the business. Perhaps the webmail site is blocked by some client network you're visiting, or important list emails aren't reaching your inbox and you can't work out why. Maybe the hosting provider has made some arbitrary decisions about content and some of your new employees find their names are unacceptable and they cannot sign up. When this happens, and if your provider is unhelpful or unresponsive, you will find yourself revisiting the decision to run your own server.

Should you decide to run your own, however, you will lose all the beneficial content filtering that your provider was doing — specifically, slimming the torrent of unsolicited commercial email that bedevils us all. There are many content filters, some commercial but many free, which can be inserted into the delivery pipeline of most mail servers to examine each incoming mail. This, however, is the point where you discover that the internet can deliver rubbish to your server more quickly than you can easily and cheaply scan it, and the departmental budget is carefully examined to see if it can accommodate a couple of big compute servers to run the spam filtering.

The cheapest, easiest tests are those that are done during the early phases of the SMTP transaction, specifically the EHLO, MAIL FROM, and RCPT TO phases, before the DATA phase is entered and the body of the email is transmitted. Any rejection that can be done at this early stage is cheap, both in network terms (because the message has not yet been transmitted) and in compute terms (because processing the few tens of bytes the client has thus far sent generally requires less hardware than processing an 8MB MIME-encoded email). There are tests that can be done at the EHLO phase, such as those relating to the resolvability and validity of the client hostname, and at the MAIL FROM phase, such as checking the client IP address against any Sender-Policy-Framework records advertised for the sender's declared domain name. Other tests can be done here, such as realtime blackhole list (RBL) lookups; I do not use those, since I've found no RBL to be sufficiently reliable for use as a bright-line test for acceptance (rather than as contributors to a total spam score in the content-based filtering system that comes later in the pipeline).

But at this stage you have three pieces of information — the sender's IP address, and the envelope sender and recipient addresses — so another elegant test can now be applied. To understand this test, it is necessary to appreciate that much of the spam email on the internet does not come from expensively-hosted bulk email senders, though those do exist. Instead, perhaps as much as 85% comes from botnets — distributed networks of compromised computers, each under the control of a single malicious actor. Many such actors use their botnets to send large quantities of unsolicited commercial email, usually for pay on behalf of third-parties.

In a perfect world, SMTP would contain an exchange where the server asked Are you a real mail server?, and the client would reply "Yes, definitely", or "No, I'm part of a botnet". The client would tell the truth, and the server would accept mail only from clients that were themselves real mail servers. Sadly, we're not there yet. Until we are, one excellent way to distinguish botnet members from real mail servers is to ask the client to behave like a real mail server. One thing real mail servers are required to do is to cope with temporary disappointment, such as the receiving server not having enough free disk space or the user's mailbox being temporarily locked. When a receiving server says it has a temporary problem by means of a 400-class return code, the sender should wait for a while, then retry the transaction. Most botnet clients are senders of a type known as fire-and-forget; they just want to zip through the fifty-million recipient addresses they've been given. If the one they're on right now has a temporary problem, well, that's just too bad; they drop that email on the floor and go on to the next one.

Greylisting

More subtlety is possible. In the simplest case, IP addresses under one's direct control can be configured as automatically whitelisted. Since few botnet mailers implement SMTP STARTTLS, any email that arrives under cover of SMTP encryption can be accepted immediately. Any mail that is submitted after SMTP AUTH succeeds is clearly from a legitimate local user, and can again be accepted immediately.

Greylisting is not without its downsides. The biggest problem is that much email will be delayed, and the size of this delay is not under the control of the receiver. Although most greylisting implementations will print a human-readable message indicating how long the sending server must wait before redelivery can succeed, there is as yet no standards-based way for the sender to recognize this, and wait only that long before retrying. Instead, the client will have its own configuration telling it how often to retry and, if this does not match the greylist's expectations, severely delayed deliveries can result. Although email delivery has never, in principle, been guaranteed nor instantaneous, some users have developed expectations outside the service guarantees. As Wikipedia dryly notes:

Problems arise with service providers that maintain a bank of sending servers, and move queued emails between these servers after each delivery attempt. Since the triplet includes the sender IP, this means each redelivery attempt is seen by greylisting as being a new email; much time can elapse before an email is lucky enough to be tried a second time from a given sending server. This is not, in my experience, common, but it does happen; all one can then do is add the whole bank of servers to an IP whitelist.

How effective is it?

• There were 101,900 inbound delivery attempts
• Of these 101,900, 3,370 were SPF failures, 1,300 were from invalid sender hostnames, and 3,800 were to invalid recipients, leaving 93,400
• Of those 93,400, 71,100 attempted no redelivery and so failed greylisting, leaving 22,300 which passed greylisting and went on to local delivery
• Of those 22,300, 7,800 were list emails filed via procmail, leaving 14,500 which went to the content-based spam filtering system
• Of those 14,500, 3,600 were identified as spam by content-based filtering

Wikipedia says that greylisting typically delays email by about 15 minutes. Looking at my body of emails from September 2016, where greylisting delayed delivery of an inbound email the median delay was 900 seconds. The mean, however, was around 8,000s, due to a long-tailed distribution (a comparatively small number of very large delays). Nonetheless, assuming that all the mail that didn't pass greylisting was spam, greylisting is the single most effective antispam technique on my mail server.

If you're of a mind to give greylisting a try on your mail server, Sendmail does it most easily through the milter interface. My Sendmail uses milter-greylist, which works well and painlessly for me; the project has instructions on configuring it with Sendmail. The milter-greylist developers also note that it works with Postfix, but those instructions seem to be a lot thinner, and other implementations like postgrey may work better. Ubuntu seems to recommend postgrey for Postfix, and the CentOS people also have a HOWTO for it. Exim has always been a bit of a mystery for me, but greylisting.org has a number of suggestions for greylisting with Exim. It would be nice if greylisting capabilities shipped as a standard part of all major mail transport agents (MTAs), but until then it should be fairly easy to glue it onto your MTA of choice.

	match -q "*Windows*" "$CLIENT_SYNOS" && greylist