This is a collection of the mail related issues, such as configuring spam-assassin, reducing spam with postfix, setting up SPF and DKIM and over-coming the "guilty until proven innocent" or as it is often searched, "Yahoo blocked my mail server" or "Tesco blocked my mail server". Try those (no quotes) and see how many hits those get, 2,500,000, and 145,000 at last count!

Spam-Assassin

Virtualmin's script installs Spam-Assassin, however, when I worked through the cron scripts I found that although spam-assassin update is called in cron.daily it did not actually run. The issue was that in /etc/default/spamassassin was "CRON=0" where as it needs to be "CRON=1" to run. This might be fixed in later Virtualmin install scripts though.

The other issue is that running spam-assassin update locks up cron for up to an hour. This is caused by /etc/cron.daily/spamassassin which has a random delay of up to 1hr just after "# Sleep for up to 3600 seconds". Presumably this is so that all Virtualmin installs which all have the same cron run time don't all hit SA update servers at the same time. I think it's sensible to move the cron to another time and then reduce the maximum delay.

Note that if you get a warnings from spam-assassin update check that everything /var/lib/spamassassin and below has owner of debian-spamd and that /var/lib/spamassassin/sa-update-keys folder and its files has permission 700 (i.e. rwx only for owner).

Also, by default spam-assassin just marks spam so you need to enable filtering either globally at Servers->SpamAssassin Mail Filter->Procmail Spam Delivery or by enabling individual filtering in the enabled features setting of each virtual server. The filtering is then set by logging in to usermin and selecting mail filters or spam-assassin. By using the (procmail) mail filters you can have more selective spam control, filtering by the actually score for example.

Postfix Anti-Spam

Mail is a rather confusing so it's worthwhile explaining a bit how it works. You get mail via another server on the internet connecting to a port on the VPS. This port is owned by postfix, hence the connecting server is a client of postfix. In Servers->Postfix->SMTP Client Restrictions click radio button "Selected below..." then:

• Tick allow connections from this system
• Tick reject clients with no reverse hostname
• Tick reject if client IP address is in RBL and enter zen.spamhaus.org in the box
• Tick reject if client hostname is in RBL and enter dbl.spamhaus.org in the box
• Tick other restrictions and enter permit_sasl_authenticated in the box (if not already done)

And save. Next, in Servers->Postfix->Edit Config Files edit main.cf (or with WinSCP edit /etc/postfix/main.cf) and add these lines at the end

In Servers->Postfix stop postfix then start postfix to apply the changes.

What all this has done is to force all error codes to be rejects rather than defers which some default to. Disabling verify prevents a spammer from obtain valid emails. Helo required enforces proper handshaking protocol (which some spammers get wrong). Similarly "Reject clients with no reverse hostname" looks up the sending IP address and if it doesn't have a hostname it gets rejected. Any sending IP in zen.spamhaus.org blacklist or hostname in dbl.spamhaus.org blacklist also gets rejected. Any email for an unknown mailbox or mail alias is also rejected.

Note this also affects you when you send emails from your email program on your PC. Hence the need to tick other restriction and enter permit_sasl_authenticated which permits you when you have logged in. Without this setting sending email can be hit and miss. Many ISP's blacklist the dynamic IPs given out to users so they can't send emails from their PCs but only from the ISP's servers. Although logged in users get permitted later in the process the blacklist report can come back fast enough to beat that and cause the email to be rejected. Hence the need to repeat the permit in the client restrictions.

If you want to see how effective this is then go to System->System Logs and click view opposite "File /var/log/mail.log". In the box "Only show lines with text" enter the word "reject" (without quotes) and refresh. I have seen as many as 30/hr, depending on the time of day. And >90% of those are "cannot find your hostname" rejects rather than from blacklists.

Setting up DKIM and SPF

Virtualmin has support for DKIM built in. This is found in Virtualmin, Email Messages, DomainKeys Identified Mail. The first time you click this it will offer to install the package for you, accept that and let it install. The second time you will get the setup form. The field "Selector for DKIM record name" can be anything, such as the year which it deafults to. The field "Signing of outgoing mail enabled?" needs to be set to yes of course. It should also default to including the hub domain and a key of 2048. The field "Reject incoming email with invalid DKIM signature?" is safest set to No. Click save then open the form again. The form will now contain the records for DNS. Copy "DNS records for additional domains" field box to notepad.

Browse to wherever you set up your DNS and create a new TXT entry. The first DNS TXT entry you need is named _domainkey.Virtual_Server_Domain, e.g. _domainkey.example.com . This has content of either "o=-" (without quotes) which means all e-mails are signed, or "o=~" means emails may be signed or unsigned. The name of second DNS TXT entry is the part xxxxx._domainkey from notepad. The contents of this TXT entry is everything inside the ( ) after joining the strings at the " ", that is delete everything between the "....." to make one long string, then delete the outside (" and "). Although maybe your DNS will accept it formatted into separate strings but that did not work for me.

Setting up SPF is simpler. Again you need to add a TXT record to your DNS. This is named after the virtual domain, i.e. example.com. The contents are "v=spf1 mx ptr:hub_domain -all" (less the quotes) where hub_domain is the FQDN of the hub, e.g. hub.com. What this means any mx at this domain is allowed, so if my from address is me@example.com then the recipient looks up the DNS record for example.com and any mx there is allowed. And when doing a reverse DNS, IP number to domain name, the result of hub_domain is allowed. If those tests pass then the email is kosher otherwise reject it (-all). You can relax this with ~all.

Testing DKIM and SPF can be done at appmaildev which has tools for this. You get a random email address from this site and then send an email to that address. The reply back confirms if DKIM and SPF is set up correctly or not.

Blocked Mail Server

Unfortunately because of all the pesky spammers the internet default seems to be guilty until proven innocent. Plus, if you create a new VPS then by definition you'll will be moving even an oldtimer wbe domain to new IP for that website and also for your mail-server. Since IP4 is a limited reasource the likelihood is that in the past this IP4 address was used by a spammer, who once the blacklists caught up with him moved on to a new IP address. Most IP4 addresses assigned to a new VPS will thus have a mucky past. You can check blacklists for your VPS by using services like MX Tool Box, RBL Check, Anti-Abuse Project but Yahoo, Tesco, and probably others, have their own secret blacklists and you have no idea you're on those until your email gets bounced. And when you find you are on Yahoo or Tesco's blacklist there is no simple way to get off. Unlike BT and some others where you go to a form and report an IP address being blocked in error and have that fixed in a few hours.

It took me about 2 weeks to get Yahoo to fix their blacklist. The first time I was unlucky enough to get the Yahoo agent who didn't know what to do so kept asking silly questions or giving form (and wrong) answers until finally he just killed the report and I had to start a fresh complaint. The second time it was fixed in a few days of emailing back and forth (and believe me, after the first time that ranks as good for Yahoo, not a patch on the few hours that the good servers manage though). Tesco is even harder to find who to complaint to. I still don't know for sure although it looks like it might be Cloudmark (csi). I used their report form for a Tesco.net block and they said they replied saying they had fixed it (nice of them, replies seem not to be the norm). Since Cloudmark didn't say "why are you contacting us about Tesco.net?" I assume they had handle Tesco.net filtering, plus a test email to a Tesco.net user hasn't bounced.

Useful links for unblocking:

• Yahoo blocking error codes. Yahoo's bulk sender form.
• Hotmail (msn) unblock request form.
• Gmail delivery problems form. Gmail bulk sender guidelines and application form.
• BT blacklistchecker and report form.

BUT, before raising unblock requests check:

• Reverse DNS resolves to a valid hostname (see MX Tool Box).
• Your server’s hostname has a DNS ‘A’ record (see MX Tool Box).
• Make sure DKIM/SPF are correct (see appmaildev tools).

Note for Yahoo to unblock you have to submit Yahoo's bulk sender form. They'll then email back with suggestions. They key one in my experience is the manual SMTP test with telnet. That seems to get more reaction or maybe I just got lucky. Note this is a tricky test to do. You probably can't do this from your PC since ISP's typically block port 25 from end users to stop spam so you'll need to do it from your VPS via SSH. And secondly timeouts are pretty short so have all the instructions ready in notepad so you can just copy and paste.

It's is also a very good idea not to blindly forward mail on to gmail (and yahoo, etc). If some of that mail is spam then to gmail (and yahoo, etc) you look like a source of spam. Instead set up gmail to collect that mail from a mailbox on your server.

The 99% Solution to Blocked Mail Server

The simple solution is to signup with mailgun a service for sending bulk mail. I know that sounds a bit like the man in the street joining a shooting club to prove he's not poacher, but it works. A mailgun free account, when you verify your domain, gives you up to 10,000 emails per month which should be ample for most people. Once signed up you can use an API from your website to send emails but the simplest solution is just to use the normal server login you get as well. In my email client I can set up multiple servers to send mail, selecting one as default. So it's simple to switch servers if my emails are blocked. And for Drupal there is SMTP Authentication Support a module that allows Drupal to use an external mail server. That works with mailgun's server and it's trival to setup. In both cases the from address remains the same so replies come back as normal.

The only tricky thing with mailgun I found is setting up the DNS as their instructions are not completely clear or maybe they just don't quite match how Cloudflare does it. Assuming your domain is called example.com then you should have these records:

Mailgun will give you the exact values you need to use (and note that {very long string} is a comment, not the exact value).

Webmin-Virtualmin How-To Blog Posts