Spam is a major problem for anyone with a mail server. Many times, spam goes to email addresses that don't exist. But, it still is hitting your server even if it isn't delivered. Other times, a users inbox will be overflowing with annoying messages about Viagra, hookers, free software, and whatever else. Below is a solution. It's an ongoing accumulation of my efforts to stop spam to the best of my abilities. So far, it has a 97% success rate with over 20,000 emails (spam and ham, alike) processed. Follow the instructions. I will update/modify as necessary if things seem unclear. Feel free to ask. 1) Install Postgrey, RRD, a log parser, and Graphing tools. Code: apt-get install postgrey rrdtool mailgraph pflogsumm Postgrey will have a delay of 5 minutes by default on email going to your mailbox. If this is too long, edit the /etc/default/postgrey file by adding "--delay=120" where 120 is seconds. 2) Restart the Postgrey server. Code: /etc/init.d/postgrey restart 3) edit the Postfix main.cf. We will be adding several things including the Postgrey configuration. Code: # See /usr/share/postfix/main.cf.dist for a commented, more complete version # Debian specific: Specifying a file name will cause the first # line of that file to be used as the name. The Debian default # is /etc/mailname. #myorigin = /etc/mailname smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU) biff = no # appending .domain is the MUA's job. append_dot_mydomain = no # Uncomment the next line to generate "delayed mail" warnings #delay_warning_time = 4h readme_directory = /usr/share/doc/postfix # TLS parameters smtpd_tls_cert_file = /etc/postfix/smtpd.cert smtpd_tls_key_file = /etc/postfix/smtpd.key smtpd_use_tls = yes smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache # See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for # information on enabling SSL in the smtp client. myhostname = my.derekgordon.com alias_maps = hash:/etc/aliases alias_database = hash:/etc/aliases myorigin = /etc/mailname mydestination = my.derekgordon.com, localhost, localhost.localdomain relayhost = mynetworks = 127.0.0.0/8 [::1]/128 66.118.142.78 mailbox_command = procmail -a "$EXTENSION" mailbox_size_limit = 0 recipient_delimiter = + inet_interfaces = all html_directory = /usr/share/doc/postfix/html virtual_alias_domains = virtual_alias_maps = proxy:mysql:/etc/postfix/mysql-virtual_forwardings.cf, mysql:/etc/postfix/mysql-virtual_email2email.cf virtual_mailbox_domains = proxy:mysql:/etc/postfix/mysql-virtual_domains.cf virtual_mailbox_maps = proxy:mysql:/etc/postfix/mysql-virtual_mailboxes.cf virtual_mailbox_base = /var/vmail virtual_uid_maps = static:5000 virtual_gid_maps = static:5000 smtpd_sasl_auth_enable = yes broken_sasl_auth_clients = yes smtpd_sasl_authenticated_header = yes [B]smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, check_recipient_access mysql:/etc/postfix/mysql-virtual_recipient.cf, reject_unauth_destination, check_policy_service inet:127.0.0.1:60000, reject_rbl_client zen.spamhaus.org, reject_rbl_client smtp.dnsbl.sorbs.net, reject_rbl_client bl.spamcop.net, reject_rbl_client combined.rbl.msrbl.net, check_recipient_access regexp:/etc/postfix/spamtrap, permit[/B] smtpd_tls_security_level = may transport_maps = proxy:mysql:/etc/postfix/mysql-virtual_transports.cf relay_domains = mysql:/etc/postfix/mysql-virtual_relaydomains.cf relay_recipient_maps = mysql:/etc/postfix/mysql-virtual_relayrecipientmaps.cf virtual_create_maildirsize = yes virtual_maildir_extended = yes virtual_mailbox_limit_maps = proxy:mysql:/etc/postfix/mysql-virtual_mailbox_limit_maps.cf virtual_mailbox_limit_override = yes virtual_maildir_limit_message = "The user you are trying to reach is over quota." virtual_overquota_bounce = yes proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps $virtual_alias_domains $virtual_mailbox_maps $virtual_mailbox_domains $relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps $recipient_canonical_maps $relocated_maps $transport_maps $mynetworks $virtual_mailbox_limit_maps smtpd_sender_restrictions = check_sender_access mysql:/etc/postfix/mysql-virtual_sender.cf smtpd_client_restrictions = check_client_access mysql:/etc/postfix/mysql-virtual_client.cf maildrop_destination_concurrency_limit = 1 maildrop_destination_recipient_limit = 1 virtual_transport = maildrop header_checks = regexp:/etc/postfix/header_checks mime_header_checks = regexp:/etc/postfix/mime_header_checks nested_header_checks = regexp:/etc/postfix/nested_header_checks body_checks = regexp:/etc/postfix/body_checks content_filter = amavis:[127.0.0.1]:10024 receive_override_options = no_address_mappings message_size_limit = 0 [B]disable_vrfy_command = yes smtpd_helo_required = yes smtpd_helo_restrictions = permit_mynetworks, reject_invalid_hostname, permit[/B] From a generic main.cf found in the Debian Lenny installation, I added/modified the BOLD areas. 4) Create a file named "spamtrap" in the /etc/postfix/ directory. This serves as a filter. If spam is emailed to this address and other addresses on your machine, it will drop that email so that it doesn't get to any other mailboxes. The Spamtrap file looks like this: Code: /emailcontrol.*@derekgordon\.com/ DISCARD This is regexp so the slashes have to be used. My filter email is [email protected] so edit accordingly and place in the spamtrap file!!! Side note: Do not create this mailbox using ISPConfig. There is absolutely no reason for it to exist on your mailserver. It's a fake address meant to catch annoying spam. 5) Open up local.cf for SpamAssassin and add the following bit. It will be an extra filter designed to work with SA more than it is with general Postfix. Code: nano /etc/spamassassin/local.cf Add the following to the bottom: Code: urirhssub URIBL_BLACK multi.uribl.com. A 2 body URIBL_BLACK eval:check_uridnsbl('URIBL_BLACK') describe URIBL_BLACK Contains an URL listed in the URIBL blacklist tflags URIBL_BLACK net score URIBL_BLACK 3.0 6) Restart Postfix and Spamassassin Code: /etc/init.d/postfix restart Code: /etc/init.d/spamassassin restart 7) Copy the mailgraph CGI script to your websites CGI-BIN: Code: cp -p /usr/lib/cgi-bin/mailgraph.cgi /var/www/www.example.com/cgi-bin 8) Create and CHMOD the postfix_report.sh script: Code: nano /usr/local/sbin/postfix_report.sh Paste the following into the script: Code: #!/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin pflogsumm /var/log/mail.log | formail -c -I"Subject: Mail Statistics" -I"From: [email protected]" -I"To: [email protected]" -I"Received: from www.example.com ([ 127.0.0.1])" | sendmail [email protected] ##gzip /var/log/mail.log.0 exit 0 Code: chmod 755 /usr/local/sbin/postfix_report.sh 9) Edit the RSYSLOG file so that your mail.log rotates daily and to setup an automatic email with postfix statistics: Code: nano /etc/logrotate.d/rsyslog Delete the line that says "/var/log/mail.log" and add this at the VERY bottom of the file: Code: /var/log/mail.log { rotate 7 daily missingok notifempty delaycompress compress prerotate /usr/local/sbin/postfix_report.sh > /dev/null endscript postrotate invoke-rc.d rsyslog reload > /dev/null endscript } With this, every time the mail.log rotates (usually around 6am by default) you will get a detailed email about what Postfix has delivered, not delivered, greylisted, and so on. So now you're all done! What did you do? You installed blacklist filters, greylisting, graphing for on-the-fly information about Postfix, daily emails with detailed Postfix stats, created a spam trap, and other minor things to make your mailserver a lot more secure and less susceptible to spam. IMPORTANT: Let me know what you all do. Please respond with your choice, if you use it, and how well it worked. If there's much of a use, I will keep building upon the instructions and make it even better (hopefully). Responses are in the form of thread messages. ---------------------- BONUS INSTRUCTIONS: If you use the script I posted below, that gives you GREYLISTING SPECIFIC STATS, do the following: 1) DELETE it from having a CRONJOB if you added one originaly. Mostlikely, you did. 2) Open the RSYSLOG file again. 3) Modify the above entry so that it looks like this: Code: /var/log/mail.log { rotate 7 daily missingok notifempty delaycompress compress prerotate /path/to/the/greylist_script.sh > /dev/null /usr/local/sbin/postfix_report.sh > /dev/null endscript postrotate invoke-rc.d rsyslog reload > /dev/null endscript } Make sure that the "/path/to/the/greylist_script.sh > /dev/null" matches the exact path to the script you were using. Again, That script is found a few posts below. --------------------------------------- TIPS AND FIXES: 1) If you have people sending you emails and they get rejected because SORBS says their dynamic IP is a bad one, change the following in /etc/postfix/main.cf: "reject_rbl_client dnsbl.sorbs.net" becomes "reject_rbl_client smtp.dnsbl.sorbs.net" SORBS has a huge variation in repositories. Some AT&T DSL IP's will be found in the DNSBL it appears. So, that could cause a potential no deliver. 2) Similarly to #1, if you find out that there are some SORBS errors in delivering legit emails, you may want to experiement with the various Zones. https://www.au.sorbs.net/using.shtml lists all of them. I recommend SMTP, NOMAIL, SPAM, and ZOMBIE. 3) When upgrading ISPC3 to later versions, the main.cf for Postfix will be erased. This means you can either backup the one you created and replace the changes made by the update script for ISP3, or you can just copy the main.cf information above and use it again. This is the only step that has to be repeated. So, again if you upgrade ISPC3, you need to reconfigure the main.cf for Postfix as it will be replaced by the ISPC3 default main.cf again. 4) If you used this guide prior to 13 October 2010, I recommend removing "reject_rbl_client multihop.dsbl.org" from the main.cf for Postfix. That database has been taken offline. This was "effective" in 2009, but still worked until recently.
The next big step for fixing spam issues is to build a robust header_check and body_check file for Postfix. As you see above, I have started on the header_check. I'd like to determine how to setup a SUBJECT filter that would delete any email with more than 7 words. I have had NO legitimate email with more than 6 words in the subject. However, a lot of spam has between 7 and 12 words in the SUBJECT line. I'm not finding great details on how it works, so it's going to take a lot of time. If you have any of your own mod's please post them so I can compile a better post above. Also, filtering out all languages you DO NOT use would be sufficient. I will be filtering out all languages except ENGLISH, SPANISH, FRENCH, GERMAN, CZECH, POLISH, and PORTUGUESE when I have time.
Thats implemented in ispconfig, so do not add such a file manually to ensure that you will not get conflicts. Use the header and body filter function from ispconfig instead. By the way, I wont use such filter files for spam filtering. The normal spamassassin and blacklists filter already about 99.x % of all spam and adding such manual filters will just rise the risk of false positives. You can not win the battle against sapm with manual filter rules.
Yes, but if you have NO header blocks added in ISPCONFIG, you can use the header file directly. In cases that it's not a production server, just for personal uses, it might be easier to copy/paste the above header_check. However, I understand your point and if entries were being made it would be a conflict. So, I would recommend doing what Till says if you're already using that feature or may use that feature.
I removed the HEADER_CHECKS I created from the HOWTO as I don't want to confuse. Here they are for you expert experimenters. Last piece, and this will be updated as I get more in depth, header_check in the postfix directory should contain these: Code: /^Subject: .* / REJECT Spam Header Many Spaces 1 # Chinese, Japanese and Korean /^Content-Type:.*?charset\s*=\s*�?(Big5|gb2312|euc-cn)�?/ REJECT HDR2100: Unaccepted character set: �$1? /^Content-Type:.*?charset\s*=\s*�?(euc-kr|iso-2022-kr)�?/ REJECT HDR2110: Unaccepted character set: �$1? /^Content-Type:.*?charset\s*=\s*�?(iso-2022-\w+|euc-jp|shift_jis)�?/ REJECT HDR2120: Unaccepted character set: �$1? # Cyrrilic character sets: Russian/Ukrainian /^Content-Type:.*?charset\s*=\s*�?(koi8-(?:r|u))�?/ REJECT HDR2200: Unaccepted character set: �$1? /^Content-Type:.*?charset\s*=\s*�?(windows-(?:1250|1251))�?/ REJECT HDR2210: Unaccepted character set: �$1? I will continue to build the header_check file as time goes on. So stay tuned.
Question/Recap: - Till is saying not to perform header/body checks within main.cf and use the tools within ispc3, yes? If so, that makes perfect sense to me. I wish there was a service/blocklist that already did it on a realtime basis (somehow). - Is Postgrey good to load or no? I still need to research postgrey to figure out what it is. Sorry for my confusion and thank you for including me. By you posting your main.cf, I noticed I had limited my sorbs bl. By changing to yours I blocked a lot more immedietly.
Yes, I would do header_checks in ISPC3 if you're comfortable with that. It's pretty simple in the end. I haven't tested my new ones I wrote yet, so won't post them. But, I'm thinking that postgrey will be the best solution. POSTGREY is an easy implementation of Greylisting: Code: Greylisting is a new method of blocking significant amounts of spam at the mailserver level, but without resorting to heavyweight statistical analysis or other heuristical (and error-prone) approaches. Consequently, implementations are fairly lightweight, and may even decrease network traffic and processor load on your mailserver. Greylisting relies on the fact that most spam sources do not behave in the same way as "normal" mail systems. Although it is currently very effective by itself, it will perform best when it is used in conjunction with other forms of spam prevention. For a detailed description of the method, see the Whitepaper. The term Greylisting is meant to describe a general method of blocking spam based on the behavior of the sending server, rather than the content of the messages. Greylisting does not refer to any particular implementation of these methods. Consequently, there is no single Greylisting product. Instead, there are many products that incorporate some or all of the methods described here. Postgrey specific: Code: Postgrey is a Postfix policy server implementing greylisting. Development of Postgrey started at the ISG.EE and is now sponsored by Open Systems AG. When a request for delivery of a mail is received by Postfix via SMTP, the triplet CLIENT_IP / SENDER / RECIPIENT is built. If it is the first time that this triplet is seen, or if the triplet was first seen, less than 5 minutes ago, then the mail gets rejected with a temporary error. Hopefully spammers or viruses will not try again later, as it is however required per RFC. I have POSTGREY running. It's blocked a significant amount of unwanted email. It's only downside, if there is any, that it delays mail delivery by 5 minutes default.
Once you're using Postgrey and what to see how it's working, install the following script somewhere on your server: "postgrey_stats.sh" Code: #!/bin/sh LOGFILE=/tmp/greylist-statistics [email protected] echo "Total amount of GreyListed messages " > $LOGFILE cat /var/log/mail.log | /usr/bin/postgreyreport --delay=300 >> $LOGFILE echo -ne "-------------------------------------\n" >> $LOGFILE echo -ne "-------------------------------------\n" >> $LOGFILE echo "Get only the top 20 sources getting greylisted out " >> $LOGFILE cat /var/log/mail.log | postgreyreport | awk '{print $1}' | sort | uniq -c | sort -nr | head -n20 >> $LOGFILE echo -ne "-------------------------------------\n" >> $LOGFILE echo -ne "-------------------------------------\n" >> $LOGFILE echo "Get a list of the top 20 email address that the greylisted sources are sending email to " >> $LOGFILE cat /var/log/mail.log | postgreyreport | awk '{print $4}' | sort | uniq -c | sort -nr | head -n20 >> $LOGFILE echo -ne "-------------------------------------\n" >> $LOGFILE echo -ne "-------------------------------------\n" >> $LOGFILE cat $LOGFILE | mail -s "Greylisting Statistics of `hostname` for `date +%Y-%m-%d`" $YOURMAIL Edit the following parts of the above script: 1) Change the "YOUREMAIL = " line so that it goes to your personal mail box. This will give you details on how its working. 2) Make sure that /var/log/mail.log is the correct path to your current mail.log file. Distros are different. Set the script to chmod +700 so that it is executable. Finally, create a cronjob so that it will give you an automatic output to your email daily (or whenever you choose).
Have you seen sqlgrey? Not sure if it's better or not but it has an interface that's interesting. Any thoughts?
I have seen it. However, I will stick with Postgrey for the time being. I recommended in a REQUEST that Till research including SqlGrey and its interface in a future release. At that point, it wouldn't be necessary to use Postgrey. But, Postgrey is extremely simple (see above instructions) so it isn't worth the extra hassle in my opinion for roughly the same functionality. I'm researching an addon to Postgrey that will make it even better. Give Postgrey a chance, use that script above to see how well it works (you will be shocked after the first full day), and see what happens. The only false positive I have heard of was Paypal. But, that's unconfirmed and worked without a hitch for me. You can add whitelists for domains, ips, specific emails, clients, etc... with easy in the event there were problems. Whitelists also will skip the delay in delivery. Let me know what you all do. Please respond with your choice, if you use it, and how well it worked. If there's much of a use, I will keep building upon the instructions and make it even better (hopefully).
Yesterday, Postgrey kept 112 from reaching my inbox. This doesn't take into account the numerous blocks from the blacklists in use above or spamassassin. I had ZERO spam emails in my inbox today. That hasn't happened in a longgggg time!
postgrey, like it's name suggested is for greylisting emails. Greylisting works in such a way that emails are put in "on-hold" for 5 minutes (in default config) before delivery. The reason behind this is because if the email is valid, no more will come into the server. If it is spam, you will get more of the same email bombarding the server thus postgrey will know those are spam. It was said to be not a "permanent" solution to fight spam because the method is "raw" and soon spammer will found a way to overcome this. However, I have encountered greylisting a lot of times and found it to work great and produce very little false positives. It also works great in combination with spamassassin. Only downside to this is that emails need to wait for 5 mins before it get delivered to recipient.
The hold-time can be changed. Further, I am testing a mixture that will allow almost immediate determinations of most email. And if it seems to still be an email with problems, it will hit the 5 minute window. So I really recommend the procedure outlined above. Once there's new stuff to add, I'll update it so that we all can continue to benefit from mail filtration.
No, it does _not_ work this way, please read: http://en.wikipedia.org/wiki/Greylisting In short, greylisting temporarily rejects incoming mail and send notice about it to originating mailserver, and if retry ever comes, mail is accepted for delivery. We have been using greylisting for 5+ years now without problems. Combined with couple blacklists and amavis things are working quite well.
You misconstrue the point of that message. I understand how it operates; however, there were concerns about the loss of email and delays. A short explanation that the emails are basically held undelivered for five minutes by default is true locally. Yes a response is sent to the originator in an attempt to verify authenticity and legitimate email. I do believe I mentioned the process and how it works before as well.
Ah you know my eyes don't work very well! Sorry for that! For all of the followers, I hope to have a big howto with graph monitoring and some extra filtration options included by mid-October. Military is keeping me busy right now so not much extra time to experiment.
In the last 24 hour period my top 20 list showed roughly 5000 emails (1200 to one user!). Now...there were plenty of false positives so I have a question: in whitelisting: Do I enter the full server domain or will the main domain work? Example: my greylist shows yyy.domain.com, but I want to whitelist domain.com completely. Do I enter domain.com in the whitelist or do I need a regex to allow for *.domain.com? In your postgrey script: Is this telling me all those emails (in the first section of the script output) are ones that were blocked, or just greylisted and possibly let through? Just looking for clarification on the output of the script. Nice work!
Okay, you can whitelist either way. I have been doing "domain.com" on the few I had to add. You can get specific but no point unless it's a domain that has half legit and half spam emails, which doesn't seem likely! The email is telling you all that has been greylisted.
Anyone having problems with this setup at all? Any other thoughts? I'm about done with an upgraded HOWTO that will add some good graphs and a daily email that provides better details. Basically, I'm just combining a lot of useful things out there for everyone... So input is worth while as I continue to help us all in the spam fight.
