Help! I need some seasoned advice please. We're running the OldStable version of Debian Etch from August 2008. We've been using squirrelmail connecting through Dovecot's Imap and Pop3 servers since then to provide either SSH or TLS/SSL connections to postfix mail via squirrelmail on our server. Although the SSL capability is installed, we're really not using it -- choosing to use SSH with strong passwords instead. This configuration has given us NO problems since we started... until today. For unexplained reasons this morning the IMAP interface suddenly began refusing or failing connections to everyone trying to connect through SquirrelMail (or that's the way it looks from the outside). It also fails to send out any emails. I've tried rebooting the server but that made almost no difference. The problem was first reported by a user. I then verified it. What I was seeing BEFORE the reboot when I tried to login was an error from SquirrelMail that said: Error connecting to IMAP server: myserver.com. 11074 : but what I'm seeing since the server reboot is: Error connecting to IMAP server: myserver.com. 11087 : The IMAP server connect problem seems to be isolated to SquirrelMail. At least I ran 2 tests and found I CAN connect to the IMAP server using both Microsoft Outlook 2003 and Outlook Express and can see the contents of all folders on the server. So the IMAP problem only shows up in SquirrelMail. But it DOES prevent ANY users from loggin in through SquirrelMail However, the inability to send mail OUT from the server shows up everywhere. Mail sent internally between accounts on the server -- either within a single domain or between domains -- and even from remote users connected to the server through Outlook or Outlook express gets delivered fine. But email addressed to anyone outside the server to any domain -- whether Yahoo or or MSN or Google or whereever is all bouncing back with a "relay request denied" error. For instance, I sent an email from one of my server accounts to my yahoo inbox and it bounced back. Other things I've tried are: checked port status with nmap -qT both port 443 and 993 are reported as open -- one with imap and the other with imaps. I restarted inetd. It had no effect. I restarted postfix. It had no effect. I restarted the whole server. The IMAP login error code when attempting to login via SquirrelMail changed 11074 to 11087. That's all. All other behaviors remain the same. I also confirmed the SquirrelMail login failure problem occurs in IE 7, IE 8 and Firefox from 3 different machines in multiple geo-locations and networks AND both with and without the user's local firewall running. So the issue is definitely ON the server and seems to be isolated to squirrelmail even though no changes have been made to squirrelmail or any of its components in months. When I checked the mail.err log, I found the following series of seemingly useless error messages: Code: Oct 25 08:40:33 axx012503 dovecot: POP3(sarah): unlink(/var/mail/sarah.lock) failed: Permission denied Oct 25 09:31:48 axx012503 dovecot: POP3(mymailname): UIDs broken with partial sync in mbox file /var/mail/mymailname Oct 25 10:02:28 axx012503 dovecot: POP3(mymailname): UIDs broken with partial sync in mbox file /var/mail/mymailname Oct 26 01:01:06 axx012503 postfix/sendmail[30523]: fatal: root(0): queue file write error Oct 26 15:06:53 axx012503 dovecot: POP3(sarah): unlink(/var/mail/sarah.lock) failed: Permission denied Oct 27 01:01:04 axx012503 postfix/sendmail[6715]: fatal: root(0): queue file write error Oct 27 08:56:15 axx012503 dovecot: POP3(bigdork): unlink(/var/mail/bigdork.lock) failed: Permission denied Oct 27 08:56:16 axx012503 dovecot: POP3(bigdork): file_dotlock_delete() failed with mbox file /var/mail/bigdork: No such file or directory Oct 27 14:57:56 axx012503 dovecot: POP3(mymailname): UIDs broken with partial sync in mbox file /var/mail/mymailname That's about it. I've been racking my brains on this all day and I'm no closer to a resolution now than I was when the problem was reported 12 hours ago. At this point, any hints, helpful suggestions or questions would be appreciated! Thanks!
Are you using Maildir or mbox? What's the output of Code: ls -la /var/mail/ ? Are there any errors in your mail log? Can you check if your server has been blacklisted? http://mxtoolbox.com/blacklists.aspx
First, thanks for the reply, falko. I appreciate your efforts to help. Second, here's the result of that ls -la /var/mail/ command Code: drwxrwsr-x 2 root mail 4096 Oct 28 08:31 . drwxr-xr-x 15 root root 4096 Oct 13 06:59 .. -rw------- 1 mail mail 104 Jun 16 07:00 .bash_history -rw------- 1 mail mail 35 Jun 16 06:59 .lesshst -rw-rw---- 1 alyianna mail 538 Jun 19 16:30 alyianna -rw-rw---- 1 board mail 24928 Oct 13 15:14 board -rw-rw---- 1 doorprod mail 0 Jun 16 07:16 doorproductions -rw-rw---- 1 daemon mail 9434 Dec 15 2008 daemon -rw-rw---- 1 dave mail 3425224 Feb 25 2009 dave -rw-rw---- 1 devion mail 26275 Oct 27 03:30 devion -rw-rw---- 1 eagle mail 6044885 Oct 27 18:59 eagle -rw-rw---- 1 sarah mail 418274 Oct 28 08:30 sarah -rw-rw---- 1 sarahtv mail 538 Nov 7 2008 sarahtv -rw-rw---- 1 eric mail 205715 Mar 4 2009 eric -rw-rw---- 1 martinpr mail 538 Sep 10 2008 martinpress -rw-rw---- 1 events mail 121615204 Oct 27 18:29 events -rw-rw---- 1 winterex mail 6140 Oct 28 08:31 winterexchange -rw-rw---- 1 mike mail 538 Sep 9 2008 mike -rw-rw---- 1 mikelinh mail 538 Jun 15 09:05 mikelinhart -rw-rw---- 1 info mail 869 Oct 1 17:43 info -rw-rw---- 1 jody0309 mail 67007 Apr 15 2009 jody0309 -rw-rw---- 1 legna725 mail 86253 Oct 27 18:59 legna7259 -rw-rw---- 1 lovingfn mail 18110 Oct 17 18:52 lovingfn -rw-rw---- 1 mail mail 44542363 Oct 28 08:22 mail -rw-rw---- 1 masterr mail 538 Oct 28 07:08 masterr -rw-rw---- 1 pianoman mail 20110 Oct 7 10:07 pianomancollette -rw-rw---- 1 pianoman mail 1015593 Oct 28 08:30 pianomanshop -rw-rw---- 1 mistyblu mail 22925 Mar 4 2009 mistyblue -rw-rw---- 1 mornings mail 0 Jun 16 07:10 morningstarentjobs -rw-rw---- 1 nobody mail 63284373 Oct 28 07:17 nobody -rw-rw---- 1 noohpyt0 mail 38444 Oct 17 18:52 noohpyt0348 -rw-rw---- 1 noom2567 mail 40084 Sep 23 21:43 noom2567 -rw-rw---- 1 oatdrol3 mail 81547 Oct 27 18:59 oatdrol3665 -rw-rw---- 1 ojoc4957 mail 8674 Mar 4 2009 ojoc4957 -rw-rw---- 1 penmark mail 538 Oct 28 08:31 penmark -rw-rw---- 1 blackcon mail 538 Sep 22 2008 blackconsultants -rw-rw---- 1 blacklab mail 538 Sep 22 2008 blacklabs -rw-rw---- 1 rednib00 mail 10481 Apr 26 2009 rednib0006 -rw-rw---- 1 markgold mail 538 Oct 19 07:13 markgoldstein -rw-rw---- 1 sandy mail 265579 Oct 27 20:10 sandy -rw-rw---- 1 savether mail 538 Sep 15 2008 savethehorse -rw-rw---- 1 shannon0 mail 111760 Apr 5 2009 shannon0309 -rw-rw---- 1 candylan mail 0 Jun 16 07:12 candyland -rw-rw---- 1 tgblack mail 538 Oct 28 08:30 tgblack -rw-rw---- 1 theinfor mail 3486 Oct 28 08:30 ourinformalchateau -rw-rw---- 1 theweste mail 3902 Sep 9 2008 theeasternmotel -rw-rw---- 1 uucp mail 21696 May 16 12:49 uucp -rw-rw---- 1 webcandy mail 3674 Oct 28 08:31 webcandy -rw-rw---- 1 webwoods mail 3480 Oct 28 08:31 webwoodscraft -rw-rw---- 1 whipping mail 538 Oct 28 06:49 whippingboy -rw-rw---- 1 wwphost mail 849129 Oct 28 07:17 wwphost -rw-rw---- 1 www-data mail 238754025 Oct 28 08:06 www-data -rw-rw---- 1 yesac387 mail 19819 Mar 17 2009 yesac3876 I also checked the mxtoolbox site. It shows the server's primary IP address has been blacklisted by the Barracuda BRBL spam engine. The secondary IP address is not listed there. Barracuda is the ONLY blacklist that identifies any problems with the server and the confusing part is that when I checked the reputations of every one of the individual domains hosted on the server, not ONE Of them shows any issue at BRBL! ??!!? How's that again? How is it possible for the server's IP address to be listed as a spam source if all domains on the server are innocent? As is common these days, all domains on the server do share the same IP address. As you saw, I checked the mail error log and did find some issues there - although I couldn't make heads nor tails out of what they were telling me. I have not checked the mail log yet. I'll do that and see what I find. I admit though that I'm not sure exactly what to look for there. For the record, I watch emails and email bounces on this server pretty close and I DO see some email bounces I can't explain which purport to be bouncing emails from sites I KNOW aren't sending those emails. I control all domains on the server. I KNOW what emails are sent out by those domains and the email loads aren't heavy, Frankly, the number of weird bounces I've seen hasn't been large enough to be a big concern to me or produce a full scale investigation into how those emails are happening to begin with. I'm very aggressive about trying to keep spammers and click-phishers out of the two sets of forums hosted on our server... manually reviewing and approving all join requests, and running queries every day that are designed to identify and remove the dozens of bots that try to register in those forums daily. In short, I'm using every standard technique I know of to prevent server hacking -- ssh-secured logins, hard-to-guess usernames, strong and hard to guess 12 - 25 character passwords, limited telnet/putty access, limited email accounts, etc. But I'll admit I'm NOT using IPTables. I couldn't see the benefit to that. What the heck can an outsider do with a port that's not being used by the server? Or conversely, if the port is being used for outgoing smtp mail, I don't see why it's a security concern? I don't mean to seem stupid or ignorant here. But I don't get it. What am I missing? Still, I want to STOP the blasted spam as much as anyone else does. So, I'll gladly listen to suggestions on how to further tighten security on my server... and how to chase down, isolate and kill the source of those unexplained outbound emails. I'm NOT averse to fighting the spam wars. I'm just not sure how to isolate and kill the mysterious sources of "how-the-heck-did-that-happen?" spam that seems to occur on many servers despite the best efforts of the admins to stop it. Like most server admins, I do have my limits. I can't spend my whole life to fighting spam either. Thanks!
One more piece to the puzzle... I just realized there is another possibly-relevant piece to this puzzle. There is a single domain on this server for which I host a mailman mailing list. When I first moved that site and its list to my server a couple of years ago it was receiving a huge load of junk mail every hour of evey day. Worse yet the inbound email address most of the spam was being sent to was the same one the mailman list had long been hooked to. So, when I installed mailman, I configured it to only accept mail from known list members and configured things so the server would bounce all mail that wasn't accepted by mailman back to the sender. Later I realized that this so-called junk mail "back-scatter" was an issue that caused some Black lists to rate the server poorly. So, I changed that bounce solution to accept all such mail and deliver it to >null instead. As far as I know, it's still operating that way on that mailman domain. That approach shouldn't cause a black-list problem, should it? Thanks.
Some blacklists list whole subnets if one or more IPs from that subnet send out spam. But usually blacklists write something about their policy on their web sites, so maybe you can find out if this is the case.
