HI, I am without resolve on this one, out of the blue, one of my servers thats been running for weeks now, suddenly become almost impossible to log into. the login process from the consile / ssh and even all ports 110 ans 25 seem tot timeout to a couple minutes!! My initial thoughts was that his server run out of some resource or another. however I can not detect the resource nor the process in error. By using top i can indicate an running system - althow slow it seem to function top - 14:25:40 up 26 min, 2 users, load average: 115.32, 103.99, 69.81 Tasks: 307 total, 2 running, 289 sleeping, 0 stopped, 16 zombie Cpu(s): 20.9%us, 1.3%sy, 0.0%ni, 0.0%id, 77.2%wa, 0.3%hi, 0.3%si, 0.0%st Mem: 515896k total, 510960k used, 4936k free, 1416k buffers Swap: 1510068k total, 1478364k used, 31704k free, 13020k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5264 nyl-cass 17 0 27508 2084 584 D 8.6 0.4 0:13.59 clamscan 6712 top-henn 17 0 10192 8512 1616 D 7.3 1.6 0:00.44 spamassassin 4638 kliek-au 17 0 27244 1792 572 R 2.0 0.3 0:08.93 clamscan 6484 lim-lind 16 0 14908 9084 1680 D 1.7 1.8 0:00.69 spamassassin 5583 root 16 0 2580 1232 788 R 0.7 0.2 0:06.39 top 6414 nyl-flip 16 0 17628 12m 1728 D 0.7 2.4 0:00.70 spamassassin 6514 am-ambro 16 0 18284 10m 1752 D 0.7 2.0 0:00.76 spamassassin 6759 nyl-bouw 16 0 10208 8516 1616 D 0.7 1.7 0:00.41 spamassassin 6767 djd-joha 16 0 10204 8512 1616 D 0.3 1.6 0:00.40 spamassassin If i reboot the server with Code: shutdown -r now It take 30-40 minutes ands some services like MYSQL fail to stop, eventually it seem to reboot. If i stop postfix, no change. What can I test for, how can I seek the problem. I had this issue with an smaller server (mandriva+mailbox), replaced it eventually with (ubuntu 7.04+maildir), but I cant keep on re-building and replacing servers, instead of repairing some missbehaving part.
1) Do not write in big red letters. This might lead the one or other user to not answer to your post. We are all able to read the normal small black letters 2) The problem is that your server receives to much mail for the currently installed spam scanning and antivirus solution with clamscan and spamassassin. Check your mail log if you get higher email traffic, e.g. a spam.
I have done an Code: shutdown -r now and let it do its thing - analysing the top on an terminal and looking at the logfiles to see what processes are closing off, obvious that the once the process causing the issue shuts down the re-boot will speed-up. All the services stopped, took loooong time from 15:13 to last entry in syslog at 16:01, almost an hour (48 minutes!) the last standing processes was called from ISPConfig clamscan and authdaemond All the time the HDD seem to run non-stop!
Spam Oops big letters came as I did not preview, ancase fixed. I have received the same mail 3 x during the day, normally indicating pop3 disconnectin and huge mail on bad lines - however this case i have <1ms latency to server on same LAN segment. Thus I was thinking an re-lapse in the bug of clam were it re-serviced already scanned mails over and over again! This last scenario seem more likely as It takes AGES to work through the mail Q, Yes it could be the case - what is the solution? this has not happened before on this site as I can indicate with Stats this is an Small site! Normal last week this time's figures. Grand Totals ------------ messages 2619 received 2787 delivered 67 forwarded 4 deferred (29 deferrals) 2 bounced 2268 rejected (44%) 0 reject warnings 0 held 0 discarded (0%) 237532k bytes received 259623k bytes delivered 838 senders 449 sending hosts/domains 210 recipients 36 recipient hosts/domains Today's figures (early edition 2/3 day) Grand Totals ------------ messages 1443 received 1501 delivered 26 forwarded 7 deferred (7 deferrals) 39 bounced 704 rejected (31%) 0 reject warnings 0 held 0 discarded (0%) 146215k bytes received 133408k bytes delivered 508 senders 273 sending hosts/domains 191 recipients 42 recipient hosts/domains I am convinced that Sumething "Broke" either an update came in or some other misshap. as the traffic to the server seem to be normal etc.
I left the server to work off the load, as another server of mine seem to have had the same issue was re-booted and that sorted it. This is for sure and spam attack of some sort! it kills clamscan and spamassassin, eventually mysql process gets killed as the server run out of memory! and all get grinded to an halt! I still have no idea how to get rid of this, were to find the "queue to scan as i stop both postfix and ispconfig_server, clamscan keep on spawning scans. Were is that backlog, how do i remove the heaped-up mail to be scaned! It also seem to scan the same mail over and over, as i got some mail 4-5x already. DARn!
First, clamscan will stop spawning when you stop postfix. But it may take some time, mostly about 5 - 10 minutes. The emails to be scanned are most likely in the mail queue. You can manage the mailqueue with the postqueue and postsuper commands. You should try this to get down the load from clamav: http://www.howtoforge.com/forums/showthread.php?t=12860 clamd is much faster then the current clamscan solution. Sinvce the clamav developers introduced clamav 0.90.x, the antivirus scanning is producing a much higher load. the problem seems to be known according to the clamav mailing lists and the clamav developers recommend to use clamd. There is also some patch which I hope will be integrated in some of the next clamav releases.
Short HOWTO switching to clamd/clamdscan in ISPConfig My Set-Up consist of: ubuntu 7.04 and ISPConfig Version: 2.2.13 Code: apt-get install clamav-daemon inside /home/admispconfig/ispconfig/tools/clamav/bin/clamassassin change the CLAMSCAN line to Code: CLAMSCAN=/usr/bin/clamdscan inside /etc/clamav/clamd.conf check the line ScanMail Code: ScanMail true inside /etc/clamav/freshclam.conf AND also in /home/admispconfig/ispconfig/tools/clamav/etc/freshclam.conf check NotifyClamd Code: NotifyClamd /etc/clamav/clamd.conf Short and sweet!
The Fix/solution It seems that clamscan could never eat the chunks of mail sent to it, the server become 100% unresponsive and login took hours. clientsatisfaction became Zero, while you see the faxes arriving from customers cancelling the service and shouting "nOOb" all over. I Backed up the old server after I managed to quickly stop services postfix and ispconfig_server as well as killed all processes to do with clamscan (It dint go away after even 30 minutes as predicted by Till) It took me exactly 3H15 minutes from the moment I rebooted on the CD to re-install the server till up and running with "Perfect ubuntu 7.04", ISPConfig and Backup restored / user passwd/shadow/group/gshadow files fixed. However only to be confronted with Exactly the same problem! At that stage Till's feedback arrived and I was confused, he told me to edit the scritpt /home/admispconfig/ispconfig/tools/clamav/bin/clamassassin and fix it to work with clamd/clamdscan. Well it took me to again untar the ISPConfig installation files and search through install_ispconfig directory and found the README for clamassassin, in there was an nice indication of what to do. The lessons learned here is that it does not always help to use the "Microsoft Fix" i.e. - Reload, but rather seek the real issue, then start digging/googling.
Hi, Thanks for the great post. I followed the instructions, however, i can't seem to get clamdscan to work on Centos 4.4. Apparently, after yum install, /usr/bin/clamdscan does not exist. I found that executable file in ISPCONFIG's directory, but when i use that path, clamdscan doesn't seem to get invoked. Could you point me in the right direction?
Switching to CLAMD on Centos 4.x with Ispconfig 2.2.13 hi guys, here's what i found. In Centos 4.x, there's no need to do an yum install of clamav-daemon. clamd executable is available in /home/admispconfig/ispconfig/tools/clamav/sbin/clamd Therefore, following Morons's instructions above, we just need to do the follows: inside /home/admispconfig/ispconfig/tools/clamav/bin/clamassassin change the CLAMSCAN line to Code: CLAMSCAN=/home/admispconfig/ispconfig/tools/clamav/bin/clamdscan inside /home/admispconfig/ispconfig/tools/clamav/etc/clamd.conf check the line ScanMail Code: ScanMail true inside /home/admispconfig/ispconfig/tools/clamav/etc/freshclam.conf check the line NotifyClamd Code: NotifyClamd /home/admispconfig/ispconfig/tools/clamav/etc/clamd.conf Thanks again Moron =)
Have you tested your solution with the eicar virus test pattern and does it really filter viruses? You need a running clamd daemon for this and I do not see where you configuare and start clamd automatically as ISPConfig does not ahs a clamd daemon.
Carefull cjc81, Althow there is an clamd script installed with the ispconfig (comes with clamassasin) it require and clamd daemon to be running. Code: updatedb locate clamd ps ax |grep clamd check whether the daemon is running, as till sais use eicar test! If not running switch on the yum installed clamd daemon and change the config to use it according to the locate you did above.
I would like to ask ISPConfig to perhaps consider using clamd instead of clamscan by default. I only relised now that I had to upgrade hardware on 2 other installs before this that would have been unnessesary if we switched! Maybe add an option to switch the two methods.
Yes sir, i tried the test pattern and the mail got filtered out. (ie. its working properly) Also, clamd daemon is running. Do i need to add that to chkconfig to make sure that clamd daemon is running when system starts? I am also monitoring my running processes via "top". I only see clamd coming active there are mail activities.
Yes this is were the issue comes in. Currently you have 2 installations of clamd, one installev via ISPC / clamassasin part and one using yum, Also 2 x freshclam. each updating its database residing in diferent places. Now, that being the case the versions might be diffrent! also make sure the you inspect the headers of an mail comming through to see the scan successfull! I have this in my header. The No mean there is no virus found! Code: X-Virus-Status: No X-Virus-Checker-Version: clamassassin 1.2.4 with clamdscan / ClamAV 0.90.2/3334/Thu May 31 10:54:25 2007 Not all distro's have the chkconfig command, it seems to be maintained by the "redhat" derivatives and yes to make sure the yum installed version is running use Code: chkconfig clamd on The ISPC version will not start automatically, if my analysis is correct. Till can perhaps comment on this.
I was wondering if i could copy /home/admispconfig/ispconfig/tools/clamav/sbin/clamd to /etc/init.d then add it to chkconfig to get it to run when system start?
This will not work because in /etc/init.d are the start scripts for the daemins and not the daemin binaries itself. If you want to run the clamd daemon /home/admispconfig/ispconfig/tools/clamav/sbin/clamd, you will have to craete a start script for it in /etc/init.d.
Hi all, I post my little contribution on how I did on my FC5 server first I followed the steps posted by cjc81 (thanx man) then : I enabled logging in /home/admispconfig/ispconfig/tools/clamav/etc/clamd.conf Code: LogFile /var/log/clamd.log I created and changed the owner of the log file to admispconfig Code: touch /var/log/clamd.log chmod admispconfig.admispconfig /var/log/clamd.log I've put a starting script in /etc/init.d/clamd that I hacked to run the clamd from ispconfig Code: #!/bin/sh # # clamd Script to start/stop clamd. # # chkconfig: - 61 39 # description: clamd is an antivirus daemon. # # processname: clamd # config: /home/admispconfig/ispconfig/tools/clamav/etc/clamd.conf # pidfile: /var/run/clamd.pid # # Source function library . /etc/rc.d/init.d/functions # Get network config . /etc/sysconfig/network # test -f /home/admispconfig/ispconfig/tools/clamav/etc/clamd.conf || exit 0 # RETVAL=0 # start() { echo -n $"Starting Clam AV daemon: " daemon /home/admispconfig/ispconfig/tools/clamav/sbin/clamd RETVAL=$? echo [ $RETVAL -eq 0 ] && touch /var/lock/subsys/clamd return $RETVAL } # stop() { echo -n $"Stopping Clam AV daemon: " killproc clamd RETVAL=$? echo [ $RETVAL -eq 0 ] && rm -f /var/run/clamd.pid /var/lock/subsys/clamd return $RETVAL } # restart() { stop start } reload() { echo -n $"Reloading DB: " killproc clamd -USR2 RETVAL=$? echo return $RETVAL } # case "$1" in start) start ;; stop) stop ;; status) status clamd ;; restart) restart ;; condrestart) [ -f /var/lock/subsys/clamd ] && restart || : ;; reload) reload ;; *) echo $"Usage: $0 {start|stop|status|restart|condrestart|reload}" exit 1 esac # exit $? and execute Code: chmod 700 /etc/init.d/clamd chkconfig --add clamd chkconfig --level 2345 clamd on service clamd start
