Hey there So as the title says, clamav-daemon keeps crashing on my server. and when it crashes, ***UNCHECKED*** gets appended to the subject line in outgoing email messages. The logs don't report anything, the just have database checks up until the point of crashing. This is the most recent status on the daemon: Any ideas as to why this keeps happening, or what else to check.
Try starting clamav-daemon to get error messages why it fails to start. Code: systemctl start clamav-daemon.service
Hi, apologies for waiting so long, the issue isn't that it doesn't start, it's that it starts, and in roughly a month's time, it stops, and I'm unable to trace why. The daemon starts without an error code.; although, while running, there is this line in the status check: Code: Process: 22027 ExecStartPre=/bin/mkdir /run/clamav (code=exited, status=1/FAILURE) The part in brackets appears in red. Is this a problem?
If you do not care about this occasional crashing, restart clamav every week. If you want to find out why crashing happens, increase log level and wait for the crash to happen. That hopefully gives more info on what happens.
Hello again, so I'm not sure i increased the appropriate log level or whether i'm looking at the appropriate logs, but there are still no errors in the logs for clamav-daemon. Are these the appropriate levels to catch any errors: Code: 7 4 1 7 I am seeing memory allocation errors when i checked the status after a recent crash. clamav-daemon status: Code: ● clamav-daemon.service - Clam AntiVirus userspace daemon Loaded: loaded (/lib/systemd/system/clamav-daemon.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/clamav-daemon.service.d └─extend.conf Active: failed (Result: signal) since Tue 2019-09-24 11:36:08 UTC; 1h 20min ago Docs: man:clamd(8) man:clamd.conf(5) https://www.clamav.net/documents/ Process: 5016 ExecStart=/usr/sbin/clamd --foreground=true (code=killed, signal=KILL) Process: 5011 ExecStartPre=/bin/chown clamav /run/clamav (code=exited, status=0/SUCCESS) Process: 5008 ExecStartPre=/bin/mkdir /run/clamav (code=exited, status=1/FAILURE) Main PID: 5016 (code=killed, signal=KILL) Sep 24 11:12:04 opensrvr01 clamd[5016]: LibClamAV Warning: fmap: map allocation failed Sep 24 11:12:04 opensrvr01 clamd[5016]: LibClamAV Error: CRITICAL: fmap() failed Sep 24 11:12:04 opensrvr01 clamd[5016]: Tue Sep 24 11:12:04 2019 -> ~/var/lib/amavis/tmp/amavis-20190923T160521-02854-_8g63HPN/parts/p001: Can't allocate memory ERROR Sep 24 11:12:04 opensrvr01 clamd[5016]: LibClamAV Warning: fmap: map allocation failed Sep 24 11:12:04 opensrvr01 clamd[5016]: LibClamAV Error: CRITICAL: fmap() failed Sep 24 11:12:04 opensrvr01 clamd[5016]: Tue Sep 24 11:12:04 2019 -> ~/var/lib/amavis/tmp/amavis-20190923T181201-06827-x0LHl9d9/parts/p001: Can't allocate memory ERROR Sep 24 11:36:03 opensrvr01 clamd[5016]: LibClamAV Warning: fmap: map allocation failed Sep 24 11:36:08 opensrvr01 systemd[1]: clamav-daemon.service: Main process exited, code=killed, status=9/KILL Sep 24 11:36:08 opensrvr01 systemd[1]: clamav-daemon.service: Unit entered failed state. Sep 24 11:36:08 opensrvr01 systemd[1]: clamav-daemon.service: Failed with result 'signal'.
I second @florian030 . Add swap partition or add more RAM. See memory situation with command Code: free -h Related recent thread: https://www.howtoforge.com/community/threads/email-delivery-very-slow.82875/
free -h reports: Code: total used free shared buff/cache available Mem: 2.0G 1.5G 85M 164M 399M 182M Swap: 0B 0B 0B Definitely going to look into more RAM, but i'm unable to right now. I will look into adding a swap partition for now and see if that solves the issue
I have the same issue except its daily when Freshclam gets a new update. I believe it to be a lack of RAM but paradoxically increasing RAM is not the solution. I have five ISPConfig servers installed using the same Perfect Server tutorial so should be identical. The issue only occurs on two - a lightly loaded 2Gb system and a medium 4Gb. I shouldn't need 4Gb as the other servers are happy with 2Gb. What is characteristic of the two servers is that when running normally the free RAM is always significantly smaller than the other systems so, I guess, when update time occurs the clamav-daemon just runs out of RAM and is killed. I have to assume it was my user error when installing ISPConfig which makes these two servers misbehave. But I can't find what I might have done wrong. Current workaround is crontab restarting clamav-daemon daily at 13:10. (Freshclam appears to randomly update just after 12:00 or 13:00). Servers are Debian 10/Apache.
What about swap? Is there swap and how much is it used? If you have 2 GB ram and 4 GB swap, it is 6 GB virtual memory space for the system.
Seemingly identical systems work flawlessly on 2GB, no swap. If I need more than 4GB something is very wrong but I can't put my finger on it. I have VMSTed TOPed etc but I can't find where the memory is leaking.
What are the logs telling you why clamd exits? It could be that freshclam needs some additional ram when doing the update. Since you don't have swap, the system kills the process it thinks it should to free memory.
code=killed, status=9/KILL But the point I'm making is that something is soaking up any memory available on two servers out of five. A 2GB & 4GB run out of memory for 2 x 2GB and a 1 x 4GB do not. So increasing RAM/swap isn't the solution. Finding and stopping the memory leak is. The Perfect Server setup for all five servers should be identical. The probable solution is I did something wrong on these two - but there is no obvious app swallowing more on the two servers - but the amount of Free memory under identical conditions is significantly less than the good servers so its disappering somewhere. ClamAV & Freshclam are playing the role of the canary in the coal mine methinks. Note that restarting clamav-daemon on fail works instantly.
I've started having this problem too. Code: $ free -h total used free shared buff/cache available Mem: 3.8G 1.7G 347M 702M 1.8G 1.2G Swap: 511M 511M 0B Happens around once a month. Will dig a little more. Ubuntu 16.04.6 LTS
this is probably more useful, if late, for brainsys. but still applicable to anyone else. the perfect sever install may well be identical. but the users files on each server aren't. if you have a webserver with postfix to send out website sourced mail, and no mailboxes, and a mailserver with no websites but lots of mailboxes, they're going to have very different load patterns and memory usage. and the amount of memory that clamav will need for itself when doing a full scan is going to be completely different on each server. add a mailserver experiencing a slightly higher load than normal due to user activity, a very recently updated virus library, and a full clamav scan all coinciding at the same time and say goodbye to all your free ram.... also it's debatable if a swap partition is helpful, at least on a vps. you're trying to write memory contents to a disk partition that is itself a file on a disk partition. that could just be too slow, more processes get backed up waiting... more ram gets needed, more swap gets used, more slowness..... could be a vicious circle that just takes out your vps. @tfboy, when did you run free -h? if it's after the processes have already been killed it's too late, if it's while your system is working ok, it's too early. to properly identify any culprits, you'll need something that will regularly monitor your system and alert you before the critical point. then you may be able to see some useful information. cosider looking at munin, monit, Nagios, icinga and many others like those.
Well, your swap was full. So the system killed some process to free up ram (otherwise the system would have crashed). Swapping in general is bad for performance. Reduce your memory footprint (find out which program uses all your memory) or upgrade your server with more memory. If you don't care about performance, just increase your swap.
I'm not too familiar with the swap management but as I had 1.2G of RAM free, should this be an issue? I can force a purge of the swap by turning off and back on again, but should I have to do that?
At the time you made the "free" command, you had 1.2G RAM free. However, swap is only used when necessary (when RAM is full). When your swap is full too, your system kills applications to free some RAM. Usually, swap is not moved back to RAM if not needed. So your swap being full suggests, that in the past all your RAM+SWAP was completely used and your system hat to kill applications to keep the system running. This freed up some ram, but killed your clamav. So the question is how to prevent this in the future: 1) Install more ram 2) Decrease ram needed by applications (you need to investigate which applications used up your ram and why. Maybe you can decrease the needed memory or have a memory leak..) 3) Increase swap. If swap does not get full, no applications will be killed. However, your system performance will degrade
If the problem is running out of memory. Purging swap does not help. You wrote , but that was when you ran free -h. Previously the host may have had memory completely used and so swap was used to help that.
