clamav / clamscan high CPU usage spikes and debugging

Discussion in 'ISPConfig 3 Priority Support' started by curiousadmin, Oct 13, 2019.

  1. curiousadmin

    curiousadmin Member HowtoForge Supporter

    Hello Team,
    After lessons learned on clamav being RAM hungry the best I could do with the budget provided was to increase the swap so the server now has 2GB RAM and 4 GB swap (before was 2+2).

    So now the server is running with 2+4 GB of RAM but I see surge in CPU usage and looking at htop it's clamscan:
    Uptime: 01:02:50
      PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
    23277 amavis     20   0  848M  475M   932 R 15.2 24.1  0:57.60 /usr/bin/clamscan --stdout --no-summary -r --tempdir=/var/lib/amavis/tmp /var/lib/amavis/tmp/amavis-20191013T191629-01876-1HU_EX2v/parts
    23381 amavis     20   0  375M  286M   608 R 69.7 14.5  0:15.21 /usr/bin/clamscan --stdout --no-summary -r --tempdir=/var/lib/amavis/tmp /var/lib/amavis/tmp/amavis-20191013T191803-01877-IKHIsAjj/parts
    Since the server was before clearly RAM&swap exhausted and eventually probably even gave up on checking (emails started to come in as ***UNCHECKED***) I did shutdown, increased the SWAP and now nail biting watching the CPU usage spike.

    What is the default configuration (or behavior) of clamav? Does it now try to catch up scanning on the previously unscanned emails or what is going on?

    Is there any way to check the progress of the clamscan and to see what is it actually doing?

    Thank you all very much in advance.
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    The problem you have is that clamav server is not running. That's why the slower clamscan is used an emergency fallback. Start the clamav server to fix it.

    It scans an incoming or outgoing email for viruses, you can see incoming and outgoing emails in the mail.log file.
  3. curiousadmin

    curiousadmin Member HowtoForge Supporter

    Hi till,
    thank you very much for the super fast followup.

    I did:
    /etc/init.d/clamav-daemon start
    But a few seconds later it died again after checking:
    /etc/init.d/clamav-daemon status
    Which is what I was suspecting - the server was restarted multiple times before - no way it would run out of memory in a few hours.

    The output from that status message actually pointed to the issue:
    Oct 14 06:48:14 clamd[15489]: Mon Oct 14 06:48:14 2019 -> !LOCAL: Socket file /var/run/clamav/clamd.ctl could not be bound: Permission denied
    With help of Taleman's post I narrowed it down to issue with permission as the file did not exist:
    # ls -l /var/run/clamav
    total 0
    Fixed the permissions:
    # chown clamav:root /var/run/clamav
    After starting the service all good again (20+ minutes):
    # systemctl start clamav-daemon.service
    # systemctl clamav-daemon status
    Is it fair to assume that the issue was with permission?

    Should I put on my tinfoil hat assuming somebody tried to mess with the server
    or the permission issue happens from time to time with server updates?
    (apt-get update && apt-get dist-upgrade)
    The server has been installed for a while - it's been installed in early 2017 (and still running Debian 8)
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    I don't think that your server has been hacked, probably an issue with a package update or something similar. When clamav daemon is working again, then you should try to restart amavis too.

    Amavis should start using the clamav daemon then as it uses way fewer resources, the processes should read 'clamdscan' then instead of 'clamscan', clamdscan is the program which helps amavis to communicate with the clamav daemon process while clamscan is just clamav without daemon process.

Share This Page