Using CentOS 7.4 perfect server guide. This issue with clamd was occurring before upgrade from 7.4 to 7.5. Have found that clamd is consuming memory under the amavis user. Currently memory consumption of clamd is 30% and has been bouncing up and down, but stays high rather than low. Server uptime is 4 days. I cannot run the server for more than 7 or 8 days without having out of memory issues in the logs, and the only item that seems to consistently consume excessive amounts of memory is clamd. If it starts having out of memory issues in the logs it will eventually go till it brings most of the server down. This system has 2GB ram, 2 cores, and SSD based storage - it's a VM on a 3rd party provider. CPU generally is not high utilization, though there are some spikes. I have found that if I kill the clamd process, then I free up enormous amounts of memory and can extend the server uptime. I would like to keep running clamd for mail scanning, but need to keep the server and email running stable. I have already modifying settings per this with little success in resolving this: https://www.centos.org/forums/viewtopic.php?t=45552 Here's a little from my logs where I log top output with it organized per Mem: Code: Sun Aug 12 14:39:02 EDT 2018 top - 14:39:02 up 4 days, 4:32, 1 user, load average: 0.76, 0.58, 0.48 Sun Aug 12 14:39:02 EDT 2018 Tasks: 177 total, 5 running, 172 sleeping, 0 stopped, 0 zombie Sun Aug 12 14:39:02 EDT 2018 %Cpu(s): 23.1 us, 30.8 sy, 0.0 ni, 3.8 id, 19.2 wa, 0.0 hi, 0.0 si, 23.1 st Sun Aug 12 14:39:02 EDT 2018 KiB Mem : 1015296 total, 199156 free, 546440 used, 269700 buff/cache Sun Aug 12 14:39:02 EDT 2018 KiB Swap: 1048572 total, 267592 free, 780980 used. 260472 avail Mem Sun Aug 12 14:39:02 EDT 2018 Sun Aug 12 14:39:02 EDT 2018 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND Sun Aug 12 14:39:02 EDT 2018 8031 amavis 20 0 742752 351664 0 S 0.0 34.6 0:00.00 clamd Sun Aug 12 14:39:02 EDT 2018 8041 web1 20 0 514352 67788 50052 S 0.0 6.7 0:01.20 php-cgi Sun Aug 12 14:39:02 EDT 2018 3138 web1 20 0 333096 20552 1648 S 0.0 2.0 0:01.70 otrs.Daemo+ Sun Aug 12 14:39:02 EDT 2018 8108 root 20 0 490860 17120 10820 R 31.2 1.7 0:00.05 php Sun Aug 12 14:39:02 EDT 2018 8094 root 20 0 490860 17116 10820 R 37.5 1.7 0:00.06 php Sun Aug 12 14:39:02 EDT 2018 1351 mysql 20 0 1653960 15412 1808 S 0.0 1.5 19:58.02 mysqld Sun Aug 12 14:39:02 EDT 2018 4542 web1 20 0 324320 8788 1232 S 0.0 0.9 0:01.50 otrs.Daemo+ Sun Aug 12 14:39:02 EDT 2018 4284 root 20 0 154680 6692 6556 S 12.5 0.7 19:45.12 systemd-jo+ Sun Aug 12 14:39:02 EDT 2018 6165 web1 20 0 322032 6284 1004 S 0.0 0.6 0:01.42 otrs.Daemo+ Sun Aug 12 14:39:02 EDT 2018 4550 web1 20 0 321904 5220 840 S 0.0 0.5 0:01.41 otrs.Daemo+ Sun Aug 12 14:40:01 EDT 2018 top - 14:40:01 up 4 days, 4:33, 1 user, load average: 0.56, 0.58, 0.49 Sun Aug 12 14:40:01 EDT 2018 Tasks: 186 total, 10 running, 176 sleeping, 0 stopped, 0 zombie Sun Aug 12 14:40:01 EDT 2018 %Cpu(s): 33.3 us, 55.6 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 11.1 st Sun Aug 12 14:40:01 EDT 2018 KiB Mem : 1015296 total, 74120 free, 662512 used, 278664 buff/cache Sun Aug 12 14:40:01 EDT 2018 KiB Swap: 1048572 total, 273736 free, 774836 used. 141572 avail Mem Sun Aug 12 14:40:01 EDT 2018 Sun Aug 12 14:40:01 EDT 2018 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND Sun Aug 12 14:40:01 EDT 2018 8031 amavis 20 0 742752 351664 0 S 0.0 34.6 0:00.00 clamd Sun Aug 12 14:40:01 EDT 2018 8041 web1 20 0 514664 68176 50056 S 0.0 6.7 0:01.42 php-cgi Sun Aug 12 14:40:01 EDT 2018 8227 web3 20 0 452416 45060 14696 S 0.0 4.4 0:00.53 php-cgi Sun Aug 12 14:40:01 EDT 2018 8222 web3 20 0 450520 41740 14772 S 0.0 4.1 0:01.05 php-cgi Sun Aug 12 14:40:01 EDT 2018 3138 web1 20 0 333100 20632 1652 S 0.0 2.0 0:01.77 otrs.Daemo+ Sun Aug 12 14:40:01 EDT 2018 1351 mysql 20 0 1653960 18272 2480 S 0.0 1.8 19:58.25 mysqld Sun Aug 12 14:40:01 EDT 2018 4284 root 20 0 154680 11560 11420 R 35.3 1.1 19:45.42 systemd-jo+ Sun Aug 12 14:40:01 EDT 2018 8308 root 20 0 340056 10180 6896 D 17.6 1.0 0:00.03 php Sun Aug 12 14:40:01 EDT 2018 4542 web1 20 0 324320 8788 1232 S 0.0 0.9 0:01.58 otrs.Daemo+ Sun Aug 12 14:40:01 EDT 2018 6165 web1 20 0 322032 6696 1304 S 0.0 0.7 0:01.55 otrs.Daemo+ Sun Aug 12 14:41:01 EDT 2018 top - 14:41:01 up 4 days, 4:34, 1 user, load average: 0.50, 0.60, 0.50 Sun Aug 12 14:41:01 EDT 2018 Tasks: 183 total, 4 running, 179 sleeping, 0 stopped, 0 zombie Sun Aug 12 14:41:01 EDT 2018 %Cpu(s): 24.1 us, 37.9 sy, 0.0 ni, 13.8 id, 13.8 wa, 0.0 hi, 0.0 si, 10.3 st Sun Aug 12 14:41:01 EDT 2018 KiB Mem : 1015296 total, 66032 free, 742304 used, 206960 buff/cache Sun Aug 12 14:41:01 EDT 2018 KiB Swap: 1048572 total, 249672 free, 798900 used. 64232 avail Mem Sun Aug 12 14:41:01 EDT 2018 Sun Aug 12 14:41:01 EDT 2018 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND Sun Aug 12 14:41:01 EDT 2018 8031 amavis 20 0 742752 310084 0 S 0.0 30.5 0:00.00 clamd Sun Aug 12 14:41:01 EDT 2018 8227 web3 20 0 482044 65000 6856 S 0.0 6.4 0:01.83 php-cgi Sun Aug 12 14:41:01 EDT 2018 8222 web3 20 0 480472 63276 6848 S 0.0 6.2 0:02.44 php-cgi Sun Aug 12 14:41:01 EDT 2018 8041 web1 20 0 515040 62200 43812 S 0.0 6.1 0:01.73 php-cgi Sun Aug 12 14:41:01 EDT 2018 8569 web3 20 0 440084 32236 5124 S 0.0 3.2 0:00.61 php-cgi Sun Aug 12 14:41:01 EDT 2018 1351 mysql 20 0 1653960 31956 2388 S 0.0 3.1 19:58.76 mysqld Sun Aug 12 14:41:01 EDT 2018 3138 web1 20 0 333080 20608 1648 S 0.0 2.0 0:01.83 otrs.Daemo+ Sun Aug 12 14:41:01 EDT 2018 8649 root 20 0 396084 11896 7948 R 25.0 1.2 0:00.04 php Sun Aug 12 14:41:01 EDT 2018 8639 root 20 0 391568 11716 7828 R 25.0 1.2 0:00.04 php Sun Aug 12 14:41:01 EDT 2018 4542 web1 20 0 324320 8768 1212 S 0.0 0.9 0:01.68 otrs.Daemo+ I was thinking a script to run daily to restart amavsid. Looking for some expert guidance. What do you think? Recommendations?
You wrote But top output included shows the host has 1 GB. Have you followed swap usage? If also swap is full host really is out of memory. You can make larger swap, that should get rid of the out of memory condition. If 100% uptime is not required, create cron job to reboot nightly.
Taleman, I was mistaken, it indeed has 1GB. Nightly reboots are not an option due to the fact there is some sort of issue that causes the Kernel to crash at reboot about once out of 7 or 8 reboots; I have a thread about it here: https://www.centos.org/forums/viewtopic.php?f=47&t=67928 I will look into swap, thanks.
Well, looks like I'm using quite a bit of swap and that I'm hitting swap pretty regular. So def a mem issue. The concern though is if I add more memory, is that going to fix this or is clamd just going to consume more and more memory? It seems it's consuming more and more memory due to the fact that it has no issue for 5 days. I killed it yesterday and it has remained low so far. I'll wait a few more days and see if it starts creeping up again. How much memory does clamd consume on your system after running for 10+ days? Code: # free total used free shared buff/cache available Mem: 1015296 599536 124544 84568 291216 158764 Swap: 1048572 987340 61232 # vmstat 3 100 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 1011976 124084 0 364300 35 20 441 100 27 23 2 1 97 0 1 7 3 1011976 111888 0 368712 21 0 1577 1 415 373 2 2 96 0 1 0 0 1013056 75880 0 430732 128 359 173613 5453 1474 1296 5 11 75 6 4 0 0 1013056 114588 0 391140 304 0 476 592 490 515 4 2 93 1 1 0 0 1013056 116172 0 389556 0 0 0 0 192 167 0 0 100 0 0 0 0 1013056 116172 0 389560 0 0 0 1 184 166 0 0 99 0 1 0 0 1013056 116172 0 389560 0 0 0 0 198 177 0 0 100 0 0 0 0 1013056 116048 0 389848 0 0 96 0 226 206 0 0 99 0 1 0 0 1013056 115924 0 389848 0 0 37 70 222 205 0 0 100 0 0 0 0 1013056 113692 0 391560 0 0 571 0 301 256 1 0 98 0 1 0 0 1013056 113612 0 391560 0 0 5 0 260 234 1 0 99 0 0 0 0 1013056 113596 0 391560 0 0 0 0 220 198 0 0 100 0 0 0 0 1013056 113612 0 391508 21 0 21 5 234 205 0 0 100 0 0 0 0 1013056 113484 0 391684 0 0 0 600 260 334 0 0 99 0 0 0 0 1013056 113508 0 391684 0 0 0 0 237 213 0 0 99 0 0 0 0 1013056 116452 0 391860 0 0 77 1 232 208 0 0 100 0 0 0 0 1013056 116220 0 391908 0 0 52 0 345 346 2 0 98 0 0 0 0 1013056 116220 0 391908 0 0 0 0 230 228 0 0 99 0 0 0 0 1013056 116220 0 392060 0 0 56 17 234 212 0 0 100 0 0 0 0 1013056 116220 0 392060 0 0 0 0 227 215 0 0 100 0 0 0 0 1013056 114112 0 394232 11 0 735 4 429 433 3 1 96 0 0 4 0 1013056 91112 0 406096 0 0 9997 1 962 1110 7 5 87 0 1 0 0 1013056 98852 0 405964 0 0 24 1 371 452 1 1 98 0 0 0 0 1013056 99056 0 405936 0 0 1 505 452 440 5 1 93 0 0 0 0 1013056 89008 0 411888 0 0 1833 1 620 524 2 1 96 0 1 0 0 1013056 81788 0 413192 0 0 0 1024 249 322 0 0 99 0 0 0 0 1013056 85972 0 413248 27 0 197 82 332 287 1 1 98 0 0 0 0 1012800 84868 0 413292 53 0 112 39 332 296 1 0 99 0 0 0 0 1012800 87228 0 413072 0 0 75 189 263 243 0 0 99 0 0 0 0 1012800 93912 0 412608 0 0 0 0 309 240 2 0 98 0 0 0 0 1012800 93928 0 412604 0 0 0 0 280 271 1 0 99 0 0 3 0 1012800 93804 0 412600 43 0 65 0 260 229 0 0 99 0 0 0 0 1012800 89440 0 413024 0 0 116 33 368 300 0 0 99 0 1 0 0 1012800 89068 0 413024 11 0 11 0 268 252 0 0 99 0 0 0 0 1011264 94784 0 412588 137 0 137 0 271 230 1 1 98 0 0 0 0 1011264 94548 0 412856 0 0 89 1 222 200 0 0 100 0 0 0 0 1011264 94548 0 412856 0 0 0 0 252 231 1 0 99 0 0 0 0 1011264 69516 0 424852 81 0 2195 287 638 647 3 1 94 1 1 0 0 1011264 78312 0 380632 43 0 1643 387 582 498 4 2 93 0 1 0 0 1011264 79256 0 379688 0 0 0 0 237 220 0 0 100 0 0 0 0 1010496 81636 0 379532 52 0 56 0 467 445 3 1 95 0 0 0 0 1010496 81292 0 379532 0 0 0 1 295 266 1 0 99 0 1 0 0 1010496 94696 0 365820 0 0 5 65 751 895 7 5 87 0 1
I think it mainly depends on how high is your mail trafiic. If it is not, you can try reading and implement this idea. Also, I think for heavy mail user, having a mail service together with web and other services sometimes may not be a very good idea. You can try separating your mail server from the rest and set it up to be on its own. ISPConfig 3.1 is ever ready for multi server setup. Or you can use third party mail server too.
Memory used on my system. Testing on Debian GNU/Linux 9.5, ISPConfig 3.1.11. Host is virtual machine with 6 CPU and 12 G memory. Code: oot@ispchost:/var/www/clients# uptime 12:49:05 up 7 days, 2:20, 1 user, load average: 0,83, 1,11, 1,06 root@ispchost:/var/www/clients# free total used free shared buff/cache available Mem: 12307736 3643032 1856032 480232 6808672 7862356 Swap: 2096124 0 2096124 root@ispchost:/var/www/clients# Host has somewhat slow disk system, I compensate by adding ample memory so most reads happen from cache. Code: root@ispchost:/var/www/clients# ps -ef | egrep "(clam|STIME)" UID PID PPID C STIME TTY TIME CMD clamav 1022 1 0 elo08 ? 00:01:32 /usr/bin/freshclam -d --foreground=true clamav 1045 1 0 elo08 ? 00:06:57 /usr/sbin/clamd --foreground=true root 27318 1824 0 12:52 root@ispchost:/var/www/clients# top -p 1045 -p 1022 -n 2 top - 13:10:06 up 7 days, 2:41, 1 user, load average: 1,28, 0,97, 1,01 Tasks: 2 total, 0 running, 2 sleeping, 0 stopped, 0 zombie %Cpu(s): 15,4 us, 2,7 sy, 0,0 ni, 75,8 id, 6,0 wa, 0,0 hi, 0,2 si, 0,0 st KiB Mem : 12307736 total, 1879116 free, 3638196 used, 6790424 buff/cache KiB Swap: 2096124 total, 2096124 free, 0 used. 7897552 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1045 clamav 20 0 980676 587524 32044 S 0,0 4,8 6:57.93 clamd 1022 clamav 20 0 152876 26516 21672 S 0,0 0,2 1:32.20 freshclam
It's not mail heavy, but there is semi-regular mail flowing, so I have setup Cron jobs to restart several services to see if that helps. If it does not, I will try the guide you provided ahrasis, thank you. That said, I still may have to bite the bullet and increase mem; I'm hesitant to because the cost doubles for the server. I'm curious on adding another server, however I'm not clear on best method or how it would avoid memory issues? Would I follow the perfect server guide, install all same services, and then what? Won't I see the same issue? I assume I can't disable httpd/php (the other most memory consuming processes) due to ISPConfig...? So where would I have any memory usage reduction?
You can disable httpd, but you will not be able to host websites on that server and not be able to access ISPConfig interface. E.g. when you have a multiserver setup, then the email node(s) normally don't run apache as the ISPConfig UI is on a separate server or on the web server.
Thanks again Till. So I just noticed that clam started pegging my CPUs and the PID kept changing every few seconds, and my IO skyrocketed. This started about the same time as these logs, and I found past logs with the same CPU and IO spike and same logs, which was after restarting amavisd and httpd so I'll be increasing the swap and watching the logs for a few days again: Code: Aug 15 12:07:19 s02 clamd[6290]: Received 0 file descriptor(s) from systemd. Aug 15 12:07:19 s02 clamd[6290]: clamd daemon 0.100.1 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64) Aug 15 12:07:19 s02 clamd[6290]: Running as user amavis (UID 994, GID 991) Aug 15 12:07:19 s02 clamd[6290]: Log file size limited to 1048576 bytes. Aug 15 12:07:19 s02 clamd[6290]: Reading databases from /var/lib/clamav Aug 15 12:07:19 s02 clamd[6290]: Not loading PUA signatures. Aug 15 12:07:19 s02 clamd[6290]: Bytecode: Security mode set to "TrustSigned". Aug 15 12:07:32 s02 clamd[6290]: Loaded 6608882 signatures. Aug 15 12:07:33 s02 clamd[6290]: LOCAL: Unix socket file /var/run/clamd.amavisd/clamd.sock Aug 15 12:07:33 s02 clamd[6290]: LOCAL: Setting connection queue length to 200 Aug 15 12:07:33 s02 clamd[6290]: daemonize() failed: Cannot allocate memory Aug 15 12:07:33 s02 clamd[6290]: Socket file removed.
Looks like increasing the swap did the trick. It's running great with 9 days of uptime now. Thanks all!
