Reoccurring problem each day near midnight and a few other times throughout the day. 1GB RAM, 1 CPU. (below is syslog) (Perfect server tutorial, Debian 10, ISPconfig(3.1), NginX(1.14), Postfix(3.4.7), Roundcube, Dovecot(2.3.4.1), PHP(7.3), Digital Ocean hosting, and lots of altering/tinkering) **Monitoring the "top" command when this is happening: It shows a bunch of Phpfpm-7.3 processes using 100% of cpu/memory while CRON jobs and some other things are runnig (as seen below Below is some syslog, before "Error establishing connection with database" is what the URL displays. Note: Rebooting server temporarily fixes issue. Note: I didnt install amavisd and clam av. Code: Apr 2 02:58:01 nyc CRON[9758]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done) Apr 2 02:58:01 nyc CRON[9759]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done) Apr 2 02:59:01 nyc CRON[9775]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done) Apr 2 02:59:01 nyc CRON[9776]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done) Apr 2 03:00:01 nyc CRON[9793]: (getmail) CMD (/usr/local/bin/run-getmail.sh > /dev/null 2>> /dev/null) Apr 2 03:00:01 nyc CRON[9795]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done) Apr 2 03:00:01 nyc CRON[9794]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done) Apr 2 03:00:04 nyc dovecot: imap-login: Disconnected (disconnected before auth was ready, waited 0 secs): user=<>, rip=127.0.0.1, lip=127.0.0.1, secured, session=<QwqyWUmidLZ/AAAB> Apr 2 03:00:04 nyc dovecot: pop3-login: Disconnected (no auth attempts in 0 secs): user=<>, rip=127.0.0.1, lip=127.0.0.1, secured, session=<3x2zWUminpR/AAAB> Apr 2 03:00:04 nyc postfix/smtpd[9856]: connect from localhost[127.0.0.1] Apr 2 03:00:04 nyc postfix/smtpd[9856]: lost connection after CONNECT from localhost[127.0.0.1] Apr 2 03:00:04 nyc postfix/smtpd[9856]: disconnect from localhost[127.0.0.1] commands=0/0 Code: Apr 2 03:06:27 nyc postfix/smtps/smtpd[10414]: connect from unknown[45.133.99.6] Apr 2 03:06:34 nyc postfix/smtps/smtpd[10414]: warning: unknown[45.133.99.6]: SASL LOGIN authentication failed: UGFzc3dvcmQ6 Apr 2 03:06:34 nyc postfix/smtps/smtpd[10414]: lost connection after AUTH from unknown[45.133.99.6] Apr 2 03:06:34 nyc postfix/smtps/smtpd[10414]: disconnect from unknown[45.133.99.6] ehlo=1 auth=0/1 commands=1/2 Apr 2 03:06:34 nyc postfix/smtps/smtpd[10414]: connect from unknown[45.133.99.6] Apr 2 03:06:42 nyc postfix/smtps/smtpd[10414]: lost connection after AUTH from unknown[45.133.99.6] Apr 2 03:06:42 nyc postfix/smtps/smtpd[10414]: disconnect from unknown[45.133.99.6] ehlo=1 auth=0/1 commands=1/2 Apr 2 03:06:42 nyc postfix/smtps/smtpd[10414]: connect from unknown[45.133.99.6] Apr 2 03:06:50 nyc postfix/smtps/smtpd[10414]: lost connection after AUTH from unknown[45.133.99.6] Apr 2 03:06:50 nyc postfix/smtps/smtpd[10414]: disconnect from unknown[45.133.99.6] ehlo=1 auth=0/1 commands=1/2 Apr 2 03:06:50 nyc postfix/smtps/smtpd[10420]: connect from unknown[45.133.99.6] Apr 2 03:06:58 nyc postfix/smtps/smtpd[10420]: warning: unknown[45.133.99.6]: SASL LOGIN authentication failed: UGFzc3dvcmQ6 Apr 2 03:06:59 nyc postfix/smtps/smtpd[10420]: lost connection after AUTH from unknown[45.133.99.6] Apr 2 03:06:59 nyc postfix/smtps/smtpd[10420]: disconnect from unknown[45.133.99.6] ehlo=1 auth=0/1 commands=1/2 Apr 2 03:07:20 nyc kernel: [14135.791974] fail2ban-server invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 Apr 2 03:07:20 nyc kernel: [14135.794601] fail2ban-server cpuset=/ mems_allowed=0 Apr 2 03:07:20 nyc kernel: [14135.795700] CPU: 0 PID: 1001 Comm: fail2ban-server Not tainted 4.19.0-6-cloud-amd64 #1 Debian 4.19.67-2+deb10u2 Apr 2 03:07:20 nyc kernel: [14135.797908] Hardware name: DigitalOcean Droplet, BIOS 20171212 12/12/2017 Apr 2 03:07:20 nyc kernel: [14135.799419] Call Trace: Apr 2 03:07:20 nyc kernel: [14135.800053] dump_stack+0x5c/0x80 Apr 2 03:07:20 nyc kernel: [14135.800844] dump_header+0x6b/0x283 Apr 2 03:07:20 nyc kernel: [14135.801754] ? do_try_to_free_pages+0x2ec/0x370 Apr 2 03:07:20 nyc kernel: [14135.802865] oom_kill_process.cold.30+0xb/0x1cf Apr 2 03:07:20 nyc kernel: [14135.803938] ? oom_badness+0x23/0x140 Apr 2 03:07:20 nyc kernel: [14135.804822] out_of_memory+0x1a5/0x430 Apr 2 03:07:20 nyc kernel: [14135.805739] __alloc_pages_slowpath+0xbd8/0xcb0 Apr 2 03:07:20 nyc kernel: [14135.806796] __alloc_pages_nodemask+0x28b/0x2b0 Apr 2 03:07:20 nyc kernel: [14135.807843] filemap_fault+0x3bd/0x780 Apr 2 03:07:20 nyc kernel: [14135.808714] ? filemap_map_pages+0x1e7/0x3a0 Apr 2 03:07:20 nyc kernel: [14135.809738] ext4_filemap_fault+0x2c/0x40 [ext4] ... ... Apr 2 03:07:20 nyc kernel: [14135.870073] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Apr 2 03:07:20 nyc kernel: [14135.872116] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Apr 2 03:07:20 nyc kernel: [14135.874122] 28324 total pagecache pages Apr 2 03:07:20 nyc kernel: [14135.875087] 0 pages in swap cache Apr 2 03:07:20 nyc kernel: [14135.875977] Swap cache stats: add 0, delete 0, find 0/0 Apr 2 03:07:20 nyc kernel: [14135.878785] Free swap = 0kB Apr 2 03:07:20 nyc kernel: [14135.879581] Total swap = 0kB Apr 2 03:07:20 nyc kernel: [14135.880390] 262039 pages RAM Apr 2 03:07:20 nyc kernel: [14135.881445] 0 pages HighMem/MovableOnly Apr 2 03:07:20 nyc kernel: [14135.882616] 8621 pages reserved Apr 2 03:07:20 nyc kernel: [14135.883689] Tasks state (memory values in pages): Apr 2 03:07:20 nyc kernel: [14135.885041] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name Apr 2 03:07:20 nyc kernel: [14135.887258] [ 214] 0 214 10099 594 110592 0 0 systemd-journal Apr 2 03:07:20 nyc kernel: [14135.889859] [ 294] 0 294 2021 798 53248 0 0 haveged Apr 2 03:07:20 nyc kernel: [14135.892378] [ 542] 0 542 4980 208 57344 0 -1000 systemd-udevd Apr 2 03:07:20 nyc kernel: [14135.894655] [ 580] 0 580 1839 59 53248 0 0 inetd Apr 2 03:07:20 nyc kernel: [14135.896756] [ 581] 0 581 2110 70 57344 0 0 cron Apr 2 03:07:20 nyc kernel: [14135.898811] [ 583] 115 583 102021 174 114688 0 0 memcached Apr 2 03:07:20 nyc kernel: [14135.900830] [ 584] 0 584 1461 58 49152 0 0 incrond Apr 2 03:07:20 nyc kernel: [14135.902852] [ 586] 105 586 629 49 45056 0 0 nscd Apr 2 03:07:20 nyc kernel: [14135.904786] [ 589] 998 589 121371 2625 143360 0 -900 do-agent Apr 2 03:07:20 nyc kernel: [14135.906819] [ 590] 0 590 56455 308 86016 0 0 rsyslogd Apr 2 03:07:20 nyc kernel: [14135.910117] [ 612] 0 612 1050 81 49152 0 0 dovecot Apr 2 03:07:20 nyc kernel: [14135.912148] [ 632] 116 632 39143 3168 139264 0 0 named Apr 2 03:07:20 nyc kernel: [14135.940153] [ 788] 5002 788 72580 2369 237568 0 0 php-fpm7.3 Apr 2 03:07:20 nyc kernel: [14135.942360] [ 789] 5002 789 72580 2370 237568 0 0 php-fpm7.3 Apr 2 03:07:20 nyc kernel: [14135.944373] [ 790] 5003 790 72762 4889 274432 0 0 php-fpm7.3 Apr 2 03:07:20 nyc kernel: [14135.946440] [ 791] 5003 791 72907 5068 278528 0 0 php-fpm7.3 Apr 2 03:07:20 nyc kernel: [14135.948478] [ 792] 5004 792 72439 2370 241664 0 0 php-fpm7.3 Apr 2 03:07:20 nyc kernel: [14135.950618] [ 793] 5004 793 72439 2370 241664 0 0 php-fpm7.3 Apr 2 03:07:20 nyc kernel: [14135.952747] [ 798] 5013 798 72439 2370 241664 0 0 php-fpm7.3 Apr 2 03:07:20 nyc kernel: [14135.954842] [ 800] 5013 800 72439 2370 241664 0 0 php-fpm7.3 Apr 2 03:07:20 nyc kernel: [14135.956893] [ 801] 5014 801 72439 3317 270336 0 0 php-fpm7.3 Apr 2 03:07:20 nyc kernel: [14136.182253] [ 10431] 0 10431 31215 1913 237568 0 0 php Apr 2 03:07:20 nyc kernel: [14136.184485] Out of memory: Kill process 745 (mysqld) score 171 or sacrifice child Apr 2 03:07:20 nyc kernel: [14136.188287] Killed process 745 (mysqld) total-vm:1277072kB, anon-rss:172924kB, file-rss:0kB, shmem-rss:0kB Apr 2 03:07:20 nyc kernel: [14136.193311] oom_reaper: reaped process 745 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
reason: out_of_memory Assign more ram to the virtual machine or add SWAP file or partition, if digitalocean supports that. But RAM is better, increase the droplet to e.g. 2GB.
If you installed amavis, you could try switching to rspamd which has a significantly lower memory footprint: https://www.howtoforge.com/replacing-amavisd-with-rspamd-in-ispconfig/ Additionally, deactivating virus scanning saves a lot of memory (because all the signatures need to be hold in memory). You have to make sure to scan vor viruses on the clients or on a relay server.
