Hello, I am running ISPConfig on Ubuntu servers. I have upgraded Ubuntu to 24.04 and ISPConfig to 3.2.12p1 and ever since then the webpages are loading slowly. I noticed that db servers are overloading with mariadb process being the dominant process. Do you have any suggestions for troubleshooting and can I safely tweak MariaDB settings to optimize performance? Code: ##### SERVER ##### IP-address (as per hostname): ***.***.***.*** [WARN] could not determine server's ip address by ifconfig [INFO] OS version is Ubuntu 24.04.1 LTS [INFO] uptime: 16:36:48 up 1 day, 9:12, 4 users, load average: 28.52, 23.97, 18.94 [INFO] memory: total used free shared buff/cache available Mem: 9.7Gi 9.5Gi 166Mi 84Ki 207Mi 122Mi Swap: 3.8Gi 3.6Gi 277Mi [INFO] systemd failed services status: UNIT LOAD ACTIVE SUB DESCRIPTION ● snap.lxd.activate.service not-found failed failed snap.lxd.activate.service Legend: LOAD → Reflects whether the unit definition was properly loaded. ACTIVE → The high-level unit activation state, i.e. generalization of SUB. SUB → The low-level unit activation state, values depend on unit type. 1 loaded units listed. [INFO] ISPConfig is installed. ##### ISPCONFIG ##### ISPConfig version is 3.2.12p1 ##### VERSION CHECK ##### [INFO] php (cli) version is 8.3.13 [INFO] php-cgi (used for cgi php in default vhost!) is version 8.3.13 ##### PORT CHECK ##### [WARN] Port 8080 (ISPConfig) seems NOT to be listening [WARN] Port 8081 (ISPConfig Apps) seems NOT to be listening [WARN] Port 143 (IMAP server) seems NOT to be listening [WARN] Port 993 (IMAP server SSL) seems NOT to be listening [WARN] Port 110 (POP3 server) seems NOT to be listening [WARN] Port 995 (POP3 server SSL) seems NOT to be listening [WARN] Port 465 (SMTP server SSL) seems NOT to be listening ##### MAIL SERVER CHECK ##### [WARN] I found no "submission" entry in your postfix master.cf [INFO] this is not critical, but if you want to offer port 587 for smtp connections you have to enable this. [WARN] I found no "smtps" entry in your postfix master.cf
ISPConfig and its version have no influence on website loading speeds, so your issue must be related to something else. Check the website's error.log and access.log. You did not run the test script as root.
I realize that ISPConfig version should not affect the loading speed. I did check access and error logs, and found nothing peculiar (how would I see speed of loading in them anyway?). But the fact is that multiple pages hosted on multiple web servers are having problems. DB pressure could explain the slow loading time I think. I was wondering if I can safely tune mariadb without breaking ISPConfig. Also I did run the test script as root, is there something in the output that points to problems with privileges?
Sure, you can tune MariaDB at any time. It's not related to ISPConfig, either. Just take care not to turn off networking in MariaDB.
Ok, then the server runs just very few services and must be a slave node with just web server without ISPConfig UI.
Code: [INFO] uptime: 16:36:48 up 1 day, 9:12, 4 users, load average: 28.52, 23.97, 18.94 [INFO] memory: total used free shared buff/cache available Mem: 9.7Gi 9.5Gi 166Mi 84Ki 207Mi 122Mi Swap: 3.8Gi 3.6Gi 277Mi Load is very high and memory is exhausted. This is why web pages load slowly. You should monitor the host to see which process uses so much memory and what is running there that uses all CPU. Is the disk system full or in degraded state so it has gotten slow?
I've determined that the problem is with db servers in multiserver setup. I added more memory, and eventually it got exhausted too. memleak shows that there is a leak in mariadb and there were a lot of these types of messages in mysql error.log: Code: 2024-11-24 8:14:14 0 [Note] InnoDB: Memory pressure event freed 199 pages After trying to find problems in mysql usage without any luck, I found this issue description, so I turned off THP https://jira.mariadb.org/browse/MDEV-33279. I was going to wait a bit more to post this as a solution, but it's been running with no problems for about ten hours now. I'm still puzzled why this kicked in after Ubuntu + ISPConfig upgrade. EDIT: Additionally I changed malloc library which mysql uses to tmalloc per instructions in this link: https://mariadb.com/kb/en/profiling-memory-usage/ except I didn't install google-perftools, but libtcmalloc-minimal4t64
It might be that older Ubuntu versions had this disabled and the new Ubuntu version did enable this. But as you see now, it's not related to the ISPConfig update, as I mentioned in my first post.