Slow website load times

All the websites on my ISPConfig server have initial slow loading times. It takes about 1-5 seconds before each website loads and after that it's still not fast but reasonable. We discovered the problem is with the server because we installed the same websites on another server and they were blazing fast.

ISPconfig is running on Ubuntu 18 and is installed on a VPS with 2 cores and 4GB RAM. The VPS itself is running in a Proxmox system on our dedicated server in the datacenter which is not experiencing peformance issues. However once i open a website i can see the CPU peaking in the proxmox panel for that VPS.

Tonight i will upgrade the VPS to 8 cores and 8GB RAM to see what happens. But i was wondering if there are any other things i can check. For instance i have my websites running on PHP 7.1 now. Is there a guide yet to install 8.1 on Ubutu? I did find this one but it's for Debian:
https://www.howtoforge.com/tutorial...fig-3-from-debian-packages-on-debian-8-and-9/

 

More cores may not help, but more RAM should make websites faster, if you are using cache that stores the pages in memory after they are accessed for the first time.
PHP 8.1 can be installed on Ubuntu 20.04, but verify or check first you website does work with PHP 8.
What kind of disks are on that Proxmox host? After I upgraded my Proxmox to SSD disks, it has been very fast on all tasks.

Run performance monitor on you web server host to see what is the bottleneck. Use for example: free -h, df -hT, top, iotop, nettop.

 

True. Other than ram type and size, disk type determines the speed where nvme is mainly in lead, then ssd, then hdd.

 

Thanks for all the tips! The proxmox server has had SSD disks from the start, the main task for these servers is handling Voip systems so they have to be fast and with no overbooking.

When i open up 'top' and load a website i see the CPU spiking to about 80% and the main processes are php-cgi and mysqld.
I used bmon for network monitoring but didn't see any alarming traffic.

 

you don't mention how long this server been in place, and what guide you followed to install it.
it may be using mod_php and mpm_prefork, so it would be worth switching to mpm_event and php-fpm.
that will be faster, and more memory efficient.
you already seem to be aware that newer releases of php should be faster, although you may want to go to 7.4 or 8.0 rather than 8.1,
i'm not sure wordpress supports 8.1 yet, many plugins/themes certainly won't.

you don't say how many cpu's, physical cores your server has, or how many vps, and how many total vcpu's are assigned.
i'm not familiar with proxmox, but pinning vcpu's to physical cores can improve performance by eliminating interrupt switching. although this level of tuning shouldn't be required normally.

 

I've not had a problem with 8.1. As you say themes and plugins are another matter. The old rule of any theme or plugin that has not been updated in the previous two years is a sure sign of trouble ahead. Change'm before it's too late.

However Wordpress's obsession with blocks is screwing a lot of our sites on even 7.4. Wordpess 5.9 added to the woes.

Either way with 7.4 going EOL in only 10 months many (most?) websites are going to have spend time and money making the PHP 8 & Wordpess 5.6+ compatible or just cross fingers that no more critical issues will be found in 2023.

PHP would do us all a favour (unless you are a hacker) by extending 7.4 EOL by another year as a mark of moving a major version is going to take the downstream apps a lot more time to adapt. In particular that so much depracated stuff has been thrown out in PHP8. Plus you couldn't start to test CMS add-ons until the CMS has itself has become compatible. Some important ones have yet to get that far.

 

The VPS has been in places since about 3-4 years. This is the guide i used to install it: https://www.howtoforge.com/tutorial...l-pureftpd-bind-postfix-doveot-and-ispconfig/

At some point i had upgraded to Ubuntu 18.

These are the server PHP settings. The settings like mod_php etc you mention are done on a website specific basis right? But i assume these general settings also matter. What do you guys recommend here?


I have already installed PHP7.4 as additional version. But maybe it's relevant to updat the main server PHP version?

I have upgraded the VPS to quite an overkill now: 8 cores and 16GB RAM. Monitoring now how it's performing.

Oh and for DNS i used the nameservers from our datacenter provider. I figured these would be fater then general server like Google.

 

Just reporting back that the server is still slow and when i work in applications on this server i notice that the slownewss occurs everytime i open a record and submit. So it looks like database issues. I ran mysqltuner:

General recommendations:
Restrict Host for user@% to user@SpecificDNSorIp
Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
Adjust your join queries to always utilize indexes
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries which have no LIMIT clause
Increase table_open_cache gradually to avoid file descriptor limits
Read this before increasing table_open_cache over 64: http://bit.ly/1mi7c4C
Beware that open_files_limit (4190) variable
should be greater than table_open_cache (2000)
Performance should be activated for better diagnostics
Consider installing Sys schema from https://github.com/mysql/mysql-sys
Variables to adjust:
query_cache_size (=0)
query_cache_type (=0)
query_cache_size (> 16M)
join_buffer_size (> 256.0K, or always use indexes with joins)
tmp_table_size (> 16M)
max_heap_table_size (> 16M)
table_open_cache (> 2000)
performance_schema = ON enable PFS
innodb_buffer_pool_size (>= 4G) if possible.
innodb_log_file_size * innodb_log_files_in_group should be equal to 1/4 of buffer pool size (=64M) if possible.
innodb_buffer_pool_instances (=1)

I'm not sure which of these recommendations would have the most impact.
While tuner was running i also noticed:
[!!] Switch to 64-bit OS - MySQL cannot currently use all of your RAM
-------- Performance Metrics -----------------------------------------------------------------------
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[!!] Query cache prunes per day: 37877
[!!] Joins performed without indexes: 2363
[!!] Temporary tables created on disk: 55% (117K on disk / 211K total)
[!!] Table cache hit rate: 0% (2K open / 4M opened)
-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 11.6% (1M used / 16M cache)
[!!] Write Key buffer hit rate: 65.4% (481K cached / 314K writes)
-------- InnoDB Metrics ----------------------------------------------------------------------------
[!!] InnoDB buffer pool / data size: 128.0M/4.1G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (75 %): 48.0M * 2/128.0M should be equal 25%
[!!] InnoDB buffer pool <= 1G and Innodb_buffer_pool_instances(!=1).
[!!] InnoDB Read buffer efficiency: 62.58% (1096236777 hits/ 1751816963 total)
[!!] InnoDB Write Log efficiency: 69.34% (2018159 hits/ 2910627 total)
-------- AriaDB Metrics ----------------------------------------------------------------------------
[!!] Aria pagecache hit rate: 91.1% (1M cached / 95K reads)

What should i look at first?

 

i'm not an expert on mysql tuning. but this caught my eye:

not sure if that's just the script not updated / not getting the info it expects, or it you really are running a 32bit version of ubuntu.

if you are, i'd say the first step in fixing anything would be to create a new vps, and install the latest ubuntu/debian (64 bit) on it, and install a clean new ispconfig on that. that'll immediately give a big performance boost, and allow you to actually use more than 4Gb of ram.

then migrate everything from your old vps to this new one.

then come back for help if you're still experiencing slow loading times with the new vps.

 

I noticed that too and investigated it immediately and found out it's indeed running a 32 bit OS. Thanks for the confirmation i will start moving it to a new server. Just to be sure, is this the right guide to follow?
https://www.howtoforge.com/tutorial...-confixx-plesk-to-ispconfig-31-single-server/

 

https://www.howtoforge.com/ispconfig-autoinstall-debian-ubuntu/

and then use the migration tool. it should come with instructions.

 

I'm migrating my server now with the migration tool from server1.mydomain to server2.mydomain. I suppose that when it's down i either have to modify all the DNS records for all my websites or after the migration is down i turn off server1 and change the hostname and iP settings of server2 to that of server1. Which do you recommend?

I could not find a good guide for changing the hostname and IP address apart from this old thread:
https://www.howtoforge.com/community/threads/prepairing-for-hosting.2760/
And this blogpost:
https://mindwatering.com/SupportRef.nsf/webpg/A5E76CE93EAC82E885257F1B005B070F

 

Well after all migration was done i had succesfully turned off the old server and renamed the target server but unfortunately letsencrypt wouldn't work.

The migration guide says:

• The old and new server must use the same Let's Encrypt client. If your old server uses certbot, then ensure that the new system is using certbot too and not acme.sh.

But how do i install certbot in the correct way and remove acme.sh?
edit:
These commands seem to have worked:

 

You should be able to install certbot with snap then remove acme.sh and its folders. Update ispconfig again should allow you to use certbot to create LE certs.

Alternatively, you can always create new server, install certbot using snap and use ISPConfig Auto Installer.

 

What will snap do differently from the commands i have already used? So if i understand you correctly the autoinstaller won't work in my current situation? I just ran the ISPConfig update script, how do i verify what my current SSL client is?

 

The developer of acme.sh suggest this at its github to remove:

Code:

acme.sh --uninstall
rm -r  ~/.acme.sh

So, I don't think certbot will work properly if you do not remove acme.sh and all its folders properly. Plus, the way to install certbot properly on Ubuntu and few other OS is now using snap and you can check this in its instructions page. My guess is the certbot you installed via apt-get will work but not properly and it may face problem in the future like in LE certs renewal etc.

 

To remove acme i did:

Code:

acme.sh --uninstall
rm -r  ~/.acme.sh

And to install certbot i did:

Code:

sudo apt update
sudo apt install snapd
sudo snap install core; sudo snap refresh core
sudo apt-get remove certbot
sudo sed -i '/certbot-auto/d' /etc/crontab
sudo rm -rf /opt/eff.org
sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot

The migration of the sites went fine after that. However for most sites i did have to disable SSL and enable it again through the ISPConfig portal.

How can i check if renewal will work?

 

You shouldn't install LE certs for your server via that certbot ---apache command if you are using ISPConfig.

Since you have done that it is best that you first remove the LE certs issued and its config totally (including its archive, live and renewal folders) and then force update ISPConfig and select create SSL during that process.

For testing renewal you should run command "certbot renew --dry-run" but do note ISPConfig covers only the renewal of LE certs created by its installer / updater or through its control panel.

However, LE certs created through other means e.g. like the one you issued via certbot ---apache command above may or may not be renewed automatically by ISPConfig.