The website with http or that I use certificate from another company, load in 2 seconds on average. Those who use let's take about 20 to 30 seconds. Does anyone know what it could be?
Maybe a problem with OCSP? If your server can't contact lets encrypt's OCSP servers, or if they are down again, might do that? Ensure you have OCSP stapling setup, and maybe look look into mod_md mentioned here and this caching ocsp proxy. (I have no experience with either.)