*** SOLVED *** All sites working well for a long time... until today. Updated to latest and realised that my sites were using the wrong certs. Tried resyncing sites and switching off/on ssl in the gui, but each time, the ssl and letsencrypt box goes empty afterwards. Checking in site.vhost the ssl section isn't there anymore. I will add that I had yesterday made some custom changes to test something, vhosts in conf-custom and this could also have potentially brought on the issue? (those customs are no longer there) but it was to do with TLS. Certs seem up to date (and I tried forcing one) and acme.sh reports no errors, not does server.sh to try to debug ispconfig. Where else to troubeshoot? Do custom edits write anywhere else that they may have remained? or is it a known issue?
Thanks, but as I mentioned, been through the standard troubleshooting. DNS working fine. acme.sh is present and working, no errors in /var/log/ispconfig/acme ispconfig is the latest. server.sh simply reports finished. Should it be reporting more? (it does hang a good while after clicking save in the gui (much longer hang than I seem to remember from past). I had recently tested forcing TLSv1.3 by adding '-TLSv1.2' by adding a vhost.conf.master file in conf-custom, adding to the line: SSLProtocol -TLSv1.2 - I then removed this and again resynced sites in the panel. (since this is ssl related I feel I have to mention this as a possible cause). Only other change is update to the current ispconfig version - no errors etc were generated. Where else to troubleshoot this? I've look in all the /var/log/ispconfig/ but can't see the cause. Since a lot og the related material about ispconfig relates to letsencrypt rather than acme.sh, it makes it more awkward to troubleshoot too. Thanks Steve
This means you either missed enabling debug mode (as described in the link that you find in the FAQ) or you missed enabling Let's Encrypt checkbox before running server.sh It's quite easy to troubleshoot, just follow the FAQ from beginning to end and finally post the output of server.sh if you did not found the issue on your own. We know what to look for and what you might not know, that's why you should always post the output as also the absence of certain information can give a hint on what's wrong with your setup.
EDIT: You were right, I hadn't fully enabled debug :/ Standby ---------------------- All I'm getting is: /usr/local/ispconfig/server/server.sh finished server.php. What should I get on a working ispconfig setup? I'm running bullseye btw in case this could be a php issue? /var/log/ispconfig/acme.log: [Thu 2 Mar 13:30:35 GMT 2023] Cert success. [Thu 2 Mar 13:30:35 GMT 2023] Your cert is in: /root/.acme.sh/domain.com/domain.com.cer [Thu 2 Mar 13:30:35 GMT 2023] Your cert key is in: /root/.acme.sh/domain.com/domain.com.key [Thu 2 Mar 13:30:35 GMT 2023] The intermediate CA cert is in: /root/.acme.sh/domain.com/ca.cer [Thu 2 Mar 13:30:35 GMT 2023] And the full chain certs is there: /root/.acme.sh/domain.com/fullchain.cer [Thu 2 Mar 13:30:35 GMT 2023] Your pre-generated next key for future cert key change is in: /root/.acme.sh/domain.com/domain.com.key.next [Thu 2 Mar 13:30:36 GMT 2023] Installing key to: /var/www/clients/client0/web31/ssl/domain.com-le.key [Thu 2 Mar 13:30:36 GMT 2023] Installing full chain to: /var/www/clients/client0/web31/ssl/domain.com-le.crt [Thu 2 Mar 13:30:36 GMT 2023] Run reload cmd: systemctl force-reload apache2.service [Thu 2 Mar 13:30:36 GMT 2023] Reload success [Thu 2 Mar 13:30:36 GMT 2023] _on_issue_success [Thu 2 Mar 13:30:36 GMT 2023] The NOTIFY_HOOK is empty, just return /var/log/ispconfig/ispconfig.log: (empty) /var/log/ispconfig/cron.log: Thu 2 Mar 16:23:01 GMT 2023 finished server.php. Thu 2 Mar 16:24:01 GMT 2023 finished server.php. Thu 2 Mar 16:25:01 GMT 2023 finished server.php. Thu 2 Mar 16:26:01 GMT 2023 finished server.php. Thu 2 Mar 16:27:01 GMT 2023 finished server.php. Thu 2 Mar 16:28:01 GMT 2023 finished server.php. Thu 2 Mar 16:29:01 GMT 2023 finished server.php. Thu 2 Mar 16:30:01 GMT 2023 finished server.php. Thu 2 Mar 16:31:02 GMT 2023 finished server.php. Thu 2 Mar 16:32:01 GMT 2023 finished server.php. Thu 2 Mar 16:33:01 GMT 2023 finished server.php. Thu 2 Mar 16:34:01 GMT 2023 finished server.php. Thu 2 Mar 16:35:02 GMT 2023 finished server.php. Thu 2 Mar 16:36:01 GMT 2023 finished server.php. Thu 2 Mar 16:37:01 GMT 2023 finished server.php. Thu 2 Mar 16:38:01 GMT 2023 finished server.php. Thu 2 Mar 16:39:01 GMT 2023 finished server.php. etc
/usr/local/ispconfig/server/server.sh 02.03.2023-17:13 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'. 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.4/fpm/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.4/cgi/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/5.6/cgi/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/5.6/fpm/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.0/cgi/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.0/fpm/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.1/cgi/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.1/fpm/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.2/cgi/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.2/fpm/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.3/cgi/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.3/fpm/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/8.0/cgi/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/8.0/fpm/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/8.1/cgi/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/8.1/fpm/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/8.2/cgi/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/8.2/fpm/php.ini' - return code: 0 02.03.2023-17:13 - DEBUG [server:177] - Found 1 changes, starting update process. 02.03.2023-17:13 - DEBUG [plugins.inc:118] - Calling function 'ssl' from plugin 'apache2_plugin' raised by event 'web_domain_update'. 02.03.2023-17:13 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'apache2_plugin' raised by event 'web_domain_update'. 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: chattr -i '/var/www/clients/client0/web31' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: chattr +i '/var/www/clients/client0/web31' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: df -T '/var/www/clients/client0/web31'|awk 'END{print $2,$NF}' - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: which 'setquota' 2> /dev/null - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: setquota -u 'web31' '0' '0' 0 0 -a &> /dev/null - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: setquota -T -u 'web31' 604800 604800 -a &> /dev/null - return code: 0 02.03.2023-17:13 - DEBUG [system.inc:2399] - safe_exec cmd: chattr +i '/var/www/clients/client0/web31' - return code: 0 02.03.2023-17:14 - WARNING - Could not verify domain domain.com, so excluding it from letsencrypt request. 02.03.2023-17:14 - WARNING - Could not verify domain02.03.2023-17:15 - WARNING - Could not verify domain www.domain.com, so excluding it from letsencrypt request. 02.03.2023-17:15 - WARNING - Let's Encrypt SSL Cert for: domain.com could not be issued. 02.03.2023-17:15 - WARNING - 02.03.2023-17:15 - DEBUG [db mysql.inc:523] - NON-String given in escape function! (boolean) 02.03.2023-17:15 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0 02.03.2023-17:15 - DEBUG [apache2 plugin.inc:1874] - Writing the vhost file: /etc/apache2/sites-available/domain.com.vhost 02.03.2023-17:15 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0 02.03.2023-17:15 - DEBUG [apache2 plugin.inc:3445] - Writing the PHP-FPM config file: /etc/php/8.2/fpm/pool.d/web31.conf 02.03.2023-17:15 - DEBUG [services.inc:56] - Calling function 'restartPHP_FPM' from module 'web_module'. 02.03.2023-17:15 - DEBUG [system.inc:2082] - Trying to use Systemd to restart service 02.03.2023-17:15 - DEBUG [system.inc:2399] - safe_exec cmd: systemctl is-enabled 'php8.2-fpm' 2>&1 - return code: 0 02.03.2023-17:15 - DEBUG [web module.inc:316] - Restarting php-fpm: systemctl reload php8.2-fpm.service 02.03.2023-17:15 - DEBUG [apache2 plugin.inc:1992] - Apache status is: running 02.03.2023-17:15 - DEBUG [services.inc:56] - Calling function 'restartHttpd' from module 'web_module'. 02.03.2023-17:15 - DEBUG [system.inc:2082] - Trying to use Systemd to restart service 02.03.2023-17:15 - DEBUG [system.inc:2399] - safe_exec cmd: systemctl is-enabled 'apache2' 2>&1 - return code: 0 02.03.2023-17:15 - DEBUG [web module.inc:246] - Restarting httpd: systemctl restart apache2.service 02.03.2023-17:15 - DEBUG [apache2 plugin.inc:1995] - Apache restart return value is: 0 02.03.2023-17:16 - DEBUG [apache2 plugin.inc:2006] - Apache online status after restart is: running 02.03.2023-17:16 - DEBUG [modules.inc:240] - Processed datalog_id 8557 02.03.2023-17:16 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock finished server.php. So I tried: acme.sh --renew -d domain.com --force [Thu 2 Mar 17:30:45 GMT 2023] Renew: 'domain.com' [Thu 2 Mar 17:30:45 GMT 2023] Renew to Le_API=https://acme-v02.api.letsencrypt.org/directory [Thu 2 Mar 17:30:46 GMT 2023] Using CA: https://acme-v02.api.letsencrypt.org/directory [Thu 2 Mar 17:30:46 GMT 2023] Using pre generated key: /root/.acme.sh/domain.com/domain.com.key.next [Thu 2 Mar 17:30:46 GMT 2023] Generate next pre-generate key. [Thu 2 Mar 17:30:47 GMT 2023] Multi domain='DNS:domain.com,DNS:www.domain.com' [Thu 2 Mar 17:30:47 GMT 2023] Getting domain auth token for each domain [Thu 2 Mar 17:30:49 GMT 2023] Getting webroot for domain='domain.com' [Thu 2 Mar 17:30:50 GMT 2023] Getting webroot for domain='www.domain.com' [Thu 2 Mar 17:30:50 GMT 2023] domain.com is already verified, skip http-01. [Thu 2 Mar 17:30:50 GMT 2023] www.domain.com is already verified, skip http-01. [Thu 2 Mar 17:30:50 GMT 2023] Verify finished, start to sign. [Thu 2 Mar 17:30:50 GMT 2023] Lets finalize the order. [Thu 2 Mar 17:30:50 GMT 2023] Le_OrderFinalize='https://acme-v02.api.letsencrypt.org/acme/finalize/254188440/167795802156' [Thu 2 Mar 17:30:51 GMT 2023] Downloading cert. [Thu 2 Mar 17:30:51 GMT 2023] Le_LinkCert='https://acme-v02.api.letsencrypt.org/acme/cert/04f8db54654deda5229ae9fa93eae92eeb6a' [Thu 2 Mar 17:30:52 GMT 2023] Cert success. -----BEGIN CERTIFICATE----- xyxyxyxyx cert was here -----END CERTIFICATE----- [Thu 2 Mar 17:30:52 GMT 2023] Your cert is in: /root/.acme.sh/domain.com/domain.com.cer [Thu 2 Mar 17:30:52 GMT 2023] Your cert key is in: /root/.acme.sh/domain.com/domain.com.key [Thu 2 Mar 17:30:52 GMT 2023] The intermediate CA cert is in: /root/.acme.sh/domain.com/ca.cer [Thu 2 Mar 17:30:52 GMT 2023] And the full chain certs is there: /root/.acme.sh/domain.com/fullchain.cer [Thu 2 Mar 17:30:52 GMT 2023] Your pre-generated next key for future cert key change is in: /root/.acme.sh/domain.com/domain.com.key.next [Thu 2 Mar 17:30:52 GMT 2023] Installing key to: /var/www/clients/client0/web31/ssl/domain.com-le.key [Thu 2 Mar 17:30:52 GMT 2023] Installing full chain to: /var/www/clients/client0/web31/ssl/domain.com-le.crt [Thu 2 Mar 17:30:52 GMT 2023] Run reload cmd: systemctl force-reload apache2.service [Thu 2 Mar 17:30:52 GMT 2023] Reload success So it seems that ispconfig isn't seeing that the cert is handed out, OR that the cert is not due for update so doesn't need 're-enabled'... I could probably put the info into the vhosts file in sites-available manually but I'd rather get to the bottom of this issue and how it came about. Thanks Steve
No, the cause is different, the reason is shown here: Code: 02.03.2023-17:14 - WARNING - Could not verify domain domain.com, so excluding it from letsencrypt request. 02.03.2023-17:14 - WARNING - Could not verify domain02.03.2023-17:15 - WARNING - Could not verify domain www.domain.com, so excluding it from letsencrypt request. 02.03.2023-17:15 - WARNING - Let's Encrypt SSL Cert for: domain.com could not be issued. This means that ISPConfig could not reach the domain. This can either mean that your system is behind a router that blocks traffic from the server itself to that domain or that the domain does not have an A record pointing to the server's IP address or that you use some kind of rewrite or proxy rules that block requests the the acme well known folder. Try disabling Let's encrypt check in ISPConfig under system > server config > web, then tick Let#s encrypt checkbox again in the website, run server.sh and post the result.
This worked: Try disabling Let's encrypt check in ISPConfig under system > server config > web, then tick Let#s encrypt checkbox again in the website, run server.sh Domains now work and it stays enabled. What was the root cause though, and can I safely enable the lets encrypt check again? Or is there something else underlying? server.sh Log as follows: 02.03.2023-18:42 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'. 02.03.2023-18:42 - DEBUG [server:177] - Found 1 changes, starting update process. 02.03.2023-18:42 - DEBUG [plugins.inc:118] - Calling function 'ssl' from plugin 'apache2_plugin' raised by event 'web_domain_update'. 02.03.2023-18:42 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'apache2_plugin' raised by event 'web_domain_update'. 02.03.2023-18:42 - DEBUG [system.inc:2399] - safe_exec cmd: chattr -i '/var/www/clients/client0/web1' - return code: 0 02.03.2023-18:42 - DEBUG [system.inc:2399] - safe_exec cmd: chattr +i '/var/www/clients/client0/web1' - return code: 0 02.03.2023-18:42 - DEBUG [system.inc:2399] - safe_exec cmd: df -T '/var/www/clients/client0/web1'|awk 'END{print $2,$NF}' - return code: 0 02.03.2023-18:42 - DEBUG [system.inc:2399] - safe_exec cmd: which 'setquota' 2> /dev/null - return code: 0 02.03.2023-18:42 - DEBUG [system.inc:2399] - safe_exec cmd: setquota -u 'web1' '0' '0' 0 0 -a &> /dev/null - return code: 0 02.03.2023-18:42 - DEBUG [system.inc:2399] - safe_exec cmd: setquota -T -u 'web1' 604800 604800 -a &> /dev/null - return code: 0 02.03.2023-18:42 - DEBUG [system.inc:2399] - safe_exec cmd: chattr +i '/var/www/clients/client0/web1' - return code: 0 02.03.2023-18:42 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0 02.03.2023-18:42 - DEBUG [system.inc:2082] - Trying to use Systemd to restart service 02.03.2023-18:42 - DEBUG [system.inc:2399] - safe_exec cmd: systemctl is-enabled 'apache2' 2>&1 - return code: 0 02.03.2023-18:42 - DEBUG [letsencrypt.inc:431] - Create Let's Encrypt SSL Cert for: domain.com 02.03.2023-18:42 - DEBUG [letsencrypt.inc:432] - Let's Encrypt SSL Cert domains: 02.03.2023-18:42 - DEBUG [system.inc:1819] - exec: R=0 ; C=0 ; /root/.acme.sh/acme.sh --issue -d domain.com -d www.domain.com -w /usr/local/ispconfig/interface/acme --always-force-new-domain-key --keylength 4096; R=$? ; if [ $R -eq 0 -o $R -eq 2 ] ; then /root/.acme.sh/acme.sh --install-cert -d domain.com -d www.domain.com --key-file '/var/www/clients/client0/web1/ssl/domain.com-le.key' --fullchain-file '/var/www/clients/client0/web1/ssl/domain.com-le.crt' --reloadcmd 'systemctl force-reload apache2.service' --log '/var/log/ispconfig/acme.log'; C=$? ; fi ; if [ $C -eq 0 ] ; then exit $R ; else exit $C ; fi 02.03.2023-18:42 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0 02.03.2023-18:42 - DEBUG [apache2 plugin.inc:1828] - Enable SSL for: domain.com 02.03.2023-18:42 - DEBUG [apache2 plugin.inc:1874] - Writing the vhost file: /etc/apache2/sites-available/domain.com.vhost 02.03.2023-18:42 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0 02.03.2023-18:42 - DEBUG [apache2 plugin.inc:3445] - Writing the PHP-FPM config file: /etc/php/8.1/fpm/pool.d/web1.conf 02.03.2023-18:42 - DEBUG [services.inc:56] - Calling function 'restartPHP_FPM' from module 'web_module'. 02.03.2023-18:42 - DEBUG [system.inc:2082] - Trying to use Systemd to restart service 02.03.2023-18:42 - DEBUG [system.inc:2399] - safe_exec cmd: systemctl is-enabled 'php8.1-fpm' 2>&1 - return code: 0 02.03.2023-18:42 - DEBUG [web module.inc:316] - Restarting php-fpm: systemctl reload php8.1-fpm.service 02.03.2023-18:42 - DEBUG [apache2 plugin.inc:1992] - Apache status is: running 02.03.2023-18:42 - DEBUG [services.inc:56] - Calling function 'restartHttpd' from module 'web_module'. 02.03.2023-18:42 - DEBUG [system.inc:2082] - Trying to use Systemd to restart service 02.03.2023-18:42 - DEBUG [system.inc:2399] - safe_exec cmd: systemctl is-enabled 'apache2' 2>&1 - return code: 0 02.03.2023-18:42 - DEBUG [web module.inc:246] - Restarting httpd: systemctl restart apache2.service 02.03.2023-18:42 - DEBUG [apache2 plugin.inc:1995] - Apache restart return value is: 0 02.03.2023-18:42 - DEBUG [apache2 plugin.inc:2006] - Apache online status after restart is: running 02.03.2023-18:42 - DEBUG [modules.inc:240] - Processed datalog_id 8562 02.03.2023-18:42 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
The root cause is that the domains are unreachable from your server. You can not re-enable it as long as your system is not able to connect to the domains.
Okay - and how to troubleshoot that? I've disabled all firewall, stopped fail2ban and checked A records etc. You're correct though, even using lynx from the cli they don't show. /var/log/ispconfig/httpd/domain.com/error.log ----------- [Thu Mar 02 13:32:02.877313 2023] [authz_core:error] [pid 14501:tid 139960115734272] [client x.y.z.z:51366] AH01630: client denied by server configuration: /var/www/clients/client0/web31/web/server-status
Is your system behind a NAT router? or in other words, does the server has a public IP assigned directly on its internal network card, or does it has an internal IP assigned and the external IP is configured on a router or another network device?
Actually, ignore that, from localhost, lynx works and brings up the sites, albeit slower than from outside.. dns appears to respond timely... at least for ipv4... I think I may have an ipv6 issue. Tnanks for all the help, first time I've had to ask and you were straight back with answers. I also learned how useful debug mode can be. Maybe an up to date wiki on where logs are located for ispconfig functions would help others? Thanks
NOT ispconfig to blame it seems IPv6 routing - for some reason my server IPv6 had stopped working.. and that seems likely to have been the cause. I've sorted that (I think maybe some changes at my vps host maybe the issue for that). unticked the skip letsencrypt check, renabled firewall, fail2ban etc and all is good I'll do the crontab later after I've finished watching some cogs turning. I'm really enjoying the debug mode (after my ignorance at not enabling in the gui :/ ) Moral of the story, just cos there's an upgraded version available, and you made some recent custom changes.... That's NOT the reason your refrigerator went off. Thanks again Till
