I thought I'd finished migrating everything OK, but this morning, I realise one of my websites isn't running SSL (although I had checked it post migration and it WAS working). I click on the LE checkbox, but the SSL configuration isn't applied. The vhost file does not include the 443 SSL section. Looking at the website's logs, I do have (domain name changed to domain.com): Code: [Thu Jul 29 14:51:07.239232 2021] [ssl:warn] [pid 812] AH01909: domain.com:443:0 server certificate does NOT include an ID which matches the server name [Thu Jul 29 14:53:06.486774 2021] [ssl:warn] [pid 812] AH01909: domain.com:443:0 server certificate does NOT include an ID which matches the server name [Thu Jul 29 14:55:04.165611 2021] [ssl:warn] [pid 812] AH01909: domain.com:443:0 server certificate does NOT include an ID which matches the server name [Thu Jul 29 14:56:03.167401 2021] [proxy_fcgi:error] [pid 424134] [client 146.90.72.152:50744] AH01067: Failed to read FastCGI header The certificate is present on the system (it was copied across from my previous server running ISPconfig) and was renewed a week ago. The LE debug log in /car/log/letsencrypt/letsencrypt.log doesn't show anything. htf-common-issues output: Code: ##### SERVER ##### IP-address (as per hostname): ***.***.***.*** [WARN] could not determine server's ip address by ifconfig [INFO] OS version is Ubuntu 20.04.2 LTS [INFO] uptime: 15:22:24 up 1 day, 16:32, 1 user, load average: 0.02, 0.03, 0.00 [INFO] memory: total used free shared buff/cache available Mem: 7.8Gi 2.0Gi 1.3Gi 124Mi 4.4Gi 5.3Gi Swap: 511Mi 22Mi 489Mi [INFO] systemd failed services status: UNIT LOAD ACTIVE SUB DESCRIPTION 0 loaded units listed. [INFO] ISPConfig is installed. ##### ISPCONFIG ##### ISPConfig version is 3.2.5 ##### VERSION CHECK ##### [INFO] php (cli) version is 7.4.3 [INFO] php-cgi (used for cgi php in default vhost!) is version 7.4.3 ##### PORT CHECK ##### ##### MAIL SERVER CHECK ##### ##### RUNNING SERVER PROCESSES ##### [INFO] I found the following web server(s): Apache 2 (PID 812) [INFO] I found the following mail server(s): Postfix (PID 1588) [INFO] I found the following pop3 server(s): Dovecot (PID 990) [INFO] I found the following imap server(s): Dovecot (PID 990) [INFO] I found the following ftp server(s): PureFTP (PID 889) ##### LISTENING PORTS ##### (only () Local (Address) [localhost]:10025 (1588/master) [localhost]:10026 (1288/amavisd-new) [localhost]:10027 (1588/master) [anywhere]:587 (1588/master) [localhost]:11211 (641/memcached) [anywhere]:110 (990/dovecot) [anywhere]:143 (990/dovecot) [anywhere]:8080 (812/apache2) [anywhere]:465 (1588/master) [anywhere]:21 (889/pure-ftpd) ***.***.***.***:53 (642/named) ***.***.***.***:53 (642/named) [localhost]:53 (642/named) ***.***.***.***:53 (540/systemd-resolve) [anywhere]:22 (1002/sshd:) [anywhere]:25 (1588/master) [localhost]:953 (642/named) [anywhere]:993 (990/dovecot) [anywhere]:995 (990/dovecot) [localhost]:10023 (813/postgrey) [localhost]:10024 (1288/amavisd-new) *:*:*:*::*:3306 (386173/mysqld) *:*:*:*::*:10026 (1288/amavisd-new) *:*:*:*::*:587 (1588/master) [localhost]10 (990/dovecot) [localhost]43 (990/dovecot) *:*:*:*::*:80 (812/apache2) *:*:*:*::*:465 (1588/master) *:*:*:*::*:8081 (812/apache2) *:*:*:*::*:21 (889/pure-ftpd) *:*:*:*::*:53 (642/named) *:*:*:*::*:22 (1002/sshd:) *:*:*:*::*:25 (1588/master) *:*:*:*::*:953 (642/named) *:*:*:*::*:443 (812/apache2) *:*:*:*::*:993 (990/dovecot) *:*:*:*::*:995 (990/dovecot) *:*:*:*::*:10023 (813/postgrey) *:*:*:*::*:10024 (1288/amavisd-new) ##### IPTABLES ##### Chain INPUT (policy DROP) target prot opt source destination f2b-pure-ftpd tcp -- [anywhere]/0 [anywhere]/0 multiport dports 21 f2b-sshd tcp -- [anywhere]/0 [anywhere]/0 multiport dports 22 ufw-before-logging-input all -- [anywhere]/0 [anywhere]/0 ufw-before-input all -- [anywhere]/0 [anywhere]/0 ufw-after-input all -- [anywhere]/0 [anywhere]/0 ufw-after-logging-input all -- [anywhere]/0 [anywhere]/0 ufw-reject-input all -- [anywhere]/0 [anywhere]/0 ufw-track-input all -- [anywhere]/0 [anywhere]/0 Chain FORWARD (policy DROP) target prot opt source destination ufw-before-logging-forward all -- [anywhere]/0 [anywhere]/0 ufw-before-forward all -- [anywhere]/0 [anywhere]/0 ufw-after-forward all -- [anywhere]/0 [anywhere]/0 ufw-after-logging-forward all -- [anywhere]/0 [anywhere]/0 ufw-reject-forward all -- [anywhere]/0 [anywhere]/0 ufw-track-forward all -- [anywhere]/0 [anywhere]/0 Chain OUTPUT (policy ACCEPT) target prot opt source destination ufw-before-logging-output all -- [anywhere]/0 [anywhere]/0 ufw-before-output all -- [anywhere]/0 [anywhere]/0 ufw-after-output all -- [anywhere]/0 [anywhere]/0 ufw-after-logging-output all -- [anywhere]/0 [anywhere]/0 ufw-reject-output all -- [anywhere]/0 [anywhere]/0 ufw-track-output all -- [anywhere]/0 [anywhere]/0 Chain f2b-pure-ftpd (1 references) target prot opt source destination RETURN all -- [anywhere]/0 [anywhere]/0 Chain f2b-sshd (1 references) target prot opt source destination REJECT all -- ***.***.***.*** [anywhere]/0 reject-with icmp-port-unreachable REJECT all -- ***.***.***.*** [anywhere]/0 reject-with icmp-port-unreachable REJECT all -- ***.***.***.*** [anywhere]/0 reject-with icmp-port-unreachable REJECT all -- ***.***.***.*** [anywhere]/0 reject-with icmp-port-unreachable RETURN all -- [anywhere]/0 [anywhere]/0 Chain ufw-after-forward (1 references) target prot opt source destination Chain ufw-after-input (1 references) target prot opt source destination ufw-skip-to-policy-input udp -- [anywhere]/0 [anywhere]/0 udp dpt:137 ufw-skip-to-policy-input udp -- [anywhere]/0 [anywhere]/0 udp dpt:138 ufw-skip-to-policy-input tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:139 ufw-skip-to-policy-input tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:445 ufw-skip-to-policy-input udp -- [anywhere]/0 [anywhere]/0 udp dpt:67 ufw-skip-to-policy-input udp -- [anywhere]/0 [anywhere]/0 udp dpt:68 ufw-skip-to-policy-input all -- [anywhere]/0 [anywhere]/0 ADDRTYPE match dst-type BROADCAST Chain ufw-after-logging-forward (1 references) target prot opt source destination LOG all -- [anywhere]/0 [anywhere]/0 limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW BLOCK] " Chain ufw-after-logging-input (1 references) target prot opt source destination LOG all -- [anywhere]/0 [anywhere]/0 limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW BLOCK] " Chain ufw-after-logging-output (1 references) target prot opt source destination Chain ufw-after-output (1 references) target prot opt source destination Chain ufw-before-forward (1 references) target prot opt source destination ACCEPT all -- [anywhere]/0 [anywhere]/0 ctstate RELATED,ESTABLISHED ACCEPT icmp -- [anywhere]/0 [anywhere]/0 icmptype 3 ACCEPT icmp -- [anywhere]/0 [anywhere]/0 icmptype 11 ACCEPT icmp -- [anywhere]/0 [anywhere]/0 icmptype 12 ACCEPT icmp -- [anywhere]/0 [anywhere]/0 icmptype 8 ufw-user-forward all -- [anywhere]/0 [anywhere]/0 Chain ufw-before-input (1 references) target prot opt source destination ACCEPT all -- [anywhere]/0 [anywhere]/0 ACCEPT all -- [anywhere]/0 [anywhere]/0 ctstate RELATED,ESTABLISHED ufw-logging-deny all -- [anywhere]/0 [anywhere]/0 ctstate INVALID DROP all -- [anywhere]/0 [anywhere]/0 ctstate INVALID ACCEPT icmp -- [anywhere]/0 [anywhere]/0 icmptype 3 ACCEPT icmp -- [anywhere]/0 [anywhere]/0 icmptype 11 ACCEPT icmp -- [anywhere]/0 [anywhere]/0 icmptype 12 ACCEPT icmp -- [anywhere]/0 [anywhere]/0 icmptype 8 ACCEPT udp -- [anywhere]/0 [anywhere]/0 udp spt:67 dpt:68 ufw-not-local all -- [anywhere]/0 [anywhere]/0 ACCEPT udp -- [anywhere]/0 ***.***.***.*** udp dpt:5353 ACCEPT udp -- [anywhere]/0 ***.***.***.*** udp dpt:1900 ufw-user-input all -- [anywhere]/0 [anywhere]/0 Chain ufw-before-logging-forward (1 references) target prot opt source destination Chain ufw-before-logging-input (1 references) target prot opt source destination Chain ufw-before-logging-output (1 references) target prot opt source destination Chain ufw-before-output (1 references) target prot opt source destination ACCEPT all -- [anywhere]/0 [anywhere]/0 ACCEPT all -- [anywhere]/0 [anywhere]/0 ctstate RELATED,ESTABLISHED ufw-user-output all -- [anywhere]/0 [anywhere]/0 Chain ufw-logging-allow (0 references) target prot opt source destination LOG all -- [anywhere]/0 [anywhere]/0 limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW ALLOW] " Chain ufw-logging-deny (2 references) target prot opt source destination RETURN all -- [anywhere]/0 [anywhere]/0 ctstate INVALID limit: avg 3/min burst 10 LOG all -- [anywhere]/0 [anywhere]/0 limit: avg 3/min burst 10 LOG flags 0 level 4 prefix "[UFW BLOCK] " Chain ufw-not-local (1 references) target prot opt source destination RETURN all -- [anywhere]/0 [anywhere]/0 ADDRTYPE match dst-type LOCAL RETURN all -- [anywhere]/0 [anywhere]/0 ADDRTYPE match dst-type MULTICAST RETURN all -- [anywhere]/0 [anywhere]/0 ADDRTYPE match dst-type BROADCAST ufw-logging-deny all -- [anywhere]/0 [anywhere]/0 limit: avg 3/min burst 10 DROP all -- [anywhere]/0 [anywhere]/0 Chain ufw-reject-forward (1 references) target prot opt source destination Chain ufw-reject-input (1 references) target prot opt source destination Chain ufw-reject-output (1 references) target prot opt source destination Chain ufw-skip-to-policy-forward (0 references) target prot opt source destination DROP all -- [anywhere]/0 [anywhere]/0 Chain ufw-skip-to-policy-input (7 references) target prot opt source destination DROP all -- [anywhere]/0 [anywhere]/0 Chain ufw-skip-to-policy-output (0 references) target prot opt source destination ACCEPT all -- [anywhere]/0 [anywhere]/0 Chain ufw-track-forward (1 references) target prot opt source destination Chain ufw-track-input (1 references) target prot opt source destination Chain ufw-track-output (1 references) target prot opt source destination ACCEPT tcp -- [anywhere]/0 [anywhere]/0 ctstate NEW ACCEPT udp -- [anywhere]/0 [anywhere]/0 ctstate NEW Chain ufw-user-forward (1 references) target prot opt source destination Chain ufw-user-input (1 references) target prot opt source destination ACCEPT tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:21 ACCEPT tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:22 ACCEPT tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:25 ACCEPT tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:53 ACCEPT tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:80 ACCEPT tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:110 ACCEPT tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:143 ACCEPT tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:443 ACCEPT tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:465 ACCEPT tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:587 ACCEPT tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:993 ACCEPT tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:995 ACCEPT tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:3306 ACCEPT tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:4190 ACCEPT tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:8080 ACCEPT tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:8081 ACCEPT tcp -- [anywhere]/0 [anywhere]/0 multiport dports 40110:40210 ACCEPT udp -- [anywhere]/0 [anywhere]/0 udp dpt:53 Chain ufw-user-limit (0 references) target prot opt source destination LOG all -- [anywhere]/0 [anywhere]/0 limit: avg 3/min burst 5 LOG flags 0 level 4 prefix "[UFW LIMIT BLOCK] " REJECT all -- [anywhere]/0 [anywhere]/0 reject-with icmp-port-unreachable Chain ufw-user-limit-accept (0 references) target prot opt source destination ACCEPT all -- [anywhere]/0 [anywhere]/0 Chain ufw-user-logging-forward (0 references) target prot opt source destination Chain ufw-user-logging-input (0 references) target prot opt source destination Chain ufw-user-logging-output (0 references) target prot opt source destination Chain ufw-user-output (1 references) target prot opt source destination ##### LET'S ENCRYPT ##### Certbot is installed in /usr/bin/letsencrypt Where do I start? Thanks!
Disable the ssl checkbox, ensure the is no certificate under the ssl tab (delete it if so), remove any certificate files in the website ssl/ directory, then enable the ssl and let's encrypt checkbox again. If it doesn't work, check the log file and attempt again with server debugging enabled.
Thank you Jesse. No certificate unfo under the ssl tab. I deleted the files (links) in the ssl directory, tried re-enabling the ssl and LE checkboxes and still nothing. The links in /ssl are not re-created, and still no SSL of course. Which / how do you enable "server debugging" ?
Since this is a migration case, do ensure that if old server used certbot, the new server also uses the same and there is no acme.sh script installed at the same time. Ensure that you have the latest certbot installed which is most probably with snap for Ubuntu 20.04 and no more than one LE account on that server. I just mentioned few common ones but the best is to read and follow the sticky LE FAQ as stated above.
Yes, no acme installed. I have version 0.40.0-1ubuntu0.1 Interestingly, in the letsencrypt logs, there is a warning of more recent config files, for example: Code: 2021-07-29 16:46:03,578:INFO:certbot.storage:Attempting to parse the version 1.9.0 renewal configuration file found at /etc/letsencrypt/renewal/www.domainxyz.com.conf with version 0.40.0 of Certbot. This might not work. but these are different domains to the one with the issue. It does appear to do something with the problematic domain; I have this: Code: 2021-07-29 16:36:02,177:DEBUG:certbot.main:certbot version: 0.40.0 2021-07-29 16:36:02,177:DEBUG:certbot.main:Arguments: ['--domains', 'domain.com', '--domains', 'www.domain.com'] 2021-07-29 16:36:02,177:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot) 2021-07-29 16:36:02,186:DEBUG:certbot.log:Root logging level set at 20 2021-07-29 16:36:02,187:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log Thanks, having been through that, I cannot see anything wrong. If I disable the server.sh in crontab and run manually, I don't seem to see much, I just get this: Code: 29.07.2021-16:38 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'. 29.07.2021-16:38 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock finished server.php.
Then Let's encrypt has not been activated for the website again. Go to website settings, enable let's encrypt and press save, then run server.sh and post the output.
Aha. Found the problem! In the problematic website, the let's encrypt renewal conf file was missing one of the two website URLs ! Code: # renew_before_expiry = 30 days version = 0.31.0 archive_dir = /etc/letsencrypt/archive/www.domain.com cert = /etc/letsencrypt/live/www.domain.com/cert.pem privkey = /etc/letsencrypt/live/www.domain.com/privkey.pem chain = /etc/letsencrypt/live/www.domain.com/chain.pem fullchain = /etc/letsencrypt/live/www.domain.com/fullchain.pem # Options used in the renewal process [renewalparams] authenticator = webroot rsa_key_size = 4096 account = 8aa4c72191c2d2af31e99fc4eed42c4d server = https://acme-v02.api.letsencrypt.org/directory post_hook = echo '1' > /usr/local/ispconfig/server/le.restart [[webroot_map]] www.domain.com = /usr/local/ispconfig/interface/acme domain.com = /usr/local/ispconfig/interface/acme <--- this line was missing!
And if I remove that line and manually run the server.sh command (thanks Till, I feel silly for not trying what you said!), I do get an error: Code: 29.07.2021-17:10 - DEBUG - Let's Encrypt Cert file: does not exist. Now that conf file was last modified on 22nd July matching when the certificate was renewed. I then migrated using the migration tool a few days later. What I'm trying to understand is what changed in my configuration during or rather post migration that would have broken the configuration where it now required both tld.com and www.tld.com It's a mystery. But note taken on how to debug in future. Thank you all for your help!
Here you go with the missing line in the conf file. Code: 29.07.2021-17:10 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'. 29.07.2021-17:10 - DEBUG - Found 1 changes, starting update process. 29.07.2021-17:10 - DEBUG - Calling function 'ssl' from plugin 'apache2_plugin' raised by event 'web_domain_update'. 29.07.2021-17:10 - DEBUG - Calling function 'update' from plugin 'apache2_plugin' raised by event 'web_domain_update'. 29.07.2021-17:10 - DEBUG - safe_exec cmd: chattr -i '/var/www/clients/client11/web5' - return code: 0 29.07.2021-17:10 - DEBUG - safe_exec cmd: chattr +i '/var/www/clients/client11/web5' - return code: 0 29.07.2021-17:10 - DEBUG - safe_exec cmd: df -T '/var/www/clients/client11/web5'|awk 'END{print $2,$NF}' - return code: 0 29.07.2021-17:10 - DEBUG - safe_exec cmd: which 'setquota' 2> /dev/null - return code: 0 29.07.2021-17:10 - DEBUG - safe_exec cmd: setquota -u 'web5' '0' '0' 0 0 -a &> /dev/null - return code: 0 29.07.2021-17:10 - DEBUG - safe_exec cmd: setquota -T -u 'web5' 604800 604800 -a &> /dev/null - return code: 0 29.07.2021-17:10 - DEBUG - safe_exec cmd: chattr +i '/var/www/clients/client11/web5' - return code: 0 29.07.2021-17:10 - DEBUG - LE version is 0.40.0, so using certificates command and --cert-name instead of --expand 29.07.2021-17:10 - DEBUG - Migration mode active, skipping Let's Encrypt SSL Cert creation for: domain.com 29.07.2021-17:10 - DEBUG - LE CERT OUTPUT: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 29.07.2021-17:10 - DEBUG - LE CERT OUTPUT: Found the following matching certs: 29.07.2021-17:10 - DEBUG - LE CERT OUTPUT: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 29.07.2021-17:10 - DEBUG - LE CERT OUTPUT: 29.07.2021-17:10 - DEBUG - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0 29.07.2021-17:10 - DEBUG - Let's Encrypt Cert file: does not exist. 29.07.2021-17:10 - DEBUG - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0 29.07.2021-17:10 - DEBUG - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0 29.07.2021-17:10 - DEBUG - Writing the vhost file: /etc/apache2/sites-available/domain.com.vhost 29.07.2021-17:10 - DEBUG - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0 29.07.2021-17:10 - DEBUG - Writing the PHP-FPM config file: /etc/php/7.4/fpm/pool.d/web5.conf 29.07.2021-17:10 - DEBUG - Calling function 'restartPHP_FPM' from module 'web_module'. 29.07.2021-17:10 - DEBUG - Restarting php-fpm: systemctl reload php7.4-fpm.service 29.07.2021-17:10 - DEBUG - Processed datalog_id 424 29.07.2021-17:10 - DEBUG - Calling function 'restartHttpd' from module 'web_module'. 29.07.2021-17:10 - DEBUG - Restarting httpd: systemctl reload apache2.service 29.07.2021-17:10 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock finished server.php. All sorted now.
The Migration mode on your system is still active, please go to System > Server config in ISPConfig and deactivate migration mode. As long as it's active, no new LE certs will be issued. Normally, this mode is switched off automatically at the end of the migration, might be that this has failed e.g. when the webserver was offline due to an error at the end so the tool could not reach the API.
