I've spent several hours debugging, and I've tracked down a problem with ISPConfig's handling of vhosts with LetsEncrypt. Specifically, if a certificate _already exists_, ISPConfig will refuse to modify the vhost file. It will only modify the vhost file if it can issue its own certificate. This is a problem if you already _have_ a cert, and had to rework a site for some reason. If you're not ready for renewal, ISPConfig spits on you. Here's a clip from the log, plus what happened when I ran the command manually. The vhost in question is marked as having SSL enabled, but not Let's Encrypt - despite there being no actual errors. There is no SSL section in the vhost. Is this normal behaviour, and is there a way to work around it? 06.03.2019-06:08 - DEBUG - Create Let's Encrypt SSL Cert for: exampledomain.com 06.03.2019-06:08 - DEBUG - Let's Encrypt SSL Cert domains: --domains exampledomain.com --domains www.exampledomain.com 06.03.2019-06:08 - DEBUG - exec: /usr/bin/letsencrypt certonly -n --text --agree-tos --expand --authenticator webroot --server https://acme-v02.api.let sencrypt.org/directory --rsa-key-size 4096 --email [email protected] --domains exampledomain.com --domains www.exampledomain.com --webroot-path /usr/local /ispconfig/interface/acme 06.03.2019-06:08 - DEBUG - Let's Encrypt Cert file: does not exist. 06.03.2019-06:08 - DEBUG - Writing the vhost file: /etc/apache2/sites-available/exampledomain.com.vhost 06.03.2019-06:08 - DEBUG - Apache status is: running 06.03.2019-06:08 - DEBUG - Calling function 'restartHttpd' from module 'web_module'. 06.03.2019-06:08 - DEBUG - Restarting httpd: systemctl restart apache2.service 06.03.2019-06:08 - DEBUG - Apache restart return value is: 0 06.03.2019-06:08 - DEBUG - Apache online status after restart is: running 06.03.2019-06:08 - DEBUG - Processed datalog_id 6452 root@heimdall:/usr/local/ispconfig/server# /usr/bin/letsencrypt certonly -n --text --agree-tos --expand --authenticator webroot --server https://acme-v02.api.letsencrypt.org/directory --rsa-key-size 4096 --email [email protected] --domains exampledomain.com --domains www.exampledomain.com --webroot-path /usr/local/ispconfig/interface/acme Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator webroot, Installer None Cert not yet due for renewal Keeping the existing certificate - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Certificate not yet due for renewal; no action taken. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (Edited - deleted out the failed paste command line. )
This might be related to this: https://git.ispconfig.org/ispconfig/ispconfig3/issues/5258 The question is how you created that LE cert that already exists, did you use the exact same command that ISPconfig is using, or did you use a different method? E.g. when you would have used certbot with apache option, then certbot tries to create a copy of the vhost file which makes it impossible for ispconfig to further manage that website due to the duplicate manual config. You might want to check the apache sites-available and sites-enabled config directories to see if there are any config files with -le in their name, if that's the case, then you must have run certbot manually with apache plugin and this broke your setup.
Even more cute. I purged _all_ of my Let's Encrypt certs, and ran the utility. It happily created the certificate, then wrote a vhost file that still contains nothing to do with SSL.
I used the _exact_ command that ISPConfig displayed in the debug logs - as you can see in the first posting. I had run certbot manually in the past, but all of those files were purged.
Really strange. When the debug log says: 06.03.2019-06:08 - DEBUG - Let's Encrypt Cert file: does not exist. then ISPConfig did not found the cert and that's why SSL is not activated. The question is why it did not found the cert, certs are not selected by their file name (as they can change from time to time), the cert is selected based on the renew config files. Please post the renew config file from /etc/letsencrypt/renewal/ folder of this domain. Compare the domains listed in there if they match the domains of the site and ensure that all subdomains are listed in there too.
# renew_before_expiry = 30 days version = 0.31.0 archive_dir = /etc/letsencrypt/archive/exampledomain.com cert = /etc/letsencrypt/live/exampledomain.com/cert.pem privkey = /etc/letsencrypt/live/exampledomain.com/privkey.pem chain = /etc/letsencrypt/live/exampledomain.com/chain.pem fullchain = /etc/letsencrypt/live/exampledomain.com/fullchain.pem # Options used in the renewal process [renewalparams] account = 945e6da4eb0490598b05570f4d486ad1 server = https://acme-v02.api.letsencrypt.org/directory authenticator = webroot rsa_key_size = 4096 webroot_path = /usr/local/ispconfig/interface/acme, [[webroot_map]]
It still keeps repeating that the file doesn't exist - when it clearly does. I've deleted it twice, and it happily recreates it, then claims that the cert doesn't exist. So, something's broken. Could you tell me how to bypass the cert check and force it to write the SSL changes? I'm certain I can find it myself, but it could take several hours.
and there is really nothing after [[webroot_map]] line in that file? Here is a file from my server, as you can see, it lists the web roots: Code: # renew_before_expiry = 30 days version = 0.21.1 archive_dir = /etc/letsencrypt/archive/domain.tld cert = /etc/letsencrypt/live/domain.tld/cert.pem privkey = /etc/letsencrypt/live/domain.tld/privkey.pem chain = /etc/letsencrypt/live/domain.tld/chain.pem fullchain = /etc/letsencrypt/live/domain.tld/fullchain.pem # Options used in the renewal process [renewalparams] account = xxxxxxxxxxxxxxxxxxxxxx authenticator = webroot rsa_key_size = 4096 installer = None [[webroot_map]] domain.tld = /usr/local/ispconfig/interface/acme www.domain.tld = /usr/local/ispconfig/interface/acme
Nope, that's it. All of the files in that directory end at [[webroot_map]] (There are four. I have some manual certs for postfix, etc. )
I need to build a new server and manually migrate all of my hosted sites to it; I guess this drives home that I _have_ to do that ASAP. I just hate doing all that work.
Instead of trying to change the ispconfig code, alter the certbot config files and add the missing web root paths in the way it is shown in my example file and then tick the le and ssl checkboxes in the site again and see how it goes. We will try to find out why certbot fails to add the domains in its config file, maybe its a bug in a specific certbot version or certbot stores this info somwhere else now.
That forced ISPconfig to realize the cert was there, and that site is now functioning with SSL. I'll do the other two/three I need to have functioning, and finish my other work for the night.
The issue is really strange, two ISPConfig developers tested this now independently on different servers, installed latest certbot 0.31 and the current ISPConfig version 3.1.13p1 and when we create a new SSL cert, all domains that belong to that cert are listed in the [[webroot_map]] section of the newly generated cert.
Even with an old cert (0.23) that was created as standalone, once I added the webroot_map section, ispconfig would use it. Why does ISPConfig refuse to use certs maintained differently? Just curious, now. I'm not going to suggest putting a lot of effort into it. I'm going to build a new server and work on gradually merging my two servers into one, then creating a second just to be a slave for DNS.
The problem is that certbot tends to rename certs, adds suffixes to cert files and also to use other domains which are in the cert as file name. So you can not assume that just because the cert is named domain.tld.crt that this belongs to a website domain.tld.crt, it can be for a different site or different subset of domains, certbot has a long history of such issues. That's why ispconfig reads in all cert config files and checks all the domains inside against the domains, subdomains and alias domains and only if that matches, the cert will be used. If you created a cert manually with a wrong method and this results in certbot to fail to write the webroot map, then this cert is excluded from being used as it basically contains no usable domains. I explained it many many times to NEVER create LE certs manually on the shell on an ISPConfig system if you plan to have a site with the same name as this cert will not get used and you will not be able to use SSL in a website with that domain anymore in ISPConfig.
Well - the two certs I was trying to use _weren't_ created manually. I did find a glitch, in that I had to use a newer version of certbot, to get ISPConfig to recognize that it was even installed (0.23 is apparently too old) - but the instructions for using/installing Certbot/LetsEncrypt also need to be updated, because the options mentioned in the post don't match up with what certbot-auto does anymore. I've noticed that in the last few years, as Certbot and LE have changed - they REALLY change stuff very version, rather than only changing what's necessary. It's almost as bad as Microsoft or Firefox. "Hey, let's change it because our marketing people say it's going to be great!" Yeesh.
The way you install certbot does not matter, you can install it from packages of your Linux distribution or with certbot-auto program, ISPConfig recognizes it in all common places, even the original letsencrypt client is recognized and used if no certbot is installed. As you can see in my post above, the first cert was created with certbot 0.21 with ispconfig, so the version was even older than yours and worked fine with ISPConfig. Why certbot on your server failed to add a webroot map when not executed manually, I don't know. It is not reproducible here, tested by 3 developers on 3 different servers now.
That's why I said it's not worth chasing. My server is ... rather old now. Well, the hardware has been upgraded, but ISPConfig has been upgraded and updated over at least one full major revision, plus multiple minor revisions of both, plus OS upgrades. So, there's probably stuff scattered in my ISPConfig that doesn't match anyone else's.
