I have several ISPConfig boxes in production and LetsEncrypt works well on all of them except one. It was working properly, but at some point in the last three months, certificates stopped renewing properly. The LetsEncrypt log shows a few lines saying that certbot was referenced, but ISPConfig doesn't seem to be initiating the certbot process. The box in question is: - Ubuntu 20.04 - ISPConfig 3.2.5 - PHP7.4.20 - Certbot 0.40.0 from the Ubuntu Repo When the LetsEncrypt box is checked in the ISPConfig GUI, this is the output in /var/log/letsencrypyt/letsencrypt.log: Code: 2021-06-30 10:34:01,954:DEBUG:certbot.main:certbot version: 0.40.0 2021-06-30 10:34:01,954:DEBUG:certbot.main:Arguments: ['--domains', 'testbox.example.org'] 2021-06-30 10:34:01,954:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot) 2021-06-30 10:34:01,961:DEBUG:certbot.log:Root logging level set at 20 2021-06-30 10:34:01,961:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log On my other ISPConfig boxes, this section is followed by responses showing certbot activity. On this ISPConfig box, there is no other information written to the log. I have 'apt remove --purge certbot' and reinstalled. No difference. I have 'ispconfig_upate.sh --force', but that hasn't helped either. The 'Server Config - Web - Skip Lets Encrypt Check' box is checked. Thank you for your help.
Only if you are behind router or have other related problems but other than that what say your log at /var/log/letsencrypt/letsencrypt.log? Let’s Encrypt Error FAQ
Your certbot version is most likely too old, not too old for ISPConfig, so there is no issue how ISPConfig calls certbot as you mentioned in the headline, the issue is that Let's encrypt might not issue certs anymore when contacted by such an old certbot version as Ubuntu does not keep up with certbot updates for a long time. Install certbot as described here: https://certbot.eff.org/ You can use the method to install it via snap.
Thank you both for your suggestions. As an experiment, I upgraded one of our other ISPConfig servers to the same level of certbot, ISPConfig and Ubuntu 20.04 upgrades. LetsEncrypt has continue to work properly. That seems to indicate that the problem with my original ISPConfig server is local to that box and is not an ISPConfig issue. I will migrate all the websites to one of the other ISPConfig servers and rebuild it. Thank you again for your time.
I would like to give one more update to this. I built a new ISPConfig server on Ubuntu 20.04. It has identical specs to the ISPConfig server that was not renewing LetsEncrypt certificates properly. I built a test website on the new ISPConfig server and it pulled the certificate properly. Everything worked as it should. I then took the database backup from the bad ISPConfig server and applied it to the new ISPConfig server. No other changes were made. After applying the database backup to the new ISPConfig server, certificates stopped deploying properly. I noticed the same messages in the letsencrypt.log that I mentioned in my first post. This seems to indicate that something in the Database is keeping the LetsEncrypt certs from being pulled from the LetsEncrypt servers. Is there a spot in the database you would recommend I look to try to resolve this issue with LetsEncrypt certificates? Is there a table that keeps track of whether a cert is due for renewal? Or is there some other table I might look at? Thanks again for your help.
Most issues I read in here is about the migration that didn't work because target server is actually using acme.sh, are you sure that is not among your other problems that caused this issue?
No. But as ahrasis mentioned, you probably mixed certbot and acme.sh setups. And please read Letsencrypt error faq, as referred to in each thread that deals with letsencrypt, to find out which issue your server has.
Thank you both again for your advice. The answer is no and yes. I manage a dozen ISPConfig webservers and we have standardized on certbot from the Ubuntu repos. It makes it easier to keep things consistent and updated. When certificates stopped renewing, only certbot was on this machine. I verified this by searching the entire machine for acme.sh. It wasn't in /root/ or any of the other places it could have been installed. That said, once certificates started failing, I attempted to switch to acme.sh since I saw it referenced in the ISPConfig documentation and scripts. But acme.sh also failed to renew certificates properly, so I abandoned that option and removed it. As I mentioned in my initial post, when I check the LetsEncrypt box for a Site, those 5 lines are added to the /var/log/letsencrypt/letsencrypt.log, but LetsEncrypt does not run. It doesn't even attempt to call out to the LetsEncrypt servers. That is what makes all this so confusing. As I mentioned in my last post, I rebuilt the ISPConfig box to the same specs as the previous box. I put a test site onto it, and it pulled a LetsEncrypt certificate normally. I then restored the database from the faulty ISPConfig server and it stopped pulling certificates. There seems to be some connection between DB and the LetsEncrypt issue, but I have no idea what that might be. Thank you again for your time.
Use ispconfig debug mode and check logs from certbot to see what is happening in detail. And take care that you use the current ISPConfig version 3.2.5 and latest certbot version, which is 1.xx and not 0.40. These lines are added by certbot, so certbot is run and certbot is the tool that interacts with the Let's encrypt servers. Also, certbot is in no way connected to the ISPConfig database, it works based on its own config files, which it creates itself and which are not handled altered by ISPConfig. That restoring the ISPconfig database is related is very unlikely, it is probably just a timely coincidence or you got locked out by LE, might be just for the relevant domain, which explains that it happens again when you restore the original database. And certbot version 0.4.0 is most likely too old, I mentioned that earlier. I understand that you like to keep packages from ubuntu, but if these packages are so outdated that they cause issues and there will be no new updates, then you have to install a more recent version.
I have several other ISPConfig servers with the same version of Ubuntu, certbot and ISPConfg that do not have this issue. I took a snapshot of the rebuilt ISPConfig server before I imported the DB. After the import and the LE failure, I rolled the snapshot back to before the DB import and LetsEncrypt does work properly. If you say that LetsEncrypt issues and the DB are not related then I will keep digging through logs to see if I can find other clues. Thank you again for your time and expertise.
Try using ispconfig debug mode to get the exact certbot command that ispconfig is calling and then use that command to run manually, maybe you get some more info on the screen then.
Unfortunately, turning on Debug mode didn't give me extra onscreen information. It also didn't give me anything new in /var/log/letsencrypt/letsencrypt.log. I checked all the log entries in /var/log/ispconfig/ and none of them mentioned certbot or letsencrypt. One interesting thing I discovered, when letsencrypt tries a renewal on our functioning ISPConfig machines, it logs an entry in /var/log/ispconfig/httpd/site.name/access.log. When I try to enable letsEncrypt on the faulty machine, no connection attempts are logged. That seems to confirm that letsencrypt is not being called properly.
Well, I have an update that changes the conversation a little. I found out a colleague at some point used the certbot cli rather than the web interface to try to fix a certificate issue. I have read on other ISPConfig forum posts that doing this causes problems. Could this be the source of my problem? And if so, what would be the fix? Thanks again for your patience as we dig through this.
Please post the debug output that you get on the screen. Depends on how he used it. If he used the same commands that ispconfig is using, then there is no issue, if he has let certbot modify apache config files, then there is an issue. Please take a look at the apache sites-enabled directory, look for config files with '-le' in the file name. is there a config file of one of the sites that has -le in it's name?
I'd rather say yes and to fix it is to fully remove those certs created via cli and use ISPConfig panel to reissue for respective websites.
@till and @ahrasis Thank you both for your help. The suggestion that I turn on DEBUG mode lead me to the solution. In DEBUG mode, in /var/log/ispconfig/ispconfig.log, it told me that the server was in migration mode, which prohibits LetsEncrypt from checking certificates. My colleague had migrated a few sites in, but forgot to uncheck the Migration Mode check box. For others in this situation, Server Config - Server - Server Migration Mode. Make sure it is unchecked after you migrate sites! Thank you both again. I couldn't have found it without your help.
