Let's encrypt certs not renewing after ispconfig update

Hey there!
I'm having problem renewing my LE certificates and I would need your kind help.
I have updated my debian 10 to 11 few weeks ago, everything went smoothly, and after I was made sure everything is working, I have updated ispconfig to 3.2.9. I always followed the perfect setup tutorials they are always worked for me without issues.
Now one of my certificates is expired and is not renewing.
I tried to debug, and discovered that those are still valid are working fine (for some more days), but iscpconfig does not renew the expired ones.
When I try a certbot renew --dry-run I get the very same error message for all of my domains, for example:

Code:

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: mydomain.com
  Type:   unauthorized
  Detail: IP4address: Invalid response from http://mydomain.com/.well-known/acme-challenge/YysiI1jxG6v-I7Ik6OhkuAG2QCzS_7EQxpM9qx3dPBw

The domains work perfectly, if I put a test.txt file under acme-challenge, I can view it from browser.

I had an old version of certbot that came with the distro, but I updated to the latest certbot (2.3) from snap, but the issue remained the same.

I tried to check the "Skip Letsencrypt domain check" in ispconfig, it did not help.
When I check the "Let's encrypt SSL" checkbox, it wont stay checked of course, because of the above error.
I also tried ispconfig_update --force script, but it also did not help.
However it gave me 2 scary warnings, not sure if they are related to the issue:

Code:

Service 'mail_server' has not been detected (strongly recommended, currently enabled) do you want to disable it?  (yes,no) [yes]: no                                                                               
Service 'web_server' has not been detected (strongly recommended, currently enabled) do you want to disable it?  (yes,no) [yes]: no

WARNING: If ISPConfig-Interface (Webfrontend) is installed on this Server we will configure the Web Server anyways but will not enable it in ISPConfig.

Both Apache and postfix are working just fine, knock-knock, but I'm very afraid of this issue, as my most important domains' certificate will also expire in a few days.

thanks a million if any hint can be provided...
have a good evening,
fmarton

 

Make sure you remove old certbot thoroughly before installing a new one via snap. There could be problem if you don't do this right.

Regarding your error, check the dns record for that website domain.

 

Hey Ahrasis,
Many thanks for your answer!
The DNS record for this domain is fine. It's served via browser as well as all my other domains, they are live sites and I have also checked with a test.txt (in .well-known/acme-challange folder)
Is there a way I can make sure no leftovers remained of my old certbot?
I have tried to make a 'certbot certonly' for this domain, (because it does not have any important content,) and the certificate is generated successfully from command line, and the new cert is working, I can see the updated dates if I open the website and check its certificate.
However if I check Let's encrypt SSL for this domain in ISPConfig (after new cert generation from command line) it wont stay on, I guess it's because I made the cert from command line and not in ISPC...?
After this the domain shows cert error again in browser, it refers to common name error, and displays another domain name (of mine)...
Any more ideas please?
Many thanks!
fmarton

 

I try to clean things up a bit, let's just forget about my tests with mydomain.com, and get back to my root problem:
My "main" certificate is expired yesterday. By "main" I mean its like sub.mydomain.com and it is the same as my host name.
My mail server uses this server name to send and receives messages, and my ispconfig is also accessible from this domain with high port number. It worked like this for many years.
After the debian update (from 10 to 11) and ispconfig update (3.1 to 3.2.9) I realized that the certificate for this "main" domain is not renewing anymore. (Thus I made the dry-run tests when all failed with same error)
In ISPC both SSL and Let's encrypt SSL checkboxes are checked.
I tried to remove/recheck Let's encrypt SSL checkbox, it looks like it worked, but it is not renewing.
I attach the full log of ispconfig and letsencrypt.
I think if I could solve this one, that would also solve all the others.
It is not a problem if I have to do something manually as there are not many virtual hosts on my server, only about a dozen.
Thanks a million for any help!

 

Hello Till,
Thank you very much and sorry for the late reply, I was away, but now I'm fully on the subject again.
I have checked it now, and no, there is no '-le' in any of the vhost file names under /etc/apache2/sites-enabled.

 

It gives the very same error messages for all domains (real domain is replaced with domain1.com below, I hope it is ok).

here is the first one:

Code:

# certbot -n renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/domain1.com-0010.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for domain1.com and www.domain1.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: domain1.com
  Type:   unauthorized
  Detail: 80.77.123.98: Invalid response from http://domain1.com/.well-known/acme-challenge/a9gPJynn9VRreFxPHTeXv6tKhyRzIfOIQKtkuBDYM9g: 404

  Domain: www.domain1.com
  Type:   unauthorized
  Detail: 80.77.123.98: Invalid response from http://www.domain1.com/.well-known/acme-challenge/l6m4w7FB9GOn5bb6d0N1bCCp_n7siSgu9zVZYfzlGfc: 404

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Failed to renew certificate domain1.com-0010 with error: Some challenges have failed.

 

I have already done this, but I gave it another shot now. It made no difference, certificates are not generated. Here is the output of the update script:

Code:

# ispconfig_update.sh --force


--------------------------------------------------------------------------------
 _____ ___________   _____              __ _
|_   _/  ___| ___ \ /  __ \            / _(_)
  | | \ `--.| |_/ / | /  \/ ___  _ __ | |_ _  __ _
  | |  `--. \  __/  | |    / _ \| '_ \|  _| |/ _` |
 _| |_/\__/ / |     | \__/\ (_) | | | | | | | (_| |
 \___/\____/\_|      \____/\___/|_| |_|_| |_|\__, |
                                              __/ |
                                             |___/
--------------------------------------------------------------------------------


>> Update

Please choose the update method. For production systems select 'stable'.
WARNING: The update from GIT is only for development systems and may break your current setup. Do not use the GIT version on servers that host any live websites!
Note: On Multiserver systems, enable maintenance mode and update your master server first. Then update all slave servers, and disable maintenance mode when all servers are updated.

Select update method (stable,nightly,git-develop) [stable]:

Downloading ISPConfig update.
Unpacking ISPConfig update.


--------------------------------------------------------------------------------
 _____ ___________   _____              __ _         ____
|_   _/  ___| ___ \ /  __ \            / _(_)       /__  \
  | | \ `--.| |_/ / | /  \/ ___  _ __ | |_ _  __ _    _/ /
  | |  `--. \  __/  | |    / _ \| '_ \|  _| |/ _` |  |_ |
 _| |_/\__/ / |     | \__/\ (_) | | | | | | | (_| | ___\ \
 \___/\____/\_|      \____/\___/|_| |_|_| |_|\__, | \____/
                                              __/ |
                                             |___/
--------------------------------------------------------------------------------


>> Update

Operating System: Debian 11.0 (Bullseye) or compatible

This application will update ISPConfig 3 on your server.

Shall the script create a ISPConfig backup in /var/backup/ now? (yes,no) [yes]:

Creating backup of "/usr/local/ispconfig" directory...
Creating backup of "/etc" directory...
Creating backup of "/etc/letsencrypt" directory...
Checking ISPConfig database .. OK
Starting incremental database update.
Loading SQL patch file: /tmp/update_runner.sh.pehAfTCwS1/install/sql/incremental/upd_dev_collection.sql
Reconfigure Permissions in master database? (yes,no) [no]:

Service 'mail_server' has not been detected (strongly recommended, currently enabled) do you want to disable it?  (yes,no) [yes]: no

Service 'web_server' has not been detected (strongly recommended, currently enabled) do you want to disable it?  (yes,no) [yes]: no

WARNING: If ISPConfig-Interface (Webfrontend) is installed on this Server we will configure the Web Server anyways but will not enable it in ISPConfig.

Reconfigure Services? (yes,no,selected) [yes]:

Configuring Postfix
sh: line 1: postalias: command not found
sh: line 1: postmap: command not found
Configuring Spamassassin
Configuring Getmail
Configuring Apache
Configuring vlogger
Configuring Apps vhost
Configuring Jailkit
Configuring Database
Updating ISPConfig
Certificate exists. Not creating a new one.
Reconfigure Crontab? (yes,no) [yes]:

Updating Crontab
Restarting services ...
Update finished.

(The domains are all working and serving pages, shall I create the folder '.well-known/acme-challenge/' under webroots or shall they created automatically?)

 

DO NOT create the folder manually!
It's an alias located elsewhere.
Do you do some kind of redirecting?

 

okay, just asking
Yes, 2 of my sites are redirected, each of them to another site, like domain1->domain3, domain2->domain 4
SSL and Let's encrypt checkboxes are not checked for these 2 redirected sites, and the redirection is made as apache directives:

RedirectMatch 301 "(.*)$" "https://domain3.com$1"

Besides this I only have aliasdomains, but I checked the "Don't create Let's encrypt cert" for each.

 

OMG thank you so much, guys! You are amazing!
This was the root of my problem!
Don't ask me why I did not have this acme.conf file in my sites-available directory, I never touch generated files manually (unless I know what i'm doing, which is not the case here )
I just mentioned because it was suspicious that I have never seen such folders in webroots, and creating them temporarily by a script then remove them along with its directories looked so unrealistic...

Just added the conf file and its simlink to sites-enabled, restarted apache, and voilá, certs are generated from ISPC immediately.

I really want to express my gratitude! Can I donate or something?

Last question: for the remaining certificates, shall I switch them Off then On on the frontend, or wait and see if they are renewing automagically? The closest expiration is 1 March.

Thanks again a million times, you really are a lifesaver!

 

Yes. You can do that but you don't have to do it on all since running certbot renew as suggested by @till shows only a few that is not renewed, so focus on them only. Plus, I think it is also good if you can use ISPConfig tool to resync all files and websites.

 

Okay, thank you!
All of the necessary certificates automatically renewed by morning, so everything seems to work perfectly fine now.
Read about the resync tool many times just never dared to do it.
What does resync do?

 

resync recreates/adjusts user accounts / config files / files & folders etc based on the account data/configs in the database.

eg, you accidentally deleted a clients vhost config or crontab or dns zone file or user entry in /etc/passwd, a resync will recreate the files, or settings in those files according to the data in the dbispconfig database

 

Cool, thats good to know, thanks very much!!

 

Hello, a while ago I updated with the "Update the ISPConfig Perfect Server from Debian 10 to Debian 11" tutorial to be able to use the latest version of WordPress, everything was ok until today when I saw that Let's encrypt did not update the certificates of the 3 hosted pages, I tried to renew them manually and it gives the following error, I do not understand the solution that fmarton applied. I ran ispconfig.update.sh-force and it gives the same error when generating the certificate. In etc/apache2/ssites-enabled/ there is no file with '-le'
Do you have any idea how to fix it?
Thank you very much Cesar

Code:

root@host:~# certbot -n renew
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.31.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 474, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2846, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2450, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2456, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 18, in <module>
    from certbot import account
  File "/usr/lib/python3/dist-packages/certbot/account.py", line 21, in <module>
    from certbot import constants
  File "/usr/lib/python3/dist-packages/certbot/constants.py", line 53, in <module>
    tls_sni_01_port=challenges.TLSSNI01Response.PORT,
AttributeError: module 'acme.challenges' has no attribute 'TLSSNI01Response'

 

Your certbot is 0.31.0 and seems very old to me. I would follow the upgrade instructions on certbot official page before retrying, but using certbot command on cli is not recommended. Just use the UI and do untick and retick first after you upgrade your certbot.