Debian 11, certbot not renewing certificates anymore

Host upgraded from Debian 10 to Debian 11 last December. Worked until recently. Now certbot fails renewing certificates, even ispconfig_update.sh --force switches to making self-signed certificate.
There was problem with emailverification.info for some domain names, it took time to get that sorted out and domains name service was not working for a while. So for those domains certificate renewal failed since name service was not working.
Certbot log shows among other stuff

Code:

2024-04-14 01:47:28,945:DEBUG:certbot._internal.storage:Should renew, less than 30 days befo
re certificate expiry 2024-04-14 00:06:32 UTC.

Code:

2024-04-14 01:47:31,617:WARNING:certbot._internal.auth_handler:Challenge failed for domain mail.company.tld
. . .
Invalid response from http://mail.company.tld/.well-known/acme-challenge/81J34_yTDnCbS-KJKUgYuEjzUj8Jf3QyzDfvjoTDPUY: 403

I have followed Let's Encrypt FAQ, domain name does point to the correct server, which I can reach in browser but the certificate is self-signed made with ispconfig_update.sh. Can't access ISPConfig Panel.
Certbot is installed from Debian repository:

Code:

root@mail:/etc/letsencrypt# apt policy certbot
certbot:
  Asennettu: 1.12.0-2
  Ehdokas:   1.12.0-2
  Versiotaulukko:
 *** 1.12.0-2 500
        500 https://deb.debian.org/debian bullseye/main amd64 Packages
        100 /var/lib/dpkg/status
root@mail:/etc/letsencrypt#

If I stop apache2 and run

Code:

certbot --dry-run certonly -d mail.company.tld

choosing temporary web server it works and answers The dry run was successful. If I choose "place files in webroot directory" then not working.
Is certbot version too old and I should install snapd and then certbot via snap? If yes, should the now installed certbot be removed with --purge? Does snapd certbot just work if I apt remove certbot and snap install certbot?

 

Maybe related to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1055330 Not sure as i don't use certbot in my setup and cannot validate this myself right now

Can you try running it with the "--webroot-path" flags and check the outcome?

 

I tried again, now a website domain:

Code:

certbot -v --dry-run certonly --webroot-path /var/www/clients/client1/web12/web/ -d somedomain.tld

I renamed .htaccess, it had lots of redirects so may confuse. Different output but still failing.

Code:

  Type:   unauthorized
   Detail: 11.222.33.44: Invalid response from
   http://somedomain.tld/.well-known/acme-challenge/XTGfMA1rRBKM19Lk_bgRmvCgINmuk3_Ow2RNfQFWzJA:
   403

Meanwhile, I can access ISPConfig Panel now. Trying to get sertificate from there results in the Let's Encrypt tick being removed. Nothing appears in letsencrypt.log, which I find curious.

 

Pass the --verbose option and see if that gives any clues

 

I already had -v, and it does provide lots, debatable if any of that is clue.

 

It can not be accessed.

Code:

[Tue Apr 16 14:42:12.741938 2024] [authz_core:error] [pid 25039] [client my.ip.he.re:59080] AH01630: client denied by server configuration: /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/test.txt

Maybe I broke this when trying to get ISPConfig Panel working, Apache did not start then.

 

Links and filenames match, but contents is different. Timestamp is today, so I do not know how or why it seems to have changed. I'll try copy paste from #8.
No change.
I noticed in /etc/apache2/sites_available ls showed

Code:

-rw------- 1 root root  320 16. 4. 15:08 acme.conf

I changed that with chmod go+r acme.conf, but no help.

Code:

$ LANG=C wget http://mail.company.tld/.well-known/acme-challenge/test.txt
--2024-04-16 15:15:50--  http://mail.company.tld/.well-known/acme-challenge/test.txt
Resolving mail.company.tld (mail.company.tld)... 94.237.37.94
Connecting to mail.company.tld(mail.company.tld)|94.237.37.94|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2024-04-16 15:15:50 ERROR 403: Forbidden.

 

I tried that first after rebooting the host. Seems to work except certificate becomes the self-signed type.
I do not understand where comes that "client denied by server configuration". Is it because the acme.conf is behind symbolic link?

 

I did

Code:

a2ensite mail.company.tld

when trying to get ISPConfig Panel working.

 

I added

Code:

RewriteEngine on
        RewriteCond %{REQUEST_URI} ^/\.well-known/acme-challenge/
        RewriteRule ^ - [END]
        RewriteCond %{HTTPS} off
                RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L,NE]

to sites-available/mail.company.tld.vhost, I noticed it was missing by comparing with a working ISPConfig host. Now works better.

Yeah, bad move from me. I try not modify anything outside Panel, but sometimes things happen.

 

No answer when site is not active.

Code:

Connecting to heartofnepal.com (heartofnepal.com)|94.237.37.94|:80... connected.
HTTP request sent, awaiting response...

Trying ispconfig_update.sh --force to get new cert for Panel, fails. From apache access log:

Code:

16.16.156.243 - - [16/Apr/2024:16:20:48 +0300] "GET /.well-known/acme-challenge/OJFrm3b52LjhFmDFNWQVBjHGCRuJAoymSePJULKluTQ HTTP/1.1" 403 363 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
52.26.187.41 - - [16/Apr/2024:16:20:49 +0300] "GET /.well-known/acme-challenge/OJFrm3b52LjhFmDFNWQVBjHGCRuJAoymSePJULKluTQ HTTP/1.1" 403 363 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
23.178.112.202 - - [16/Apr/2024:16:20:49 +0300] "GET /.well-known/acme-challenge/OJFrm3b52LjhFmDFNWQVBjHGCRuJAoymSePJULKluTQ HTTP/1.1" 403 363 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

 

Tried in ISPConfig Panel Tools | Sync Tools : Resync all services. Then ispconfig_update, and still can not create new certificate.
In apache error log I find:

Code:

[Tue Apr 16 21:07:42.009886 2024] [authz_core:error] [pid 57963] [client 16.170.233.61:17686] AH01630: client denied by server configuration: /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/jFwBKFc3fe3ErQwVUadKeFjzEnyY88KpHM_RmAwkTeQ
[Tue Apr 16 21:07:42.341951 2024] [authz_core:error] [pid 57964] [client 23.178.112.101:62249] AH01630: client denied by server configuration: /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/jFwBKFc3fe3ErQwVUadKeFjzEnyY88KpHM_RmAwkTeQ
[Tue Apr 16 21:07:42.415215 2024] [authz_core:error] [pid 57965] [client 18.191.174.211:39480] AH01630: client denied by server configuration: /usr/local/ispconfig/interface/acme/.well-known/acme-challenge/jFwBKFc3fe3ErQwVUadKeFjzEnyY88KpHM_RmAwkTeQ

How can I see what is the server configuration that denies the client?

 

It's not that easy. You have to check each "Allow from ..." or in newer version "Require all ..." settings of the vHost. Check that any DocumentRoot, Location and Webroot settings are correct. Double-check that all permissions and ownerships on these directories AND the whole path have the correct settings.

You can add "LogLevel debug" to the vHost which maybe provides some usefull information

 

This host was dist upgraded from Debian 10 to Debian 11 in december, and it was Debian 9 originally. So there can be old syntax lurking somewhere.
I'll see what I can find.