letsencrypt renewal fails

hi.
i have a somewhat old (but frequently updated) installation of ispconfig (3.2.9p1). i followed the perfect server guide for debian 10.
everything ran smooth and stable, but today i found out i have issues with certificate renewal.
certbot-auto seems to work, but tells me it's outdated and won't receive updates anymore.
the old guide now does not mention certbot-auto anymore.
so i tried to reinstall certbot, following the instructions on certbot.eff.org (snapd), but that didn't solve my problem.
right now, one of my websites isn't working. the checkboxes for "ssl" and "let's encrypt ssl" are unchcecked, and if i try to change that it just reverts to the unchecked state. the website itself doesn't work at all atm, ispconfig panel works fine, other websites work as well (i guess until the renewal is due).
certbot-renew --dry-run fails, (invalid response, failed to renew, some challenges failed)

i really hope someone can help me troubleshoot this issue, i'm not sure how to proceed and i don't want to f it up any more.
please advise.

 

thanks, i followed the steps in the list:
- certbot is installed, and updated via the certbot-auto script. i also installed the snapd-version like described on eff.org (i hope this wasn't a mistake).
- ispconfig is up-to-date (i also re-ran it several times with --force parameter)
- points 4-6 are probably not relevant, since it worked up until now and there were no network changes
- apache 2.4
- redid the update with "reconfigure services"
- server migration mode is off

here is the le-logfile:
(the line "could not find a usable 'nginx' binary", perhaps? i installed exactly like instructed on eff.org for apache/debian10)

Code:

2023-02-08 13:52:04,943:DEBUG:urllib3.connectionpool:http://localhost:None "GET /v2/connections?snap=certbot&interface=content HTTP/1.1" 200 97
2023-02-08 13:52:05,281:DEBUG:certbot._internal.main:certbot version: 1.32.2
2023-02-08 13:52:05,282:DEBUG:certbot._internal.main:Location of certbot entry point: /snap/certbot/2618/bin/certbot
2023-02-08 13:52:05,282:DEBUG:certbot._internal.main:Arguments: ['--preconfigured-renewal']
2023-02-08 13:52:05,282:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2023-02-08 13:52:05,312:DEBUG:certbot._internal.log:Root logging level set at 30
2023-02-08 13:52:05,313:DEBUG:certbot._internal.plugins.selection:Requested authenticator None and installer None
2023-02-08 13:52:06,026:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.38
2023-02-08 13:52:06,505:DEBUG:certbot._internal.plugins.disco:No installation (PluginEntryPoint#nginx): Could not find a usable 'nginx' binary. Ensure nginx exists, the binary is executable, and your PATH is set correctly.
Traceback (most recent call last):
  File "/snap/certbot/2618/lib/python3.8/site-packages/certbot/_internal/plugins/disco.py", line 160, in prepare
    self._initialized.prepare()
  File "/snap/certbot/2618/lib/python3.8/site-packages/certbot_nginx/_internal/configurator.py", line 194, in prepare
    raise errors.NoInstallationError(
certbot.errors.NoInstallationError: Could not find a usable 'nginx' binary. Ensure nginx exists, the binary is executable, and your PATH is set correctly.
2023-02-08 13:52:06,507:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: Installer, Authenticator, Plugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_debian.DebianConfigurator object at 0x7fcd18791700>
Prep: True
2023-02-08 13:52:06,508:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot_apache._internal.override_debian.DebianConfigurator object at 0x7fcd18791700> and installer <certbot_apache._internal.override_debian.DebianConfigurator object at 0x7fcd18791700>
2023-02-08 13:52:06,508:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator apache, Installer apache
2023-02-08 13:52:06,840:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-v02.api.letsencrypt.org/acme/acct/63766465', new_authzr_uri=None, terms_of_service=None), 2b199d1e48949b9a3500a16cf4b15937, Meta(creation_dt=datetime.datetime(2019, 8, 19, 12, 16, 11, tzinfo=<UTC>), creation_host='web.zbit.at', register_to_eff=None))>
2023-02-08 13:52:06,841:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2023-02-08 13:52:06,843:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2023-02-08 13:52:07,365:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 756
2023-02-08 13:52:07,366:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Wed, 08 Feb 2023 12:52:07 GMT
Content-Type: application/json
Content-Length: 756
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "KReuyExC1jw": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "renewalInfo": "https://acme-v02.api.letsencrypt.org/get/draft-ietf-acme-ari-00/renewalInfo/",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}
2023-02-08 13:52:19,026:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/snap/certbot/2618/bin/certbot", line 8, in <module>
    sys.exit(main())
  File "/snap/certbot/2618/lib/python3.8/site-packages/certbot/main.py", line 19, in main
    return internal_main.main(cli_args)
  File "/snap/certbot/2618/lib/python3.8/site-packages/certbot/_internal/main.py", line 1744, in main
    return config.func(config, plugins)
  File "/snap/certbot/2618/lib/python3.8/site-packages/certbot/_internal/main.py", line 1436, in run
    domains, certname = _find_domains_or_certname(config, installer)
  File "/snap/certbot/2618/lib/python3.8/site-packages/certbot/_internal/main.py", line 508, in _find_domains_or_certname
    raise errors.Error("Please specify --domains, or --installer that "
certbot.errors.Error: Please specify --domains, or --installer that will help in domain names autodiscovery, or --cert-name for an existing certificate name.
2023-02-08 13:52:19,028:ERROR:certbot._internal.log:Please specify --domains, or --installer that will help in domain names autodiscovery, or --cert-name for an existing certificate name.

and finally, i enabled debug logging but not more for now. should i generate and post the debug output?

enabling the "let's encrypt ssl"-checkbox also doesn't generate loglines in /var/log/letsencrypt/letsencrypt.log, but it should, right?

update!
if i run certbot manually, i can select the defunct site and it sucessfully renews and deploys the cert! so my website is up and running again, at least for now. it seems i only have issues with the connection between ispconfig and the certbot command.

update2.
hmm, but certbot renew --dry-run still fails, unfortunately. type: unauthorized, invalid response from <my-website-url>/.well-known/..

 

Installing snapd version is right but you may need to remove the old version of certbot and certbot-auto before installing certbot via snapd to ensure that the latest working certbot is used instead of the old one.

As an extra note, I noticed that some people have safely migrated from certbot to acme.sh as as the later is preferred by ISPConfig and works very well but I am not sure whether this is advisable as there is no official tutorial that supports such migration.

 

thank you for your quick replies!
i'll do what till suggested as soon as possible.
but.. does that mean i will break all my websites as soon as i restart apache?
i am feeling slightly panicky right now

 

thank you so much for your support. you're a lifesaver.
i did what you suggested, here is the output of /usr/local/ispconfig/server/server.sh.
i could not spot any problems, other than the failed le-requests (i changed the correct url to "the_client.com").

Code:

09.02.2023-15:46 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
09.02.2023-15:46 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.3/apache2/php.ini' - return code: 0
09.02.2023-15:46 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.3/fpm/php.ini' - return code: 0
09.02.2023-15:46 - DEBUG [system.inc:2399] - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.3/cgi/php.ini' - return code: 0
09.02.2023-15:46 - DEBUG [server:177] - Found 1 changes, starting update process.
09.02.2023-15:46 - DEBUG [plugins.inc:118] - Calling function 'ssl' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
09.02.2023-15:46 - DEBUG [plugins.inc:118] - Calling function 'update' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
09.02.2023-15:46 - DEBUG [system.inc:2399] - safe_exec cmd: chattr -i '/var/www/clients/client0/web6' - return code: 0
09.02.2023-15:46 - DEBUG [system.inc:2399] - safe_exec cmd: chattr +i '/var/www/clients/client0/web6' - return code: 0
09.02.2023-15:46 - DEBUG [system.inc:2399] - safe_exec cmd: df -T '/var/www/clients/client0/web6'|awk 'END{print $2,$NF}' - return code: 0
09.02.2023-15:46 - DEBUG [system.inc:2399] - safe_exec cmd: which 'setquota' 2> /dev/null - return code: 0
09.02.2023-15:46 - DEBUG [system.inc:2399] - safe_exec cmd: setquota -u 'web6' '0' '0' 0 0 -a &> /dev/null - return code: 0
09.02.2023-15:46 - DEBUG [system.inc:2399] - safe_exec cmd: setquota -T -u 'web6' 604800 604800 -a &> /dev/null - return code: 0
09.02.2023-15:46 - DEBUG [system.inc:2399] - safe_exec cmd: chattr +i '/var/www/clients/client0/web6' - return code: 0
09.02.2023-15:46 - WARNING - Could not verify domain the_client.com, so excluding it from letsencrypt request.
09.02.2023-15:46 - WARNING - Could not verify domain www.the_client.com, so excluding it from letsencrypt request.
09.02.2023-15:46 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
09.02.2023-15:46 - WARNING - Let's Encrypt SSL Cert for: the_client.com could not be issued.
09.02.2023-15:46 - WARNING -
09.02.2023-15:46 - DEBUG [db mysql.inc:523] - NON-String given in escape function! (boolean)
09.02.2023-15:46 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
09.02.2023-15:46 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
09.02.2023-15:46 - DEBUG [system.inc:2399] - safe_exec cmd: which 'apache2ctl' 2> /dev/null - return code: 0
09.02.2023-15:46 - DEBUG [system.inc:2399] - safe_exec cmd: chattr -i '/var/www/php-fcgi-scripts/web6/.php-fcgi-starter' - return code: 0
09.02.2023-15:46 - DEBUG [apache2 plugin.inc:1602] - Creating fastcgi starter script: /var/www/php-fcgi-scripts/web6/.php-fcgi-starter
09.02.2023-15:46 - DEBUG [system.inc:2399] - safe_exec cmd: chattr +i '/var/www/php-fcgi-scripts/web6/.php-fcgi-starter' - return code: 0
09.02.2023-15:46 - DEBUG [apache2 plugin.inc:1874] - Writing the vhost file: /etc/apache2/sites-available/the_client.com.vhost
09.02.2023-15:46 - DEBUG [apache2 plugin.inc:1992] - Apache status is: running
09.02.2023-15:46 - DEBUG [services.inc:56] - Calling function 'restartHttpd' from module 'web_module'.
09.02.2023-15:46 - DEBUG [system.inc:2082] - Trying to use Systemd to restart service
09.02.2023-15:46 - DEBUG [system.inc:2399] - safe_exec cmd: systemctl is-enabled 'apache2' 2>&1 - return code: 0
09.02.2023-15:46 - DEBUG [web module.inc:246] - Restarting httpd: systemctl restart apache2.service
09.02.2023-15:46 - DEBUG [apache2 plugin.inc:1995] - Apache restart return value is: 0
09.02.2023-15:46 - DEBUG [apache2 plugin.inc:2006] - Apache online status after restart is: running
09.02.2023-15:46 - DEBUG [modules.inc:240] - Processed datalog_id 537
09.02.2023-15:46 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished server.php.

ah! this line
WARNING - Could not verify domain the_client.com, so excluding it from letsencrypt request.
led me to this page
https://www.niih.de/fixed-warning-could-not-verify-domain-so-excluding-it-from-letsencrypt-request/
where they suggested to turn on "skip lets encrypt check" in the server config/ssl settings.
did that, reran server.sh, now the cert is generated and applied, and the site is back up again.

i hope this means my server is somewhat stable for now.
it would be interesting how this error started in the first place. certbot? network changes?
i've pushed back the debian upgrade a few times already, i guess it's about time.
or would it be more adviseable to start from scratch with a new machine and migrate all the sites?

 

hey till, thanks for your help, it's really appreciated!
unfortunately my problem still isn't quite fixed, i was hoping you could help me troubleshoot some more.
the cert for the client site got renewed, but now i seem to have a problem with the server's own cert.

the server.sh's debug output isn't very helpful (actual url replaced with "a.server.net"):

Code:

03.03.2023-10:53 - DEBUG [system.inc:1819] - exec: /usr/bin/certbot certonly -n --text --agree-tos --cert-name a.server.net --authenticator webroot --server https://acme-v02.api.letsencrypt.org/directory --rsa-key-size 4096 --email [email protected] --webroot
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

the content of /var/log/letsencrypt/letsencrypt.log doesn't really help me either:

Code:

2023-03-03 10:53:18,561:DEBUG:urllib3.connectionpool:http://localhost:None "GET /v2/connections?snap=certbot&interface=content HTTP/1.1" 200 97
2023-03-03 10:53:18,993:DEBUG:certbot._internal.main:certbot version: 2.3.0
2023-03-03 10:53:18,994:DEBUG:certbot._internal.main:Location of certbot entry point: /snap/certbot/2772/bin/certbot
2023-03-03 10:53:18,994:DEBUG:certbot._internal.main:Arguments: ['--domains', 'a.server.net', '--preconfigured-renewal']
2023-03-03 10:53:18,994:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2023-03-03 10:53:19,055:DEBUG:certbot._internal.log:Root logging level set at 30
2023-03-03 10:53:19,170:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2023-03-03 10:53:19,176:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2023-03-03 10:53:19,176:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/live/a.server.net/cert.pem is signed by the certificate's issuer.
2023-03-03 10:53:19,177:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/live/a.server.net/cert.pem is: OCSPCertStatus.GOOD
2023-03-03 10:53:19,827:DEBUG:certbot._internal.display.obj:Notifying user: Found the following matching certs:
  Certificate Name: a.server.net
    Serial Number: xxx..
    Key Type: RSA
    Domains: a.server.net
    Expiry Date: 2023-03-07 01:04:48+00:00 (VALID: 3 days)
    Certificate Path: /etc/letsencrypt/live/a.server.net/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/a.server.net/privkey.pem

so i tried something you posted last year on this topic:

and it turns out i can't access the .txt-file in the browser!
that's with vpn, so the firewall doesn't block anything. i think this may be the main source of my issue. the whole letsencrypt-process has been working fine for years, and the network config hasn't been changed recently. i only did linux and ispconfig-updates. does anyone have an idea on how i should proceed to get it working again?
thanks in advance.

 

umm.. i guess i have to move all websites to a new server then :/ i have no idea why this happened or how to fix it.

 

no, i went pretty much by the book (perfect server guide for debian 10). i thought it might be a good idea to start fresh with debian 11 and the latest ispconfig with the snapd-version of certbot. i mean it would be awesome if i had some breathing room for the site-migrations.

the problem is, i really don't understand why it suddenly stopped working. the logs don't tell me much more than it simply failed to obtain a new cert. also,the new server gets it's le-certs without any problems, and it sits in the exact same place as the old one (same network, same firewall-rules and so on).

 

Code:

root@web:~# ls -la /etc/apache2/sites-enabled/
total 12
drwxr-xr-x 2 root root 4096 Feb  9 15:33 .
drwxr-xr-x 8 root root 4096 Mar  6 21:10 ..
lrwxrwxrwx 1 root root   39 Aug 19  2019 000-apps.vhost -> /etc/apache2/sites-available/apps.vhost
lrwxrwxrwx 1 root root   35 Aug 19  2019 000-default.conf -> ../sites-available/000-default.conf
lrwxrwxrwx 1 root root   43 Aug 19  2019 000-ispconfig.conf -> /etc/apache2/sites-available/ispconfig.conf
lrwxrwxrwx 1 root root   44 Aug 19  2019 000-ispconfig.vhost -> /etc/apache2/sites-available/ispconfig.vhost
lrwxrwxrwx 1 root root   44 Apr 13  2021 100-website1.net.vhost -> /etc/apache2/sites-available/website1.net.vhost
lrwxrwxrwx 1 root root   44 Apr 13  2021 100-website2.net.vhost -> /etc/apache2/sites-available/website2.net.vhost
lrwxrwxrwx 1 root root   44 Apr 13  2021 100-website3.net.vhost -> /etc/apache2/sites-available/website3.net.vhost
...

 

hey, sorry for the late reply, i got sick and caught a fever (very convenient)
anyways, here's the output (i had to split in in three parts, i hope i didn't leave anything sensitive in):


EDIT the output was fine, i deleted the three posts for readability

 

ok, maybe i`ll give that a shot. thank you very much for your help!

 

hey again, soooo i painstakingly migrated all my websites to my new server, in order to get the le-functionality back. which worked, for the most part. but sadly, exactly the three most important sites don't work, because their certs are not due to renewal.

how do i force the acme-script to (force-)renew my certs? please help.


update, 5 minutes after my post, one of the three sites works now. the other two are still down. it might be an issue with the dns, since the sites moved to another ip adress. i really hope this somehow works out..

update2, running acme.sh manually with --force is not an option, right? can i maybe delete the existing certs to get new ones?