Cert problems with ISPC 3.2.9

First of:
I run The Perfect server with ispc 3.2.9
it's on ubuntu 18.04.4 LTS
with apache2.4.29
dovecot

To be honest it is not running very well anymore because I had a problem with corrupted data on /var which I never was able to reconstruct 100%. For instance I have problems with roundcube that will not connect to the account. But that is not my immediate problem.

I'm experiencing some problems with my letsencrypt certificates after upgrading ISPC to latest version 3.2.9.
And I think that there may be a problem with the upgrade script. Because I run with apache2 but somewhere in the letsencrypt logfile I see that it thinks that the server is nginx.

Is there a friendly soul in here who can take a look at my letsencrypt logfile and tell me what my problem and a solution might be.
Here is the tail of the file.

root@odin:~# tail -100 /var/log/letsencrypt/letsencrypt.log
2022-12-07 05:33:16,258:INFO:certbot.main:Renewing an existing certificate
2022-12-07 05:33:18,056EBUG:certbot.crypto_util:Generating key (4096 bits): /etc/letsencrypt/keys/3581_key-certbot.pem
2022-12-07 05:33:18,088EBUG:certbot.crypto_util:Creating CSR: /etc/letsencrypt/csr/3581_csr-certbot.pem
2022-12-07 05:33:18,089EBUG:acme.client:Requesting fresh nonce
2022-12-07 05:33:18,089EBUG:acme.client:Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-nonce.
2022-12-07 05:33:18,223EBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
2022-12-07 05:33:18,224EBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Wed, 07 Dec 2022 05:33:18 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: C878LLVYH7z4EC9gPWiQbnQUetpwUYDMWGMNtokfeHFkaO8
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800


2022-12-07 05:33:18,224EBUG:acme.client:Storing nonce: C878LLVYH7z4EC9gPWiQbnQUetpwUYDMWGMNtokfeHFkaO8
2022-12-07 05:33:18,225EBUG:acme.client:JWS payload:
b'{\n "identifiers": [\n {\n "type": "dns",\n "value": "bnjpro.dk"\n },\n {\n "type": "dns",\n "value": "drop.bnjpro.dk"\n },\n {\n "type": "dns",\n "value": "ispc.bnjpro.dk"\n },\n {\n "type": "dns",\n "value": "ispconfig.bnjpro.dk"\n },\n {\n "type": "dns",\n "value": "phpmyadmin.bnjpro.dk"\n },\n {\n "type": "dns",\n "value": "webmail.bnjpro.dk"\n }\n ]\n}'
2022-12-07 05:33:18,241EBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{
"protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvODA5Mzk2NDYiLCAibm9uY2UiOiAiQzg3OExMVllIN3o0RUM5Z1BXaVFiblFVZXRwd1VZRE1XR01OdG9rZmVIRmthTzgiLCAidXJsIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL25ldy1vcmRlciJ9",
"signature": "uL16n0XpqDLGaoEDreKq2Wj54kIeRTNC_Ew34yTWPYB1OCwmKk1g-XEsRawRVzPwcs6ONgfOZ6j7Fkhf7W771t8sTM54rhnTuoWLqkxaqDMSgRnxarKvsJRqKNIMPytqH2PqkX59dzr-qdZyk8hOmJNNGEWpUTl6_cEbidx-NdmBbZWGBDUdUKGB_-yCiR8Ki3OnxnHrl06wOHT_QNgAzrEj3zlfbBYjrN-lYMFI6TRoDV-AGxvbezCDbotj8hXcmD8Ung5VfD0PBJf0GGZHwn1SSL2MeDOaHx1gzN4xb9GuV-BB32UZVuQ7UMsExZnLXGNh97QjT8gPV2chzehTaZ-p4-uIDT6h4EWbD0foJWEv3mNa4kubPzVKWqPwp0H6EGqsfB9Fw2mAdgRLCQLVFGoYlfNwWQhlozJc19_bbHQVA5_G4PI8h64HytJ6ckzI7cxwY_ty35K2eprgFUrK-un3BBTpE2ru1Y7LqcZDco_E8LI2blQyEMfUKDQDEdxVqUsK1Sr8vLTTqglhJTMKqKEjOl-KyVdvOMwJGb5geSia3fZ13NcXWKhBijapDJEvOP9-kci18XWiPmYgELwJAvqBpnZCWFlw4Te4O30ao0H86V-3YqlnSQprpNp7r--B3lvFBiU21bpvWq1ncKBZSR9yQNmnJO0UZeMz_psd8pw",
"payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogImJuanByby5kayIKICAgIH0sCiAgICB7CiAgICAgICJ0eXBlIjogImRucyIsCiAgICAgICJ2YWx1ZSI6ICJkcm9wLmJuanByby5kayIKICAgIH0sCiAgICB7CiAgICAgICJ0eXBlIjogImRucyIsCiAgICAgICJ2YWx1ZSI6ICJpc3BjLmJuanByby5kayIKICAgIH0sCiAgICB7CiAgICAgICJ0eXBlIjogImRucyIsCiAgICAgICJ2YWx1ZSI6ICJpc3Bjb25maWcuYm5qcHJvLmRrIgogICAgfSwKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogInBocG15YWRtaW4uYm5qcHJvLmRrIgogICAgfSwKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogIndlYm1haWwuYm5qcHJvLmRrIgogICAgfQogIF0KfQ"
}
2022-12-07 05:33:18,400EBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 429 213
2022-12-07 05:33:18,401EBUG:acme.client:Received response:
HTTP 429
Server: nginx
Date: Wed, 07 Dec 2022 05:33:18 GMT
Content-Type: application/problem+json
Content-Length: 213
Connection: keep-alive
Boulder-Requester: 80939646
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 1DFAy7fP-g1lIYj8dQrS3-rf2AAoi-fG5SACoKVJq7kUw7g

{
"type": "urn:ietfarams:acme:error:rateLimited",
"detail": "Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/",
"status": 429
}
2022-12-07 05:33:18,402:WARNING:certbot.renewal:Attempting to renew cert (bnjpro.dk-0004) from /etc/letsencrypt/renewal/bnjpro.dk-0004.conf produced an unexpected error: urn:ietfarams:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/. Skipping.
2022-12-07 05:33:18,403EBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 443, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File "/usr/lib/python3/dist-packages/certbot/main.py", line 1197, in renew_cert
renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
File "/usr/lib/python3/dist-packages/certbot/main.py", line 115, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 318, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
File "/usr/lib/python3/dist-packages/certbot/client.py", line 334, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File "/usr/lib/python3/dist-packages/certbot/client.py", line 366, in _get_order_and_authorizations
orderr = self.acme.new_order(csr_pem)
File "/usr/lib/python3/dist-packages/acme/client.py", line 889, in new_order
return self.client.new_order(csr_pem)
File "/usr/lib/python3/dist-packages/acme/client.py", line 672, in new_order
response = self._post(self.directory['newOrder'], order)
File "/usr/lib/python3/dist-packages/acme/client.py", line 96, in _post
return self.net.post(*args, **kwargs)
File "/usr/lib/python3/dist-packages/acme/client.py", line 1204, in post
return self._post_once(*args, **kwargs)
File "/usr/lib/python3/dist-packages/acme/client.py", line 1218, in _post_once
response = self._check_response(response, content_type=content_type)
File "/usr/lib/python3/dist-packages/acme/client.py", line 1073, in _check_response
raise messages.Error.from_json(jobj)
acme.messages.Error: urn:ietfarams:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/

2022-12-07 05:33:18,412:INFO:certbot.renewal:Cert not yet due for renewal
2022-12-07 05:33:18,413EBUG:certbot.plugins.selection:Requested authenticator webroot and installer None
2022-12-07 05:33:18,416:INFO:certbot.renewal:Cert not yet due for renewal
2022-12-07 05:33:18,417EBUG:certbot.plugins.selection:Requested authenticator webroot and installer None
2022-12-07 05:33:18,417:ERROR:certbot.renewal:All renewal attempts failed. The following certs could not be renewed:
2022-12-07 05:33:18,418:ERROR:certbot.renewal: /etc/letsencrypt/live/odin.bnjpro.dk-0001/fullchain.pem (failure)
/etc/letsencrypt/live/bnjpro.dk/fullchain.pem (failure)
/etc/letsencrypt/live/imap.bnjpro.dk/fullchain.pem (failure)
/etc/letsencrypt/live/bnjpro.dk-0006/fullchain.pem (failure)
/etc/letsencrypt/live/bnjpro.dk-0001/fullchain.pem (failure)
/etc/letsencrypt/live/odin.bnjpro.dk/fullchain.pem (failure)
/etc/letsencrypt/live/bnjpro.dk-0007/fullchain.pem (failure)
/etc/letsencrypt/live/bnjpro.dk-0008/fullchain.pem (failure)
/etc/letsencrypt/live/bnjpro.dk-0004/fullchain.pem (failure)
2022-12-07 05:33:18,418:INFO:certbot.hooks:Running post-hook command: echo '1' > /usr/local/ispconfig/server/le.restart
2022-12-07 05:33:18,421EBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File "/usr/bin/certbot", line 11, in <module>
load_entry_point('certbot==0.27.0', 'console_scripts', 'certbot')()
File "/usr/lib/python3/dist-packages/certbot/main.py", line 1364, in main
return config.func(config, plugins)
File "/usr/lib/python3/dist-packages/certbot/main.py", line 1276, in renew

 

Well that is good to know.
But what I find weird is that if I run a ssllabs test or internet.nl test, the certificate seems ok and is renewed from november 30 2022 until february 28 2023.
But my outlook connection tells me that the cert is unvalid because it is issued for a period that ran out december 4, 2022.

And if I look at the certificates for the domain, in /etc/letsencrypt/live/bnjpro.dk all have a date from september 5 2022, where I would expect them to be younger (november 30 2022).

In postfix and dovecot I have (a long time ago) changed the cert path to be the forementioned path, whilivingch seems to have worked well until now.

Why isn't the newest certificates shown to outlook. And where are the newest actually living, if not in /etc/letsencrypt/live/bnjpro.dk?

 

I know my bad, sorry.
But trying to locate the ispconfig.pem my search end up empty with "locate ispconfig.pem". but in dovecot.conf I have set the lines:
ssl_cert = </etc/letsencrypt/live/bnjpro.dk/fullchain.pem
ssl_key = </etc/letsencrypt/live/bnjpro.dk/privkey.pem
and in postfix main.conf is these lines put in:
smtpd_tls_cert_file = /etc/letsencrypt/live/bnjpro.dk/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/bnjpro.dk/privkey.pem

But since the file isn't updated in the /etc/letsencrypt/live/domain.tld I suspect a symlink is missing (but why - since it has worked) or I expect the certificate to be located in the wrong place. But I'm pretty sure this has worked and in the way described above. I was of the impression that the "live" folder was the place ispconfig/letsencrypt puts the renewed certs.

 

By the way had allready restarted dovecot anf postfix, and have done it again without a change.

 

Shouldn't /etc/letsencrypt/live/bnjpro.dk/fullchain.pem and /etc/letsencrypt/live/bnjpro.dk/privkey.pem be updated automatically?
Is there a symlink broken somewhere?

 

It is:
root@odin:~# ls -la /etc/letsencrypt/live/bnjpro.dk/
total 12
drwxr-xr-x 2 root root 4096 sep 5 03:00 .
drwx------ 25 root root 4096 dec 6 08:04 ..
lrwxrwxrwx 1 root root 34 sep 5 03:00 cert.pem -> ../../archive/bnjpro.dk/ce rt17.pem
lrwxrwxrwx 1 root root 35 sep 5 03:00 chain.pem -> ../../archive/bnjpro.dk/c hain17.pem
lrwxrwxrwx 1 root root 39 sep 5 03:00 fullchain.pem -> ../../archive/bnjpro. dk/fullchain17.pem
lrwxrwxrwx 1 root root 37 sep 5 03:00 privkey.pem -> ../../archive/bnjpro.dk /privkey17.pem
-rw-r--r-- 1 root root 682 dec 27 2020 README

 

What could be my problem?

 

My guess is the e-mail server is using wrong certificate files. If e-mail client shows different dates for the certificate from server than certificate you have now installed. then there is mixed configuration. This can happen if e-mail server is not restarted after certificate update, for example. You wrote previously you have restarted, but have you tried rebooting the whole system?
If problem persists, you need to look at the e-mail server configuration files, see what certificate files it is configured to use, then find where the updated certificates are and figure out why those are not used.

 

I have both restarted and rebooted the system, with no luck. And the paths written in my answer #5, is the path used in postfix and dovecot until I updated the ISPC to latest version (3.2.9).
But it seems like the certificates aren't put there anymore after the upgrade of ISPC to 3.2.9.

If I make a tail -f /var/log/letsencrypt/letsencrypt.log

I get this response:
2022-12-19 09:16:03,733EBUG:certbot.cert_manager:Renewal conf file /etc/letsencrypt/renewal/seafile.bnjpro.dk.conf is broken. Skipping.
2022-12-19 09:16:03,735EBUG:certbot.cert_manager:Traceback was:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/certbot/cert_manager.py", line 382, in _search_lineages
candidate_lineage = storage.RenewableCert(renewal_file, cli_config)
File "/usr/lib/python3/dist-packages/certbot/storage.py", line 420, in __init__
"file reference".format(self.configfile))
certbot.errors.CertStorageError: renewal config file {} is missing a required file reference

2022-12-19 09:16:03,743:INFO:certbot.renewal:Cert not yet due for renewal
2022-12-19 09:16:03,744:INFO:certbot.main:Keeping the existing certificate

But the files in /etc/letsencrypt/live/bnjpro.dk/xxxxxxxxx
are not renewed. And I am worried about the python script errors. What is that about?

 

It would be easier to read if you put the listing sin CODE tags.
Anyway, the error message is

Code:

2022-12-19 09:16:03,733:DEBUG:certbot.cert_manager:Renewal conf file /etc/letsencrypt/renewal/seafile.bnjpro.dk.conf is broken. Skipping.

So what is in that file?

 

It is an alternative to dropbox, but hosted on my own server. I will try to not ask for a renewal of LE cert for this particular webpage and see if it solves my problem.

 

I have tried to remove all files from the renew and the live paths, and tried renewing the other certificates, but I'm only responded with a message that the certificate not is up for renewal. Can I revoke the certificate any way other than just untick the Lets Encrypt marker for the pages in ISPC?

 

You must untick it in ISPConfig panel, otherwise ISPConfig creates the certificate again next time it resyncs files.
What was in file /etc/letsencrypt/renewal/seafile.bnjpro.dk.conf ?
When you have unticked the certificate in ISPConfig panel, and waited 2 minutes for the process to complete, you can remove the certificate files. If you use certbot (I do not know what the commands are for acme.sh), the command is

Code:

certbot revoke --cert-name seafile.bnjpro.dk

if seafile.bnjpro.dk is the domain certificate is for. If that command does not work (and it fails if certificate is already outdated), use command

Code:

certbot delete --cert-name name-of-cert-here

 

/etc/letsencrypt/renewal/seafile.bnjpro.dk.conf is an empty file. Don't know why it is there.
Well the seafile subdomain is no longer used. So it is apparently not deleted automatically by ISPC when the subdomain is deleted in ISPC.
So I have manually deleted the file (and others no longer in use).

But I still fail to understand why I can't find the certificates I should use for my mail server which used to be at /etc/letsencrypt/live/bnjpro.dk/xxxxx.pem.
The certificate for the website works, but not for the mail server. Or they are placed somwhere else. I don't even know where the cert for the web is placed. I have again run a test at qualys ssl-labs and get an A+ on my server.

 

You have had LE certificate for seafile.bnjpro.dk, that is why.

This is true. That is why I showed the certbot command for revoking and deleting the certificate and certificate files.

Those are not the same certificate. Or at least probably should not be.
E-mail server, applications postfix and dovecot, are supposed to use the certificate for ISPConfig server hostname. Have you modified the certificate setup? Have you checked the certificate for ISPConfig hostname works? If you check with

Code:

hostname -f

the ISPConfig server hostname, and have let ISPConfig create certificates, then you should still verify dovecot and postfix use that certificate. Also dovecot and postfix should use the hostname -f as mailname.
Examine what certificates the applications are using, for example with

Code:

grep -r -E '(ssl_cert|cert_file)' /etc/postfix /etc/dovecot

then work further from there.
To test you e-mail server certificate, use for example https://ssl-tools.net/mailservers
My signature has link to e-mail setup tutorial.

 

Output for hostname -f is

Code:

odin.bnjpro.dk

Output for grep -r -E '(ssl_cert|cert_file)' /etc/postfix /etc/dovecot is:

Code:

the output of grep -r -E '(ssl_cert|cert_file)' /etc/postfix /etc/dovecot is
/etc/postfix/main.cf:smtpd_tls_cert_file = /etc/letsencrypt/live/bnjpro.dk/fullchain.pem
/etc/postfix/main.cf.bnj280322:smtpd_tls_cert_file = /etc/postfix/smtpd.cert
/etc/postfix/main.cf.bnj130121:#smtpd_tls_cert_file = /etc/postfix/smtpd.cert
/etc/postfix/main.cf.bnj130121:smtpd_tls_cert_file = </etc/letsencrypt/live/bnjpro.dk/fullchain.pem
/etc/postfix/main.cf.bnj200521:#smtpd_tls_cert_file = /etc/postfix/smtpd.cert
/etc/postfix/main.cf.bnj200521:#smtpd_tls_cert_file = /etc/postfix/smtpd.cert
/etc/postfix/main.cf.bnj200521:smtpd_tls_cert_file = </etc/letsencrypt/live/bnjpro.dk/fullchain.pem
/etc/postfix/main.cf.bnj200521:#smtpd_tls_cert_file = /etc/postfix/smtpd.cert
/etc/postfix/main.cf~:smtpd_tls_cert_file = /etc/letsencrypt/live/bnjpro.dk/fullchain.pem
/etc/postfix/main.cf~3:smtpd_tls_cert_file = /etc/postfix/smtpd.cert
/etc/postfix/main.cf~2:smtpd_tls_cert_file = /etc/postfix/smtpd.cert
/etc/dovecot/dovecot-sql.conf~:#     ssl_cert, ssl_key   - For sending client-side certificates to server
/etc/dovecot/conf.d/10-ssl.conf:ssl_cert = </etc/letsencrypt/live/bnjpro.dk/fullchain.pem
/etc/dovecot/conf.d/10-ssl.conf:#ssl_cert = </etc/dovecot/private/dovecot.pem
/etc/dovecot/conf.d/10-ssl.conf:#ssl_cert_username_field = commonName
/etc/dovecot/dovecot.conf~:ssl_cert = </etc/letsencrypt/live/bnjpro.dk/fullchain.pem
/etc/dovecot/dovecot.conf~:#ssl_cert = </etc/postfix/smtpd.cert
/etc/dovecot/dovecot.conf.newest:#ssl_cert = </etc/postfix/smtpd.cert
/etc/dovecot/dovecot.conf.newest:ssl_cert = </etc/letsencrypt/live/bnjpro.dk/fullchain.pem
/etc/dovecot/dovecot-sql.conf:#     ssl_cert, ssl_key   - For sending client-side certificates to server
/etc/dovecot/dovecot.conf:#ssl_cert = </etc/postfix/smtpd.cert
/etc/dovecot/dovecot.conf:ssl_cert = </etc/letsencrypt/live/bnjpro.dk/fullchain.pem
/etc/dovecot/dovecot-sql.conf.ext:#     ssl_cert, ssl_key      - For sending client-side certificates to server

I think I will use the Christmas hollydays to make a new server with ubuntu 22.04 LTS. I think there are to much problems going on with this server. Not giving up on ISPC though. It's so amazing, and such a good work done.

 

1. What is your server hostname FQDN? I asked because all I can see you are talking about is the main domain, whereby it should be a subdomain in an ISPConfig server setup.

2. To me it is best for all of the services to symlink to ISPConfig LE SSL certs, which is the default ISPConfig server setup, and that should include postfix and dovecot, so I don't why you need to specify direct path of the LE SSL certs in them. Just check all your symlinks are working (not broken) and correct from ISPConfig LE SSL certs to other services.