update does not create LE certificate: cannot open /dev/tcp/127.0.0.1/80

OS Debian 12, ISPConfig version was 3.2.11p2, updated to 3.2.12. LE client is acme.sh. Installed with autoinstaller in january 2024.
ispconfig_update.sh did update, but creting new certificate failed, so I guess it created self signed certificate. Error message was

Code:

sh: 1: cannot open /dev/tcp/127.0.0.1/80: No such file
Using apache for certificate validation
acme.sh is installed, overriding certificate path to use /root/.acme.sh

On this host shell is dash,

Code:

# ls -lh /bin/sh
lrwxrwxrwx 1 root root 4 Jan  5  2023 /bin/sh -> dash

Is the certificate creation using dash now, but should be using bash?

When I try running ispconfig_update.sh --force again, it says:
Certificate exists. Not creating a new one.
I hoped to run update with debug or log, but now it does not try to create certificate.
The certificates are now messed up, some services show the self signed, some use old expired certificate.
The host certificate expired on tuesday, that is why I started the update now. When I can get new certificate, I need to know: how I can check all services use the correct certificate file?

 

Drat, I forgot to read the log.
Anyway, seems there is something wrong with certificate renewal. One host shows in log

Code:

276   "status": "pending",
 277   "expires": "2024-07-20T10:09:12Z",

so there is only 7 days left, the certificate should have renewed 23 days ago.
Log has 695 lines, seems going OK until line 376 where it stands

Code:

Pending, The CA is processing your order, please just wait. (1/30)

I thought it would be simple to sort certificate problem on ns2 host, but host does not have apache installed and looks like acme.sh tries dns validation, and that does not work. It may be ns2 never had working certificate.
I'll switch to reading log on main host, which does have apache.
It looks to me problem is not in Let's Encrypt, acme.sh has renewed certificate and it is still valid. The error message ispconfig_update.sh shows about /dev/tcp is strange, how is that file relevant? Anyway, file indeed does not exist on main host nor on ns2.
But webmail is using expired certificate. I think it is just wrong symlink.

 

Dovecot files:

Code:

root@posti:/etc/dovecot# ls -lh private/
total 8.0K
drwx------ 2 root root 4.0K Dec 20  2023 .
drwxr-xr-x 4 root root 4.0K Jan 12  2024 ..
lrwxrwxrwx 1 root root   38 Dec 20  2023 dovecot.key -> /etc/ssl/private/ssl-cert-snakeoil.key
lrwxrwxrwx 1 root root   36 Dec 20  2023 dovecot.pem -> /etc/ssl/certs/ssl-cert-snakeoil.pem
root@posti:/etc/dovecot#

Code:

conf.d/10-ssl.conf:ssl_cert = </etc/dovecot/private/dovecot.pem
conf.d/10-ssl.conf:ssl_key = </etc/dovecot/private/dovecot.key

That can not be right, this used to work.

Code:

dovecot.conf:ssl_cert = </etc/postfix/smtpd.cert
dovecot.conf:ssl_key = </etc/postfix/smtpd.key
dovecot.conf:ssl_dh = </etc/dovecot/dh.pem

That looks like a reasonable configuration.
And dovecot was restarted after certificate file renewal.

Code:

root@posti:/etc/dovecot# ls -lh /etc/postfix/smtpd.cert /etc/postfix/smtpd.key
lrwxrwxrwx 1 root root 48 Jul 13 13:36 /etc/postfix/smtpd.cert -> /usr/local/ispconfig/interface/ssl/ispserver.crt
lrwxrwxrwx 1 root root 48 Jul 13 13:36 /etc/postfix/smtpd.key -> /usr/local/ispconfig/interface/ssl/ispserver.key
root@posti:/etc/dovecot# ls -lh /usr/local/ispconfig/interface/ssl/ispserver.crt /usr/local/ispconfig/interface/ssl/ispserver.key
-rwxr-x--- 1 root root 3.9K Jul 13 13:35 /usr/local/ispconfig/interface/ssl/ispserver.crt
-rwxr-x--- 1 root root 3.2K Jul 13 13:35 /usr/local/ispconfig/interface/ssl/ispserver.key

Those are certificate files created today. I can not understand why webmail uses old certificate still. Maybe the ispconfig_update.sh messed something when it failed with Let's Encrypt and created the self signed certificate.
Maybe roundcube is culprit? But it does not seem to run as service so nothing to restart.
Files in /root/.acme.sh/$(hostname -f) are latest from June 9th, so not same certificate but still valid for 60 days someting.
It is time for sauna and my brain is exhausted.

 

I can access /webmail with other FQDN, all FQDN/webmail I tried work and use the LE certificate for that website.
But (hostname -f)/webmail uses old certificate. I found Firefox allows me to see the certificate information, it is created on Wed, 10 April 2024 and expired Tue 09 Jul 2024.
I just can not understand why it uses that old certificate. Is it possible something is wrong with ispconfig_update.sh when I force it to create certificate? The files in /root/.acme.sh/$(hostname -f) are dated Jun 9 10:58 so the certificate did renew at the set date. And host uptime is 13 days so it has rebooted since that certificate was created.

 

I still do not understand whatfor the error "sh: 1: cannot open /dev/tcp/127.0.0.1/80:", but otherwise I think I now understand.
This host has website named $(hostname -f), and I finally remember there was discussion on this forum about certificates if such website exists.
With firefox I could see website is using certificate created 10 April, and the files in /var/www/$(hostname -f)/ssl are from that date. Maybe something like this happened at that time, and I just turned off and back on again the LE tick box in website settings.
I symlinked the two files in ssl/ directory to acme directory, restarted apache and now it works!
The old discussion I found is this: https://forum.howtoforge.com/threads/ispconfig-3-2-ssl-for-primary-host.90030/#post-442536
I have used website hostname -f for 12 years. But then I had certbot as LE client. Now acme.sh needs this tweaking.
And @till wrote his excellent answer while I was typing this. So I reinwented the acme symlinking.

 

Hello @till !
Since it's about acme.sh topic, i would like to add (also Deb12, ISPC 3..2.11p2 - when disabling SSL, or disabling Let'sencrypt for website, the files inside acme directory are not deleted which makes some interesting problems. One of my friends had two different websites: www.domain.com and domain.com. Then he added some subdomains for www.domain.com, and i believe deleted domain.com, and enabled Auto-subdomain "www". Because one directory wasn't deleted in acme, everytime acme refreshed certificates, it was overwriting the correct ones so the domain.com and all subdomains.domain.com certificate was wrong and showed that it is made for www.domain.com.

 

Looks like having website with same URL as hostname -f is problematic no matter what. Yesterday and today after acme.sh did the renew certificates thing, apache2 could not be started. I fixed it yesterday by creating the symlinks again, it looked like name of certificate file had changed. Today apache2 was down again, so I gave up and removed certificates for the website, disabled the website and removed the certificate with acme.sh --remove and delating the certificate files. Then I ran ispconfig_update.sh --force to make sure there is certificate created for the hostname.
Apache error log gave no clue on what was wrong, besides showing time when it went down. I looked in acme log what it was doing at that time and saw it was renewing hostname -f.