ISPConfig upgrade 3.1.15 -> 3.2.2 certificate fails

Updated Debian 9 to Debian 10 previously. After that the certificates no longer renewed, did not debug much.
Now acme.sh is installed and upgrading ISPConfig 3.1.15 to 3.2.2 (replaced real domain name with mydomain. It does resolve from name service):

Code:

Create new ISPConfig SSL certificate (yes,no) [no]: yes
Checking / creating certificate for mail.mydomain.fi
Using certificate path /etc/letsencrypt/live/mail.mydomain.fi
Using apache for certificate validation
[ti 26.1.2021 10.00.31 +0200] mail.mydomain.fi:Verify error:Fetching http://mail.mydomain.fi/.well-known/acme-challenge/HoWdtIubVd-_UF1l-43fp7KIzMCS-pfEaRqgt8CkmQQ: Connection refused
[ti 26.1.2021 10.00.31 +0200] Please add '--debug' or '--log' to check more details.
[ti 26.1.2021 10.00.31 +0200] See: https://github.com/acmesh-official/acme.sh/wiki/How-to-debug-acme.sh
Issuing certificate via acme.sh failed. Please check that your hostname can be verified by letsencrypt
Could not issue letsencrypt certificate, falling back to self-signed.
genrsa: Can't open "/usr/local/ispconfig/interface/ssl/ispserver.key" for writing, No such file or directory
Can't open /usr/local/ispconfig/interface/ssl/ispserver.key for reading, No such file or directory
139673506100416:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:69:fopen('/usr/local/ispconfig/interface/ssl/ispserver.key','r')
139673506100416:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:76:
unable to load Private Key
Can't open /usr/local/ispconfig/interface/ssl/ispserver.key for reading, No such file or directory
140664533984448:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:69:fopen('/usr/local/ispconfig/interface/ssl/ispserver.key','r')
140664533984448:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:76:
unable to load Private Key
Can't open /usr/local/ispconfig/interface/ssl/ispserver.key for reading, No such file or directory
140154495259840:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:69:fopen('/usr/local/ispconfig/interface/ssl/ispserver.key','r')
140154495259840:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:76:
unable to load Private Key
PHP Warning:  rename(/usr/local/ispconfig/interface/ssl/ispserver.key.insecure,/usr/local/ispconfig/interface/ssl/ispserver.key): No such file or directory in /tmp/update_stable.sh.m9VrhEVEYE/ispconfig3_install/install/lib/installer_base.lib.php on line 3097
Reconfigure Crontab? (yes,no) [yes]:

My guess is acme.sh fails getting certificate because apache2 is not running. And apache does not start because certicate error.

Code:

root@mail:~# systemctl --state=failed
  UNIT            LOAD   ACTIVE SUB    DESCRIPTION                   
● apache2.service loaded failed failed The Apache HTTP Server       
● dovecot.service loaded failed failed Dovecot IMAP/POP3 email server

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

2 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
root@mail:~# systemctl status apache2.service
● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2021-01-26 10:04:59 EET; 12min ago
     Docs: https://httpd.apache.org/docs/2.4/
  Process: 13521 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILURE)

tammi 26 10:04:58 mail systemd[1]: Starting The Apache HTTP Server...
tammi 26 10:04:59 mail apachectl[13521]: AH00548: NameVirtualHost has no effect and will be remove
tammi 26 10:04:59 mail apachectl[13521]: AH00526: Syntax error on line 188 of /etc/apache2/sites-e
tammi 26 10:04:59 mail apachectl[13521]: SSLCertificateFile: file '/var/www/clients/client0/web1/s
tammi 26 10:04:59 mail apachectl[13521]: Action 'start' failed.
tammi 26 10:04:59 mail apachectl[13521]: The Apache error log may have more information.
tammi 26 10:04:59 mail systemd[1]: apache2.service: Control process exited, code=exited, status=1/
tammi 26 10:04:59 mail systemd[1]: apache2.service: Failed with result 'exit-code'.
tammi 26 10:04:59 mail systemd[1]: Failed to start The Apache HTTP Server.

 

I do

Code:

# ls -lh /var/www/clients/client*/web*/ssl

and see lots of key, crt and bundle files pointing to /etc/letsencrypt/live/. The /etc/letsencrypt is removed already, so all those are missing and Apache complains.

 

Not much is working, from htf_report.txt:

Code:

##### SERVER #####
IP-address (as per hostname): ***.***.***.***
[WARN] could not determine server's ip address by ifconfig
[INFO] OS version is Debian GNU/Linux 10 (buster)
 
[INFO] uptime:  10:45:32 up 9 min,  1 user,  load average: 0,03, 0,11, 0,10
 
[INFO] memory:
              total        used        free      shared  buff/cache   available
Mem:          1,9Gi       1,6Gi       106Mi        26Mi       215Mi       153Mi
Swap:         1,3Gi        13Mi       1,2Gi
 
[INFO] ISPConfig is installed.

##### ISPCONFIG #####
ISPConfig version is 3.2.2


##### VERSION CHECK #####

[INFO] php (cli) version is 7.3.26-1+0~20210112.74+debian10~1.gbpd78724
[INFO] php-cgi (used for cgi php in default vhost!) is version 5.6.40-0+deb8u2

##### PORT CHECK #####

[WARN] Port 8080 (ISPConfig) seems NOT to be listening
[WARN] Port 8081 (ISPConfig Apps) seems NOT to be listening
[WARN] Port 80 (Webserver) seems NOT to be listening
[WARN] Port 443 (Webserver SSL) seems NOT to be listening
[WARN] Port 143 (IMAP server) seems NOT to be listening
[WARN] Port 993 (IMAP server SSL) seems NOT to be listening
[WARN] Port 110 (POP3 server) seems NOT to be listening
[WARN] Port 995 (POP3 server SSL) seems NOT to be listening

##### MAIL SERVER CHECK #####

 

It seems (from your second post) that you deleted /etc/letsencrypt. You should not have done that. Apache cannot run because each vhost has missing certificates now and so you are not able to issue new ones.

What you need to do now is:

• either restore /etc/letsencrypt with all contents, so apache can start again
• or delete all vhost files from /etc/apache2/sites-enabled that are using non-existent LE cert files, then remove all the invalid symlinks from the /*/ssl/ paths, then restart apache and re-save each website from the ISPConfig UI to re-generate the certificates and vhost files.

 

Should I uninstall acme.sh and apt-get install certbot? Then reconfigure services with ispconfig?
Websites work but for some reason logging in to ISPConfig panel as admin fails.

 

Depends.
Have you restored /etc/letsencrypt? Then uninstall acme and install certbot (while the apt-get version is quite outdated …).
If you have not restored /etc/letsencrypt then stick with acme and re-create the certs.

 

I restored /etc/letsencrypt from backup, and websites work. Still can not figure out why I can not login to ISPConfig panel. So I can not recreate certificates ...
Debian 10 has certbot 0.31.0-1 but backports buster has python3-certbot-dns-gandi (1.2.5-2~bpo10+1).

 

Now logging to panel works. Go figure.
No luck today, still problems. Running ispconfig_update.sh --force. Got complaints instance of certbot was already running, so I did systemctl stop certbot.service and tried again:

Code:

Create new ISPConfig SSL certificate (yes,no) [no]: yes

Checking / creating certificate for mail.mydomain.fi
Using certificate path /etc/letsencrypt/live/mail.mydomain.fi
Using apache for certificate validation
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.mydomain.fi
Using the webroot path /usr/local/ispconfig/interface/acme for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. mail.mydomain.fi (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://mail.mydomain.fi/.well-known/acme-challenge/9duCjmYR1rEcnZC5LNISCT0ibwoXeDhj55-ZH8AjTHw: Connection refused
Issuing certificate via certbot failed. Please check log files and make sure that your hostname can be verified by letsencrypt
Could not issue letsencrypt certificate, falling back to self-signed.
Generating RSA private key, 4096 bit long modulus (2 primes)

 

Code:

grep -r '9duCjmYR1rEcnZC5LNISCT0ib' /var/log/apache2 /var/log/ispconfig/httpd

 

That command finds nothing.
My interpretation is that apache went down again. So apache is not running and LE fails to verify.
And again, apache2 fails to start.

 

I can not figure this. Apache is not running:

Code:

root@mail:/var/log/apache2# systemctl --state=failed
  UNIT            LOAD   ACTIVE SUB    DESCRIPTION                                     
● apache2.service loaded failed failed The Apache HTTP Server                         
● certbot.service loaded failed failed Certbot                                         
● dovecot.socket  loaded failed failed Dovecot IMAP/POP3 email server activation socket

And it can not start

Code:

Create new ISPConfig SSL certificate (yes,no) [no]: yes

Checking / creating certificate for mail.mydomain.fi
Using certificate path /etc/letsencrypt/live/mail.mydomain.fi
Using apache for certificate validation
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.mydomain.fi
Using the webroot path /usr/local/ispconfig/interface/acme for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. mail.mydomain.fi (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://mail.mydomain.fi/.well-known/acme-challenge/9duCjmYR1rEcnZC5LNISCT0ibwoXeDhj55-ZH8AjTHw: Connection refused
Issuing certificate via certbot failed. Please check log files and make sure that your hostname can be verified by letsencrypt
Could not issue letsencrypt certificate, falling back to self-signed.
Generating RSA private key, 4096 bit long modulus (2 primes)

ispconfig_update.sh can not issue LE certificate because apache is not running.
I could restore this host from snapshot, but I do not understand why it messed certificatet this morning so this may happen again.

 

I do not understand why apache fails to start. This from apache error log:

Code:

[ 2021-01-26 16:31:32.3224 721/7f79e52e1980 age/Cor/CoreMain.cpp:967 ]: Passenger core shutdown finished
[ 2021-01-26 16:36:29.0587 5448/7f2f9ea3d980 age/Wat/WatchdogMain.cpp:1291 ]: Starting Passenger watchdog...
[ 2021-01-26 16:36:29.0814 5451/7f8273f63980 age/Cor/CoreMain.cpp:982 ]: Starting Passenger core...
[ 2021-01-26 16:36:29.0815 5451/7f8273f63980 age/Cor/CoreMain.cpp:235 ]: Passenger core running in multi-application mode.
[ 2021-01-26 16:36:29.0833 5451/7f8273f63980 age/Cor/CoreMain.cpp:732 ]: Passenger core online, PID 5451
[ 2021-01-26 16:36:29.0928 5458/7fb68ad8f980 age/Ust/UstRouterMain.cpp:529 ]: Starting Passenger UstRouter...
[ 2021-01-26 16:36:29.0945 5458/7fb68ad8f980 age/Ust/UstRouterMain.cpp:342 ]: Passenger UstRouter online, PID 5458
AH00016: Configuration Failed

[ 2021-01-26 16:36:29.1204 5451/7f8273674700 age/Cor/CoreMain.cpp:532 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ 2021-01-26 16:36:29.1205 5451/7f8273f63980 age/Cor/CoreMain.cpp:901 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...

[ 2021-01-26 16:36:29.1206 5458/7fb68ac90700 age/Ust/UstRouterMain.cpp:422 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ 2021-01-26 16:36:29.1206 5458/7fb68ad8f980 age/Ust/UstRouterMain.cpp:492 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ 2021-01-26 16:36:29.1206 5458/7fb68a40e700 Ser/Server.h:817 ]: [UstRouterApiServer] Freed 0 spare client objects
[ 2021-01-26 16:36:29.1207 5458/7fb68a40e700 Ser/Server.h:464 ]: [UstRouterApiServer] Shutdown finished
[ 2021-01-26 16:36:29.1208 5458/7fb68ac90700 Ser/Server.h:464 ]: [UstRouter] Shutdown finished
[ 2021-01-26 16:36:29.1209 5451/7f8273674700 Ser/Server.h:817 ]: [ServerThr.1] Freed 128 spare client objects
[ 2021-01-26 16:36:29.1209 5451/7f8273674700 Ser/Server.h:464 ]: [ServerThr.1] Shutdown finished
[ 2021-01-26 16:36:29.1210 5458/7fb68ad8f980 age/Ust/UstRouterMain.cpp:523 ]: Passenger UstRouter shutdown finished
[ 2021-01-26 16:36:29.1218 5451/7f8272df2700 Ser/Server.h:817 ]: [ApiServer] Freed 0 spare client objects
[ 2021-01-26 16:36:29.1219 5451/7f8272df2700 Ser/Server.h:464 ]: [ApiServer] Shutdown finished

[ 2021-01-26 16:36:29.1253 5451/7f8273674700 age/Cor/CoreMain.cpp:532 ]: Signal received. Gracefully shutting down... (send signal 1 more time(s) to force shutdown)
[ 2021-01-26 16:36:29.1460 5451/7f8273f63980 age/Cor/CoreMain.cpp:967 ]: Passenger core shutdown finished

 

In /usr/local/ispconfig/interface/ssl the ispserver.key is ordinary file, not a symlink. I'll try making it symlink to the corresponding entry in letsencrypt files.
Copying the set of files with same datestamp from backup worked, apache starts. Now to figure out how I break this setup and not do it again, and check sites do have a valid certificate.

 

I have this same problem, how did you fix this?
i just updated from 3.1.15p3 to 3.2.4 few moments ago, now have got i have same problem, im still using ubuntu 18.04, but all sites are working but no access to control panel.

Code:

Can't open /usr/local/ispconfig/interface/ssl/ispserver.crt for writing, No such file or directory
140485853753792:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('/usr/local/ispconfig/interface/ssl/ispserver.crt','w')
140485853753792:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
writing RSA key
cat: /usr/local/ispconfig/interface/ssl/ispserver.crt: No such file or directory
Symlink ISPConfig SSL certs to Postfix? (y,n) [y]:

 

I think you have missing required lib files or something, so retrace / redo all required steps in ISPConfig PST normally fix such error(s) and, if that are all done, try run ispconfig force update again.

Before that I think it is better that you to check and remove old method of securing ispconfig panel (by undoing all of the steps) as it will be conflicting with ISPConfig 3.2++, if you followed that steps before.

 

I did forget to mention that I used your uninstall script prior to the update, so just to confirm I already did this step that you suggest, I can try to run the script again but this was not mentioned previously in this thread as a Cause

letsencrypt.log

Code:

2021-05-14 12:17:34,249:DEBUG:certbot.main:certbot version: 0.27.0
2021-05-14 12:17:34,250:DEBUG:certbot.main:Arguments: ['--agree-tos', '--non-interactive', '--expand', '--rsa-key-size', '4096', '--server', 'https://acme-v02.api.letsencrypt.org/directory', '--authenticator', 'webroot', '--webroot-path', '/usr/local/ispconfig/interface/acme', '--email', '[email protected]', '-d', 'server1.example.com', '--renew-hook', 'letsencrypt_renew_hook.sh']
2021-05-14 12:17:34,251:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2021-05-14 12:17:34,284:DEBUG:certbot.log:Root logging level set at 20
2021-05-14 12:17:34,285:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2021-05-14 12:17:34,286:DEBUG:certbot.plugins.selection:Requested authenticator webroot and installer None
2021-05-14 12:17:34,286:DEBUG:certbot.plugins.selection:Single candidate plugin: * webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator
Initialized: <certbot.plugins.webroot.Authenticator object at 0x7f30f35b4710>
Prep: True
2021-05-14 12:17:34,288:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.webroot.Authenticator object at 0x7f30f35b4710> and installer None
2021-05-14 12:17:34,288:INFO:certbot.plugins.selection:Plugins selected: Authenticator webroot, Installer None
2021-05-14 12:17:34,302:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/letsencrypt", line 11, in <module>
    load_entry_point('certbot==0.27.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1364, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1238, in certonly
    le_client = _init_le_client(config, auth, installer)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 641, in _init_le_client
    acc, acme = _determine_account(config)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 512, in _determine_account
    acc = display_ops.choose_account(accounts)
  File "/usr/lib/python3/dist-packages/certbot/display/ops.py", line 83, in choose_account
    "Please choose an account", labels, force_interactive=True)
  File "/usr/lib/python3/dist-packages/certbot/display/util.py", line 512, in menu
    self._interaction_fail(message, cli_flag, "Choices: " + repr(choices))
  File "/usr/lib/python3/dist-packages/certbot/display/util.py", line 474, in _interaction_fail
    raise errors.MissingCommandlineFlag(msg)
certbot.errors.MissingCommandlineFlag: Missing command line flag or config entry for this setting:
Please choose an account
Choices: ['server1.example.com@2019-07-20T11:19:22Z (8d53)', 'server1.example.com@2019-07-16T11:46:07Z (6193)']