We are seeing an issue on one of our ISPConfig 3 servers that when acme.sh renews, it causes httpd to get into a reloading loop where basically the apache service freezes up while reloading, and acme.sh times out trying to renew or verify the order. Here is the output when running the command manually: [11:19] [server acme.sh] # acme.sh --issue --apache -d example.com -d www.example.com --force [Wed Jan 19 11:20:03 EST 2022] Using CA: https://acme-v02.api.letsencrypt.org/directory [Wed Jan 19 11:20:04 EST 2022] Checking if there is an error in the apache config file before starting. [Wed Jan 19 11:20:04 EST 2022] OK [Wed Jan 19 11:20:04 EST 2022] JFYI, Config file /etc/httpd/conf/httpd.conf is backuped to /root/.acme.sh/httpd.conf [Wed Jan 19 11:20:04 EST 2022] In case there is an error that can not be restored automatically, you may try restore it yourself. [Wed Jan 19 11:20:04 EST 2022] The backup file will be deleted on success, just forget it. [Wed Jan 19 11:20:04 EST 2022] Creating domain key [Wed Jan 19 11:20:04 EST 2022] The domain key is here: /root/.acme.sh/example.com/example.com.key [Wed Jan 19 11:20:04 EST 2022] Multi domain='DNS:example.com,DNS:www.example.com' [Wed Jan 19 11:20:05 EST 2022] Getting domain auth token for each domain [Wed Jan 19 11:20:06 EST 2022] Getting webroot for domain='example.com' [Wed Jan 19 11:20:06 EST 2022] Getting webroot for domain='www.example.com' [Wed Jan 19 11:20:06 EST 2022] Verifying: example.com [Wed Jan 19 11:20:07 EST 2022] Pending, The CA is processing your order, please just wait. (1/30) [Wed Jan 19 11:20:10 EST 2022] Pending, The CA is processing your order, please just wait. (2/30) [Wed Jan 19 11:20:12 EST 2022] Pending, The CA is processing your order, please just wait. (3/30) [Wed Jan 19 11:20:14 EST 2022] Pending, The CA is processing your order, please just wait. (4/30) [Wed Jan 19 11:20:17 EST 2022] Pending, The CA is processing your order, please just wait. (5/30) [Wed Jan 19 11:20:19 EST 2022] Pending, The CA is processing your order, please just wait. (6/30) [Wed Jan 19 11:20:22 EST 2022] Pending, The CA is processing your order, please just wait. (7/30) [Wed Jan 19 11:20:24 EST 2022] Pending, The CA is processing your order, please just wait. (8/30) [Wed Jan 19 11:20:27 EST 2022] Pending, The CA is processing your order, please just wait. (9/30) [Wed Jan 19 11:20:29 EST 2022] Pending, The CA is processing your order, please just wait. (10/30) [Wed Jan 19 11:20:32 EST 2022] Pending, The CA is processing your order, please just wait. (11/30) [Wed Jan 19 11:20:34 EST 2022] Pending, The CA is processing your order, please just wait. (12/30) [Wed Jan 19 11:20:36 EST 2022] Pending, The CA is processing your order, please just wait. (13/30) [Wed Jan 19 11:20:39 EST 2022] Pending, The CA is processing your order, please just wait. (14/30) [Wed Jan 19 11:20:41 EST 2022] Pending, The CA is processing your order, please just wait. (15/30) [Wed Jan 19 11:20:44 EST 2022] Pending, The CA is processing your order, please just wait. (16/30) [Wed Jan 19 11:20:46 EST 2022] Pending, The CA is processing your order, please just wait. (17/30) [Wed Jan 19 11:20:48 EST 2022] Pending, The CA is processing your order, please just wait. (18/30) [Wed Jan 19 11:20:51 EST 2022] Pending, The CA is processing your order, please just wait. (19/30) [Wed Jan 19 11:20:53 EST 2022] Pending, The CA is processing your order, please just wait. (20/30) [Wed Jan 19 11:20:56 EST 2022] Pending, The CA is processing your order, please just wait. (21/30) [Wed Jan 19 11:20:58 EST 2022] example.com:Verify error:Fetching http://example.com/.well-known/acme-challenge/IXW7pyjMX4a-ogtEAdnR7Gx5PC_-7NFKsR7qtwVOZBk: Connection reset by peer [Wed Jan 19 11:20:59 EST 2022] Please check log file for more details: /var/log/ispconfig/acme.log Additionally, when checking the log file mentioned, this is what displays. https://pastebin.com/mw4ReE3M (had to put it in a PasteBin, it's too long to post) This is essentially happening for all domains on the server, and we can't renew any certs at the moment. When it attempts to reload apache it throws the following error: Redirecting to /bin/systemctl status httpd.service ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Drop-In: /usr/lib/systemd/system/httpd.service.d └─php-fpm.conf, php70-php-fpm.conf, php71-php-fpm.conf, php72-php-fpm.conf, php73-php-fpm.conf, php74-php-fpm. conf, php80-php-fpm.conf Active: reloading (reload) (Result: core-dump) since Wed 2022-01-19 11:21:20 EST; 12min ago Docs: man:httpd.service(8) Process: 123496 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS) Process: 54933 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=dumped, signal=ABRT) Main PID: 54933 (code=dumped, signal=ABRT) Status: "Reading configuration..." Tasks: 2 (limit: 49440) Memory: 77.4M CGroup: /system.slice/httpd.service ├─123823 vlogger (access log) └─123824 /usr/sbin/httpd -DFOREGROUND Jan 19 11:21:20 server httpd[54933]: AH00112: Warning: DocumentRoot [/var/www/server /web] does not exist Jan 19 11:21:20 server httpd[54933]: [Wed Jan 19 11:21:20.585931 2022] [alias:warn] [pid 54933] AH00671: The Alias directive in /etc/httpd/conf/httpd.conf at line 377 will probably never match because it overlaps an earlier Alias . Jan 19 11:21:20 server systemd[1]: Started The Apache HTTP Server. Jan 19 11:21:20 server httpd[54933]: Server configured, listening on: port 8080, port 8081, port 443, port 8 0 Jan 19 11:33:42 server systemd[1]: Reloading The Apache HTTP Server. Jan 19 11:33:42 server httpd[123496]: AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/httpd/conf/httpd.conf:360 Jan 19 11:33:42 server httpd[123496]: AH00112: Warning: DocumentRoot [/var/www/ispc1.sparkrack.net/web] does not exist Jan 19 11:33:42 server systemd[1]: Reloaded The Apache HTTP Server. Jan 19 11:33:46 server systemd-coredump[123826]: Process 54933 (httpd) of user 0 dumped core. Jan 19 11:33:46 server systemd[1]: httpd.service: Main process exited, code=dumped, status=6/ABRT Any assistance I can be given would be great appreciated.
You could try 'apachectl -t' and see if it complains of any problems, then maybe 'apachectl -X' to see what else you might find. If that doesn't find anything, my next step would be to run apache under strace and see what you find out.
Output of apachectl -t & apachectl -X AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/httpd/conf/httpd.conf:360 Syntax OK
In that case, I guess I won't reply. Just kidding. But it's nonsense to tag Till for assistance. If you want a specific person to support you, find someone you can hire to give you support. But if you post on the forum, anyone can reply and there are quite some users that help you out here while they aren't part of the ISPConfig team. Till reads all threads on this forum (in my experience), so if he wants to reply, he can do that. No need to ping him. But others can help you out just as well. By the way, Jesse already told you what you can try already: run Apache2 under strace, and I would recommend so as well. OK, now on to your original post. First off, you should never issue a certificate manually, as this will lead to problems with the acme client and the vhost for your website(s). I would recommend deleting the related acme.sh files for this domain and then try enabling it from the UI. Alright, so here's a interesting warning already: DocumentRoot [/var/www/ispc1.sparkrack.net/web] does not exist If you created this site through ISPConfig, that DocRoot should exist.
I've posted in the past, and @till and I'm pretty sure you, are usually the only two who answer (for one reason or another), not really sure why. Over the course of the night, this issue has started to affect all domains not just one, as overnight all the certificates on the server expired and are now invalid. We generally don't run the command manually, this was taken during debugging after attempts through the UI failed. That's the hostname of the server. Which doesn't need a webroot persay. But is supposed to be using the default /var/www/html directory.
I think this issue actually goes further than that, because making any changes to domains via the interface results in the following HTTPD crash. Jan 20 06:59:09 server systemd-coredump[423457]: Process 73929 (httpd) of user 0 dumped core. Jan 20 06:59:09 server systemd[1]: httpd.service: Main process exited, code=dumped, status=6/ABRT
Still no error when testing the apache config? What does not show up in an apache config test is if SSL cert files are missing, maybe some certs have been deleted?
Correct @till. Still no errors in the Apache config. What I find strange is I was just able to issue one via the interface (that previously failed last night) and now that website is showing a 503 error, but apache is running and all of the config for that site is correct (according to the vhost file). But other domains are failing now.
Hm, to further check the SSL files, run the SSL test script (Thanks @Croydon): To start the script, run the following command as root user on your server: Code: curl https://gitplace.net/pixcept/ispconfig-tools/-/raw/stable/cert_check.sh | sh Share the output here in code blocks please (insert -> code).
Curious, I must have misunderstood your initial description, I thought apache would not start (as your logs show). What do you do to start it up again, anything manually, or just wait? Or it was stopped, and it is now running again via 'apachectl -X' ?
Regarding the restart loop, you might want to diable (at least temporarily) the ISPConfig rescue module under System > server config.
Command doesn't issue an output, for some reason. Apache starts and has no issues until a certificate is requested, once a cert is requested it "reloads" and then stays in that status for about 60-120 seconds, by that time the challenge check has already expired. It appears to already be disabled.
