ERROR 503 need help debugging

I have a multiple server ISPConfig system. On one of the Webservers are two sites
One ist a regular web site, the other acts as a reverse proxy
the web server/site is just for internal testing and has no https but works fine
the reverse proxy has a valid Lets Encrypt certificate and worked fine since months at least till yesterday afternoon
since this morning it responds wit a 503 Error, while the site works fine
accessing the proxyed server directly works fine, the reason why I proxy is for security and that the backend server only has a self signed cert and can not have a Lets Encrypt cert for some other reason
server load is nearly zero, server is Debian 10 latest update yesterday morning ISPConfig 3.2.4 didn´t change anything since weeks.
systemctl status apache2 looks fine
apache access shows nothing
apache error log shows nothing
apache other.vhosts.log show 404 accessing the site and 503 accessing the proxy, cannot find further information
Rebooting the server changed nothing
I do not have any idea how to find out the reason. The access (ist a mail web frontend, no exchange OWA) is essential for my external users. need hint what to do
Thanks Rainer

 

Code:

gwmail.muekno.de:443 172.16.1.100 - - [26/Apr/2021:12:22:47 +0200] "GET /gw/webacc HTTP/1.1" 503 6183 "https://gwmail.muekno.de/gw/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36"
gwmail.muekno.de:443 172.16.1.100 - - [26/Apr/2021:12:23:07 +0200] "-" 408 4660 "-" "-"

accessing the proxy

Code:

zen.muekno.de:80 172.16.1.100 - - [26/Apr/2021:10:25:57 +0200] "GET /home HTTP/1.1" 200 2209 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36"
zen.muekno.de:80 172.16.1.100 - - [26/Apr/2021:10:25:57 +0200] "GET /fileadmin/images/img4.gif HTTP/1.1" 404 2129 "http://zen.muekno.de/fileadmin/default.css?1586694152" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36"

accessing the working site

Code:

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
SSLProxyEngine On
RequestHeader set Front-End-Https "On"
ProxyPreserveHost On
ProxyRequests Off
ServerName gwmail.muekno.de
#ServerAlias controlpanel.ispcp.domain.name
ProxyPass /.well-known !
ProxyPass / http://gw1.gerdakloos.de/  retry=0 disablereuse=On
ProxyPassReverse / http://gw1.gerdakloos.de/
<Proxy *>
Order deny,allow
Allow from all
</Proxy>

the proxy config Apache Direcive in the Option Tab of ISPConfig Site Config

Rainer

 

The ISPConfig controlled web server (the proxy) is Apache 2.4 (latest Debian 10 patches) the Apache on the Backend Server SuSE SLES/OES is 2.2 I know, for some essential reason I can not update it. It runs fine and is only reachable from my small internal net, so no security problem at all. I do not think it will stop working as it is the rest of my life too. Its is a VMware VM so if HW fail it my be moved easyly.
This should not the reason do stop working from one day to the other without changeing anything in the hole system. My wife worked via the proxy late sunday evening, while I was sleeping, sure doing nothing an the system
And yes I rebooted the backend server, and yes if I connect directly to the backend server and it works well, so I assume it is on the proxy side, but I do not know.
Just need hint who to debug, how to get more informational logs, would a packet trace help.
Sorry the late response, had been at customer yesterday.

Rainer

 

Solved by itself, while still searching it works again like before from one Moment to the other. Don't ask me why
Thank you
Rainer