Wrong Cert Served

Hello:

We host an ISP Config 3.1 server, Ubuntu 20.04, PHP 7.4, Apache2, MySQL

Currently we host 9 websites, seven of which have valid SSL certificates installed. (All Comodo, we don't use LetsEncrypt or other self-signed certificates). Of those 7 sites, 2 are suffering some SSL failure, so a 'default' cert is served up instead, causing a domain/cert name mismatch error

Code:

NET::ERR_CERT_COMMON_NAME_INVALID
This server could not prove that it is domain2.com; its security certificate is from domain1.com.

We're logging SSL activity (LogLevel debug), but the only entries we get are from Apache restarts.

Code:

[Wed Aug 26 16:12:08.948478 2020] [ssl:info] [pid 76774] AH01914: Configuring server domain1.com:443 for SSL protocol
[Wed Aug 26 16:12:08.948614 2020] [ssl:debug] [pid 76774] ssl_engine_init.c(497): AH01893: Configuring TLS extension handling
[Wed Aug 26 16:12:08.948622 2020] [ssl:debug] [pid 76774] ssl_util_stapling.c(929): AH01960: OCSP stapling initialized
[Wed Aug 26 16:12:08.948806 2020] [ssl:debug] [pid 76774] ssl_util_ssl.c(444): AH02412: [domain1.com:443] Cert matches for name 'domain1.com' [subject: CN=domain1.com / issuer: CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB / serial: 3649C77B154FF8B40987190B91D79D4B / notbefore: Aug 26 00:00:00 2020 GMT / notafter: Sep 27 00:00:00 2021 GMT]
[Wed Aug 26 16:12:08.948815 2020] [ssl:info] [pid 76774] AH02568: Certificate and private key domain1.com:443:0 configured from /var/www/clients/client1/web11/ssl/domain1.com.crt and /var/www/clients/client1/web11/ssl/domain1.com.key
[Wed Aug 26 16:12:08.948915 2020] [ssl:error] [pid 76774] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=domain1.com / issuer: CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB / serial: 3649C77B154FF8B40987190B91D79D4B / notbefore: Aug 26 00:00:00 2020 GMT / notafter: Sep 27 00:00:00 2021 GMT]
[Wed Aug 26 16:12:08.948922 2020] [ssl:error] [pid 76774] AH02604: Unable to configure certificate domain1.com:443:0 for stapling

At first we thought we were onto the problem, noting the failure to staple the certificate, and the missing CA certificate. But the other domain suffering the same symptoms, shows stapling was successful, and was fine with the exact same CA certificate

Code:

[Wed Aug 26 16:12:08.948927 2020] [ssl:info] [pid 76774] AH01914: Configuring server domain2.com:443 for SSL protocol
[Wed Aug 26 16:12:08.949058 2020] [ssl:debug] [pid 76774] ssl_engine_init.c(497): AH01893: Configuring TLS extension handling
[Wed Aug 26 16:12:08.949065 2020] [ssl:debug] [pid 76774] ssl_util_stapling.c(929): AH01960: OCSP stapling initialized
[Wed Aug 26 16:12:08.949331 2020] [ssl:debug] [pid 76774] ssl_util_ssl.c(444): AH02412: [domain2.com:443] Cert matches for name 'domain2.com' [subject: CN=domain2.com / issuer: CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB / serial: 8D15AAF1386A8702558C18627FE365CF / notbefore: Aug 20 00:00:00 2020 GMT / notafter: Sep 23 23:59:59 2020 GMT]
[Wed Aug 26 16:12:08.949339 2020] [ssl:info] [pid 76774] AH02568: Certificate and private key domain2.com:443:0 configured from /var/www/clients/client1/web5/ssl/domain2.com.crt and /var/www/clients/client1/web5/ssl/domain2.com.key

The other domains where SSL is working show identical successful certificate loading with each restart of Apache. Could not find any related errors in /var/log/apache2/error.log or log/error.log. (Is there a way to turn on debug level for the apache log?)

Happy to share real domain names if need be, I know that's otherwise avoided

Boiled down, mod_ssl is fine with the certificate when initializing at Apache startup, but https requests are failing.

Everything in the ispconfig configs for the various sites seem correct, and with no differences that we could identify.

All the cert directives in the associated .vhost files point to the correct path / file names; we've even double-checked the keys and certificates on the server, against what was actually issued. Everything checks.

Happy to provide any config detail, error logs, or the like. As always, any help or direction would be greatly appreciated.

htf_report here:

Code:

##### SERVER #####
IP-address (as per hostname): ***.***.***.***
[WARN] could not determine server's ip address by ifconfig
[INFO] OS version is Ubuntu 20.04.1 LTS

[INFO] ISPConfig is installed.
[WARN] /usr/local/ispconfig/server/lib/config.inc.php is missing.

##### VERSION CHECK #####

[INFO] php (cli) version is 7.4.3

##### PORT CHECK #####

##### MAIL SERVER CHECK #####

##### RUNNING SERVER PROCESSES #####

[WARN] I could not determine which web server is running.
[WARN] I could not determine which mail server is running.
[WARN] I could not determine which pop3 server is running.
[WARN] I could not determine which imap server is running.
[WARN] I could not determine which ftp server is running.

##### LISTENING PORTS #####
(only        ()
Local        (Address)
[anywhere]:21        (-)
***.***.***.***:53        (-)
[localhost]:53        (-)
***.***.***.***:53        (-)
[anywhere]:22        (-)
[anywhere]:25        (-)
[localhost]:953        (-)
[anywhere]:993        (-)
[anywhere]:995        (-)
[localhost]:10023        (-)
[localhost]:10024        (-)
[localhost]:10025        (-)
[localhost]:10026        (-)
[localhost]:10027        (-)
[anywhere]:587        (-)
[localhost]:11211        (-)
[anywhere]:110        (-)
[anywhere]:143        (-)
[anywhere]:465        (-)
*:*:*:*::*:21        (-)
*:*:*:*::*a6:e8ff:feb7:1:53        (-)
*:*:*:*::*:53        (-)
*:*:*:*::*:22        (-)
*:*:*:*::*:25        (-)
*:*:*:*::*:953        (-)
*:*:*:*::*:443        (-)
*:*:*:*::*:993        (-)
*:*:*:*::*:995        (-)
*:*:*:*::*:10024        (-)
*:*:*:*::*:10026        (-)
*:*:*:*::*:3306        (-)
*:*:*:*::*:587        (-)
[localhost]10        (-)
[localhost]43        (-)
*:*:*:*::*:8080        (-)
*:*:*:*::*:80        (-)
*:*:*:*::*:8081        (-)
*:*:*:*::*:465        (-)

##### IPTABLES #####

 

Thanks Till -
All good there; every site uses '*'.
Are there any other logs I could investigate? None of the php or apache logs have anything, nor do the 'custom' debug logs we configured for the various domains in the ssl.c section of the .vhost file. I thought there would be a way to turn on debug-level logging for any ssl events, but not seeing it.

(update; found how to turn on the debug firehose in Apache2; now seeing some related errors. Will update this post with results... )
Thanks!

 

Thanks guys -
The access logs are tricky to interpret, as they only shows internal IP's, e.g.:

Code:

172.31.13.170 - - [27/Aug/2020:16:16:27 +0000] "-" 408 0 "-" "-"

The http error 408 seems odd; it's certainly not timeout behavior we're seeing. And if I'm reading it right, the request header is empty, which would match what I found in the apache error log:

Code:

[Thu Aug 27 16:43:29.123075 2020] [reqtimeout:info] [pid 150686] [client 172.31.16.89:40104] AH01382: Request header read timeout

... and what should be the matching entry from the access log:

Code:

172.31.16.89 - - [27/Aug/2020:16:43:29 +0000] "-" 408 0 "-" "-"

If the header really is empty, where might I look to fix that?

Thanks as always ~

 

If you run "apache2ctl -S", is there an entry for the domain in question for both, port 80 and 443?

 

Yes - for the site in question (all sites, in fact), we see entries for both 80 and 443

 

Gentlemen:

No success as yet, but considering whacking a whole website record in ISPConfig, and starting with a fresh one. Not sure it will solve anything, but after everything else I've tried, figured it might help. I have excellent site and ISPConfig backups, so all good there. Any red flags that you see in this plan?

Thanks as always ~

 

Thanks Thom -
I have tried toggling SSL off/on, with no luck, but I will try removing & reinstalling the certs themselves. (I'm still convinced that it's something with the certs themselves, or their config, but can't prove it with any logs...)
Cheers ~

 

Thanks Thom, but unlucky - after deleting & reinstalling cert, no change. I did notice that after installing the cert, I looked on the server, and confirmed the crt file contains both certs (site cert and CA cert); I assume that's correct

intenttopay.com - logs show apache/ssl happy with the right cert, when restarting apache, but redirects
umsturz.net - logs show apache/ssl fails stapling cert, but otherwise seems to 'load' cert ok, but redirects
gcjaguar.net - fyi only; this is the default cert being served. We don't actually host a site under this domain; we us it as our server 'root'. E.g., postfix mail server is ns1.gcjaguar.net; we manage admin email under webmail.gcjaguar.net

Not ruling out desperate measures

 

Et, voila!

Thom, you nailed it - we had mis-configured our load-balancer, and once fixed, both sites are validating ssl/tls. I couldn't see the forest for the trees...

A huge thanks again to you and Till - do you guys have a tip jar?