Inconsistent SSL configurations

Hi,

I have two websites: X and Y

On X, file_get_contents(..) works fine for posting to the website itself via https.
On Y, file_get_contents(..) throws the following error when posting to the website itself via https:

PHP Warning: file_get_contents(): SSL operation failed with code 1. OpenSSL Error messages:
mod_fcgid: stderr: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed in FILENAME
mod_fcgid: stderr: PHP Warning: file_get_contents(): Failed to enable crypto in FILENAME
mod_fcgid: stderr: PHP Warning: file_get_contents(https://YYYYYYYY):$
mod_fcgid: stderr: PHP Fatal error: Uncaught Exception: Request to URL 'https://YYYYYYYYY
....

The one significant difference I can find is the two websites' certificates.

X's vhost file contains:
SSLCertificateFile /var/www/clients/client1/web4/ssl/XXX-le.crt
SSLCertificateKeyFile /var/www/clients/client1/web4/ssl/XXX-le.key

Y's vhost file contains:
SSLCertificateFile /var/www/clients/client10/web75/ssl/YYY-le.crt
SSLCertificateKeyFile /var/www/clients/client10/web75/ssl/YYY-le.key

Looks identical. However, what's interesting is the symlinks to these files.

Content of X's certificate folder (notice how XXX-le.crt points to fullchain.pem):
root@isp:/# ls -la /var/www/clients/client1/web4/ssl/
total 68
drwxr-xr-x 2 root root 4096 Jan 18 2019 .
drwxr-xr-x 11 root root 4096 Dec 4 2017 ..
lrwxrwxrwx 1 root root 44 Jan 18 2019 XXX-le.bundle -> /etc/letsencrypt/live/XXX/chain.pem
-r-------- 1 root root 1647 Dec 4 2017 XXX-le.bundle.old.20171204194502
-r-------- 1 root root 1647 Dec 4 2017 XXX-le.bundle.old.20171204194902
-r-------- 1 root root 1647 Feb 16 2018 XXX-le.bundle.old.20180216075913
-r-------- 1 root root 1647 Feb 16 2018 XXX-le.bundle.old.20180216080003
-r-------- 1 root root 1647 Jan 18 2019 XXX-le.bundle.old.20190118191504
lrwxrwxrwx 1 root root 48 Jan 18 2019 XXX-le.crt -> /etc/letsencrypt/live/XXX/fullchain.pem
-r-------- 1 root root 2159 Dec 4 2017 XXX-le.crt.old.20171204194502
-r-------- 1 root root 3806 Dec 4 2017 XXX-le.crt.old.20171204194902
-r-------- 1 root root 3830 Feb 16 2018 XXX-le.crt.old.20180216075913
-r-------- 1 root root 3830 Feb 16 2018 XXX-le.crt.old.20180216080003
-r-------- 1 root root 3944 Jan 18 2019 XXX-le.crt.old.20190118191504
lrwxrwxrwx 1 root root 46 Jan 18 2019 XXX-le.key -> /etc/letsencrypt/live/XXX/privkey.pem
-r-------- 1 root root 3272 Dec 4 2017 XXX-le.key.old.20171204194502
-r-------- 1 root root 3272 Dec 4 2017 XXX-le.key.old.20171204194902
-r-------- 1 root root 3272 Feb 16 2018 XXX-le.key.old.20180216075913
-r-------- 1 root root 3272 Feb 16 2018 XXX-le.key.old.20180216080003
-r-------- 1 root root 3272 Jan 18 2019 XXX-le.key.old.20190118191504

Content of Y's certificate folder (notice how YYY-le-crt points to cert.pem)
root@isp:/# ls -la /var/www/clients/client10/web75/ssl/
total 12
drwxr-xr-x 2 root root 4096 Sep 18 2017 .
drwxr-xr-x 11 root root 4096 Jul 29 22:24 ..
lrwxrwxrwx 1 root root 58 Sep 18 2017 YYY-le.bundle -> /etc/letsencrypt/live/YYY/chain.pem
lrwxrwxrwx 1 root root 57 Sep 18 2017 YYY-le.crt -> /etc/letsencrypt/live/YYY/cert.pem
lrwxrwxrwx 1 root root 60 Sep 18 2017 YYY-le.key -> /etc/letsencrypt/live/YYY/privkey.pem

I know close to little about SSL, but my understand is that fullchain.pem is the file contains both the certificate and intermediate certificates while cert.pem only contains the certificate - and we need the fullchain.pem file for file_get_contents(..) to work properly.

1) Why are the two websites different ?
2) Can I fix the Y website (and potentially other broken websites) without modifying the vhost files by hand or changing symlinks? This is a production server, so I can't risk breaking things - I must be able to upgrade packages and ISPConfig. Also, I'd like both old and future website to behave the same.

- Thanks in advance

Best Regards
Jimmy Thomsen

 

@till, thanks for your suggestion. I just created a new Let's Encrypt certificate for a new website, and that too does not have the full chain in the cert.pem file. But that's how it's supposed to work (https://community.letsencrypt.org/t/public-and-private-keys/25493).

root@isp:/# ls -la /etc/letsencrypt/live/ZZZZZZZ/
total 12
drwxr-xr-x 2 root root 4096 Aug 1 12:43 .
drwx------ 13 root root 4096 Aug 1 12:42 ..
lrwxrwxrwx 1 root root 31 Aug 1 12:42 cert.pem -> ../../archive/ZZZZZZZ/cert1.pem
lrwxrwxrwx 1 root root 32 Aug 1 12:42 chain.pem -> ../../archive/ZZZZZZZ/chain1.pem
lrwxrwxrwx 1 root root 36 Aug 1 12:42 fullchain.pem -> ../../archive/ZZZZZZZ/fullchain1.pem
lrwxrwxrwx 1 root root 34 Aug 1 12:42 privkey.pem -> ../../archive/ZZZZZZZ/privkey1.pem
-rw-r--r-- 1 root root 692 Aug 1 12:42 README
root@isp:/#

1) Would it make sense to have ISPConfig reference fullchain.pem in vhosts instead I wonder?
I'm not familiar with the potential security implications.
2) Rather than modifying the files, wouldn't it make better sense to change the symlink like so?:
/var/www/clients/client10/web75/ssl/YYY-le.crt => /etc/letsencrypt/live/YYY/cert.pem
to
/var/www/clients/client10/web75/ssl/YYY-le.crt => /etc/letsencrypt/live/YYY/fullchain.pem
This is what the working website is doing and it seems to have been working for a while.
Are you familiar with anything that would break ? Will ISPConfig overwrite /var/www/clients/client10/web75/ssl/YYY-le.crt
and have it point to the cert.pem file again at some point, or will ISPConfig only "ensure" these symlinks if they are not already there ?

- Thanks

Jimmy

 

@till, or would it perhaps be possible to point to a specific certificate file using the Apache Directives under Options for the website? I'm not sure how it works. If I specify
SSLCertificateFile /var/www/clients/client0/web109/ssl/custom-symlink-to-fullchain.crt
it will simply add that line at the bottom of both <VirtualHost *:80> and <VirtualHost *:443>, and crash Apache.
Can I somehow make it merge into <VirtualHost *:443> and have it replace the existing SSLCertificateFile directive added by ISPConfig?

I'm trying to figure out which solution would be the most reliable as I continue to upgrade the OS and ISPConfig.

 

Sorry @till, I only just now realized that the new website with a new certificate has its certificate symlink in the ssl directory pointing to fullchain.pem and not chain.pem. So that's good.

I tried fixing the problem with my Y website by renaming YYY-le.crt (symlink to /etc/letsencrypt/live/YYY/cert.pem) to __YYY-le.crt and created a new symlink:
YYY-le.crt -> /etc/letsencrypt/live/YYY/fullchain.pem

However, that caused SSLLabs' SSL checker (https://www.ssllabs.com/ssltest/analyze.html) to report
"This server's certificate chain is incomplete. Grade capped to B."

I finally got fed up and removed the SSL certificate (unchecked it under the website settings), gave it a couple of minutes, and checked it again, and everything now works fine. The symlink now points to fullchain.pem, and the SSL checker reports no errors. Guess I should have tried that hours ago

Jimmy