Slow browsing in HTTPS

1 /I have nothing in the "global settings" of the haproxy regarding SSL.
2 / I have nothing about SSL in /etc/apache2/mods-enabled/ssl.conf.
3 / Include Optional conf-enabled / *. Conf no SSL option.
4 / Include Optional sites-enabled
SSL configurations are found by default in vhost.

I'm doing an apache2ctl -S
the first vhost is dev.dsden60.ac-amiens.fr

Code:

192.168.236.50:443     is a NameVirtualHost
         default server dev.dsden60.ac-amiens.fr (/etc/apache2/sites-enabled/100-dev.dsden60.ac-amiens.fr.vhost:110)
         port 443 namevhost dev.dsden60.ac-amiens.fr (/etc/apache2/sites-enabled/100-dev.dsden60.ac-amiens.fr.vhost:110)
                 alias dev.dsden60.ac-amiens.fr
         port 443 namevhost gdmat.dsden60.ac-amiens.fr (/etc/apache2/sites-enabled/100-gdmat.dsden60.ac-amiens.fr.vhost:116)
                 alias gdmat.dsden60.ac-amiens.fr

I have just modified the option as follows in dev.dsden60.ac-amiens.fr:

Code:

SSLProtocol All -SSLv2 -SSLv3 -TLSv1 -TLSv1.1

this applied to all sites after checking on Qualys SSL Labs site:

Code:

Overall Rating = A & Protocol Support = TLS 1.3 TLS 1.2 and always "Protocol or cipher suite mismatch" errors with TLS 1.3 protocol

We went from 17s to 10s of SSL negotiation, capture attached.
I do not understand why by acting on the first vhost 443 by default this has repercussions on all the sites (vhost).

I understand that the default server is just the first one encountered for the port because Apache parses the config / vhost files.
But why by acting on the first vhost 443 by default this has an impact on all the sites (vhost).

 

Without having a look on the system I can't really help you. You might consider using the professional support and let them have a look on your server: https://www.ispconfig.org/get-support/?type=ispconfig
I still believe you have a problem with your HAProxy as I have never seen that behavior with apache. This problem does not occur on the standard configuration. So, you somehow have messed with your config files. Given the time you have already invested, it might almost be faster to restart from scratch. Start with a minimal system and verify the results after each step.

What have you changed in that file? In the standard configuration, there should be a lot about SSL in that file

 

there doesn't seem too much wrong with the haproxy config, it can be as simple as:

Code:

frontend https
       bind *:443
       mode tcp
       default_backend bk_app

backend bk_app
      mode tcp
      server TWXAPP01  <twxapp01IP>:<port>

maybe remove the extraneous bits you have configured : 'check-ssl verify none' and see if that makes any difference.

but the concern i have with it is that your frontend doesn't specify any backend, not even as a default. It may be that it matches them as the frontend and backend have the same name, it's been years since i've used haproxy, so my memories hazy on that, but this is what concerns me about it:

 

ok, forget that about the backend not specified, just looked at the full haproxy cfg you attached, that does specify it.

perhaps the problem is you also set 'option ssl-hello-chk'
it looks like that's an old option that only uses sslv3, and manually constructs the tcp frame.
try removing that, or replace it with the check-ssl directive, it replaces the old ssl-hello-chk. It actually uses OpenSSL

 

Hello !
I always had the default configuration from the beginning and after upgrading from 3.1.15p2 to 3.2.2. The only modification made is in the vhost and I then reset via Ispconfig> Tools> Resync your websites.

Indeed, I must have a problem with the Haproxy which takes 10s to respond and 10 for SSL.
In private browsing I practically have timed out errors



I commented the line :

Code:

        ## vérifie la connexion ainsi que sa capacité à gérer les connexions SSL (SSLv3 en particulier).
        ## option ssl-hello-chk

We always have a long time.

 

as already stated. ssl-hello-chk is SSLv3 only. that's your service health check.
sslv3 is disabled on apache on your server (if it isn't, it bloody well should be). so it's always going to fail. which probably lines up quite nicely with what you're seeing, i'm betting the timeout is set at 10 seconds.

try removing option ssl-hello-chk or replace it with check-ssl or tcp-check.

 

1/ I confirm that the ssl-hello-chk option is indeed disabled in the backend.

Code:

        ## vérifie la connexion ainsi que sa capacité à gérer les connexions SSL (SSLv3 en particulier).
        ## option ssl-hello-chk

2/ SSLv3 is indeed disabled throughout vhost:

Code:

SSLProtocol All -SSLv2 -SSLv3 -TLSv1 -TLSv1.1

3/ in /etc/apache2/mods-enabled/ssl.conf

Code:

SSLProtocol all -SSLv3

4/ I also have in the backend the option:

Code:

server panel3_https.in 192.168.236.50:443 check-ssl verify none

 

From the manual:

I would try with the most simple haproxy setup (if you can't bypass it for a test). Then add your options one after the other and observe the results.
And while you are at it, you should really upgrade your HAProxy setup. There is a valid chance, that your problem is solved then.

 

I can't understand why the first default vhost on port 443 takes over all vhost?
Let me explain, when I modify an SSL option SSLProtocol example this applies to all vhost.

Code:

192.168.236.50:443     is a NameVirtualHost
         default server dev.dsden60.ac-amiens.fr (/etc/apache2/sites-enabled/100-dev.dsden60.ac-amiens.fr.vhost:110)
         port 443 namevhost dev.dsden60.ac-amiens.fr (/etc/apache2/sites-enabled/100-dev.dsden60.ac-amiens.fr.vhost:110)

When I modify an SSL option in any vhost it is not taken into account.

Code:

         port 443 namevhost madeleine.michelis-amiens.lyc.ac-amiens.fr (/etc/apache2/sites-enabled/100-madeleine.michelis-amiens.lyc.ac-amiens.fr.vhost:111)
                 alias madeleine.michelis-amiens.lyc.ac-amiens.fr

Does this correspond to normal behavior?

 

Another behavior, without putting the option "check-ssl verify none" in the backend to access via port 8080 to the management of Ispconfig that returns an error via my browser:

Code:

Bad Request
Your browser sent a request that this server could not understand.
Reason: You're speaking plain HTTP to an SSL-enabled server port.
Instead use the HTTPS scheme to access this URL, please.

######
Without the check-ssl verify none option the display is slower. I left the conf in the lightest haproxy to no avail. I need to plan to upgrade my server and update the version of my haproxy to 2.1 or 2.3.

 

your using http mode for the 8080 port frontend and backend. that will not put an https request through unchanged.

where exactly is this information being found? this is not the format of the information in the vhost configuration or the systemctl service status.

exactly how are you creating these websites and their certificates?

this is what the start of a vhost confguration should look like:

Code:

<Directory /var/www/domainname.co.uk>
                AllowOverride None
                                Require all denied
                </Directory>
<VirtualHost *:80>
                      DocumentRoot /var/www/clients/client1/web25/web

                ServerName domainname.co.uk
                ServerAlias www.domainname.co.uk
                ServerAdmin [email protected]

Code:

<Directory /var/www/madeleine.michelis-amiens.lyc.ac-amiens.fr>
        AllowOverride None
                Require all denied
        </Directory>

<VirtualHost 192.168.236.50:80>

                    DocumentRoot /var/www/clients/client5/web85/web
          
        ServerName madeleine.michelis-amiens.lyc.ac-amiens.fr
        ServerAlias madeleine.michelis-amiens.lyc.ac-amiens.fr
        ServerAdmin [email protected]

notice that in my example. the servername and serveralias are different, with the serveralias having the www prefix.
whilst in your vhost, the servername and serveralias are identical, which is just weird. that serveralias line should not exist in your vhost configuration. i'm not aware that it, in itself, would cause any issues, but it is indicative of some fundamental misconfiguration in your system somewhere.

 

My old experiences say that too much levels of config your apache2 server plus the duplicated SSLProtocol All (which in my mind should be on either vhost or ssl.conf and not both) could lead to delay as well.

 

I constantly have errors in all the apache logs of vhost with OCSP stapling

Code:

[ssl:error] [pid 8950] AH01941: stapling_renew_response: responder error
ssl_engine_init.c(1750): AH10083: Init: (madeleine.michelis-amiens.lyc.ac-amiens.fr:443) mod_md support
is unavailable.
ssl_util_stapling.c(761): AH01951: stapling_cb: OCSP Stapling callback called

Reference to vhost: 100-madeleine.michelis-amiens.lyc.ac-amiens.fr.vhost

Code:

SSLUseStapling on

</VirtualHost>

<IfModule mod_ssl.c>
        SSLStaplingCache shmcb:/var/run/ocsp(128000)
</IfModule>

 

I checked OCSP Stapling using this command: echo QUIT | openssl s_client -connect madeleine.michelis-amiens.lyc.ac-amiens.fr:443 -status
and received this response which shows it working:

Code:

CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = madeleine.michelis-amiens.lyc.ac-amiens.fr
verify return:1
OCSP response:
======================================
OCSP Response Data:
    OCSP Response Status: trylater (0x3)
======================================
---
Certificate chain
 0 s:CN = madeleine.michelis-amiens.lyc.ac-amiens.fr
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
subject=CN = madeleine.michelis-amiens.lyc.ac-amiens.fr

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3594 bytes and written 433 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)

I checked for the cache files mentioned in the "SSLStaplingCache shmcb vhost:/var/run/ocsp" but the path does not exist (/var/run/ ocsp). Does it have to exist?

Certificate test on the Qualyb SSL Lab website:
OCSP stapling No INTOLERANT

 

i don't think the non-existence of /var/run/ocsp is a problem, i don't have it on any of my servers, and i don't have any problems with ssl or slowness, and the stapling config in the vhost configurations matches yours.

that said, for :

Code:

Server certificate
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
subject=CN = madeleine.michelis-amiens.lyc.ac-amiens.fr

are the certificate details really empty? or did you remove all that data from your post?
if i run that status check on domains on my server i get all the certificate info included there.

and for

Code:

OCSP response:
======================================
OCSP Response Data:
   OCSP Response Status: trylater (0x3)
======================================

i don't get trylater, i get successful, along with a lot more data:

Code:

OCSP response:
======================================
OCSP Response Data:
    OCSP Response Status: successful (0x0)
    Response Type: Basic OCSP Response
    Version: 1 (0x0)
    Responder Id: C = US, O = Let's Encrypt, CN = R3
    Produced At: Jan 25 03:01:00 2021 GMT
    Responses:
    Certificate ID:
      Hash Algorithm: sha1
      Issuer Name Hash: <data removed>
      Issuer Key Hash: <data removed>
      Serial Number: <data removed>
    Cert Status: good
    This Update: Jan 25 03:00:00 2021 GMT
    Next Update: Feb  1 03:00:00 2021 GMT

    Signature Algorithm: sha256WithRSAEncryption
         39:<data removed>:06
======================================

 

when you ran

Code:

echo QUIT | openssl s_client -connect madeleine.michelis-amiens.lyc.ac-amiens.fr:443 -status

where did you run it? what ip does that host resolve for that domain name?

even if you ran it on your webserver, if that's resolving it to the external ip it's still sending in back out to come in via the haproxy instance, so it's not necessarily a fault on the webserver being detected.

you may need to set the domain and it's local private ip address in the webservers hosts file so that it doesn't use dns to resolve the domain name. and then run the openssl command again from the webservers cli. see what response you get then without any involvement from haproxy.

 

You are probably tunneling all traffic, not just traffic destined for the remote LAN... Find and compare the best payday loan and cash advance https://maybeloan.com/cash-advance

 

So I disable OCSP.
Then redo a certificate test on the Qualyb SSL Lab site: I no longer have the error: "protocol or cipher suite mismatch"
I also have as info:

Code:

Revocation status Validation error
OCSP ERROR: Exception: connect timed out [http://r3.o.lencr.org]

 

Performance test with the chrome debugger:
We went from 10s to 20.12ms for the SSL test



I no longer have an error: Lighthouse returned error: failed_document_request (net :: err_times_out).
On the performance test via the site: https://developers.google.com/

 

Command executed from the haproxy:

Code:

echo QUIT | openssl s_client -connect madeleine.michelis-amiens.lyc.ac-amiens.fr:443 -status

CONNECTED(00000004)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = madeleine.michelis-amiens.lyc.ac-amiens.fr
verify return:1
OCSP response: no response sent
---
Certificate chain
 0 s:CN = madeleine.michelis-amiens.lyc.ac-amiens.fr
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGWzCCBUOgAwIBAgISBNZWcYpdbkfO7uL/OU65/5BOMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTAxMDQwODE4MDFaFw0yMTA0MDQwODE4MDFaMDUxMzAxBgNVBAMT
Km1hZGVsZWluZS5taWNoZWxpcy1hbWllbnMubHljLmFjLWFtaWVucy5mcjCCAiIw
DQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBANxSHVzUsbpKkOj+ZBiX+F4xDjMi
WnPn4n/bLIIuhiAfgfkprS61efSVarLOiK1c2avjNEgS2ZLhexSrn5SwZjx+Vcrc
u88VbM3fe/DKDkZgBeBQShBHeFcZMb0aUhe1yCnPMegKYB5Z9OG0ZDeezguN/CnC
7HOs3DWOBiouss7ctHrCsGxcqFTBSjw1VBCmNO3JFQ+agpQTt8Zyp1TP4W/PWB/v
Lv46XdNo1r1WNlXkw7MmdwRrU8DSvzZ/GIVYRZDjh+E1PiWM0riG4gpf969Il56V
PCjodnYxI8luA67tSUEKb/ZPdqr3Y1gfZyN+m8MZXNS/sRLC2Cazce4bIsL/Mgh0
QQfZeljGd6MHoA7CtXJYpbAjEJDL1tyNHOLGuNcvpR0stoxVt3qCcvhj1zsZis6R
qajIWM5OQTsSIzHOAYwX30FzhKqEVT7RFizR/MRFx0W6iWLWEJpw4gYPDpOvPjE7
C1jcrauTbESNaVLxGH1bIi82TX9ztmJbDS6oxdcbAgLa6uHPAgdcQGVLsBr3dn6T
8x/EGw6L9+N3VS0MDkRLv5c0c+zJ4y3t0YvNthqilu9pd73JMnCYb+gTYf2THXo6
c7qyMB54aurI7Olk4H73D1mvLVbCXk/dooIAgyxVp657cTMqMzrzq9hQD0yfUvZR
UGwdhOvADSCbo2hzAgMBAAGjggJmMIICYjAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0l
BBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYE
FGMUBsKWBRvTIzTqikw+OhuA13hPMB8GA1UdIwQYMBaAFBQusxe3WFbLrlAJQOYf
r52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEFBQcwAYYVaHR0cDovL3IzLm8u
bGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8vcjMuaS5sZW5jci5vcmcvMDUG
A1UdEQQuMCyCKm1hZGVsZWluZS5taWNoZWxpcy1hbWllbnMubHljLmFjLWFtaWVu
cy5mcjBMBgNVHSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsG
AQUFBwIBFhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQUGCisGAQQB1nkC
BAIEgfYEgfMA8QB2AESUZS6w7s6vxEAH2Kj+KMDa5oK+2MsxtT/TM5a1toGoAAAB
dsywaBAAAAQDAEcwRQIhAIANWgdinIg6PWyhbDWYWHXLgn60Gk5qGNpjxTfu8D6o
AiA6bOIxGCt/3gArJ1buwYEivQoD7C5VW+KdWjBJZZ64DwB3APZclC/RdzAiFFQY
CDCUVo7jTRMZM7/fDC8gC8xO8WTjAAABdsywaAAAAAQDAEgwRgIhAIqmCx8w6Ub4
rji8ttcGb4M89WLpUfb8JZ5l+AHpeudnAiEA1bkhNN5tQ3mHobGQviZWLKWAzghb
fSvu28cHuMIjdgYwDQYJKoZIhvcNAQELBQADggEBADgnrn7t6CJh+NJW/9NuMx64
rUEINRMxUYVn4tMEx7G4UGaE1kRzFd32n8YIg62OLlw0OeVGHeBvRHsJjr/TQk+a
CSi98S4LeE4nWk6w5x5jzy899XJMnvOUU6ooIcs3IDnw+8db7j/+Dcy2LnJ4fYaE
yL2d3etFosI4eVYoBxP0e+uzJDFiWLBB8PeiabFUSkjGWkISZ+I8CvhFfuth2NYm
ZxWJxmB61Y0f7D9fTrDAU3cY+2nXtmpGJWpclU4MitGfvJzGtXL9FEHuZLKB7MFM
mPEBO5FcJV4sVqf2mQee9XbER4RiTfYUPPYc0s2Cb9Gyzpi7igaIT5ebr+pCgKo=
-----END CERTIFICATE-----
subject=CN = madeleine.michelis-amiens.lyc.ac-amiens.fr

issuer=C = US, O = Let's Encrypt, CN = R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3581 bytes and written 433 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE