somethings change with php-fpm unix sockets on ubuntu 24.04

Discussion in 'General' started by nhybgtvfr, Jul 25, 2024.

  1. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    not sure exactly what's going on... but somethings changed about the php-fpm unix sockets on ubuntu 24.04

    a couple of vps's i've tried installing ispconfig on were displaying the same issue with monit continually restarting the php-fpm services.

    to confirm it's not anything i've done, i've created 2 new digital ocean droplets, same specs, one ubuntu 22.04 and one 24.04.
    all i've done on these is set the hostname, install available updates to the base install, reboot, and then install ispconfig using the auto-installer.

    the command used to install, on both droplets:
    Code:
    wget -O - https://get.ispconfig.org | sh -s -- --use-php=8.1,8.2,8.3 --use-ftp-ports=40110-40210 --no-mail --no-dns --no-roundcube --monit --monit-alert-email=<redacted> --ssh-permit-root=without-password --ssh-password-authentication=no --ssh-harden --unattended-upgrades --no-quota
    
    after installing ispconfig, i uncommented the ssl settngs in /etc/monit/conf-enabled/webui and restarted the monit service..
    no other changes have been made anywhere on the vps, and i've not even bothered logging into the ispconfig interface.

    after logging in to the monit interfaces on both vps', on 22.04 everything shows fine for the php8.(1|2|3)-fpm services.. connection is ok..
    on 24.04, after 1 minute, it shows connection failed and monit restarts the service, repeated every minute until monit unmonitors the services

    going into each php-fpm service shows the problem to be the same issue:
    FAILED to /var/run/php/php8.1-fpm.sock type TCP protocol DEFAULT
    FAILED to /var/run/php/php8.2-fpm.sock type TCP protocol DEFAULT
    FAILED to /var/run/php/php8.3-fpm.sock type TCP protocol DEFAULT

    if i comment out that check in the /etc/monit/conf-enabled/php8.(1|2|3)-fpm config file:
    # if failed unixsocket /var/run/php/php8.3-fpm.sock then restart
    and reload monit, and start monitoring those services again, it shows the connection ok for those services, even after several minutes, with no restarts.

    i can't find any differences in the file/socket permissions/locations.
    Code:
    lrwxrwxrwx 1 root     root     30 Jul  7 06:13 php-fpm.sock -> /etc/alternatives/php-fpm.sock
    -rw-r--r-- 1 root     root      3 Jul  7 06:13 php8.1-fpm.pid
    srw-rw---- 1 www-data www-data  0 Jul  7 06:13 php8.1-fpm.sock
    -rw-r--r-- 1 root     root      3 Jul  7 06:13 php8.2-fpm.pid
    srw-rw---- 1 www-data www-data  0 Jul  7 06:13 php8.2-fpm.sock
    -rw-r--r-- 1 root     root      3 Jul  7 06:13 php8.3-fpm.pid
    srw-rw---- 1 www-data www-data  0 Jul  7 06:13 php8.3-fpm.sock
    lsb_release -a
    No LSB modules are available.
    Distributor ID:    Ubuntu
    Description:    Ubuntu 22.04.4 LTS
    Release:    22.04
    Codename:    jammy
    
    
    lrwxrwxrwx 1 root     root     30 Jun 27 15:41 php-fpm.sock -> /etc/alternatives/php-fpm.sock
    -rw-r--r-- 1 root     root      6 Jul 17 06:59 php8.1-fpm.pid
    srw-rw---- 1 www-data www-data  0 Jul 17 06:59 php8.1-fpm.sock
    -rw-r--r-- 1 root     root      7 Jul 24 17:13 php8.2-fpm.pid
    srw-rw---- 1 www-data www-data  0 Jul 24 17:13 php8.2-fpm.sock
    -rw-r--r-- 1 root     root      7 Jul 25 12:11 php8.3-fpm.pid
    srw-rw---- 1 www-data www-data  0 Jul 25 12:11 php8.3-fpm.sock 
    lsb_release -a
    No LSB modules are available.
    Distributor ID:    Ubuntu
    Description:    Ubuntu 24.04 LTS
    Release:    24.04
    Codename:    noble
    
    and the /var/run/php folder on both servers is set to 755 and owned by www-data:www-data

    so i can't see anything that's actually set different anywhere.. thought it might be some new syntax format in monit, but the version of monit on both 22.04 and 24.04 is 5.33.0

    and it doesn't actually appear to be any problem in php itself.. if i install anything the needs php, wordpress, phpinfo.php etc, they all work fine.
    the only thing that seems to be a problem is the monit unix-socket check, a small problem, but i don't want to update production servers to 24.04 and not have the php services properly monitored.... the screenshots below are from another server i recently installed ispconfig to, running 24.04 and displaying the same problems with the monit php-fpm service checks, but test sites on the server are able to use php with no problem.

    i'm at a complete loss now as to what's causing this.



    Screenshot from 2024-07-25 13-27-21.png Screenshot from 2024-07-25 13-28-22.png Screenshot from 2024-07-25 13-30-30.png
     
  2. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    very weird.. it's just a standard auto-install. get the same problem with 24.04 on aws and hetzner..

    found some more info in the monit logs, but again, i can't see any permission, location, or configuration that is in any way any different than on ubuntu 22.04 servers....

    Code:
    [2024-07-25T14:10:32+0000] info     : Starting Monit 5.33.0 daemon with http interface at [*]:2812
    [2024-07-25T14:10:32+0000] info     : 'taw-test1' Monit 5.33.0 started
    [2024-07-25T14:10:33+0000] error    : 'php8.3-fpm' process is not running
    [2024-07-25T14:10:33+0000] info     : 'php8.3-fpm' trying to restart
    [2024-07-25T14:10:33+0000] info     : 'php8.3-fpm' start: '/usr/bin/systemctl start php8.3-fpm'
    [2024-07-25T14:10:33+0000] error    : 'php8.2-fpm' process is not running
    [2024-07-25T14:10:33+0000] info     : 'php8.2-fpm' trying to restart
    [2024-07-25T14:10:33+0000] info     : 'php8.2-fpm' start: '/usr/bin/systemctl start php8.2-fpm'
    [2024-07-25T14:10:33+0000] error    : 'php8.1-fpm' process is not running
    [2024-07-25T14:10:33+0000] info     : 'php8.1-fpm' trying to restart
    [2024-07-25T14:10:33+0000] info     : 'php8.1-fpm' start: '/usr/bin/systemctl start php8.1-fpm'
    [2024-07-25T14:11:34+0000] info     : 'php8.3-fpm' process is running with pid 4143
    [2024-07-25T14:11:34+0000] error    : Unix socket /var/run/php/php8.3-fpm.sock connection error -- Permission denied
    [2024-07-25T14:11:34+0000] error    : 'php8.3-fpm' failed protocol test [DEFAULT] at /var/run/php/php8.3-fpm.sock -- Cannot create unix socket for /var/run/php/php8.3-fpm.sock
    [2024-07-25T14:11:34+0000] info     : 'php8.3-fpm' trying to restart
    [2024-07-25T14:11:34+0000] info     : 'php8.3-fpm' stop: '/usr/bin/systemctl stop php8.3-fpm'
    [2024-07-25T14:11:34+0000] info     : 'php8.3-fpm' start: '/usr/bin/systemctl start php8.3-fpm'
    [2024-07-25T14:11:35+0000] info     : 'php8.2-fpm' process is running with pid 4152
    [2024-07-25T14:11:35+0000] error    : Unix socket /var/run/php/php8.2-fpm.sock connection error -- Permission denied
    [2024-07-25T14:11:35+0000] error    : 'php8.2-fpm' failed protocol test [DEFAULT] at /var/run/php/php8.2-fpm.sock -- Cannot create unix socket for /var/run/php/php8.2-fpm.sock
    [2024-07-25T14:11:35+0000] info     : 'php8.2-fpm' trying to restart
    [2024-07-25T14:11:35+0000] info     : 'php8.2-fpm' stop: '/usr/bin/systemctl stop php8.2-fpm'
    [2024-07-25T14:11:35+0000] info     : 'php8.2-fpm' start: '/usr/bin/systemctl start php8.2-fpm'
    [2024-07-25T14:11:35+0000] info     : 'php8.1-fpm' process is running with pid 4160
    [2024-07-25T14:11:35+0000] error    : Unix socket /var/run/php/php8.1-fpm.sock connection error -- Permission denied
    [2024-07-25T14:11:35+0000] error    : 'php8.1-fpm' failed protocol test [DEFAULT] at /var/run/php/php8.1-fpm.sock -- Cannot create unix socket for /var/run/php/php8.1-fpm.sock
    [2024-07-25T14:11:35+0000] info     : 'php8.1-fpm' trying to restart
    [2024-07-25T14:11:35+0000] info     : 'php8.1-fpm' stop: '/usr/bin/systemctl stop php8.1-fpm'
    [2024-07-25T14:11:35+0000] info     : 'php8.1-fpm' start: '/usr/bin/systemctl start php8.1-fpm'
    [2024-07-25T14:12:36+0000] error    : Unix socket /var/run/php/php8.3-fpm.sock connection error -- Permission denied
    [2024-07-25T14:12:36+0000] error    : 'php8.3-fpm' failed protocol test [DEFAULT] at /var/run/php/php8.3-fpm.sock -- Cannot create unix socket for /var/run/php/php8.3-fpm.sock
    [2024-07-25T14:12:36+0000] info     : 'php8.3-fpm' trying to restart
    [2024-07-25T14:12:36+0000] info     : 'php8.3-fpm' stop: '/usr/bin/systemctl stop php8.3-fpm'
    [2024-07-25T14:12:36+0000] info     : 'php8.3-fpm' start: '/usr/bin/systemctl start php8.3-fpm'
    [2024-07-25T14:12:36+0000] error    : Unix socket /var/run/php/php8.2-fpm.sock connection error -- Permission denied
    [2024-07-25T14:12:36+0000] error    : 'php8.2-fpm' failed protocol test [DEFAULT] at /var/run/php/php8.2-fpm.sock -- Cannot create unix socket for /var/run/php/php8.2-fpm.sock
    [2024-07-25T14:12:36+0000] info     : 'php8.2-fpm' trying to restart
    [2024-07-25T14:12:36+0000] info     : 'php8.2-fpm' stop: '/usr/bin/systemctl stop php8.2-fpm'
    [2024-07-25T14:12:37+0000] info     : 'php8.2-fpm' start: '/usr/bin/systemctl start php8.2-fpm'
    [2024-07-25T14:12:37+0000] error    : Unix socket /var/run/php/php8.1-fpm.sock connection error -- Permission denied
    [2024-07-25T14:12:37+0000] error    : 'php8.1-fpm' failed protocol test [DEFAULT] at /var/run/php/php8.1-fpm.sock -- Cannot create unix socket for /var/run/php/php8.1-fpm.sock
    [2024-07-25T14:12:37+0000] info     : 'php8.1-fpm' trying to restart
    [2024-07-25T14:12:37+0000] info     : 'php8.1-fpm' stop: '/usr/bin/systemctl stop php8.1-fpm'
    [2024-07-25T14:12:37+0000] info     : 'php8.1-fpm' start: '/usr/bin/systemctl start php8.1-fpm'
    
     
    ahrasis likes this.
  3. till

    till Super Moderator Staff Member ISPConfig Developer

    I have not heard of that issue yet, but my own systems use Debian. I can try to make a test setup to see if I can reproduce it here in a few days unless we find a solution for that until then.
     
    ahrasis and nhybgtvfr like this.
  4. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    just checking back to see if anyone else has been able to verify this problem exists on ubuntu 24.04..

    i get it on a local virtualbox install, and using aws, digitalocean, and hetzner vps's, and i'm only setting the hostname, installing base OS updates immediately after vps creation, and then running an automated ispconfig install using standard options, so i can't believe it's something specific that i'm doing, or that no-one else encounters the same issue.
     
  5. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    They are using their own repo right? Mine is downloaded directly from ubuntu and setup following minimal setup. Php-fpm are all still working fine.
     
  6. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    yeah, they all use their own repo's.. although the virtualbox install used a iso image downloaded directly from ubuntu..
    i also tried using an iso image to install onto a hetzner vps, although they also supplied the iso, so i don't know if they modified that at all.

    and as far as i can tell, php-fpm is working fine. i can still install php sites and they appear to work fine, regardless of the php version used (8.1, 8.2 or 8.3)..

    the only issue is that monit keeps trying to restart each php-fpm version each minute because it doesn't seem to have permission to connect to /var/run/php/php#-#.sock, which is what i can't work out..
    i can't find any difference in folder/file locations, ownership, or permissions compared to ubuntu 22.04
    the actual file contents all seem to be identical. and the monit version installed on both 22.04 and 24.04 is 5.33.0, so no difference there either.

    it's happened on too many installs on different hosting providers to be a coincidence, and i can, and have done exactly the same install on ubuntu 22.04 and 24.04 with aws and digital ocean. absolutely no difference in any command used, and in ubuntu 22.04, monit can connect to the php-fpm sockets fine, 24.04 it can't..

    i just can't find any reason for it.
     
  7. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    Did you restart monit after that? And monitor what happened thereafter?
     
    Last edited: Aug 1, 2024
  8. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    yep. restarted monit several times, rebooted the vps's.
    nothing seems to make any difference..

    in the monit log:
    Code:
    [2024-08-01T17:15:55+0000] error    : Unix socket /var/run/php/php8.2-fpm.sock connection error -- Permission denied
    [2024-08-01T17:15:55+0000] error    : 'php8.2-fpm' failed protocol test [DEFAULT] at /var/run/php/php8.2-fpm.sock -- Cannot create unix socket for /var/run/php/php8.2-fpm.sock
    [2024-08-01T17:15:55+0000] info     : 'php8.2-fpm' trying to restart
    [2024-08-01T17:15:55+0000] info     : 'php8.2-fpm' stop: '/usr/bin/systemctl stop php8.2-fpm'
    [2024-08-01T17:15:55+0000] info     : 'php8.2-fpm' start: '/usr/bin/systemctl start php8.2-fpm'
    [2024-08-01T17:15:55+0000] error    : Unix socket /var/run/php/php8.1-fpm.sock connection error -- Permission denied
    [2024-08-01T17:15:55+0000] error    : 'php8.1-fpm' failed protocol test [DEFAULT] at /var/run/php/php8.1-fpm.sock -- Cannot create unix socket for /var/run/php/php8.1-fpm.sock
    [2024-08-01T17:15:55+0000] info     : 'php8.1-fpm' trying to restart
    [2024-08-01T17:15:55+0000] info     : 'php8.1-fpm' stop: '/usr/bin/systemctl stop php8.1-fpm'
    [2024-08-01T17:15:55+0000] info     : 'php8.1-fpm' start: '/usr/bin/systemctl start php8.1-fpm'
    
    and as i've said, the entire paths to the socket files are identical paths, ownership and permissions as on an identically installed 22.04 instance, which works without issues. the monit configuration files are identical. i'm just completely stumped on this one.
     
  9. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    I am really sorry but I must be checking a wrong server.

    I can confirm this as a bug. For now the issue for php-fpm is permission which will be gone if the owner is root instead of www-data but I do not know for sure why.
     
    Last edited: Aug 2, 2024
    Taleman likes this.
  10. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    great.. at least now i know i'm not just going mad. it's been driving me nuts
     
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    Are the tests run against the default socket of the php-fpm process (the one for the www-data user)?
     
  12. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    should be, monit config doesn't suggest anything else...

    /etc/monit/conf-enabled/php8.3-fpm
    Code:
    check process php8.3-fpm with pidfile /var/run/php/php8.3-fpm.pid
            group php-fpm
            start program = "/usr/bin/systemctl start php8.3-fpm" with timeout 60 seconds
            stop program = "/usr/bin/systemctl stop php8.3-fpm"
            if failed unixsocket /var/run/php/php8.3-fpm.sock then restart
            if 5 restarts within 5 cycles then timeout
            depend php8.3-fpm_bin
    
    check file php8.3-fpm_bin with path /usr/sbin/php-fpm8.3
            group php-fpm
            include /etc/monit/templates/rootbin
    
    you can see the monit checks yourself, at https://apachetest.theaccessibleweb.co.uk:2812
    login admin, pass 3XFFQUeP95ct

    a working wordpress is installed at apachewp.theaccessibleweb.co.uk, and the phpinfo for that vhost is accessible at
    https://apachewp.theaccessibleweb.co.uk/phpinfo.php

    it's a standalone install with nothing except a new ispconfig install and a new default wp install on it, so feel free to restart the services, i'm only going to wipe the vps out anyway.. nothing needed on there.
    i can give you root access to the vps as well if you want to check any config files for yourself, just pm me your public ssh-key.
     
  13. till

    till Super Moderator Staff Member ISPConfig Developer

    The ownership of the socket is (or at least can be) defined in the pool file. the www pool files come with the PHP Debian/Ubuntu packages, so its not from ISPConfig. My guess is that the settings in the www pool file have been changed, or nothing is set there and the defaults have been changed. As the packages you use are likely the ones from Ondrej, it might be Ondrej who changed something in the defaults of his packages for Ubuntu 24.04. So you might want to look into the www pool file of the affected PHP versions.
     
  14. till

    till Super Moderator Staff Member ISPConfig Developer

    You can find the files in /etc/php/VERSION/php-fpm/pool.d/ Replace VERSION with the PHP version number.
     
  15. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    run a diff on all the www.conf files from the 24.04 test server with the equivalents from a working 22.04 install. both using the ondrej repo.

    the www.conf files on both servers for each corresponding php version are identical.
     
  16. till

    till Super Moderator Staff Member ISPConfig Developer

    Ok, so the second option I mentioned might be the case, that's that the compiled-in defaults have changed. So we can try to get the original setup back by altering the pool file that ships with the ondrej packages. If I understand it correctly from what @ahrasis posted, the issue comes from change of owner of the file from www-data to root user?
     
  17. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    you mean changing /etc/php/*.*/fpm/pool.d/www.conf file ownership from www-data to root?

    they're already owned by root:root on both 22.04 and 24.04
     
  18. till

    till Super Moderator Staff Member ISPConfig Developer

    no. I mean to change the listen owner, mode and group of the socket in that file. The settings are:

    listen.owner =
    listen.group =
    listen.mode =

    so likely, we have to add this line:

    listen.owner = www-data
     
  19. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    Both already set to use www-data.
    What I was trying to say is, currently in Ubuntu 24.04 the owner www-data fo all /var/run/php/php*-fpm.sock have no permission to connect to unix socket resulting to error "Unix socket /var/run/php/php{version}-fpm.sock connection error -- Permission denied". It may be temporarily worked out for monit by changing the owner for the same to root though this is most probably not a fix.

    To test the work around, I ran:
    Code:
    chown root:root /var/run/php/php*-fpm.sock
    To restore:
    Code:
    chown www-data:www-data /var/run/php/php*-fpm.sock
     
  20. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    yeah, changing the owner of the .sock files works, at least in that it makes monit happy.
    the problem then is global aliases like /phpmyadmin and /webmail won't work.. as the service is unavailable.

    i don't think there is anything wrong with php itself with the default installed settings.. it all seems to work.
    the fact that the global aliases work when the socket files are owned by www-data but not root further indicate this.
    it's just the monit check itself which fails, i assume trying to connect as root.
     

Share This Page