Hello guys, I found myself with some issues that I'm not very sure what could have cause them, My only idea for now is some update broke something, I have an ISPConfig 3.2.2 (latest at the time of this post) Ubuntu Server 20.04 Focal PHP 7.4 FPM from Sury PPA 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 « here the possible update issue. Set manually to 7.4 (update-alternatives --config php) since it will install bits of PHP8.0 Apache 2 Apache/2.4.41 (Ubuntu) http2 using mpm_event Any way, now when trying to access the ISPConfig interface I get a white page, /va/log/apache2/error.log says, Code: [Mon Feb 15 23:20:46.086988 2021] [proxy_fcgi:error] [pid 2617081] [client 111.222.333.444:41182] AH01071: Got error 'PHP message: PHP Warning: require_once(/usr/local/ispconfig/interface/lib/config.inc.php): failed to open stream: Permission denied in /usr/local/ispconfig/interface/web/index.php on line 31PHP message: PHP Fatal error: require_once(): Failed opening required '../lib/config.inc.php' (include_path='.:/usr/share/php') in /usr/local/ispconfig/interface/web/index.php on line 31' Which is very strange, I have found this old thread where they describe a very similar behavior where permissions were not working properly, by changing them (last two post) they could get a login screen. Code: chmod 660 /usr/local/ispconfig/interface/lib/config.inc.php chmod 760 /usr/local/ispconfig/interface/temp But that's not the end of the permissions issues, so I want to stop here and ask for some advice on how to debug/fix this issue. I've being searching and found no further info. Is this some known issue? Another side symptom is that some sites are complaining for directories permission too. What I've tried with no result, Downgrade to Focal's default PHP Reinstall ISPConfig to reset permissions from installer (just in case) Also just in case here the changelog from the latest recorded changes on the php ppa, Any suggestion is appreciated, regards.
Just found something that describes very closely the behavior that I'm facing, just with an legacy PHP version, https://stackoverflow.com/questions...n-after-updating-php56w-from-5-6-31-to-5-6-35 In this case the cause was SELinux, which I don't have, but I looking into this I just realize that I had AppArmor installed It was supposed to be removed since I installed the system, but I guess it got reintroduced when upgrading among releases xenial > bionic > focal. Code: The following packages will be REMOVED: apparmor lxd lxd-client snapd 0 upgraded, 0 newly installed, 4 to remove and 0 not upgraded. After this operation, 128 MB disk space will be freed. Do you want to continue? [Y/n] If apparmor would had something to do with it, what would be the best way to approach this issue? Regards
Seems as if somehow suexec is not working anymore on your system as the ispconfig vhost apparently runs as www-data user and group now instead of the user and group ispconfig. I've noticed another thraed on that issue and what both threads have in common that you use mpm_event, so standard setups with mpm_prefork do not seem to be effected. I guess apparmor might be required by snap, so removing it might cause issues if you run any snaps. Maybe you can try to disable apparmor instead for a test and not remove it.
Thanks till, I've tried switching back to prefork, using the following, Code: a2dismod mpm_event http2 a2enmod mpm_prefork php7.4 systemctl restart apache2 Still not luck.
Hello Th0m, thanks for the hints, I was hoping stopping/removing/disabling apparmor would restore a normal behavior, but it didn't. Also php-cgi, 7.4 flavor has being the default selection with no change there. Still I applied update-alternatives, restarted services, and why not, tried turning it off and on again. Seems like what till mentioned, is the core issue, since re-installing ISPConfig to confirm correct permissions won't work. I'm wondering what could have triggered this new behavior, since last week everything was working fine.
Hi again, here the output. Code: drwxr-x--- 5 ispconfig ispconfig 4096 feb 15 03:59 . drwxr-x--- 9 ispconfig ispconfig 4096 ene 8 2019 .. -rwxr-x--- 1 ispconfig ispconfig 15813 feb 16 00:50 app.inc.php drwxr-x--- 5 ispconfig ispconfig 4096 feb 15 04:35 classes -rw------- 1 ispconfig ispconfig 6783 feb 16 00:50 config.inc.php -rwxr-x--- 1 ispconfig ispconfig 6783 feb 16 00:50 config.inc.php~ drwxrwx--- 2 ispconfig ispconfig 4096 ene 8 2019 lang drwxr-x--- 2 ispconfig ispconfig 4096 jul 9 2019 plugins -rwxr-x--- 1 ispconfig ispconfig 341 feb 16 00:50 server_conf.master -rwxr-x--- 1 ispconfig ispconfig 226 feb 16 00:50 shelluser_blacklist
I was pointed out by a caring user that even when mpm_prefork and php7.4 mod was enabled maybe some other php-fpm version might be stealing the connection, and he was right, by checking the debug output (also instructed to check) I found the thief, Code: [Wed Feb 17 01:15:35.327351 2021] [proxy:debug] [pid 168839] mod_proxy.c(1251): [client 111.222.333.444:43362] AH01143: Running scheme unix handler (attempt 0) [Wed Feb 17 01:15:35.327366 2021] [proxy_fcgi:debug] [pid 168839] mod_proxy_fcgi.c(1030): [client 111.222.333.444:43362] AH01076: url: fcgi://localhost/var/www/ispconfig/index.php proxyname: (null) proxyport: 0 [Wed Feb 17 01:15:35.327375 2021] [proxy_fcgi:debug] [pid 168839] mod_proxy_fcgi.c(1039): [client 111.222.333.444:43362] AH01078: serving URL fcgi://localhost/var/www/ispconfig/index.php [Wed Feb 17 01:15:35.327380 2021] [proxy:debug] [pid 168839] proxy_util.c(2336): AH00942: FCGI: has acquired connection for (*) [Wed Feb 17 01:15:35.327388 2021] [proxy:debug] [pid 168839] proxy_util.c(2392): [client 111.222.333.444:43362] AH00944: connecting fcgi://localhost/var/www/ispconfig/index.php to localhost:8000 [Wed Feb 17 01:15:35.327397 2021] [proxy:debug] [pid 168839] proxy_util.c(2428): [client 111.222.333.444:43362] AH02545: fcgi: has determined UDS as /run/php/php8.0-fpm.sock [Wed Feb 17 01:15:35.327557 2021] [proxy:debug] [pid 168839] proxy_util.c(2615): [client 111.222.333.444:43362] AH00947: connected /var/www/ispconfig/index.php to httpd-UDS:0 [Wed Feb 17 01:15:35.327658 2021] [proxy:debug] [pid 168839] proxy_util.c(2982): AH02823: FCGI: connection established with Unix domain socket /run/php/php8.0-fpm.sock (*) [Wed Feb 17 01:15:35.329548 2021] [proxy_fcgi:error] [pid 168839] [client 111.222.333.444:43362] AH01071: Got error 'PHP message: PHP Warning: require_once(/usr/local/ispconfig/interface/lib/config.inc.php): Failed to open stream: Permission denied in /usr/local/ispconfig/interface/web/index.php on line 31PHP message: PHP Fatal error: Uncaught Error: Failed opening required '../lib/config.inc.php' (include_path='.:/usr/share/php') in /usr/local/ispconfig/interface/web/index.php:31\nStack trace:\n#0 {main}\n thrown in /usr/local/ispconfig/interface/web/index.php on line 31' [Wed Feb 17 01:15:35.329605 2021] [proxy:debug] [pid 168839] proxy_util.c(2351): AH00943: FCGI: has released connection for (*) For the record update-alternatives --config php/php-cgi are both set to php7.4, anyway. So by removing the package, php8.0-fpm apache2 was able to select the correct php proxy channel so the permissions match resolving all the permissions issues and bringing back the ISPConfig interface, ufff close call . Moral lesson here, Don't trust (yet) that sneaky php8.0 when using Sury's PHP PPA. Now with php8.0-fpm out of the way, i tried again http2 and mpm_event just to confirm the behavior that till mentioned before. And yes, the behavior is consistent, when using mpm_event seems php connection messes the channel permissions and takes the whole thing down. So if this could be of any use here the debug output section, Code: [Wed Feb 17 01:40:56.745608 2021] [proxy:debug] [pid 176083:tid 140021859481344] mod_proxy.c(1251): [client 111.222.333.444:44040] AH01143: Running scheme unix handler (attempt 0) [Wed Feb 17 01:40:56.745616 2021] [proxy_fcgi:debug] [pid 176083:tid 140021859481344] mod_proxy_fcgi.c(1030): [client 111.222.333.444:44040] AH01076: url: fcgi://localhost/var/www/ispconfig/index.php proxyname: (null) proxyport: 0 [Wed Feb 17 01:40:56.745622 2021] [proxy_fcgi:debug] [pid 176083:tid 140021859481344] mod_proxy_fcgi.c(1039): [client 111.222.333.444:44040] AH01078: serving URL fcgi://localhost/var/www/ispconfig/index.php [Wed Feb 17 01:40:56.745628 2021] [proxy:debug] [pid 176083:tid 140021859481344] proxy_util.c(2336): AH00942: FCGI: has acquired connection for (*) [Wed Feb 17 01:40:56.745636 2021] [proxy:debug] [pid 176083:tid 140021859481344] proxy_util.c(2392): [client 111.222.333.444:44040] AH00944: connecting fcgi://localhost/var/www/ispconfig/index.php to localhost:8000 [Wed Feb 17 01:40:56.745642 2021] [proxy:debug] [pid 176083:tid 140021859481344] proxy_util.c(2428): [client 111.222.333.444:44040] AH02545: fcgi: has determined UDS as /run/php/php7.4-fpm.sock [Wed Feb 17 01:40:56.745845 2021] [proxy:debug] [pid 176083:tid 140021859481344] proxy_util.c(2615): [client 111.222.333.444:44040] AH00947: connected /var/www/ispconfig/index.php to httpd-UDS:0 [Wed Feb 17 01:40:56.745886 2021] [proxy:debug] [pid 176083:tid 140021859481344] proxy_util.c(2982): AH02823: FCGI: connection established with Unix domain socket /run/php/php7.4-fpm.sock (*) [Wed Feb 17 01:40:56.746784 2021] [proxy_fcgi:error] [pid 176083:tid 140021859481344] [client 111.222.333.444:44040] AH01071: Got error 'PHP message: PHP Warning: require_once(/usr/local/ispconfig/interface/lib/config.inc.php): failed to open stream: Permission denied in /usr/local/ispconfig/interface/web/index.php on line 31PHP message: PHP Fatal error: require_once(): Failed opening required '../lib/config.inc.php' (include_path='.:/usr/share/php') in /usr/local/ispconfig/interface/web/index.php on line 31' [Wed Feb 17 01:40:56.746855 2021] [proxy:debug] [pid 176083:tid 140021859481344] proxy_util.c(2351): AH00943: FCGI: has released connection for (*) [Wed Feb 17 01:40:56.747448 2021] [http2:debug] [pid 176083:tid 140021859481344] h2_task.c(83): [client 111.222.333.444:44040] AH03348: h2_task(79-15): open output to GET srv.domain.com:8080 / [Wed Feb 17 01:40:56.747593 2021] [http2:debug] [pid 176083:tid 140021344454400] h2_session.c(1384): [client 111.222.333.444:44040] AH03073: h2_stream(79-15,HALF_CLOSED_REMOTE): submit response 500, REMOTE_WINDOW_SIZE=12582912 [Wed Feb 17 01:40:56.747702 2021] [http2:debug] [pid 176083:tid 140021344454400] h2_session.c(1539): [client 111.222.333.444:44040] AH02936: h2_stream(79-15,HALF_CLOSED_REMOTE): resumed [Wed Feb 17 01:40:56.747854 2021] [http2:debug] [pid 176083:tid 140021344454400] h2_session.c(589): [client 111.222.333.444:44040] AH03068: h2_session(79,BUSY,1): sent FRAME[HEADERS[length=253, hend=1, stream=15, eos=0]], frames=10/4 (r/s) [Wed Feb 17 01:40:56.747916 2021] [http2:debug] [pid 176083:tid 140021344454400] h2_session.c(589): [client 111.222.333.444:44040] AH03068: h2_session(79,BUSY,1): sent FRAME[DATA[length=0, flags=1, stream=15, padlen=0]], frames=10/5 (r/s) [Wed Feb 17 01:40:56.748056 2021] [http2:debug] [pid 176083:tid 140021344454400] h2_bucket_beam.c(1275): [client 111.222.333.444:44040] beam(79-15,output,closed=1,aborted=1,empty=1,buf=0): AH03385: h2_task_destroy, reuse secondary [Wed Feb 17 01:40:56.748198 2021] [http2:debug] [pid 176083:tid 140021344454400] h2_session.c(1694): [client 111.222.333.444:44040] AH03078: h2_session(79,IDLE,0): transit [BUSY] -- no io (keepalive) --> [IDLE] [Wed Feb 17 01:40:56.830584 2021] [http2:debug] [pid 176083:tid 140021344454400] h2_session.c(337): [client 111.222.333.444:44040] AH03066: h2_session(79,IDLE,0): recv FRAME[SETTINGS[ack=1, stream=0]], frames=10/5 (r/s) [Wed Feb 17 01:41:02.836117 2021] [http2:debug] [pid 176083:tid 140021436774144] h2_session.c(589): [client 111.222.333.444:44040] AH03068: h2_session(79,IDLE,0): sent FRAME[GOAWAY[error=0, reason='timeout', last_stream=15]], frames=11/6 (r/s) [Wed Feb 17 01:41:02.836399 2021] [http2:debug] [pid 176083:tid 140021436774144] h2_session.c(751): [client 111.222.333.444:44040] AH03069: h2_session(79,IDLE,0): sent GOAWAY, err=0, msg=timeout [Wed Feb 17 01:41:02.836438 2021] [http2:debug] [pid 176083:tid 140021436774144] h2_session.c(1694): [client 111.222.333.444:44040] AH03078: h2_session(79,DONE,0): transit [IDLE] -- local goaway --> [DONE] [Wed Feb 17 01:41:02.836541 2021] [http2:debug] [pid 176083:tid 140021436774144] h2_session.c(1694): [client 111.222.333.444:44040] AH03078: h2_session(79,CLEANUP,0): transit [DONE] -- pre_close --> [CLEANUP] Thanks again for all the interest and help, I think I owe someone a coffe or a beer.
I suspect this was not the case; unless things changed within apache so that there proxy takes priority over the type handling of mod-php (not a change within the php8.0-fpm conf file), this has never been the case, it would be new behavior in apache internals. Would it simply be that both your old php7.4-fpm conf and the new php8.0-fpm conf were enabled at the same time? Only one would effectively be used, and if it were 8.0, things would break. Thinking out loud, I wonder if php8.0-fpm doesn't work with suexec, and that causes the permission problems? Edit: It comes to mind that at least on debian the ispconfig vhost does not use fpm mode at all, but fastcgi, so even having conf for multiple fpm versions enabled shouldn't have mattered, and the requests should never have hit a fpm daemon. Maybe ubuntu 20.04 is different?
@Ark74 what does 'apt policy's show for suexec and other php meta packages? Eg. see https://www.howtoforge.com/community/threads/roundcube-1-4-11-and-multiple-php-versions.86397/ for a recent php packaging issue - maybe you have something similar that got changed?
Hey Jesse thanks for the follow up. In order to enable http2 and mpm_event is required to disable php7.4, not sure if that could influence the behavior. Indeed, it's my understanding that to be the case, but as put on the log, seems that the sites responded to a fpm connection. Here the output, Code: apt-cache policy apache2-suexec-pristine php php-fpm php-cgi php-cli php-intl php-json apache2-suexec-pristine: Installed: 2.4.41-4ubuntu3.1 Candidate: 2.4.41-4ubuntu3.1 Version table: *** 2.4.41-4ubuntu3.1 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages 100 /var/lib/dpkg/status 2.4.41-4ubuntu3 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/universe amd64 Packages php: Installed: (none) Candidate: 2:8.0+80~exp2+ubuntu20.04.1+deb.sury.org+1 Version table: 2:8.0+80~exp2+ubuntu20.04.1+deb.sury.org+1 500 500 http://ppa.launchpad.net/ondrej/php/ubuntu focal/main amd64 Packages 2:7.4+75 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/main amd64 Packages php-fpm: Installed: (none) Candidate: 2:8.0+80~exp2+ubuntu20.04.1+deb.sury.org+1 Version table: 2:8.0+80~exp2+ubuntu20.04.1+deb.sury.org+1 500 500 http://ppa.launchpad.net/ondrej/php/ubuntu focal/main amd64 Packages 2:7.4+75 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/universe amd64 Packages php-cgi: Installed: (none) Candidate: 2:8.0+80~exp2+ubuntu20.04.1+deb.sury.org+1 Version table: 2:8.0+80~exp2+ubuntu20.04.1+deb.sury.org+1 500 500 http://ppa.launchpad.net/ondrej/php/ubuntu focal/main amd64 Packages 2:7.4+75 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/main amd64 Packages php-cli: Installed: (none) Candidate: 2:8.0+80~exp2+ubuntu20.04.1+deb.sury.org+1 Version table: 2:8.0+80~exp2+ubuntu20.04.1+deb.sury.org+1 500 500 http://ppa.launchpad.net/ondrej/php/ubuntu focal/main amd64 Packages 2:7.4+75 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/main amd64 Packages php-intl: Installed: (none) Candidate: 2:8.0+80~exp2+ubuntu20.04.1+deb.sury.org+1 Version table: 2:8.0+80~exp2+ubuntu20.04.1+deb.sury.org+1 500 500 http://ppa.launchpad.net/ondrej/php/ubuntu focal/main amd64 Packages 2:7.4+75 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/universe amd64 Packages php-json: Installed: (none) Candidate: 2:8.0+80~exp2+ubuntu20.04.1+deb.sury.org+1 Version table: 2:8.0+80~exp2+ubuntu20.04.1+deb.sury.org+1 500 500 http://ppa.launchpad.net/ondrej/php/ubuntu focal/main amd64 Packages 2:7.4+75 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/universe amd64 Packages Hmm interesting, none of the above metapackages are installed, mostly all php packages are focused to the os default php version (7.4) unless stated on the Perfect Guide. https://www.howtoforge.com/tutorial...l-pureftpd-bind-postfix-doveot-and-ispconfig/ Let me know if there is any other info I can provide. Edit: I'm also adding the php7.4 packages policy, Code: apt-cache policy apache2-suexec-pristine php7.4 php7.4-fpm php7.4-cgi php7.4-cli php7.4-intl php7.4-json apache2-suexec-pristine: Installed: 2.4.41-4ubuntu3.1 Candidate: 2.4.41-4ubuntu3.1 Version table: *** 2.4.41-4ubuntu3.1 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages 100 /var/lib/dpkg/status 2.4.41-4ubuntu3 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/universe amd64 Packages php7.4: Installed: 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 Candidate: 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 Version table: *** 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 500 500 http://ppa.launchpad.net/ondrej/php/ubuntu focal/main amd64 Packages 100 /var/lib/dpkg/status 7.4.3-4ubuntu2.4 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages 7.4.3-4ubuntu1 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/main amd64 Packages php7.4-fpm: Installed: 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 Candidate: 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 Version table: *** 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 500 500 http://ppa.launchpad.net/ondrej/php/ubuntu focal/main amd64 Packages 100 /var/lib/dpkg/status 7.4.3-4ubuntu2.4 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages 7.4.3-4ubuntu1 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/universe amd64 Packages php7.4-cgi: Installed: 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 Candidate: 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 Version table: *** 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 500 500 http://ppa.launchpad.net/ondrej/php/ubuntu focal/main amd64 Packages 100 /var/lib/dpkg/status 7.4.3-4ubuntu2.4 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages 7.4.3-4ubuntu1 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/main amd64 Packages php7.4-cli: Installed: 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 Candidate: 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 Version table: *** 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 500 500 http://ppa.launchpad.net/ondrej/php/ubuntu focal/main amd64 Packages 100 /var/lib/dpkg/status 7.4.3-4ubuntu2.4 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages 7.4.3-4ubuntu1 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/main amd64 Packages php7.4-intl: Installed: 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 Candidate: 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 Version table: *** 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 500 500 http://ppa.launchpad.net/ondrej/php/ubuntu focal/main amd64 Packages 100 /var/lib/dpkg/status 7.4.3-4ubuntu2.4 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages 7.4.3-4ubuntu1 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/universe amd64 Packages php7.4-json: Installed: 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 Candidate: 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 Version table: *** 7.4.15-3+ubuntu20.04.1+deb.sury.org+1 500 500 http://ppa.launchpad.net/ondrej/php/ubuntu focal/main amd64 Packages 100 /var/lib/dpkg/status 7.4.3-4ubuntu2.4 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages 7.4.3-4ubuntu1 500 500 http://nova.clouds.archive.ubuntu.com/ubuntu focal/main amd64 Packages
