After upgrade Debian 10 to 11, apache2 fails to start

I upgraded Debian 10 to Debian 11, following instructions from https://www.howtoforge.com/update-the-ispconfig-perfect-server-from-debian-10-to-debian-11/. Also from Debian 11 Release Notes.
After full upgrade, Apach2 can not be started.
I have already used update-alternatives to set default php to 7.4. I have rebooted. I have a2disconf php7.3-fpm and enabled php7.4-fpm, no luck.

Code:

root@mail:/etc/mysql/mariadb.conf.d# systemctl status apache2.service | more
● apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2023-11-27 17:25:59 EET; 3min 12s ago
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 3364 ExecStart=/usr/sbin/apachectl start (code=exited, status=139)
        CPU: 35ms

marras 27 17:25:59 mail systemd[1]: Starting The Apache HTTP Server...
marras 27 17:25:59 mail apachectl[3367]: [Mon Nov 27 17:25:59.587817 2023] [so:warn] [pid 3367] AH01574: module php7_module is already loaded, skipping
marras 27 17:25:59 mail apachectl[3367]: [Mon Nov 27 17:25:59.594381 2023] [so:warn] [pid 3367] AH01574: module php_module is already loaded, skipping
marras 27 17:25:59 mail apachectl[3367]: [Mon Nov 27 17:25:59.594793 2023] [so:warn] [pid 3367] AH01574: module php_module is already loaded, skipping
marras 27 17:25:59 mail apachectl[3364]: /usr/sbin/apachectl: line 150:  3367 Segmentation fault      $HTTPD ${APACHE_ARGUMENTS} -k "$ARGV"
marras 27 17:25:59 mail apachectl[3364]: Action 'start' failed.
marras 27 17:25:59 mail apachectl[3364]: The Apache error log may have more information.
marras 27 17:25:59 mail systemd[1]: apache2.service: Control process exited, code=exited, status=139/n/a
marras 27 17:25:59 mail systemd[1]: apache2.service: Failed with result 'exit-code'.
marras 27 17:25:59 mail systemd[1]: Failed to start The Apache HTTP Server.

Apache error log:

Code:

[Mon Nov 27 16:16:28.476563 2023] [mpm_event:notice] [pid 85086:tid 140652045153408] AH00493: SIGUSR1 received.  Doing graceful restart


[ 2023-11-27 16:16:28.6117 121822/7f9215553700 age/Cor/CoreMain.cpp:532 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ 2023-11-27 16:16:28.6121 121822/7f9215e42980 age/Cor/CoreMain.cpp:901 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ 2023-11-27 16:16:28.6131 121827/7f6125a6c700 age/Ust/UstRouterMain.cpp:422 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ 2023-11-27 16:16:28.6138 121827/7f6125b6b980 age/Ust/UstRouterMain.cpp:492 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ 2023-11-27 16:16:28.6139 121822/7f9214cd1700 Ser/Server.h:817 ]: [ApiServer] Freed 0 spare client objects
[ 2023-11-27 16:16:28.6143 121822/7f9214cd1700 Ser/Server.h:464 ]: [ApiServer] Shutdown finished
[ 2023-11-27 16:16:28.6154 121827/7f6125a6c700 Ser/Server.h:464 ]: [UstRouter] Shutdown finished
[ 2023-11-27 16:16:28.6154 121827/7f61251ea700 Ser/Server.h:817 ]: [UstRouterApiServer] Freed 0 spare client objects
[ 2023-11-27 16:16:28.6155 121827/7f61251ea700 Ser/Server.h:464 ]: [UstRouterApiServer] Shutdown finished
[ 2023-11-27 16:16:28.6156 121822/7f9215553700 Ser/Server.h:817 ]: [ServerThr.1] Freed 128 spare client objects
[ 2023-11-27 16:16:28.6161 121822/7f9215553700 Ser/Server.h:464 ]: [ServerThr.1] Shutdown finished
[ 2023-11-27 16:16:28.6186 121827/7f6125b6b980 age/Ust/UstRouterMain.cpp:523 ]: Passenger UstRouter shutdown finished
[ 2023-11-27 16:16:28.6494 121822/7f9215e42980 age/Cor/CoreMain.cpp:967 ]: Passenger core shutdown finished
apache2: Syntax error on line 146 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/autoindex.load: Cannot load /usr/lib/apache2/modules/mod_autoindex.so into server: /usr/lib/apache2/modules/mod_autoindex.so: undefined symbol: ap_normalize_path

I tried disabling autonidex module, but it warns me sternly not to disable essential module.

Code:

oot@mail:/etc/mysql/mariadb.conf.d# apachectl configtest
[Mon Nov 27 17:34:10.241715 2023] [so:warn] [pid 4038] AH01574: module php7_module is already loaded, skipping
[Mon Nov 27 17:34:10.248694 2023] [so:warn] [pid 4038] AH01574: module php_module is already loaded, skipping
[Mon Nov 27 17:34:10.248902 2023] [so:warn] [pid 4038] AH01574: module php_module is already loaded, skipping
/usr/sbin/apachectl: line 150:  4038 Segmentation fault      $HTTPD ${APACHE_ARGUMENTS} -t
Action 'configtest' failed.
The Apache error log may have more information.

Two hours of Internet Search engines with the error messages has given no progress.

 

apache2ctl configtest segfaults at line 150, shown below.

Code:

    146
    147 [ ! -d ${APACHE_RUN_DIR:-/var/run/apache2} ] && mkdir -p ${APACHE_RUN_DIR:-/var/run/apache2}
    148 [ ! -d ${APACHE_LOCK_DIR:-/var/lock/apache2} ] && mkdir_chown ${APACHE_RUN_USER:-www-data} ${APACHE_LOCK_DIR:-/var/lock/apache2}
    149
    150 case "$ARGV" in
    151 start)
    152     # ssl_scache shouldn't be here if we're just starting up.
    153     # (this is bad if there are several apache2 instances running)
    154     rm -f ${APACHE_RUN_DIR:-/var/run/apache2}/*ssl_scache*

It looks to me it might be the calling actor that fails to give proper arguments, if $ARGV is faulty. Or it is the previous line that segfaults but it is reported at line 150.

 

Good idea! But I added before line 150 the echo statement.

Code:

root@mail:~# apache2ctl help
Usage: /usr/sbin/apache2ctl start|stop|restart|graceful|graceful-stop|configtest|status|fullstatus|help
       /usr/sbin/apache2ctl <apache2 args>
       /usr/sbin/apache2ctl -h            (for help on <apache2 args>)
root@mail:~# apache2ctl configtest
configtest
[Tue Nov 28 09:36:27.002402 2023] [so:warn] [pid 68041] AH01574: module php7_module is already loaded, skipping
[Tue Nov 28 09:36:27.007540 2023] [so:warn] [pid 68041] AH01574: module php_module is already loaded, skipping
[Tue Nov 28 09:36:27.007668 2023] [so:warn] [pid 68041] AH01574: module php_module is already loaded, skipping
/usr/sbin/apache2ctl: line 151: 68041 Segmentation fault      $HTTPD ${APACHE_ARGUMENTS} -t
Action 'configtest' failed.
The Apache error log may have more information.

It echoes the parameter, configtest in this case. stop and graceful-stop make same segfault. I think I need to see what

Code:

$HTTPD ${APACHE_ARGUMENTS} -k "$ARGV"

is.

 

dpkg can verify installed files, it found the apache2ctl that I modified with echo statements, and two config files.

Code:

root@mail:~# dpkg --verify apache2
??5?????? c /etc/apache2/apache2.conf
??5?????? c /etc/apache2/ports.conf
??5??????   /usr/sbin/apache2ctl
root@mail:~#

Reinstall did not make things better.

Code:

apt --reinstall install apache2

 

You could strace the apache startup at this point and see if whatever happends before the segfault provides any hints. I've had this issue before on Debian after an upgrade months ago but i can't remember what it was, but i will see if i can find the ticket.

Code:

strace apache2

//EDIT: Oh and if you got time on your hand it's always good advice to disable module after module and check if any of the modules causes the issue

 

Tried strace, it showed error

Code:

H00111: Config variable ${APACHE_RUN_DIR} is not defined
) = 116
stat("/etc/apache2/${APACHE_RUN_DIR}", 0x7fff95b79990) = -1 ENOENT (Tiedostoa tai hakemistoa ei ole)

It is defined in /etc/apache2/envvars, trying source that file and export the variable still makes segfault when strace apache2.
But this error message I can find in the Internet, so maybe some solution is eventually found.
Althoug I think it should stat /var/run/apache and not subdir of /etc/apache2.

 

What are these php modules?

Code:

[Tue Nov 28 12:33:39.754987 2023] [so:warn] [pid 84174] AH01574: module php7_module is already loaded, skipping
[Tue Nov 28 12:33:39.755234 2023] [so:warn] [pid 84174] AH01574: module php7_module is already loaded, skipping
[Tue Nov 28 12:33:39.757283 2023] [so:warn] [pid 84174] AH01574: module php_module is already loaded, skipping
[Tue Nov 28 12:33:39.757404 2023] [so:warn] [pid 84174] AH01574: module php_module is already loaded, skipping
They are not in mods enabled. Is something loading them twice?

 

Now Apache2 starts after a2dismod all other php exept php7.4, the default in Debian 11.
I find it strange apache segfault because of PHP module, apache is not written in PHP.
I'll try one by one those php modules, may be only one of them breaks.
Edit: it is php7.2. that breaks apache2. I enabled back other PHP modules, they do work. I'll just leave php7.2 off for now.

 

Don't get things mixed up. The segfault appears while loading the php extension itself. Neither apache2 nor the extension itself is "written in php".

Regarding the "is already loaded, skipping" this usually happends when multiple php.ini files are loaded which call the same extension, but should not be an issue.

Check step by step which module causes the exception and then purge all the packages for this version as long as it is not the system default and reinstall it clean.

 

I did dpkg --verify to all php7.2 packages, nothing found. So I'll leave this here unless something enlightening comes up to see what is wrong.

 

No website uses Mod-PHP.