Server down

Discussion in 'Installation/Configuration' started by Marcio Urakawa, Dec 11, 2023.

Tags:
  1. Marcio Urakawa

    Marcio Urakawa Member HowtoForge Supporter

    Sorry for the published location.

    Today the server crashed and I'm trying to get it up but I noticed that when I enable or disable Apache's mod_ssl it comes back. I need SSL to work. I use the most up-to-date Ispconfig as well as Debian 12.

    Code:
    [Mon Dec 11 11:19:19.201905 2023] [ssl:info] [pid 484710:tid 140686001518464] AH01883: Init: Initialized OpenSSL library
    [Mon Dec 11 11:19:19.204574 2023] [ssl:trace2] [pid 484710:tid 140686001518464] ssl_engine_rand.c(125): Init: Seeding PRNG with 656 bytes of entropy
    [Mon Dec 11 11:19:19.204601 2023] [ssl:debug] [pid 484710:tid 140686001518464] ssl_engine_init.c(353): AH01886: OpenSSL has FIPS mode disabled
    [Mon Dec 11 11:19:19.204674 2023] [ssl:info] [pid 484710:tid 140686001518464] AH01887: Init: Initializing (virtual) servers for SSL
    
     
  2. pyte

    pyte Well-Known Member HowtoForge Supporter

    What exactly does not work?

    Is it possible that you've changed a vhost and have some configuration that is not working in there?

    Try running "apachectl configtest" does it provide any hints?
     
    Marcio Urakawa likes this.
  3. Marcio Urakawa

    Marcio Urakawa Member HowtoForge Supporter

    Syntax OK
    When I disable Apache's mod_ssl, it is able to start.

    These previous days I had restarted the machine several times and had not encountered this problem. It was only this morning that the problem occurred.

    In the log it doesn't show anything significant.
     
  4. pyte

    pyte Well-Known Member HowtoForge Supporter

    Ok. Does the service fail to start? if thats the case check the journal of the service to see what causes the issue:
    Code:
    journalctl -u apache2
     
  5. Marcio Urakawa

    Marcio Urakawa Member HowtoForge Supporter

    Code:
    nov 30 21:54:20 ispconfig systemd[1]: Stopping apache2.service - The Apache HTTP Server...
    nov 30 21:54:38 ispconfig systemd[1]: apache2.service: Killing process 2492995 (PassengerAgent) with signal SIGKILL.
    nov 30 21:54:38 ispconfig systemd[1]: apache2.service: Killing process 1102863 (php-cgi) with signal SIGKILL.
    nov 30 21:54:38 ispconfig systemd[1]: apache2.service: Killing process 1109779 (PassengerAgent) with signal SIGKILL.
    nov 30 21:54:38 ispconfig systemd[1]: apache2.service: Killing process 2493004 (PassengerAgent) with signal SIGKILL.
    nov 30 21:54:38 ispconfig systemd[1]: apache2.service: Deactivated successfully.
    nov 30 21:54:38 ispconfig systemd[1]: Stopped apache2.service - The Apache HTTP Server.
    nov 30 21:54:38 ispconfig systemd[1]: apache2.service: Consumed 4d 8h 35min 47.674s CPU time.
    -- Boot 09fe5ee17dc94a1590cc12a6f815c0a1 --
    nov 30 21:56:51 ispconfig systemd[1]: Starting apache2.service - The Apache HTTP Server...
    nov 30 21:57:13 ispconfig systemd[1]: Started apache2.service - The Apache HTTP Server.
    dez 01 00:01:24 ispconfig systemd[1]: Reloading apache2.service - The Apache HTTP Server...
    dez 01 00:01:33 ispconfig systemd[1]: Reloaded apache2.service - The Apache HTTP Server.
    dez 01 00:52:42 ispconfig systemd[1]: Reloading apache2.service - The Apache HTTP Server...
    dez 01 00:52:51 ispconfig systemd[1]: Reloaded apache2.service - The Apache HTTP Server.
    dez 01 00:53:11 ispconfig systemd[1]: Reloading apache2.service - The Apache HTTP Server...
    dez 01 00:53:20 ispconfig systemd[1]: Reloaded apache2.service - The Apache HTTP Server.
    dez 01 09:33:03 ispconfig systemd[1]: Stopping apache2.service - The Apache HTTP Server...
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 3194 (apache2) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 478061 (vlogger (access) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 478070 (apache2) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 478071 (apache2) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 478073 (PassengerAgent) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 478081 (PassengerAgent) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 779711 (apache2) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 797623 (apache2) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 1311945 (apache2) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 1375231 (php-cgi) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 1375232 (php-cgi) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 1375233 (php-cgi) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 1375275 (php-cgi) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 1375276 (php-cgi) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 1375277 (php-cgi) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 478080 (PassengerAgent) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 478089 (PassengerAgent) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 478090 (PassengerAgent) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 478097 (PassengerAgent) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 478102 (PassengerAgent) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 478103 (PassengerAgent) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 478105 (PassengerAgent) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 478108 (PassengerAgent) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 478109 (PassengerAgent) with signal SIGKILL.
    dez 01 09:33:08 ispconfig systemd[1]: apache2.service: Killing process 478113 (PassengerAgent) with signal SIGKILL.
     
  6. pyte

    pyte Well-Known Member HowtoForge Supporter

    Please at least read the output for yourself before replying. This is the first page of that output scroll to the bottom to go to today(check the date and time) and see what is going on.
     
    Marcio Urakawa likes this.
  7. Marcio Urakawa

    Marcio Urakawa Member HowtoForge Supporter

    Code:
    dez 11 12:06:01 ispconfig apachectl[562710]: [Mon Dec 11 12:06:01.873970 2023] [core:trace3] [pid 562710:tid 140602761009024] core.c(3471): Setting LogLevel>
    dez 11 12:06:01 ispconfig apachectl[562710]: [Mon Dec 11 12:06:01.874019 2023] [core:trace3] [pid 562710:tid 140602761009024] core.c(3498): Setting LogLevel>
    dez 11 12:06:06 ispconfig apachectl[562572]: Segmentation fault
    dez 11 12:06:06 ispconfig apachectl[562572]: Action 'start' failed.
    dez 11 12:06:06 ispconfig apachectl[562572]: The Apache error log may have more information.
    dez 11 12:06:06 ispconfig systemd[1]: apache2.service: Control process exited, code=exited, status=139/n/a
    dez 11 12:06:06 ispconfig systemd[1]: apache2.service: Failed with result 'exit-code'.
    dez 11 12:06:06 ispconfig systemd[1]: Failed to start apache2.service - The Apache HTTP Server.
    dez 11 12:06:06 ispconfig systemd[1]: apache2.service: Consumed 4.254s CPU time.
    dez 11 12:07:01 ispconfig systemd[1]: Starting apache2.service - The Apache HTTP Server...
    dez 11 12:07:01 ispconfig apachectl[564332]: [Mon Dec 11 12:07:01.217926 2023] [core:trace3] [pid 564332:tid 139938400458624] core.c(3471): Setting LogLevel>
    dez 11 12:07:01 ispconfig apachectl[564332]: [Mon Dec 11 12:07:01.217985 2023] [core:trace3] [pid 564332:tid 139938400458624] core.c(3498): Setting LogLevel>
    dez 11 12:07:05 ispconfig apachectl[564191]: Segmentation fault
    dez 11 12:07:05 ispconfig apachectl[564191]: Action 'start' failed.
    dez 11 12:07:05 ispconfig apachectl[564191]: The Apache error log may have more information.
    dez 11 12:07:05 ispconfig systemd[1]: apache2.service: Control process exited, code=exited, status=139/n/a
    dez 11 12:07:05 ispconfig systemd[1]: apache2.service: Failed with result 'exit-code'.
    dez 11 12:07:05 ispconfig systemd[1]: Failed to start apache2.service - The Apache HTTP Server.
    dez 11 12:07:05 ispconfig systemd[1]: apache2.service: Consumed 4.092s CPU time.
    
     
  8. pyte

    pyte Well-Known Member HowtoForge Supporter

  9. Marcio Urakawa

    Marcio Urakawa Member HowtoForge Supporter

    dmesg:

    Code:
    [ 2719.402835] apache2[79916]: segfault at 138 ip 00007f35062af890 sp 00007ffe47f12c18 error 4 in libssl.so.3[7f3506295000+5e000] likely on CPU 1 (core 0, socket 2)
    [ 2779.541430] apache2[81698]: segfault at 138 ip 00007f6d8d98f890 sp 00007fff72bc9328 error 4 in libssl.so.3[7f6d8d975000+5e000] likely on CPU 2 (core 0, socket 4)
    [ 2839.721584] apache2[83442]: segfault at 138 ip 00007f10fe672890 sp 00007fff23fa6678 error 4 in libssl.so.3[7f10fe658000+5e000] likely on CPU 2 (core 0, socket 4)
    [ 2900.065267] apache2[85862]: segfault at 138 ip 00007efcff14e890 sp 00007ffd5b3e86c8 error 4 in libssl.so.3[7efcff134000+5e000] likely on CPU 7 (core 0, socket 14)
    [ 2960.139936] apache2[87682]: segfault at 138 ip 00007fd0d9e14890 sp 00007ffc6f56c548 error 4 in libssl.so.3[7fd0d9dfa000+5e000] likely on CPU 3 (core 0, socket 6)
    [ 3020.217944] apache2[89331]: segfault at 138 ip 00007f1cc458f890 sp 00007ffd7843bfe8 error 4 in libssl.so.3[7f1cc4575000+5e000] likely on CPU 6 (core 0, socket 12)
    
     
  10. till

    till Super Moderator Staff Member ISPConfig Developer

    Maybe one ssl cert or key on your system is damaged in a way that it crashes mod_ssl. What you can try is this: move all symlinks in /etc/apache2/sites-enabled folder to a different folder, e.g. create a folder /root/apache-backup and move them this location. Then restart apache2, does it start now? If yes, move one symlink after aonother from backup folder back to /etc/apache2/sites-enabled and restart apache2 in between to see if it fails. If it starts to fail, then you know which site is causing this and most likely the SSL cert from that site is damaged.
     
    Marcio Urakawa likes this.
  11. Marcio Urakawa

    Marcio Urakawa Member HowtoForge Supporter


    It worked!

    Thank you very much! I don't even know how to thank you.
     
  12. Marcio Urakawa

    Marcio Urakawa Member HowtoForge Supporter

    Do you suggest any way to debug what this certificate or key would be?

    I was looking at the log /var/log/ispconfig/acme.log and saw that moments before stopping the server in the early hours of the morning. There is a Vhost that is no longer hosted on that machine but DNS points to it.

    Could this cause this whole problem?
     
  13. Marcio Urakawa

    Marcio Urakawa Member HowtoForge Supporter

    Well, after moving the sites-enabled vhosts, I reactivated them one by one until I discovered which site was causing the problem.
    The annoying thing is that even with the apachectl configtest command, the vhost that caused the problem on the server did not indicate the problem.

    After finding out what the website was. I noticed his certificate was broken. Even deleting the site using ispconfig and creating it again was using the broken certificate.

    The solution was to manually delete the /root/.acme directory the certificate that belonged to that problem website.
     
  14. till

    till Super Moderator Staff Member ISPConfig Developer

    That's right, the Apache config test is only able to detect invalid (wrongly written) config files, but it's not able to detect invalid or missing SSL certs.
     

Share This Page