FPM instance seems to already listen on socket - ispconfig mirror setup - multiple php versions

Discussion in 'Installation/Configuration' started by bern lehn, Jul 24, 2018.

  1. bern lehn

    bern lehn New Member

    hi,
    I think there is a bug in ISPC regarding changing between fpm versions.
    tested on ubuntu 16.04 - nginx - ISPC3.11 and 3.12 (found an old post on stackoverflow that seemed to have the same issue but was not well detailed - so it could be that this is an issue for a long time. It is not the most common situation tbh...)

    effect: fpm unit will not start with error:
    systemd[1]: Starting The PHP 7.1 FastCGI Process Manager...
    php-fpm7.1[451]: [23-Jul-2018 08:05:04] ERROR: An another FPM instance seems to already listen on /var/lib/php7.0-fpm/web140.sock
    php-fpm7.1[451]: [23-Jul-2018 08:05:04] ERROR: FPM initialization failed
    systemd[1]: php7.1-fpm.service: Main process exited, code=exited, status=78/n/a
    systemd[1]: Failed to start The PHP 7.1 FastCGI Process Manager.

    steps to reproduce:
    have a mirror setup of ispc running multiple php versions
    create a webhost and set a php version (tested only with php-fpm)
    change php version to a different value than before

    the problem will only occur on the mirror server not on the master
    (which can have a devious effect in a hot spare situation if you dont notice..)

    the change of php version causes a new pool file to be created in /etc/php/v.m/fpm/pool.d/webx.conf
    there we define which socket file should be used

    the bug seems to be that after php version is changed to a new value the old pool file is only removed on the master but not on the mirror, so the new file which is created with the same socket file will cause the problem.
    the first fpm process wins the socket file the other one can not have it.
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Try to compare the debug output on both nodes. The code that handles the fpm pool creation on master and slave is the same, so it should do the exact same thing on both nodes.
     
  3. bern lehn

    bern lehn New Member

    hi,
    I compared the logs.
    master is logging: Removed PHP-FPM config file: /etc/php/7.1/fpm/pool.d/web36.conf
    while the mirror log simply does not show this line at all.

    I was able to narrow it down to the non default versions. If i change to 7.0 (default in 16.04) the log will show that
    Removed PHP-FPM config file: /etc/php/7.0/fpm/pool.d/web36.conf
    is beeing removed.
    If the change is from a non default version the line will not show.

    Could it be the case that Ispc relies on an OS function that is not aware of the addtional versions?
    But why is it only happening on the mirror ^^

    At least this is a reliable pattern now. I hope you can investigate on these grounds...

    The fix is easy, just remove the file from pool.d manually - this will not be something a lot of people will ever notice.

    btw we use Ondřej Surý´s ppa

    the whole operation from both sides:

    master:
    24.07.2018-14:58 - DEBUG - Writing the vhost file: /etc/nginx/sites-available/dummytest2..vhost
    24.07.2018-14:58 - DEBUG - Writing the PHP-FPM config file: /etc/php/7.0/fpm/pool.d/web36.conf
    24.07.2018-14:58 - DEBUG - Removed PHP-FPM config file: /etc/php/7.1/fpm/pool.d/web36.conf
    24.07.2018-14:58 - DEBUG - Calling function 'restartPHP_FPM' from module 'web_module'.
    24.07.2018-14:58 - DEBUG - Restarting php-fpm: systemctl reload php7.1-fpm.service
    24.07.2018-14:58 - DEBUG - Calling function 'restartPHP_FPM' from module 'web_module'.
    24.07.2018-14:58 - DEBUG - Restarting php-fpm: systemctl reload php7.0-fpm.service
    24.07.2018-14:58 - DEBUG - nginx status is: running
    24.07.2018-14:58 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
    24.07.2018-14:58 - DEBUG - Checking nginx configuration...
    24.07.2018-14:58 - DEBUG - nginx configuration ok!
    24.07.2018-14:58 - DEBUG - Restarting httpd: systemctl restart nginx.service
    24.07.2018-14:58 - DEBUG - nginx restart return value is: 0
    24.07.2018-14:58 - DEBUG - nginx online status after restart is: running
    24.07.2018-14:58 - DEBUG - Processed datalog_id 1380

    mirror:
    24.07.2018-14:58 - DEBUG - Writing the vhost file: /etc/nginx/sites-available/dummytest2.vhost
    24.07.2018-14:58 - DEBUG - Writing the PHP-FPM config file: /etc/php/7.0/fpm/pool.d/web36.conf
    24.07.2018-14:58 - DEBUG - Calling function 'restartPHP_FPM' from module 'web_module'.
    24.07.2018-14:58 - DEBUG - Restarting php-fpm: systemctl reload php7.0-fpm.service
    24.07.2018-14:58 - DEBUG - nginx status is: running
    24.07.2018-14:58 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
    24.07.2018-14:58 - DEBUG - Checking nginx configuration...
    24.07.2018-14:58 - DEBUG - nginx configuration ok!
    24.07.2018-14:58 - DEBUG - Restarting httpd: systemctl restart nginx.service
    24.07.2018-14:58 - DEBUG - nginx restart return value is: 0
    24.07.2018-14:58 - DEBUG - nginx online status after restart is: running
    24.07.2018-14:58 - DEBUG - Processed datalog_id 1380
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    It might be that your issue is somehow related to this problem, which we could not reproduce yet. I'll add a link to your thread to the bug report, hopefully, we will get a scenario setup where we can reproduce it on our dev servers now.

    https://git.ispconfig.org/ispconfig/ispconfig3/issues/4816
     
  5. bern lehn

    bern lehn New Member

    well at least that part looks pretty much exactly alike:
    Michael Scholz @mscholz commented · 8 months ago
    Today we had a similar problem again.
    After switching a site from php 5.6.30 fpm to php 7.0.xx fpm the socket wasn't readable anymore.
    The first thing we realized was that there was still the webXXXX.conf file in both php version (5 and 7).
    After saving again the site in ispconfig the php5 conf gets removed but the socket wasn't readable cause it was still in use.
    (from the comments of the issue you linked)
     
  6. bern lehn

    bern lehn New Member

    oh and both the systems I tested on where mirrors added to previously existing single masters...

    so one scenario could be that the database entry for the default version is created by installer - but additional versions are not copied from the master DB to mirror? would explain why it is working with changes from 7.0
     
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    If that would be the case, then you should be able to see it in the database on the mirror. are all PHP versions present in the server_php table on both servers?
     
  8. bern lehn

    bern lehn New Member

    the table is indeed completly empty on the mirror while on master I see all my versions fine...
     
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    That's strange, the table is set to be mirrored in the code. Was your server installed as mirror system right from the beginning, or did you add a second mirror server at a later time?
     
  10. bern lehn

    bern lehn New Member

    both the systems I tested on where mirrors added to previously existing single masters...
     
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    Then your problem is an installation issue. When mirroring is turned on between two servers, both servers must contain the same data in their databases (normally both servers are empty at that time, so the 'same data' here is that both are empty and have the exact same config state). If data would exist in one database but not the other, then this data will not get mirrored as the mirroring system mirrors only changes and does not copy over all data from one database to the other on each change. If you create a mirror of an existing single server at a later stage, then you basically have to copy everything over from the first database except of the server and server_ip table before starting to use it.
     
  12. ztk.me

    ztk.me ISPConfig Developer ISPConfig Developer

    do all your php version have a www.conf or something so they start even if no ISPConfig web assigned? Because that's where ISPConfig has an issue with and not starting php-fpm like that
     
    Taleman likes this.
  13. till

    till Super Moderator Staff Member ISPConfig Developer

    This explains why I never had that issue. I use the additional PHP versions compiled from my various PHP guides and they all have a www.conf by default.
     
  14. ztk.me

    ztk.me ISPConfig Developer ISPConfig Developer

    btw I'd prefer having no service run if not needed some day maybe :)
     
  15. bern lehn

    bern lehn New Member

    < If you create a mirror of an existing single server at a later stage, then you basically have to copy everything over from the first database except of the server and server_ip table before starting to use it.>

    ok, that is valuable information. Not sure if I will have this situation ever again since atm I am building a new cluster from scratch to replace the old servers we have in use.
    But if one is creating a mirror to an existing server (we did that to have backup load moved away from the previously single master) that is definitely something to keep in mind.
    Is there any documentation for this?

    Do you think it is a bad idea to restore a dump (excluding server and server_ip table) from the master DB to mirror now?
    I am not sure if I would even bother with it, since what I want working atm. But just in case...

    also again I have to mention how amazed I am by the reaction time and effort you guys provide!
     
  16. bern lehn

    bern lehn New Member

    FYI - just inserting the lines into dbispconfig.server_php like they exist on the master server did not fix the issue for me.
    I was wondering why there was not even an entry for 7.0 but still the pool file corresponding to 7.0 was removed from the mirror contrary to the others - my whole theory is in danger again ;)
     
  17. till

    till Super Moderator Staff Member ISPConfig Developer

    No, I won't do that as it always imposes the risk that something went wrong. Just copy over the content of the php version table.

    The most likely reason for the whole issue is what @ztk.me pointed out. When the custom php versions don't have a default www.conf pool, then the reload might fail when there are no other pools which in the end will make php-fpm to fail for that site.
     
  18. bern lehn

    bern lehn New Member

    ok, as said adding those entries into server_php did not change the issue for me.
    also my problem is a bit different and definitely related to the existence of more than one pool config pointing to the same socket file.
     
  19. ztk.me

    ztk.me ISPConfig Developer ISPConfig Developer

Share This Page