Resync hangs

Discussion in 'Installation/Configuration' started by rsteuer, Oct 16, 2018.

  1. rsteuer

    rsteuer Member

    We have a multi-server install with separate DNS, Web, DB and Panel servers based on ISPConfig Manual. Recently, the System State was indicating incorrect versions of ISPConig installed on the various servers. We ran a resync of all services and the information is still incorrect. In addition, there are 134 sync operations that are now hung and will not complete.

    Also, one server needs to be completely deleted from the system manually, as the physical server is now out of service and cannot be brought back online. A new server was added to the cluster and it and is functioning.

    1. How best to start troubleshooting the sync issues?
    2. How do we remove the server that is no longer in service?

    Thank you for your help.
     
  2. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Maybe the syncs are to the server that is no longer up?

    There was in the previous few days thread about removing jobs from the queue.
     
  3. rsteuer

    rsteuer Member

    Thanks for the reply. The sync items all appear to be for the web server. The server that is no longer available was a DNS server, so I don't believe that they are related. I'm reluctant to remove the jobs from the queue, as there may be updates that need to be applied. These are the pending updates:
    upload_2018-10-16_8-13-0.png
     
  4. rsteuer

    rsteuer Member

    Anyone?
     
  5. Taleman

    Taleman Well-Known Member HowtoForge Supporter

  6. rsteuer

    rsteuer Member

    Thanks Taleman, the link was helpful.
    For the sync issue, I think I've finally figured out what the issue is, but not sure how to correct it. during the last ISPConfig upgrade, must have enabled the control panel on the web server. The ISPConfig control panel I now functional on the web server and the admin server. I'm assuming there's a conflict that's preventing the sync operations. Is there a way to verify that this is the issue? If so, how do I correct the problem?
    Thanks I advance for your help.
     
  7. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    Check /usr/local/ispconfig/server/lib/config.inc.php (and config.inc.local.php if you have one), the dbmaster_* settings should be different than the db_* settings on a slave server. I imagine the server_id (set in config.inc.php) wouldn't change by enabling the control panel; I do not know offhand if the "dbispconfig" database on your slave server would now be out of sync; perhaps it would be ok if you didn't use the ispconfig interface there, but definitely back up that database first thing, then see how it goes (eg. cron will start checking the master database for tasks again - see if they go through ok and changes are applied).

    You'd have to remove the ispconfig vhost manually I imagine (/etc/apache2/sites-available/ispconfig.*).

    In a worst case, that the database or things are out of sync, hopefully you answered 'yes' to performing a backup as the first step of the ispconfig update; if so, your database and config files at that time will be found under /var/backup/ispconfig_....something...
     
  8. rsteuer

    rsteuer Member

    Thanks Jesse. What I'm seeing is that, on the slave (web) server, there is a section in the config.inc.php file, as follows:

    //** Database settings for the master DB. This setting is only used in multiser$
    $conf['dbmaster_type'] = 'mysql';
    $conf['dbmaster_host'] = '';
    $conf['dbmaster_port'] = '';
    $conf['dbmaster_database'] = 'dbispconfig';
    $conf['dbmaster_user'] = '';
    $conf['dbmaster_password'] = 'hidden_password';
    $conf['dbmaster_new_link'] = false;

    Seeing the host, port, etc., with blank entries suggested to me that the info was overwritten because the web server was no longer part of a multi-server config. At least that was my assumption.

    I looked at the config file on another member server in the cluster and the info was there so the install of the control panel must override the settings because it becomes its own master. Unfortunately, the local backups performed during the upgrades are too new to be able to grab the settings. I'll restore a machine from our remote backup and hope I'll be able to gather the info needed. I'm assuming it will resolve the issue (if I have the info available).

    Pointing me to the config file was extremely helpful. Thank you.
     
  9. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    You can re-run ISPConfig's update.php, providing the correct info for your master server, and answer 'yes' to reconfigure the master database, and I think it should fix those dbmaster_ entries for you. All it really is is username/password/hostname, which you could just manually (re)set as well.
     
  10. rsteuer

    rsteuer Member

    Already tried that but it indicates there are no updates available. Tried selecting the git-stable and it gets to the point where it tries to connect to the master's db and hangs. Keep running into road blocks.
     
  11. rsteuer

    rsteuer Member

    The saga continues! I cannot connect to the master's db from the webserver through mysql, so there's something else that's happened.
     
  12. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    if it's actually "can't connect" at a tcp level, ensure mysql is listening on the network, and that firewall rules allow your webserver to that port (probably 3306). if it's mysql refusing to complete a connection, check standard mysql tables (mysql.user and mysql.db).
     
  13. rsteuer

    rsteuer Member

    Thanks Jesse, it was a mysql issue that has been resolved. But the sync from the panel to the web server is still an issue. I've not been able to resolve the problem as mentioned above (Thurs, 3:29PM). In the master database, I've found at least 10 different entries for the web server (no idea why that many, as we've only ever had two) and can find entries for the dbmaster_user but substituting them in the slave server does nothing to resolve the issue. Still stuck in limbo.
     
  14. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    Check (or change) what you're using for server_id in /usr/local/ispconfig/server/lib/config.inc.php on the slave, then check and make sure that id is consistently used in db tables in both the master server's dbispconfig database as well as the dbispconfig database on the slave server itself. Make sure you don't just check the dbispconfig.server table, but also web_domain, etc.

    as far as the sync goes, that involves the sys_datalog table (so a bad server_id there will have tasks queued up for a server which will never check/complete them).
     
    Last edited: Oct 30, 2018
  15. rsteuer

    rsteuer Member

    Thank you, again.

    I must be missing something. I've checked the ispcsrv* setting on both the slave server (config.inc.php), the mysql.user table and they match. I seem to recall finding the table in ispdbconfig db that included the ispcsrv* values but cannot locate it again. Do you know which table includes it? I'm not sure if its entry matches everything else. It's the only thing I can think of.
     
  16. till

    till Super Moderator Staff Member ISPConfig Developer

    The ispcsrv* users are not stored in dbispconfig. The exist just in the mysql user database on the master and in the server config.inc.php file of the slave server.
     
  17. rsteuer

    rsteuer Member

    Thanks Till. I'm at a loss. It looks like everything matches but the control panel indicates the incorrect version of ISPConfig and the sync is still hung. Any other suggestions?
     
  18. rsteuer

    rsteuer Member

    is there a way to determine which server (ispcsrv*) the sync is destined for? As mentioned previously there are numerous ispcsrv* listed for the web server that cannot be updated. I still have no idea why there are so many, as we've only ever had two machines. I'm wondering if the ispcrv that is listed is incorrect.
     
  19. rsteuer

    rsteuer Member

    Or, might there be a log file that ight give info relating to the error?
     
  20. rsteuer

    rsteuer Member

    I ran the server.sh script from the webserver and got a access denied error. I then tried logging in with the ispcsrv* user and realized the password was auto-generated, so I have no idea what it is. Is there a way to reset the password in the db on the control panel server and on the web server?
     

Share This Page