Mutiserver cannot replicate and jobqueue does not finish

Discussion in 'Server Operation' started by [email protected], Jun 13, 2023.

  1. Hi
    My setup is 1 Master that is DNS, Webserver, FTP. I have all mail on Microsoft or Google and 1 slave dns in same location diffrent ip and 2 dns servers in differnt data ceter that has an VPN tunnel to master. All servers fully updated and runnin latest Ispconfig
    After I upgraded master server from Debian10 to 11 this started and I have stuck in job que;
    The following changes are not yet populated to all servers:
    • Delete server: 1
    • Update server settings: 13
    Once I disable server.sh cron I get following errors in log in debug mode:
    13.06.2023-09:45 - DEBUG [plugins.inc:155] - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    13.06.2023-09:45 - DEBUG [server:217] - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    finished server.php.
    It looks like all is running as expected but not completing server changes.
    Job que is stuc with the following:
    Jobqueue


    Date Server Action DB Table
    15


    2023-06-10 13:48 keiko.martolvan.is Update server
    2023-06-10 13:48 keiko.martolvan.is Update server
    2023-05-30 15:04 flipper.martolvan.is Update server
    2023-05-30 15:04 flipper.martolvan.is Update server
    2023-05-30 15:03 flipper.martolvan.is Update server
    2023-05-30 15:03 flipper.martolvan.is Update server
    2023-05-23 19:44 flipper.martolvan.is Update server
    2023-05-23 19:44 flipper.martolvan.is Update server
    2023-05-23 19:44 flipper.martolvan.is Update server
    2023-05-23 19:44 flipper.martolvan.is Update server
    2023-05-19 14:07 flipper.martolvan.is Update server
    2022-03-31 18:23 flipper.martolvan.is Update server
    2021-10-15 11:05 flipper.martolvan.is Delete server
    2021-08-25 22:50 flipper.martolvan.is Update server

    This seems to inhibit updates to slave DNS servers witch is getting some heat since we host several domains, is there a way to thw out older jobs one at a time an see if it will run the rest ?

    Regards Stefan
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    The debug output means that nothing needs to be processed, so there is no error and nothing is stuck on this system. Are you sure you ran server.sh on the right server? It needs to be run on the system that does not process the changes, not on the master.
     
  3. michelangelo

    michelangelo Active Member

    Do you have any servers running with Nginx?
    If so, make sure that the nginx_reverseproxy_plugin.inc.php is not enabled in /usr/local/ispconfig/server/plugins-enabled - unless you need this plugin.

    I had it on two of my Nginx servers happening, that the datalog entries were processed but due to the enabled reverse proxy plugin along with the regular nginx plugin, the master server could not be updated by the slaves.
     
  4. Yes that is a big mistake on my part, so I ran server.sh on slaves and I'm getting all servers with an failed data base connection, I can using mysql cli log into databases both local and remote with credetials used in setup (I set up same credetials for both hosname, fhostname and ip address). Next question is how can I find what data base, local or remote, is failing and what credentials are used, and influence that. What would be the best way to go about correcting this issue?
    I just moved master setver to fresh install og Debian 11 and created salves from ground up using auto installer.
     
  5. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    the slave servers log in to the master database using the logins ispcsrv#@*.*.*.*
    where # should match the server_id in the dbispconfig.server table, and *.*.*.* should be the ip or fqdn of that slave server
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    You can try to run an ispconfig update on the slave system using:

    ispconfig_update.sh --force

    and choose to reconfigure permissions on master database during update. if this does not fix the issue, then you can further test it manually by using mysql command. The login details that ispconfig is using can be found in /usr/local/ispconfig/server/lib/config.inc.php. The details are the one for dbmaster (not the ones for db, which is localhost).
     
  7. Hi quite some time, passed an lots of trials and errors, an still slave DNS servers not working, resync and .../server.sh script run error less on all servers still the 3 slaves are not creating dns records. and no comaprable zonefiles being created.
    So what have I done ?
    1. Recrated and migrated master server to an fresh install of Debian 11.
    2. Deleted and recreated all slave DNS servers, using ispconfig autoinstall per directions in muliserver setup.
    3. Run server resync ticked in all All services/All active servers and DNS records, runs all the way without error.
    4. Run manually /usr/local/ispconfig/server/server.sh on all slaves, runs and completed without error.
    5. All servers are fully upgraded and so is ispconfig too
    6. Rebooted all servers
    7. All mysql connectiona seem to work using credetials found in /usr/local/ispconfig/server/lib/config.inc.php.
    Now I getting to the point of creating some alternative way like using rsync on the /etc/bind/ folder synking both pri.* and slave.* from master folder to other 3 servers, this would be dirty way an totally prone to break ispconfig.
    So there must be a better way and most likely an small tick or typo somwhere I how ever am leanig to an bug since all works but dns records are not created.
     
    Last edited: Jun 21, 2023
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    There is no bug, I use DNS mirror myself, and it works perfectly, plus there are thousands of other installations that use DNS mirroring successfully too. So it's just an issue on your setup that we must deal with here. Maybe you manually edited or deleted data in the database or something similar. E.g. manually removing data from sys_datalog table on master destroys replication for all slave servers. Repliactions is really simple, the slave node asks the master for all records in sys_datalog where datalog_id is higher that the value in the updated column of the server, so when you delete data from sys_datalog manually or even empty the whole table, data will not be synced anymore. Or you maybe configured mirroring wrong. "Is mirror of server" must be set to the first DNS server (primary DNS), not the ISPConfig master.
     
  9. till

    till Super Moderator Staff Member ISPConfig Developer

  10. Hi
    I have been usign ISPconfig for an decade with few problems an using slave servers for 2 years, all was working great.
    However now I have the situation that despite all I do /etc/bind/named.config is generated blank.an no zone files in /etc/bind on slave servers. They do however resolve using dig on localhost an one I copied the content from named.conf from master to slaves they started answering elsewere, what is verified is:
    • All DB connections and credetials work
    • sever.sh runs with out errors in Debug mode
    • Firewall is disbled since all are on an firewalled server networks.
    • Cron seems to run flawlessly.
    • slaves are only DNS.
    • ufw and ipv6 are disabled
    one of the slaves is not resoving like the others an only error I can find is that ntp daemon was stuck on ipv6 that fixed but no change, so I'm out of ideas what to check, or likely I just do not have enough depth of knowlege how system is rigged up.
    Next is setting up server manually not using the auto install script and add isp config on top, to rule out the auto install script.
     
  11. It seems that thigs started working after I updated ip settings in ispconfig server config for the slaves this was some how on local host and once I changed the ip to the internal ip address of server it seems to have started working, updates are being replicated and so on.
    Thank you guys for the help.
     
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    This should point to localhost and not the internal IP of the local server, so the original setting was right and there is an issue wth your mysql/mariadb setup or the resolving of localhost heght system. Check your /etc/hosts file, maybe the entry for localhost is missing or messed up there.
     
  13. Hi
    I thik I have found the culpit named is trying to resolve using ip6v even though I have disabled ipv6 on network and my networ is unforunaetly not ipv6 enabled. I 'm how ever havong trouble figuring out graceful way of disabling ipv6 gracefully in named I found out that I needto add option -4 to the start command og named, just not finding where
     
  14. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Perhaps /etc/default/bind9 ?
    Did you already edit /etc/bind/named.conf.options and comment listen-on-v6 { any; };
     

Share This Page