Update 3.1.15p3 to 3.2.7p1

Discussion in 'Installation/Configuration' started by Entangled, Mar 10, 2022.

  1. Entangled

    Entangled Member

  2. till

    till Super Moderator Staff Member ISPConfig Developer

    The page is online again, thanks for the notice.
     
  3. Entangled

    Entangled Member

    Thanks, but, I thought there was another page that explains in detail what to check or verify on the server before attempting to upgrade ISPConfig 3.1.15p3 to 3.2.7.p1.

    ispconfig_update.sh

    I have always used this command to update between versions.

    Are the server requirements the same between 3.1 and 3.2? I don't mean to whine, but, I got some 30+ Debian 9-10 servers in a Multiserver ISPConfig environment, does the update script check or verify that all the requirements are met before attempting to update?

    Obviously, I have not done an update in awhile so I am a little paranoid about performing this "major" version update.

    Thanks again.
     
  4. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    There are risks in running old versions of ISPConfig, no security updates applied for example.
    I have not had problems from ispconfig_update.sh, but I try to keep my setups plain and simple. Note that all servers in ISPConfig multiserver setup must run the same version of ISPConfig, so once you start upgrading you have to upgrade all hosts.
    Check your OS are all suppoted by ISPConfig 3.2: https://www.howtoforge.com/updating-ispconfig-3-1-to-ispconfig-3-2/
     
  5. Entangled

    Entangled Member

    Thanks, Taleman ... yes, I know, I learned this "lesson" years ago: all servers have to run the same version of ISPConfig ... and like you, all our ISPConfig managed servers are plain and simple following the ISPConfig Perfect Multiserver setup guide.
     
  6. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    The biggest concern when updating releases is custom templates which need updated as well. I think support for goaccess was added in there, which you would have to install in order to use. Nothing else comes immediately to mind.
     
    Gwyneth Llewelyn likes this.
  7. Entangled

    Entangled Member

    Thanks Jesse ... no custom templates for us, at least at this time ... Just installed goaccess on the CP server ... again, when 3.2 first came out, I thought I saw a page with a list of "verify before updating" ... was there ever such a page?

    I'll goes though the Perfect server setup guide and make sure all the requirements have been installed.
     
  8. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    That's probably a pretty safe bet to catch things. The biggest concerns I know of were in custom templates, some of the email stuff would break if you didn't update your templates to be compatible with changes. There also seemed to be a fair number of inconsistent sql updates happening at that time, which I don't know why offhand, but those have died down for some time.
     
  9. Entangled

    Entangled Member

  10. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Is that not the link I posted in #4?
     
  11. Entangled

    Entangled Member

    Embarrassed ... yes it is Taleman, some how I overlooked it ... geeeez, I need to slow down and READ ... thanks again.
     
  12. Entangled

    Entangled Member

    Well, this upgrade has been nothing but a nightmare ... and I only have myself to blame ... the ISPConfig update failed for update "stuff" ... I got a bunch of errors on apt update and upgrade ... so I decided to take a backup of /etc/, /usr/local/ispconfig, /var/backup/ and did mysqldump on mysql / dbispconfig.

    Once the backups were done, I reinstalled Debian 10. On the fresh server, I installed ISPConfig using the automated script ... worked great. I tried to restore dbsipconfig using the backup from the 1st attempt (on the screwed up OS), but, it failed for "duplicate" records/keys (or something like that). The db backup sql file created from an update does not have a DROP ... so I used the mysqldump backups instead for both dbispconfig and mysql ...

    After this, I got mySQL Warnings about some tables (or whatever) so I did a mysql_upgrade which solved the Warnings ... since I restored dbispconfig from a 3.1.15p3 mysqldump, I did a ispconfig_update.sh which did a patch on some tables ... All was well, until I attempted to "talk" with the slave servers, the Jobqueue was stuck. So i brought down mysql, restarted with --skip-grant-tables and the Jobqueue started running ... all is well, once again, accept for the mysql running with skip-grant-tables.

    Being it was 3am, I went to sleep ... when I returned some 3-4 hours later, the dbispconfig db was hacked and was being held for ransom of some 0.0008 in Bitcoins (or whatever ... basically $5K+). Needless to say, I just reinstalled the OS and went through the same "routine" as above.

    However, this time, the "routine" did not work ... I have screwed around with this for day ... I am clueless at this time on how to fix it ... and now ISPConfig 3.2.8p1 has been released. I got to get 3.2.7p1 fix first, right?

    The problem: the master is running 3.2.7p1 and all but 1 slave are running 3.2.7p1. Currently, there are 29 tasks in the Jobqueue. Under the Monitor some of the slaves show 3.1.15p3 and some show 3.2.7p1 running ... I can mysql from Slave to Master using the info in /usr/local/ispconfig/server/lib/config.inc.php ... however, I can not mysql from Master to Slave unless I --skip-grant-tables on the Slave.

    I would think once the Master can connect to the Slave, the Jobqueue would have been processed for that Slave ... nope, still have 29 tasks.

    I have Debug turned on ... all I get is on the Slave:

    Fri 25 Mar 2022 04:14:01 PM CDT finished server.php.
    Fri 25 Mar 2022 04:15:01 PM CDT 25.03.2022-16:15 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
    Fri 25 Mar 2022 04:15:01 PM CDT finished server.php.

    And on the Master:

    25.03.2022-16:16 - DEBUG - Calling function 'check_phpini_changes' from plugin 'webserver_plugin' raised by action 'server_plugins_loaded'.
    25.03.2022-16:16 - DEBUG - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.0/fpm/php.ini' - return code: 0
    25.03.2022-16:16 - DEBUG - safe_exec cmd: grep ^opcache.validate_root '/etc/php/7.0/cgi/php.ini' - return code: 0
    25.03.2022-16:16 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock

    Don't know why the I am getting the above only on the Master ... anyway, the Master has 7.0 ... and opcache.validate_root = 1 is set on Master and Slave.

    I have even attempted to get 3.1.15p3 running again, but, that failed too, with the same issue ... even for the lone 3.1.15p3 slave. I have another 3.2.7p1 multi-server environment running at another location without any problems.

    I wish ISPConfig Jobqueue would give more info, like: "hey stupid, I am having problems in this routine: ...". All tasks for the Master run without any problem, it is the tasks for the Slaves that are being queued.

    Lost without a clue ... any advice, suggestions to get my out of this pickle are very much welcomed.

    Appreciate it in advance,

    Craig
     
  13. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    This is a weird but concerning part of your reply. How did this server randomly get hacked? That's weird...

    Go through these steps on each slave server listed under System in the panel: https://www.faqforge.com/linux/debugging-ispconfig-3-server-actions-in-case-of-a-failure/
     
  14. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    No, just use the latest version (3.2.8p1).
    This sounds correct, the master server does not make any mysql connections to the slaves.
    Ensure the default PHP version in correct, I think 7.3. 7.0 can be installed, too, but only as an additional PHP version.
    Each job in the job queue has an id number, and each server in the server table (in the master dB) records the highest job number which that server has executed - if those are out of sync, the slave servers can think they have already performed the outstanding jobs, so they never get executed and changed to completed status. I don't remember the column name in the server table, but hopefully that is enough that you can find it; then just set the number in the server table back to one less that the id of the first unfinished task.
     
    Last edited: Mar 28, 2022
  15. Entangled

    Entangled Member


    Appreciate the link ... I think I have already done that, but, I will look through it again.

    The IP has been the same for years ... I am sure the hack was all automated ... They "saw" that mySQL was wide open, deleted dbispconfig and created readme_first and create a canned ransom note.
     
  16. Entangled

    Entangled Member

    Master has 5.6 - 8.1 installed ... PHP 7.4.28 (cli) (built: Feb 17 2022 16:17:19) ( NTS ) on Debian 11.2.

    Already checked dbispconfig:server:updated = 55053 (all slaves) ... dbispconfig:sys_datalog:datalog_id = 55143 ... I now have 30 tasks ... dbispconfig:sys_datalog:server_id shows what servers these tasks are for.

    I can mysql into the Master from any of these servers using the info in the config.inc.php ... using the same data, I can mysql into local dbispconfig ...

    I rather get .7p1 working before upgrading to .8p1 ... the fix is going to be something really stupid that I have over-looking.

    Appreciate the input,

    Craig
     
  17. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    The server.updated value sounds fine; maybe ensure the slaves can read that table and the server_id is correct. Ie. "grep dbmaster_ /usr/local/ispconfig/server/lib/config.inc.php" on a slave, and grep for 'server_id' in the same file - then login to master with those dbmaster_* credentials (and ensure dbmaster_host is correct) and 'select * from server where server_id = {server_id}' (but use the actual number for "{server_id}").
     

Share This Page