ISPConfig 3.1.15p2 loses configuration of web server, db server, mail server, and firewall server

My old server uses 3.1.13 and does not exhibit this issue. Ubuntu 18.04 nginx mysql My new server exhibits the issue. Ubuntu 18.04 nginx mariadb

On my new server, I used the perfect server guide for Ubuntu 18.04, nginx, and ispconfig 3. Then I took a snapshot of the old server and migrated it over - client directories/sites, ISPConfig files/configurations, and other configurations. I used this site's step as a guideline, but made alterations along the way (mostly what got backed up and what got restored)
www
zvonimir-buzanic.
from.
hr/
migration-of-ispconfig-3-to-new-server

I also migrated from MySQL to Mariadb during the move, by not making backups of the Mysql schema and config and reinstalling that portion from scratch instead.

I then upgraded to the latest version of ispconfig 3.1.15p2 on the new server.

I've noticed very frequently (around once a day) I can no longer log into the ISPConfig control panel. Running through all the checks I've found on various howtoforge forum threads, none of them appeared to work. So I reinstall ISPConfig using the update.php script. Here is the output:

root@server3:/tmp/update_stable.sh.CmXozREuP2/ispconfig3_install/install# php -q update.php
--------------------------------------------------------------------------------
_____ ___________ _____ __ _ ____
|_ _/ ___| ___ \ / __ \ / _(_) /__ \
| | \ `--.| |_/ / | / \/ ___ _ __ | |_ _ __ _ _/ /
| | `--. \ __/ | | / _ \| '_ \| _| |/ _` | |_ |
_| |_/\__/ / | | \__/\ (_) | | | | | | | (_| | ___\ \
\___/\____/\_| \____/\___/|_| |_|_| |_|\__, | \____/
__/ |
|___/
--------------------------------------------------------------------------------
>> Update
Operating System: Ubuntu 18.04.3 LTS (Bionic Beaver)
This application will update ISPConfig 3 on your server.
Shall the script create a ISPConfig backup in /var/backup/ now? (yes,no) [yes]:
Creating backup of "/usr/local/ispconfig" directory...
Creating backup of "/etc" directory...
mysqldump: Got error: 1153: "Got a packet bigger than 'max_allowed_packet' bytes" when using LOCK TABLES
Checking ISPConfig database .. OK
Starting incremental database update.
Loading SQL patch file: /tmp/update_stable.sh.CmXozREuP2/ispconfig3_install/install/sql/incremental/upd_dev_collection.sql
Reconfigure Permissions in master database? (yes,no) [no]:
Service 'mail_server' has been detected (currently disabled) do you want to enable and configure it? (yes,no) [no]: yes
Service 'web_server' has been detected (currently disabled) do you want to enable and configure it? (yes,no) [no]: yes
Service 'firewall_server' has been detected (currently disabled) do you want to enable and configure it? (yes,no) [no]: yes
Service 'db_server' has been detected (currently disabled) do you want to enable and configure it? (yes,no) [no]: yes
Reconfigure Services? (yes,no,selected) [yes]:
Configuring Postfix
Configuring Dovecot
Configuring Spamassassin
Configuring Amavisd
Configuring Getmail
Configuring Pureftpd
Configuring nginx
Skipping config of Apps vhost
Configuring Jailkit
Configuring Ubuntu Firewall
Configuring Database
Updating ISPConfig
ISPConfig Port [8080]:
Create new ISPConfig SSL certificate (yes,no) [no]:

Reconfigure Crontab? (yes,no) [yes]:
Updating Crontab
Restarting services ...
Update finished.
root@server3:/tmp/update_stable.sh.CmXozREuP2/ispconfig3_install/install# uptime 16:23:42 up 11 days, 22:30, 1 user, load average: 0.80, 1.03, 0.95
As you can see, the web_server, db_server, firewall_server, and mail_server appeared to lose their configuration - as they're showing disabled in the update. Once I run this, they're enabled and I can log in for another day... and then I have to do the update process again in order to access the control panel.

Can anyone shed some light into why these settings are being lost? I am happy to provide any logs, etc that are needed to troubleshoot, I'm just not sure which logs are relevant for the issue. When I view nginx error log and mysql's log, nothing shows out of the ordinary - looks like the usual behavior from the server.

 

What's interesting is that after I run the update script, everything looks normal within ISPConfig - I can log in just fine, edit / add clients/websites etc... works like normal. But if I come back to it tomorrow, no longer able to login and all those server services show disabled in the update script.

I can confirm when its borked, that if I attempt to login the sessions table is updated for dbispconfig to show the login session was authorized and allowed, so it doesn't seem to be a db connectivity problem (at least for that small portion).

Is it possible to do a fresh reinstall of ispconfig but without losing the clients/websites that are configured and their mapping? That might be the best way forward for me to be honest. Is it as simple as exporting those tables from the db, backing up the folder paths/symlinks... then reinstall ISPConfig, restore those tables in the db, and restore the folders?

 

Thanks. Here's the output (after the recent run of update.php)
`mail_server`, `web_server`, `dns_server`, `file_server`, `db_server`, `vserver_server`, `proxy_server`, `firewall_server`, `xmpp_server`
1, 1, 0, 1, 1, 0, 0, 0, 0,

As you can see, currently mail_server, web_server, db_server are enabled - firewall still shows disabled for some reason, but I'm less concerned about that as I did see some issues in the tracker about it. More importantly is the web/mail/db servers for this specific box.

I will wait for the issue to show up again and get you the new values in a separate post in this thread.

There is no error present on the GUI, and there is no error in the logs either - it's as if the login was actually approved (proven in auth log) , but looping back to the login screen. I thought at first maybe it was a client-side cookie issue, so I have tried in several different browsers (safari, firefox, chrome) both in normal and incognito/anonymous mode, and all browsers exhibit the same behavior - unable to login, no error message provided. I have also tried logging in with incorrect credentials and I get the appropriate "username or password incorrect" message.

 

OK, the issue just occurred again. Unable to log in, no error messages at the login screen, auth log says successful login, sys_sessions table shows the session added there w/ the same session ID as in the auth log.

Nothing was done on the server from the last message, only the 2 cron jobs that run every minute for ISPConfig were run (server.sh & cron.sh)

Here's the relevant output of the server table:
`mail_server`, `web_server`, `dns_server`, `file_server`, `db_server`, `vserver_server`, `proxy_server`, `firewall_server`, `xmpp_server`
1, 1, 0, 1, 1, 0, 0, 0, 0,
You can see it is unchanged.

Running optimize got me "OK" status for all 80 tables except for dns_ssl_ca, which offers a note that says the Table does not support optimize, doing recreate +... (message is truncated)

Similarly, doing a repair also got me "OK" status for all 80 tables except for dns_ssl_ca, which says the storage engine for the table doesn't support it (also truncated)

After performing these two operations, I attempted to log in again and was greeted w/ the same behavior - login page with no errors, auth log says successful login, sys_sessions table reflects the same session ID referenced in auth log.

After doing all that, I ran the update.php script again... here's the output

After the update script, I can log in again.
Any other thoughts on where the issue might be?