Every now and again, my IPSConfig 3 server (Ubuntu 14.04) stops responding, and I cannot access any website or SSH onto it, so I'm trying to dig into what could be causing the issue... I've noticed a lot of the following in my syslog: Code: Oct 30 06:33:03 local kernel: [1492778.616857] init: mysql main process (14216) terminated with status 1 Oct 30 06:33:03 local kernel: [1492778.616869] init: mysql main process ended, respawning Oct 30 06:33:03 local /etc/mysql/debian-start[14788]: Upgrading MySQL tables if necessary. Oct 30 06:33:03 local /etc/mysql/debian-start[14791]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored Oct 30 06:33:03 local /etc/mysql/debian-start[14791]: Looking for 'mysql' as: /usr/bin/mysql Oct 30 06:33:03 local /etc/mysql/debian-start[14791]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck Oct 30 06:33:03 local /etc/mysql/debian-start[14791]: Error: Server version (5.5.44-0ubuntu0.14.04.1) does not match with the version of Oct 30 06:33:03 local /etc/mysql/debian-start[14791]: the server (5.5.46) with which this program was built/distributed. You can Oct 30 06:33:03 local /etc/mysql/debian-start[14791]: use --skip-version-check to skip this check. Oct 30 06:33:03 local /etc/mysql/debian-start[14791]: FATAL ERROR: Upgrade failed Oct 30 06:33:03 local /etc/mysql/debian-start[14815]: Checking for insecure root accounts. Oct 30 06:33:03 local /etc/mysql/debian-start[14820]: Triggering myisam-recover for all MyISAM tables If I run mysql_upgrade manually, I get the following error: Code: mysql_upgrade -u root -p --verbose Looking for 'mysql' as: mysql Looking for 'mysqlcheck' as: mysqlcheck FATAL ERROR: Upgrade failed How do I solve / correct this please?
Thats strange. Do you use any third party repositories? Maybe you shoud reinstall mysql. 1) Make a backup of /var/lib/mysql and /etc/mysql, just to be sure. 2) Reinstall mysql with: apt-get install --reinstall mysql-server mysql-client
I haven't seen before that an Ubuntu MySQl update failed this way, so I guess it will not happen that often and you probably won't encounter the issue again.
Thanks @till It seems to have fixed that issue, but the server just blipped again, so unfortunately not that that's causing the downtime...
Do you find any errors in /var/log/syslog after you rebooted the server for the timepoint it stopped responding? If there are no relevant errors, then this might be a hardware error like mainboard or flaky RAM. I had a similar issue at ispconfig.org, the data center replaced the RAM and updated the BIOS and the server runs stable again afterward.
The syslog seems to be clean now, so maybe it is down to a hardware issue - unless it's network load?
If its network load then there should be new entry's in the syslog (even if they are not errors). If the log just stopped and no new entrys, then it is most likely a hardware issue.
I've just had another blip whilst I was in the crontab, so quickly jumped back on and looked at the syslog (I've changed some URLs to protect my sites): One thing I do notice is some of the log lines have an incorrect date - note 'Nov 1' and 'Nov 2'. You can see when I was in the crontab, and then shortly after I was kicked out and couldn't SSH back in for a minute or so, and sites went down.
There's also an old line from an ISPConfig cron: Code: Nov 2 00:45:01 local CRON[7907]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 > /dev/null | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cr$
