I have an Ubuntu 16.04 server with ISPConfig 3.1. For some reason the server was working too much on a process with high load, so I powered off and restarted the machine. After that, Mysql stopped working and couldn't be restarted anymore. I checked the error log at /var/log/mysql/error.log but it's empty. When I tried to restart the service and I got the following messages: Code: Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details. systemctl status mysql.service: Code: ● mysql.service - LSB: Start and stop the mysql database server daemon Loaded: loaded (/etc/init.d/mysql; bad; vendor preset: enabled) Active: failed (Result: exit-code) since jue 2019-08-22 18:58:05 -04; 41s ago Docs: man:systemd-sysv-generator(8) Process: 12020 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS) Process: 12056 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE) journalctl -xe: Code: ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0x955d0c] ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0x93b08e] ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0x93cd7f] ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0xa1dd9d] ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0xa670b6] ago 22 18:58:32 servidor1 mysqld[12230]: /usr/sbin/mysqld[0x9ba375] ago 22 18:58:32 servidor1 mysqld[12230]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f8ba90256ba] ago 22 18:58:32 servidor1 mysqld[12230]: x86_64/clone.S:111(clone)[0x7f8ba84cc41d] ago 22 18:58:32 servidor1 mysqld[12230]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains ago 22 18:58:32 servidor1 mysqld[12230]: information that should help you find out what is causing the crash. ago 22 18:58:52 servidor1 dovecot[904]: auth-worker(12400): Error: sql([email protected],152.172.252.195): Password query failed: Not connected to database ago 22 18:58:52 servidor1 dovecot[904]: auth: Error: auth worker: Aborted PASSV request for [email protected]: Lookup timed out ago 22 18:58:52 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug ago 22 18:58:53 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug ago 22 18:58:54 servidor1 dovecot[904]: imap-login: Aborted login (auth failed, 1 attempts in 62 secs): user=<[email protected]>, method=PLAIN, rip=152.172.252.19 ago 22 18:58:58 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug ago 22 18:59:01 servidor1 CRON[12687]: pam_unix(cron:session): session opened for user root by (uid=0) ago 22 18:59:01 servidor1 CRON[12686]: pam_unix(cron:session): session opened for user root by (uid=0) ago 22 18:59:01 servidor1 CRON[12688]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log ago 22 18:59:01 servidor1 CRON[12689]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.l ago 22 18:59:01 servidor1 CRON[12686]: pam_unix(cron:session): session closed for user root ago 22 18:59:23 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug ago 22 18:59:25 servidor1 su[12004]: pam_unix(su:session): session closed for user root ago 22 18:59:25 servidor1 sudo[12003]: pam_unix(sudo:session): session closed for user root ago 22 18:59:27 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug ago 22 18:59:28 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug ago 22 18:59:33 servidor1 dovecot[904]: auth-worker(12673): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server throug ago 22 18:59:34 servidor1 sudo[12790]: surempresa : TTY=pts/20 ; PWD=/home/surempresa ; USER=root ; COMMAND=/bin/su ago 22 18:59:34 servidor1 sudo[12790]: pam_unix(sudo:session): session opened for user root by (uid=0) ago 22 18:59:34 servidor1 su[12809]: Successful su for root by root ago 22 18:59:34 servidor1 su[12809]: + /dev/pts/20 root:root ago 22 18:59:34 servidor1 su[12809]: pam_unix(su:session): session opened for user root by (uid=0) ago 22 18:59:34 servidor1 su[12809]: pam_systemd(su:session): Cannot create session: Already running in a session However, I managed to restart the service after I tried the solution from this post: https://askubuntu.com/questions/1008298/cant-start-mysql-server-database-corruption So I set: Code: [mysqld] innodb_force_recovery = 9 But, it doesn't work well since some database tables are now read only. What should I do instead? Thank you.
Did you do the power off with the power button? Do not touch the power button on a running server. Log in and do shutdown from root terminal session. If you now check those systemctl and journalctl commands, do they still show errors?
Yes I did the power off with the power button. I guess that was the problem. This is what I got from those commands, with innodb_force_recovery = 9: systemctl status mysql.service: Code: mysql.service - LSB: Start and stop the mysql database server daemon Loaded: loaded (/etc/init.d/mysql; bad; vendor preset: enabled) Active: active (running) since jue 2019-08-22 20:41:44 -04; 15h ago Docs: man:systemd-sysv-generator(8) Process: 1098 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS) CGroup: /system.slice/mysql.service ├─1212 /bin/bash /usr/bin/mysqld_safe ├─1366 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --skip-log-error --pid-file=/var/run/mysqld/my └─1367 logger -t mysqld -p daemon error ago 23 12:07:04 servidor1 mysqld[1367]: 190823 12:07:04 [ERROR] InnoDB: Failed to find tablespace for table '"phpmyadmin"."pma__export_templates"' in the cache. Attempti Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable. journalctl -xe: Code: lines 991-1013/1013 (END) ago 23 12:31:14 servidor1 postfix/smtpd[24059]: disconnect from mail-eopbgr680124.outbound.protection.outlook.com[40.107.68.124] ehlo=2 starttls=1 mail=1 rcpt=1 data=1 q ago 23 12:31:20 servidor1 postfix/smtpd[28032]: connect from localhost[127.0.0.1] ago 23 12:31:20 servidor1 postfix/smtpd[28032]: A4AAFA6DCE8: client=localhost[127.0.0.1] ago 23 12:31:20 servidor1 postfix/cleanup[28002]: A4AAFA6DCE8: message-id=<DM6PR18MB3113A2829C2647D642D89E21EBA40@DM6PR18MB3113.namprd18.prod.outlook.com> ago 23 12:31:20 servidor1 postfix/qmgr[1773]: A4AAFA6DCE8: from=<[email protected]>, size=28575, nrcpt=1 (queue active) ago 23 12:31:20 servidor1 postfix/smtpd[28032]: disconnect from localhost[127.0.0.1] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5 ago 23 12:31:20 servidor1 amavis[22471]: (22471-06) Passed CLEAN {RelayedOpenRelay}, [40.107.68.124]:38726 [40.107.68.124] <[email protected]> -> <k.iosifidis@ ago 23 12:31:20 servidor1 postfix/smtp[28015]: 655B1A6DCE7: to=<[email protected]>, relay=127.0.0.1[127.0.0.1]:10024, delay=8.3, delays=1.6/0.35/0.45/5.8, ds ago 23 12:31:20 servidor1 postfix/qmgr[1773]: 655B1A6DCE7: removed ago 23 12:31:21 servidor1 postfix/smtpd[26126]: warning: unknown[80.82.77.18]: SASL LOGIN authentication failed: UGFzc3dvcmQ6 ago 23 12:31:21 servidor1 dovecot[28034]: lda([email protected]): sieve: msgid=? <DM6PR18MB3113A2829C2647D642D89E21EBA40@DM6PR18MB3113.namprd18.prod.outlook. ago 23 12:31:21 servidor1 postfix/pipe[28033]: A4AAFA6DCE8: to=<[email protected]>, relay=dovecot, delay=0.59, delays=0.1/0.05/0/0.44, dsn=2.0.0, status=sent ago 23 12:31:21 servidor1 postfix/qmgr[1773]: A4AAFA6DCE8: removed ago 23 12:31:22 servidor1 postfix/smtpd[26126]: disconnect from unknown[80.82.77.18] ehlo=1 auth=0/1 rset=1 quit=1 commands=3/4 ago 23 12:31:26 servidor1 sudo[28035]: surempresa : TTY=pts/5 ; PWD=/home/surempresa ; USER=root ; COMMAND=/bin/su ago 23 12:31:26 servidor1 sudo[28035]: pam_unix(sudo:session): session opened for user root by (uid=0) ago 23 12:31:26 servidor1 su[28037]: Successful su for root by root ago 23 12:31:26 servidor1 su[28037]: + /dev/pts/5 root:root ago 23 12:31:26 servidor1 su[28037]: pam_unix(su:session): session opened for user root by (uid=0) ago 23 12:31:26 servidor1 su[28037]: pam_systemd(su:session): Cannot create session: Already running in a session ago 23 12:31:27 servidor1 dovecot[894]: pop3-login: Login: user=<[email protected]>, method=PLAIN, rip=209.85.167.4, lip=192.168.0.11, mpid=28049, session=<kQV7V ago 23 12:31:28 servidor1 dovecot[894]: pop3([email protected]): Disconnected: Logged out top=0/0, retr=0/0, del=0/8, size=267944 ago 23 12:31:51 servidor1 postfix/smtpd[24059]: connect from unknown[80.82.77.18]
Are you a serious system administrator?!? Your data is severely corrupted! Restore it to an earlier healthy copy then! It is the only way! The best way is to restore it to the latest system image!
