Mysql down after updates

Discussion in 'Server Operation' started by axelcon, Sep 10, 2017.

  1. axelcon

    axelcon New Member HowtoForge Supporter

    Hi,

    just ran the outstanding updates via apt-get and found that the mysql server is down and cannot be startet anymore (same for fail2ban...).
    The information I get dooesn't really help me, as there's no visible error explanation/cause (for me).

    Code:
    [....] Starting mysql (via systemctl): mysql.serviceJob for mysql.service failed because the control process exited with error code.
    See "systemctl status mysql.service" and "journalctl -xe" for details.
    
    systemctl status mysql.service:
    Code:
    ● mysql.service - LSB: Start and stop the mysql database server daemon
       Loaded: loaded (/etc/init.d/mysql; generated; vendor preset: enabled)
       Active: failed (Result: exit-code) since Sun 2017-09-10 12:55:13 CEST; 1min 4s ago
         Docs: man:systemd-sysv-generator(8)
      Process: 10899 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)
    
    Sep 10 12:54:43 srv1 systemd[1]: Starting LSB: Start and stop the mysql database server daemon...
    Sep 10 12:55:13 srv1 mysql[10899]: Starting MariaDB database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!
    Sep 10 12:55:13 srv1 systemd[1]: mysql.service: Control process exited, code=exited status=1
    Sep 10 12:55:13 srv1 systemd[1]: Failed to start LSB: Start and stop the mysql database server daemon.
    Sep 10 12:55:13 srv1 systemd[1]: mysql.service: Unit entered failed state.
    Sep 10 12:55:13 srv1 systemd[1]: mysql.service: Failed with result 'exit-code'.
    journalctl -xe:
    Code:
    Sep 10 12:58:41 srv1 sshd[13756]: rexec line 18: Deprecated option KeyRegenerationInterval
    Sep 10 12:58:41 srv1 sshd[13756]: rexec line 19: Deprecated option ServerKeyBits
    Sep 10 12:58:41 srv1 sshd[13756]: rexec line 30: Deprecated option RSAAuthentication
    Sep 10 12:58:41 srv1 sshd[13756]: rexec line 37: Deprecated option RhostsRSAAuthentication
    Sep 10 12:58:42 srv1 sshd[13756]: reprocess config line 30: Deprecated option RSAAuthentication
    Sep 10 12:58:42 srv1 sshd[13756]: reprocess config line 37: Deprecated option RhostsRSAAuthentication
    Sep 10 12:58:42 srv1 sshd[13756]: Invalid user minecraft from 46.101.104.252 port 42032
    Sep 10 12:58:42 srv1 sshd[13756]: input_userauth_request: invalid user minecraft [preauth]
    Sep 10 12:58:42 srv1 sshd[13756]: pam_unix(sshd:auth): check pass; user unknown
    Sep 10 12:58:42 srv1 sshd[13756]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=46.101.104.252
    Sep 10 12:58:42 srv1 dovecot[26700]: auth-worker(13684): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory")
    Sep 10 12:58:42 srv1 dovecot[26700]: auth-worker(13684): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory")
    Sep 10 12:58:42 srv1 /etc/init.d/mysql[13777]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
    Sep 10 12:58:42 srv1 /etc/init.d/mysql[13777]: [61B blob data]
    Sep 10 12:58:42 srv1 /etc/init.d/mysql[13777]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory")'
    Sep 10 12:58:42 srv1 /etc/init.d/mysql[13777]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
    Sep 10 12:58:42 srv1 /etc/init.d/mysql[13777]:
    Sep 10 12:58:42 srv1 mysql[13145]: Starting MariaDB database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!
    Sep 10 12:58:42 srv1 systemd[1]: mysql.service: Control process exited, code=exited status=1
    Sep 10 12:58:42 srv1 systemd[1]: Failed to start LSB: Start and stop the mysql database server daemon.
    -- Subject: Unit mysql.service has failed
    -- Defined-By: systemd
    -- Support: https://www.debian.org/support
    --
    -- Unit mysql.service has failed.
    --
    -- The result is failed.
    Sep 10 12:58:42 srv1 systemd[1]: mysql.service: Unit entered failed state.
    Sep 10 12:58:42 srv1 systemd[1]: mysql.service: Failed with result 'exit-code'.
    
    Any ideas/hints where to start?

    Thanks in advance!
    Axel
     
  2. HSorgYves

    HSorgYves Active Member HowtoForge Supporter

    How about /var/log/syslog?
     
  3. axelcon

    axelcon New Member HowtoForge Supporter

    Not very much different, here after a server reboot:
    Code:
    Sep 10 13:23:52 srv1 monit[1233]: 'mysql' failed to start (exit status -1) -- '/etc/init.d/mysql start': Program timed out -- Starting mysql (via systemctl): mysql.service
    Sep 10 13:23:52 srv1 monit[1233]: 'apache' process is not running
    Sep 10 13:23:52 srv1 monit[1233]: 'apache' trying to restart
    Sep 10 13:23:52 srv1 monit[1233]: 'apache' start: '/etc/init.d/apache2 start'
    Sep 10 13:23:53 srv1 /etc/init.d/mysql[3244]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
    Sep 10 13:23:53 srv1 /etc/init.d/mysql[3244]: #007/usr/bin/mysqladmin: connect to server at 'localhost' failed
    Sep 10 13:23:53 srv1 /etc/init.d/mysql[3244]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory")'
    Sep 10 13:23:53 srv1 /etc/init.d/mysql[3244]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
    Sep 10 13:23:53 srv1 /etc/init.d/mysql[3244]:
    Sep 10 13:23:53 srv1 mysql[2577]: Starting MariaDB database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!
    Sep 10 13:23:53 srv1 systemd[1]: mysql.service: Control process exited, code=exited status=1
    Sep 10 13:23:53 srv1 systemd[1]: Failed to start LSB: Start and stop the mysql database server daemon.
    Sep 10 13:23:53 srv1 systemd[1]: mysql.service: Unit entered failed state.
    Sep 10 13:23:53 srv1 systemd[1]: mysql.service: Failed with result 'exit-code'.
    Sep 10 13:23:58 srv1 dovecot: auth-worker(2845): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory") - waitin
    g for 125 seconds before retry
    Sep 10 13:23:58 srv1 dovecot: auth-worker(2845): Error: mysql(localhost): Connect failed to database (dbispconfig): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2 "No such file or directory") - waitin
    g for 125 seconds before retry
    Sep 10 13:24:00 srv1 CRON[3272]: (root) CMD (/usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 10 13:24:00 srv1 CRON[3273]: (root) CMD (/usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done)
    Sep 10 13:24:23 srv1 monit[1233]: 'apache' failed to start (exit status 0) -- '/etc/init.d/apache2 start': Starting apache2 (via systemctl): apache2.service.
    Sep 10 13:24:23 srv1 postfix/smtpd[2006]: connect from localhost.localdomain[127.0.0.1]
    Sep 10 13:24:23 srv1 postfix/smtpd[2006]: disconnect from localhost.localdomain[127.0.0.1] ehlo=1 quit=1 commands=2
    Sep 10 13:24:27 srv1 dovecot: auth: Error: auth worker: Aborted PASSV request for [email protected]: Lookup timed out
    Sep 10 13:24:27 srv1 dovecot: auth: Error: Aborting auth request that was queued for 109 secs, 22 left in queue
    Sep 10 13:24:27 srv1 dovecot: auth: Error: Aborting auth request that was queued for 109 secs, 21 left in queue
    Sep 10 13:24:27 srv1 dovecot: auth: Error: Aborting auth request that was queued for 109 secs, 20 left in queue
    Sep 10 13:24:27 srv1 dovecot: auth: Error: Aborting auth request that was queued for 109 secs, 19 left in queue
    Sep 10 13:24:27 srv1 dovecot: auth: Error: Aborting auth request that was queued for 109 secs, 18 left in queue
    Sep 10 13:24:27 srv1 dovecot: auth: Error: Aborting auth request that was queued for 109 secs, 17 left in queue
    Sep 10 13:24:27 srv1 dovecot: auth: Error: Aborting auth request that was queued for 109 secs, 16 left in queue
    Sep 10 13:24:27 srv1 dovecot: auth: Error: Aborting auth request that was queued for 109 secs, 15 left in queue
    Sep 10 13:24:27 srv1 dovecot: auth: Error: Aborting auth request that was queued for 109 secs, 14 left in queue
    
     
  4. HSorgYves

    HSorgYves Active Member HowtoForge Supporter

    Run "mysqld --help" and check if it reports problem with config.
     
  5. axelcon

    axelcon New Member HowtoForge Supporter

    Code:
     mysqld --help
    2017-09-10 16:28:37 140445392642624 [Note] Using unique option prefix 'key_buffe                                                       r' is error-prone and can break in the future. Please use the full name 'key_buf                                                       fer_size' instead.
    2017-09-10 16:28:37 140445392642624 [Warning] options --log-slow-admin-statement                                                       s, --log-queries-not-using-indexes and --log-slow-slave-statements have no effec                                                       t if --log_slow_queries is not set
    mysqld  Ver 10.1.26-MariaDB-0+deb9u1 for debian-linux-gnu on x86_64 (Debian 9.1)
    Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
    
    Starts the MariaDB database server.
    
    Usage: mysqld [OPTIONS]
    
    For more help options (several pages), use mysqld --verbose --help.
    
     
  6. axelcon

    axelcon New Member HowtoForge Supporter

    When trying several things with apt, this came to the surface:
    Code:
    dpkg: Abhängigkeitsprobleme verhindern Konfiguration von default-mysql-server:
     default-mysql-server hängt ab von mariadb-server-10.1; aber:
      Paket mariadb-server-10.1 ist noch nicht konfiguriert.
    
    dpkg: Fehler beim Bearbeiten des Paketes default-mysql-server (--configure):
     Abhängigkeitsprobleme - verbleibt unkonfiguriert
    dpkg: Abhängigkeitsprobleme verhindern Konfiguration von mysql-server:
     mysql-server hängt ab von default-mysql-server; aber:
      Paket default-mysql-server ist noch nicht konfiguriert.
    
    dpkg: Fehler beim Bearbeiten des Paketes mysql-server (--configure):
     Abhängigkeitsprobleme - verbleibt unkonfiguriert
    
    Code:
    dpkg-reconfigure default-mysql-server
    /usr/sbin/dpkg-reconfigure: default-mysql-server ist kaputt oder nicht komplett installiert
    
    apt-get install default-mysql-server --reinstall
    Paketlisten werden gelesen... Fertig
    Abhängigkeitsbaum wird aufgebaut.
    Statusinformationen werden eingelesen.... Fertig
    Die folgenden Pakete wurden automatisch installiert und werden nicht mehr benötigt:
      fonts-dejavu fonts-dejavu-extra libexiv2-13 libjs-jquery-mousewheel libjs-jquery-ui liblensfun-data liblensfun0 liblognorm1
      libprocps3 librrd4 libruby2.1 php-intl php-mail-mime php-net-sieve php-pspell php7.0-intl php7.0-pspell tinymce ttf-dejavu
      ttf-dejavu-core ttf-dejavu-extra
    Verwenden Sie »apt autoremove«, um sie zu entfernen.
    0 aktualisiert, 0 neu installiert, 1 erneut installiert, 0 zu entfernen und 0 nicht aktualisiert.
    4 nicht vollständig installiert oder entfernt.
    Nach dieser Operation werden 0 B Plattenplatz zusätzlich benutzt.
    E: Internal Error, No file name for default-mysql-server:amd64
    
    So something probably went totally wrong during installation.
     
  7. axelcon

    axelcon New Member HowtoForge Supporter

    During the night I was able to solve the main problems. Removed and reinstalled the mysql and mariadb packages after updating my package repositories (which weren't up to date as I saw from the perfect server tutorial), fixed sql-mode. Reinstalled all mysql related packages that went down with removing mysql and so far everything's backto normal. Now I just need to reinstall Roundcube and Fail2Ban, both also damaged during updates and all should be well.

    Thanks for the help!
     

Share This Page