MariaDB server is down

Discussion in 'General' started by Nawras, Oct 16, 2022.

  1. Nawras

    Nawras New Member

    Hello,
    MariaDB crashes much often. I restart it and it works fine again, then suddenly it crashes again. Here are some details on the issue:
    -----
    Server: Debian 10
    Database: MariaDB 10.3.36 database server
    ISPConfig Version 3
    -------
    service mariadb status
    Code:
    mariadb.service - MariaDB 10.3.36 database server
       Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
      Drop-In: /etc/systemd/system/mysql.service.d
               └─limits.conf
       Active: failed (Result: exit-code) since Sat 2022-10-15 01:32:23 UTC; 1 day 7h ago
         Docs: man:mysqld(8)
               https://mariadb.com/kb/en/library/systemd/
      Process: 26300 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
      Process: 26301 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
      Process: 26302 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`;
      Process: 26378 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
     Main PID: 26378 (code=exited, status=1/FAILURE)
       Status: "MariaDB server is down"
    
    ------
    System Report:
    Code:
    ##### SERVER #####
    IP-address (as per hostname): ***.***.***.***
    [WARN] could not determine server's ip address by ifconfig
    [INFO] OS version is Debian GNU/Linux 10 (buster)
     
    [INFO] uptime:  08:59:43 up 11 days, 12:13,  1 user,  load average: 0.00, 0.01, 0.00
     
    [INFO] memory:
                  total        used        free      shared  buff/cache   available
    Mem:          987Mi       244Mi       461Mi        34Mi       281Mi       568Mi
    Swap:         511Mi       508Mi       3.0Mi
     
    [INFO] systemd failed services status:
      UNIT                  LOAD   ACTIVE SUB    DESCRIPTION                   
    ● clamav-daemon.service loaded failed failed Clam AntiVirus userspace daemon
    ● logrotate.service     loaded failed failed Rotate log files               
    ● mariadb.service       loaded failed failed MariaDB 10.3.36 database server
    
    LOAD   = Reflects whether the unit definition was properly loaded.
    ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
    SUB    = The low-level unit activation state, values depend on unit type.
    
    3 loaded units listed. Pass --all to see loaded but inactive units, too.
    To show all installed unit files use 'systemctl list-unit-files'.
    
    [INFO] ISPConfig is installed.
    
    ##### ISPCONFIG #####
    ISPConfig version is 3.2dev20220915
    
    [WARN] Could not connect to mysql with given user data.
    [WARN] Could not use mysql command line to connect to the database. Check your mysql config!
    
    ##### VERSION CHECK #####
    
    [INFO] php (cli) version is 5.6.40-63+0~20220929.69+debian10~1.gbp639d4c
    [INFO] php-cgi (used for cgi php in default vhost!) is version 7.3.33
    
    ##### PORT CHECK #####
    
    
    ##### MAIL SERVER CHECK #####
    
    
    ##### RUNNING SERVER PROCESSES #####
    
    [INFO] I found the following web server(s):
        Apache 2 (PID 802)
    [INFO] I found the following mail server(s):
        Postfix (PID 1141)
    [INFO] I found the following pop3 server(s):
        Dovecot (PID 528)
    [INFO] I found the following imap server(s):
        Dovecot (PID 528)
    [INFO] I found the following ftp server(s):
        PureFTP (PID 1055)
    
    ##### LISTENING PORTS #####
    (only        ()
    Local        (Address)
    [anywhere]:465        (1141/master)
    ***.***.***.***:53        (15446/named)
    [localhost]:53        (15446/named)
    [anywhere]:21        (1055/pure-ftpd)
    [anywhere]:22        (554/sshd)
    [localhost]:953        (15446/named)
    [anywhere]:25        (1141/master)
    [anywhere]:4190        (528/dovecot)
    [anywhere]:993        (528/dovecot)
    [anywhere]:995        (528/dovecot)
    [localhost]:11332        (622/rspamd:)
    [localhost]:11333        (622/rspamd:)
    [localhost]:11334        (622/rspamd:)
    [localhost]:10023        (523/postgrey)
    [anywhere]:587        (1141/master)
    [localhost]:6379        (597/redis-server)
    [localhost]:11211        (495/memcached)
    [anywhere]:110        (528/dovecot)
    [anywhere]:143        (528/dovecot)
    *:*:*:*::*:465        (1141/master)
    *:*:*:*::*:8081        (802/apache2)
    *:*:*:*::*:53        (15446/named)
    *:*:*:*::*:21        (1055/pure-ftpd)
    *:*:*:*::*:22        (554/sshd)
    *:*:*:*::*:953        (15446/named)
    *:*:*:*::*:25        (1141/master)
    *:*:*:*::*:443        (802/apache2)
    *:*:*:*::*:4190        (528/dovecot)
    *:*:*:*::*:993        (528/dovecot)
    *:*:*:*::*:995        (528/dovecot)
    *:*:*:*::*:11332        (622/rspamd:)
    *:*:*:*::*:11333        (622/rspamd:)
    *:*:*:*::*:11334        (622/rspamd:)
    *:*:*:*::*:10023        (523/postgrey)
    *:*:*:*::*:587        (1141/master)
    *:*:*:*::*:6379        (597/redis-server)
    [localhost]10        (528/dovecot)
    [localhost]43        (528/dovecot)
    *:*:*:*::*:8080        (802/apache2)
    *:*:*:*::*:80        (802/apache2)
    
    
    
    
    ##### IPTABLES #####
    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination         
    f2b-sshd   tcp  --  [anywhere]/0            [anywhere]/0            multiport dports 22
    
    Chain FORWARD (policy ACCEPT)
    target     prot opt source               destination         
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination         
    
    Chain f2b-sshd (1 references)
    target     prot opt source               destination         
    REJECT     all  --  ***.***.***.***         [anywhere]/0            reject-with icmp-port-unreachable
    RETURN     all  --  [anywhere]/0            [anywhere]/0           
    
    
    
    
    ##### LET'S ENCRYPT #####
    acme.sh is installed in /root/.acme.sh/acme.sh
    Note : Upon running *service mariadb restart* , the service is recovered. However, this happens much often. May you please help me to stop this error from happening in the future? I appreciate your help!
    Thank you!
     
    Last edited: Oct 16, 2022
  2. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    I am not sure why that happened to this server of yours but you have wrong version of php cli for your OS.

    By the way, is this server using default mariadb that comes with the OS? Can you check its version and report?
     
    Nawras likes this.
  3. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Examine logs to see why database fails. It may be out of memory, you could make bigger swap (4 gigabytes, for example).
    I'm not sure which log files gets the out of memory notifications, but my guess is this should find them:
    Code:
    grep -i oom /var/log/syslog
    If that did not help, examine the logs in /var/log/mysql/ -directory.
     
    Nawras likes this.
  4. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    ok. it looks like you're running everything on that server, apache, postfix, mysql, pure-ftp etc.

    as @Taleman suggests, you could be running out of memory, most likely when a virus scan is being run.
    with around 1gb ram and 512mb swap, that would seem to be the most likely issue. especially since you only have 3mb of swap memory left and 568mb ram available including the cache, with mysql and clamav already failed. clamav can easily use all that on it's own.
     
    Nawras likes this.
  5. Nawras

    Nawras New Member

    I appreciate all your replies. I will stop some unnecessary services. And create more swap. Thank you for your feedback.
     
  6. pyte

    pyte Well-Known Member HowtoForge Supporter

    I would strongly recommend to upgrade the ram of the machine and not just increase the swap size if possible.
     

Share This Page