Panel not accessible after full disk

Discussion in 'Installation/Configuration' started by MaxOber, Jul 26, 2022.

  1. MaxOber

    MaxOber New Member

    Hello,
    last week i had a completly full disk (already solved) and since then i cannot access the webinterface.

    Distributor ID: Debian
    Description: Debian GNU/Linux 9.13 (stretch)
    Release: 9.13
    Codename: stretch

    PHP 7.0.33-0+deb9u12 (cli) (built: Oct 26 2021 17:51:39) ( NTS )
    Copyright (c) 1997-2017 The PHP Group
    Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies
    with Zend OPcache v7.0.33-0+deb9u12, Copyright (c) 1999-2017, by Zend Technologies


    htf_report.txt
    Code:
    ##### SERVER #####
    IP-address (as per hostname): ***.***.***.***
    [WARN] could not determine server's ip address by ifconfig
    [INFO] OS version is Debian GNU/Linux 9.13 (stretch)
    
    [INFO] uptime:  17:17:33 up 24 min,  1 user,  load average: 0,00, 0,00, 0,04
    
    [INFO] memory:
                  total        used        free      shared  buff/cache   available
    Mem:           7,8G        221M        7,1G        108M        461M        7,3G
    Swap:            0B          0B          0B
    
    [INFO] ISPConfig is installed.
    
    ##### ISPCONFIG #####
    ISPConfig version is 3.2.7p1
    
    [WARN] Could not connect to mysql with given user data.
    [WARN] Could not use mysql command line to connect to the database. Check your mysql config!
    ##### VERSION CHECK #####
    
    [INFO] php (cli) version is 7.0.33-0+deb9u12
    [INFO] php-cgi (used for cgi php in default vhost!) is version 5.6.40
    [WARN] You are using an outdated php version.
    
    ##### PORT CHECK #####
    
    [WARN] Port 143 (IMAP server) seems NOT to be listening
    [WARN] Port 993 (IMAP server SSL) seems NOT to be listening
    [WARN] Port 110 (POP3 server) seems NOT to be listening
    [WARN] Port 995 (POP3 server SSL) seems NOT to be listening
    [WARN] Port 465 (SMTP server SSL) seems NOT to be listening
    
    ##### RUNNING SERVER PROCESSES #####
    
    [INFO] I found the following web server(s):
            Apache 2 (PID 1231)
    [INFO] I found the following mail server(s):
            Postfix (PID 1178)
    [WARN] I could not determine which pop3 server is running.
    [WARN] I could not determine which imap server is running.
    [INFO] I found the following ftp server(s):
            PureFTP (PID 1217)
    
    ##### LISTENING PORTS #####
    Server)         ()
    Local           (Address)
    [localhost]:11211               (789/memcached)
    [anywhere]:111          (729/rpcbind)
    [anywhere]:21           (1217/pure-ftpd)
    [anywhere]:22           (1065/sshd)
    [localhost]:25          (1178/master)
    [localhost]:8996                (788/php-fpm:)
    [localhost]:8997                (807/php-fpm:)
    [localhost]:8998                (790/php-fpm:)
    [localhost]:8999                (791/php-fpm:)
    [localhost]11           (729/rpcbind)
    *:*:*:*::*:80           (1231/apache2)
    *:*:*:*::*:8080         (1231/apache2)
    *:*:*:*::*:8081         (1231/apache2)
    *:*:*:*::*:21           (1217/pure-ftpd)
    *:*:*:*::*:22           (1065/sshd)
    *:*:*:*::*:25           (1178/master)
    *:*:*:*::*:443          (1231/apache2)
    
    ##### IPTABLES #####
    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination
    f2b-pureftpd  tcp  --  [anywhere]/0            [anywhere]/0            multiport dports 21
    f2b-sshd   tcp  --  [anywhere]/0            [anywhere]/0            multiport dports 22
    
    Chain FORWARD (policy ACCEPT)
    target     prot opt source               destination
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination
    
    Chain f2b-pureftpd (1 references)
    
    Chain FORWARD (policy ACCEPT)
    target     prot opt source               destination
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination
    
    Chain f2b-pureftpd (1 references)
    target     prot opt source               destination
    RETURN     all  --  [anywhere]/0            [anywhere]/0
    
    Chain f2b-sshd (1 references)
    target     prot opt source               destination
    RETURN     all  --  [anywhere]/0            [anywhere]/0
    
    
    
    ##### LET'S ENCRYPT #####
    Certbot is installed in /usr/bin/letsencrypt
    
    
    

    when i try to access the panel, the apache2 logs show the following error:

    Code:
    [Tue Jul 26 16:54:33.931976 2022] [fcgid:warn] [pid 1239] [client 127.0.0.1:46456] mod_fcgid: stderr: PHP Fatal error:  Uncaught Error: Call to a member function queryOneRecord() on boolean in /usr/local/ispconfig/int$[Tue Jul 26 16:54:33.931983 2022] [fcgid:warn] [pid 1239] [client 127.0.0.1:46456] mod_fcgid: stderr: Stack trace:
    [Tue Jul 26 16:54:33.931990 2022] [fcgid:warn] [pid 1239] [client 127.0.0.1:46456] mod_fcgid: stderr: #0 /usr/local/ispconfig/interface/lib/app.inc.php(93): app->conf('interface', 'session_timeout')
    [Tue Jul 26 16:54:33.931998 2022] [fcgid:warn] [pid 1239] [client 127.0.0.1:46456] mod_fcgid: stderr: #1 /usr/local/ispconfig/interface/lib/app.inc.php(401): app->initialize_session()
    [Tue Jul 26 16:54:33.932005 2022] [fcgid:warn] [pid 1239] [client 127.0.0.1:46456] mod_fcgid: stderr: #2 /usr/local/ispconfig/interface/web/index.php(32): require_once('/usr/local/ispc...')
    [Tue Jul 26 16:54:33.932024 2022] [fcgid:warn] [pid 1239] [client 127.0.0.1:46456] mod_fcgid: stderr: #3 {main}
    [Tue Jul 26 16:54:33.932031 2022] [fcgid:warn] [pid 1239] [client 127.0.0.1:46456] mod_fcgid: stderr:   thrown in /usr/local/ispconfig/interface/lib/app.inc.php on line 169
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    Most likely there is a corruption in the MySQL database or MySQL/MariaDB is not started. Try to check and repair the dbispconfig database with e.g. phpmyadmin.
     
    Th0m likes this.
  3. MaxOber

    MaxOber New Member

    Thanks for your tip.
    Indeed the DB is not started and i get an error when starting:
    Code:
    root@web1:/var/run/mysqld# systemctl status mariadb.service
    ● mariadb.service - MariaDB 10.1.48 database server
       Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
       Active: failed (Result: exit-code) since Wed 2022-07-27 12:31:03 CEST; 10s ago
         Docs: man:mysqld(8)
               https://mariadb.com/kb/en/library/systemd/
      Process: 4519 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
      Process: 4454 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _
      Process: 4450 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
      Process: 4447 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
     Main PID: 4519 (code=exited, status=1/FAILURE)
       Status: "MariaDB server is down"
    
    Jul 27 12:31:01 web1.xxx.at mysqld[4519]: 2022-07-27 12:31:01 139646463864192 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.49-89.0 started; log sequence number 44
    Jul 27 12:31:01 web1.xxx.at mysqld[4519]: 2022-07-27 12:31:01 139645794907904 [Note] InnoDB: Dumping buffer pool(s) not yet started
    Jul 27 12:31:01 web1.xxx.at mysqld[4519]: 2022-07-27 12:31:01 139646463864192 [Note] Plugin 'FEEDBACK' is disabled.
    Jul 27 12:31:01 web1.xxx.at mysqld[4519]: 2022-07-27 12:31:01 139646463864192 [Note] Recovering after a crash using tc.log
    Jul 27 12:31:01 web1.xxx.at mysqld[4519]: 2022-07-27 12:31:01 139646463864192 [ERROR] Can't init tc log
    Jul 27 12:31:01 web1.xxx.at mysqld[4519]: 2022-07-27 12:31:01 139646463864192 [ERROR] Aborting
    Jul 27 12:31:03 web1.xxx.at systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
    Jul 27 12:31:03 web1.xxx.at systemd[1]: Failed to start MariaDB 10.1.48 database server.
    Jul 27 12:31:03 web1.xxx.at systemd[1]: mariadb.service: Unit entered failed state.
    Jul 27 12:31:03 web1.xxx.at systemd[1]: mariadb.service: Failed with result 'exit-code'.

    I think I know what the problem could be:
    because of the full disk, i deleted some website directories in /var/www and also in /clients/clientx
    do you think thats the problem ?

    do you have some tips how to troubleshoot the db ?

    Thank you very much
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    That's not related to the start issue of the database, but your web server will probably fail now as well after you removed the website directories if you have not removed the vhost files in the apache sites-enabled and sites-available folder as well.

    Search at google on the topic MariaDB InnoDB recovery, there are guidelines from MariaDB and other websites on this topic.
     
  5. MaxOber

    MaxOber New Member

    Thank you very much.
    I deleted /var/lib/mysql/tc.log and now it's working again :D
     
    Steini86 and till like this.

Share This Page