httpd is down! MySQL is down! Rescue will not help

Discussion in 'Server Operation' started by conny2540, May 21, 2019.

  1. conny2540

    conny2540 Member

    I'm suddenly having tons of errors and cannot find why.
    All is working. Sites working, no problem on database, also when I check service apache2 and mysql I get active and running without any error report. Why is ispconfig writing theses mass errors ?

    What I already did for to debug:

    1) I've already udated ispconfig, incl. SSL cert.
    2) Checked error.logs of apache2 and mysql - no errors but warnings for apache:
    AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/sites-enabled/000-ispconfig
    AH00112: Warning: DocumentRoot [/dev/null] does not exist


    What I observed is that there was an attack to port 8081 last night, but not successful (404 errors).
    How to get rid what is happening and where exactly ? Perhaps there is a changed file I've changed before and after reboot of the system the error is applying ? How to locate for changed files in the last 3 days for ubuntu 16.04. ?
     
  2. conny2540

    conny2540 Member

    Also this error I have when I monitor with ispconfig.
    connect to mysql server 127.0.0.1: Can't connect to MySQL server on '127.0.0.1' (111)
    postfix/error[19636]: warning: fast_flush_domains: mysql:/etc/postfix/mysql-virtual_relaydomains.cf: table lookup problem
    bind service is working. Was something changed on my server configuration by hacker ???
    Mails are not working also after reboot of the server.
    Mailqueue full of errors as well: (delivery temporarily suspended: connect to 127.0.0.1[127.0.0.1]:10024: Connection refused).
    And FTP connection also not possible anymore. What happened ? Hacker destroyed something in DB ?
     
    Last edited: May 21, 2019
  3. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    Topic says "httpd is down!" but then you write "All is working. Sites working".
    Topic says "MySQL is down!" but then you write "no problem on database".
    What errors?
    You wrote you are using ISPConfig, but posted in Linux forum.
    Compare what you have now to backups from 3 days ago. If you run etckeeper https://packages.ubuntu.com/xenial/etckeeper you can compare in version control what configuration files have changed.
    https://www.howtoforge.com/community/threads/please-read-before-posting.58408/
     
  4. conny2540

    conny2540 Member

    Worst case of all I cannot access to nothing anymore. Seems that apache and database crashed. How to access to backups in rescue mode ? And download them to my local computer. I intend to install ispconfig with latest ubuntu stable instead of the old 16.04.

    Yes I'm using ispconfig 3.1, but the problem is for Ubuntu 16.04.
    Ispconfig was writing the errors I mentioned in title in mass -> httpd down, mysql down on monitor tab.

    Furthermore I received mass mails with same problem and others with these errors:

    WARNING - Falsche Anfrage / Wrong QuerySQL-Query = SELECT action_id, action_type, action_param FROM sys_remoteaction WHERE server_id = 1 AND action_id > 0 ORDER BY action_id -> 2006 (MySQL server has gone away)

    WARNING - Falsche Anfrage / Wrong QuerySQL-Query = INSERT INTO sys_log (server_id,datalog_id,loglevel,tstamp,message) VALUES (1, 0, 2, UNIX_TIMESTAMP(), 'httpd is down! Rescue will not help!') -> 2006 (MySQL server has gone away)

    WARNING - Falsche Anfrage / Wrong QuerySQL-Query = SELECT server_php_id, php_fastcgi_ini_dir, php_fpm_ini_dir FROM server_php WHERE server_id = 1 -> 2006 (MySQL server has gone away)

    WARNING - Falsche Anfrage / Wrong QuerySQL-Query = SELECT config FROM server WHERE server_id = 1 LIMIT 0,1 -> 2006 (MySQL server has gone away)

    WARNING - Clearing semaphores table for user apache

    EDIT: I was able to connect to databases and make a back-up locally, BUT I'm unable to connect to FTP to do the same. Any hint on how to get in with filezilla ? I get socket error 10060. Can I get access to FTP with filezilla in rescue mode ? Or is there another method I can download FTP locally ?
     
    Last edited: May 22, 2019
  5. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    With rescue mode you mean Ubuntu boot rescue mode? That is, host does not boot normally, but you can boot in rescue mode?
    You can examine the situation in rescue mode. Start services manually and see if they start or what error messages you get if the do not start.
    You may have to start with networking,
    Code:
    systemctl start networking
    before starting services that need network.
    I do not have Ubunty handy, but in Debian GNU/Linux 9 FTP server is started with
    Code:
    systemctl start pure-ftpd-mysql.service 
    Then you can use FTP to transfer files.
    You can get a list of services with
    Code:
    systemctl list-unit-files | less
    If you do not know what caused this situation, run memory test and check disk status from smartmontools. Also read /var/log/syslog to see what happened around the time problems appeared.
     
  6. conny2540

    conny2540 Member

    Still cannot connect via FTP.

    /var/log/syslog errors:
    postfix/error[23201]: warning: connect to mysql server 127.0.0.1: Can't connect to MySQL server on '127.0.0.1' (111)
    postfix/error[23201]: warning: fast_flush_domains: mysql:/etc/postfix/mysql-virtual_relaydomains.cf: table lookup problem
    systemd[1]: Reached target Local File Systems.
    systemd[1]: Starting Tell Plymouth To Write Out Runtime Data...
    systemd[1]: Starting Create Volatile Files and Directories...
    systemd[1]: Starting Set console font and keymap...
    systemd-udevd[2266]: could not open builtin file '/lib/modules/3.10.23-xxxx-std-ipv6-64-vps/modules.builtin.bin'
    systemd[1]: Stopping MySQL Community Server...
    systemd[1]: Stopped MySQL Community Server.
    systemd[1]: Starting MySQL Community Server...
    systemd[1]: Started MySQL Community Server.
    ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^$
    rsyslogd-2222: command 'KLogPermitNonKernelFacility' is currently not permitted - did you already set it via a RainerScri$
    rsyslogd: rsyslogd's groupid changed to 104
    rsyslogd: rsyslogd's userid changed to 101
    rsyslogd-2039: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.16.0 try http://www.rsyslog.com$
    rsyslogd-2007: action 'action 10' suspended, next retry is Wed May 22 17:55:55 2019 [v8.16.0 try http://www.rsyslog.com/e$
    systemd-modules-load[2123]: could not open builtin file '/lib/modules/3.10.23-xxxx-std-ipv6-64-vps/modules.builtin.bin'
    systemd-modules-load[2123]: Failed to find module 'lp'
    systemd-sysctl[2127]: Couldn't write '2' to 'net/ipv6/conf/all/use_tempaddr', ignoring: No such file or directory
    loadkeys[2114]: Loading /etc/console-setup/cached.kmap.gz
    systemd-sysctl[2127]: Couldn't write '2' to 'net/ipv6/conf/default/use_tempaddr', ignoring: No such file or directory
    systemd-sysctl[2127]: Couldn't write '1' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory
    ufw-init[2115]: modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules$
    rsyslogd-2039: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.16.0 try http://www.rsyslog.com$
    rsyslogd-2007: action 'action 10' suspended, next retry is Wed May 22 17:55:55 2019 [v8.16.0 try http://www.rsyslog.com/e$
    systemd-modules-load[2123]: could not open builtin file '/lib/modules/3.10.23-xxxx-std-ipv6-64-vps/modules.builtin.bin'
    systemd-modules-load[2123]: Failed to find module 'lp'
    systemd-sysctl[2127]: Couldn't write '2' to 'net/ipv6/conf/all/use_tempaddr', ignoring: No such file or directory
    loadkeys[2114]: Loading /etc/console-setup/cached.kmap.gz
    systemd-sysctl[2127]: Couldn't write '2' to 'net/ipv6/conf/default/use_tempaddr', ignoring: No such file or directory
    systemd-sysctl[2127]: Couldn't write '1' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory
    ufw-init[2115]: modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules$
    ufw-init[2115]: modprobe: FATAL: Module nf_conntrack_ftp not found in directory /lib/modules/3.10.23-xxxx-std-ipv6-64-vps
    ufw-init[2115]: modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules$
    ufw-init[2115]: modprobe: FATAL: Module nf_nat_ftp not found in directory /lib/modules/3.10.23-xxxx-std-ipv6-64-vps
    ufw-init[2115]: modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules$
    ufw-init[2115]: modprobe: FATAL: Module nf_conntrack_netbios_ns not found in directory /lib/modules/3.10.23-xxxx-std-ipv6$
    systemd[1]: Starting Flush Journal to Persistent Storage...
    systemd[1]: Started Flush Journal to Persistent Storage.
    systemd[1]: Started Initial Check File System Quotas.
    systemd[1]: Started udev Kernel Device Manager.
    systemd[1]: Started udev Coldplug all Devices.

    server is disconnecting each 20 seconds... I need to reboot them from OVH Panel after each 20 seconds. In Rescue mode of OVH I can mount FTP, but I need to hit each file for to get them downloaded. So a Sysiphusarbeit and I will ready in one week...
    Seems that database is broken somewhere, although I run repair tables and there was nothing to repair (InnoDB cannot be repaired). Perhaps the attack to port 8081 had success ?? I cannot say
     
    Last edited: May 23, 2019
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    FTP login will not work when MySQL is not working as all FTP users are in mysql. To do a backup, login as root user and use sftp / scp (which is SSH) to download all files. If you have a Windows Desktop, then e.g. use WinSCP as scp client for the file download.
     
  8. conny2540

    conny2540 Member

    I also tried by disabling ufw. Cannot access to FTP. When I enable UFW again I get this errors:
    sudo ufw enable
    Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
    ERROR: problem running ufw-init
    modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules/3.10.23-xxxx-std-ipv6-64-vps/modules.builtin.bin'
    modprobe: FATAL: Module nf_conntrack_ftp not found in directory /lib/modules/3.10.23-xxxx-std-ipv6-64-vps
    modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules/3.10.23-xxxx-std-ipv6-64-vps/modules.builtin.bin'
    modprobe: FATAL: Module nf_nat_ftp not found in directory /lib/modules/3.10.23-xxxx-std-ipv6-64-vps
    modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules/3.10.23-xxxx-std-ipv6-64-vps/modules.builtin.bin'
    modprobe: FATAL: Module nf_conntrack_netbios_ns not found in directory /lib/modules/3.10.23-xxxx-std-ipv6-64-vps
    iptables-restore: line 6 failed
     
  9. conny2540

    conny2540 Member

    SOLVED: Finally I got all ispconfig tar.gz files I need with OVH rescue tools. Thank you for your support Till.:)
     

Share This Page