I'm suddenly having tons of errors and cannot find why. All is working. Sites working, no problem on database, also when I check service apache2 and mysql I get active and running without any error report. Why is ispconfig writing theses mass errors ? What I already did for to debug: 1) I've already udated ispconfig, incl. SSL cert. 2) Checked error.logs of apache2 and mysql - no errors but warnings for apache: AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/sites-enabled/000-ispconfig AH00112: Warning: DocumentRoot [/dev/null] does not exist What I observed is that there was an attack to port 8081 last night, but not successful (404 errors). How to get rid what is happening and where exactly ? Perhaps there is a changed file I've changed before and after reboot of the system the error is applying ? How to locate for changed files in the last 3 days for ubuntu 16.04. ?
Also this error I have when I monitor with ispconfig. connect to mysql server 127.0.0.1: Can't connect to MySQL server on '127.0.0.1' (111) postfix/error[19636]: warning: fast_flush_domains: mysql:/etc/postfix/mysql-virtual_relaydomains.cf: table lookup problem bind service is working. Was something changed on my server configuration by hacker ??? Mails are not working also after reboot of the server. Mailqueue full of errors as well: (delivery temporarily suspended: connect to 127.0.0.1[127.0.0.1]:10024: Connection refused). And FTP connection also not possible anymore. What happened ? Hacker destroyed something in DB ?
Topic says "httpd is down!" but then you write "All is working. Sites working". Topic says "MySQL is down!" but then you write "no problem on database". What errors? You wrote you are using ISPConfig, but posted in Linux forum. Compare what you have now to backups from 3 days ago. If you run etckeeper https://packages.ubuntu.com/xenial/etckeeper you can compare in version control what configuration files have changed. https://www.howtoforge.com/community/threads/please-read-before-posting.58408/
Worst case of all I cannot access to nothing anymore. Seems that apache and database crashed. How to access to backups in rescue mode ? And download them to my local computer. I intend to install ispconfig with latest ubuntu stable instead of the old 16.04. Yes I'm using ispconfig 3.1, but the problem is for Ubuntu 16.04. Ispconfig was writing the errors I mentioned in title in mass -> httpd down, mysql down on monitor tab. Furthermore I received mass mails with same problem and others with these errors: WARNING - Falsche Anfrage / Wrong QuerySQL-Query = SELECT action_id, action_type, action_param FROM sys_remoteaction WHERE server_id = 1 AND action_id > 0 ORDER BY action_id -> 2006 (MySQL server has gone away) WARNING - Falsche Anfrage / Wrong QuerySQL-Query = INSERT INTO sys_log (server_id,datalog_id,loglevel,tstamp,message) VALUES (1, 0, 2, UNIX_TIMESTAMP(), 'httpd is down! Rescue will not help!') -> 2006 (MySQL server has gone away) WARNING - Falsche Anfrage / Wrong QuerySQL-Query = SELECT server_php_id, php_fastcgi_ini_dir, php_fpm_ini_dir FROM server_php WHERE server_id = 1 -> 2006 (MySQL server has gone away) WARNING - Falsche Anfrage / Wrong QuerySQL-Query = SELECT config FROM server WHERE server_id = 1 LIMIT 0,1 -> 2006 (MySQL server has gone away) WARNING - Clearing semaphores table for user apache EDIT: I was able to connect to databases and make a back-up locally, BUT I'm unable to connect to FTP to do the same. Any hint on how to get in with filezilla ? I get socket error 10060. Can I get access to FTP with filezilla in rescue mode ? Or is there another method I can download FTP locally ?
With rescue mode you mean Ubuntu boot rescue mode? That is, host does not boot normally, but you can boot in rescue mode? You can examine the situation in rescue mode. Start services manually and see if they start or what error messages you get if the do not start. You may have to start with networking, Code: systemctl start networking before starting services that need network. I do not have Ubunty handy, but in Debian GNU/Linux 9 FTP server is started with Code: systemctl start pure-ftpd-mysql.service Then you can use FTP to transfer files. You can get a list of services with Code: systemctl list-unit-files | less If you do not know what caused this situation, run memory test and check disk status from smartmontools. Also read /var/log/syslog to see what happened around the time problems appeared.
Still cannot connect via FTP. /var/log/syslog errors: postfix/error[23201]: warning: connect to mysql server 127.0.0.1: Can't connect to MySQL server on '127.0.0.1' (111) postfix/error[23201]: warning: fast_flush_domains: mysql:/etc/postfix/mysql-virtual_relaydomains.cf: table lookup problem systemd[1]: Reached target Local File Systems. systemd[1]: Starting Tell Plymouth To Write Out Runtime Data... systemd[1]: Starting Create Volatile Files and Directories... systemd[1]: Starting Set console font and keymap... systemd-udevd[2266]: could not open builtin file '/lib/modules/3.10.23-xxxx-std-ipv6-64-vps/modules.builtin.bin' systemd[1]: Stopping MySQL Community Server... systemd[1]: Stopped MySQL Community Server. systemd[1]: Starting MySQL Community Server... systemd[1]: Started MySQL Community Server. ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^$ rsyslogd-2222: command 'KLogPermitNonKernelFacility' is currently not permitted - did you already set it via a RainerScri$ rsyslogd: rsyslogd's groupid changed to 104 rsyslogd: rsyslogd's userid changed to 101 rsyslogd-2039: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.16.0 try http://www.rsyslog.com$ rsyslogd-2007: action 'action 10' suspended, next retry is Wed May 22 17:55:55 2019 [v8.16.0 try http://www.rsyslog.com/e$ systemd-modules-load[2123]: could not open builtin file '/lib/modules/3.10.23-xxxx-std-ipv6-64-vps/modules.builtin.bin' systemd-modules-load[2123]: Failed to find module 'lp' systemd-sysctl[2127]: Couldn't write '2' to 'net/ipv6/conf/all/use_tempaddr', ignoring: No such file or directory loadkeys[2114]: Loading /etc/console-setup/cached.kmap.gz systemd-sysctl[2127]: Couldn't write '2' to 'net/ipv6/conf/default/use_tempaddr', ignoring: No such file or directory systemd-sysctl[2127]: Couldn't write '1' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory ufw-init[2115]: modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules$ rsyslogd-2039: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.16.0 try http://www.rsyslog.com$ rsyslogd-2007: action 'action 10' suspended, next retry is Wed May 22 17:55:55 2019 [v8.16.0 try http://www.rsyslog.com/e$ systemd-modules-load[2123]: could not open builtin file '/lib/modules/3.10.23-xxxx-std-ipv6-64-vps/modules.builtin.bin' systemd-modules-load[2123]: Failed to find module 'lp' systemd-sysctl[2127]: Couldn't write '2' to 'net/ipv6/conf/all/use_tempaddr', ignoring: No such file or directory loadkeys[2114]: Loading /etc/console-setup/cached.kmap.gz systemd-sysctl[2127]: Couldn't write '2' to 'net/ipv6/conf/default/use_tempaddr', ignoring: No such file or directory systemd-sysctl[2127]: Couldn't write '1' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory ufw-init[2115]: modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules$ ufw-init[2115]: modprobe: FATAL: Module nf_conntrack_ftp not found in directory /lib/modules/3.10.23-xxxx-std-ipv6-64-vps ufw-init[2115]: modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules$ ufw-init[2115]: modprobe: FATAL: Module nf_nat_ftp not found in directory /lib/modules/3.10.23-xxxx-std-ipv6-64-vps ufw-init[2115]: modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules$ ufw-init[2115]: modprobe: FATAL: Module nf_conntrack_netbios_ns not found in directory /lib/modules/3.10.23-xxxx-std-ipv6$ systemd[1]: Starting Flush Journal to Persistent Storage... systemd[1]: Started Flush Journal to Persistent Storage. systemd[1]: Started Initial Check File System Quotas. systemd[1]: Started udev Kernel Device Manager. systemd[1]: Started udev Coldplug all Devices. server is disconnecting each 20 seconds... I need to reboot them from OVH Panel after each 20 seconds. In Rescue mode of OVH I can mount FTP, but I need to hit each file for to get them downloaded. So a Sysiphusarbeit and I will ready in one week... Seems that database is broken somewhere, although I run repair tables and there was nothing to repair (InnoDB cannot be repaired). Perhaps the attack to port 8081 had success ?? I cannot say
FTP login will not work when MySQL is not working as all FTP users are in mysql. To do a backup, login as root user and use sftp / scp (which is SSH) to download all files. If you have a Windows Desktop, then e.g. use WinSCP as scp client for the file download.
I also tried by disabling ufw. Cannot access to FTP. When I enable UFW again I get this errors: sudo ufw enable Command may disrupt existing ssh connections. Proceed with operation (y|n)? y ERROR: problem running ufw-init modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules/3.10.23-xxxx-std-ipv6-64-vps/modules.builtin.bin' modprobe: FATAL: Module nf_conntrack_ftp not found in directory /lib/modules/3.10.23-xxxx-std-ipv6-64-vps modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules/3.10.23-xxxx-std-ipv6-64-vps/modules.builtin.bin' modprobe: FATAL: Module nf_nat_ftp not found in directory /lib/modules/3.10.23-xxxx-std-ipv6-64-vps modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules/3.10.23-xxxx-std-ipv6-64-vps/modules.builtin.bin' modprobe: FATAL: Module nf_conntrack_netbios_ns not found in directory /lib/modules/3.10.23-xxxx-std-ipv6-64-vps iptables-restore: line 6 failed
SOLVED: Finally I got all ispconfig tar.gz files I need with OVH rescue tools. Thank you for your support Till.