I have a Fedora 6/Ispconfig system that has been running perfectly for a long time now. It is on a network with several pc's and other servers. This morning we had no internet connectivity. I unplugged the Fedora computer from the switch and all traffic was restored, if I plug it back in, we lose all connectivity again. I have installed rkhunter and it said to check the /dev/.udev directory but I have no idea what I am looking for. Any ideas on where to start?
To update on this problem. I have been messing around with this all day. I cannot seem to find the problem. If I turn off networking on the server... /etc/init.d/network stop I can sit at any computer in the network and I am able to ping my router and get to the internet. As soon as I turn networking back on /etc/init.d/network start I can no longer ping the router or get to the internet on any pc in the network. I can, however, get to the server and do anything I want internally. I just cannot get to the router any longer. Anyone?
This might be a falso positive in rkhunetr, I've seen this on several computers. Any other errors in the syslog or messages log?
No errors that I can find, at least nothing that catches my eye. I have made a little headway though. I changed the root password and rebooted the server. The server has been running for the past 15 minutes without a problem. I am nervous about letting this go though. It almost seems to me like some process was sending out information so fast that it was causing my router to lock up. Perhaps some sort of trojan or something like that. Not sure what to do next.
If rkhunter is not finding anything you can additionally try chkrootkit and maybe do a scan with clamav.
Well when you are saying that all traffic breaks down in case the linux machine is active in the network. So I'd check beside for the amount of traffic the machine is generating (e.g. with iptraf) as well checking the router / switch for any possibilities in that manner. Next thing I'd check how many open connections you have and in which direction, e.g. netstat -tap Or you could also do just a tcpdump (without any parameters, except eventually -w <filename>, to write the capture in a file for later analysis with e.g. wireshark) to see the packages going through. maybe the machine is, why in the world flooding the switch so that it stops acting. Besides this you could also to a tcpdump host <ipofyourfaultylinuxclient> on any of the working clients to see if the linuxmachine as it is put back to the network sends any malicious broadcasts or whatever.
Thank you both for your comments. Last night before heading home, I tried to run a scan with clamav. But, it would only scan the local folder for some reason. So, I then noticed that I was running an older version of ISPConfig (2.2.30), so I updated it to 2.2.32. The machine did not act back up, so I decided I would continue working on it in the morning. At some point in the night, it just decided to start again. When I arrived this morning it was doing the same thing. I rebooted the machine (shutdown -r now), while it was killing processes it was unable to kill named and postfix. I am starting to lean towards one of these as a culprit. I am now waiting for it to start again so that I can try looking at the actual traffic coming from the machine. It has been stable for about an hour now. I can't figure out what sets it off.
Whatever this is, it is in full force. I have been able to run full virus scans, checked for rootkits and done about all I know how to do at this point. Now, when I reboot the server, it runs fine for about 10 minutes and then it starts locking up my router. I cannot watch any traffic going between the server and the router, I have tried tcpdump and it does nothing. I have tried shutting down services one at a time and each time I turn back on networking it locks up the router again. I have searched for logs of traffic on the router and there isnt anything coming from the IP address of the server. Is there somewhere I can see a list of all the running services so that I can start troubleshooting?
While this is happening I am running top and watching the cpu usage of the processes. I shut down networking and then ran top and watched apache using 99.7% of CPU. So, I shut down apache by: /etc/init.d/httpd stop then ran top again and have the same thing, apache using 97% of the CPU. just for kicks I went to the /var/www directory and listed the directory and found a file called Afriend.exe. it was in the /var/www directory, /var/www/localhost directory and the /var/www/html directory. I deleted all three instances of the file, but when I run locate Afriend.exe it still lists all three of these instances. If I go to the directory and run the dir command, it does not list it in those directories. Could I have stumbled onto something? The fact that I shut down apache and it was still showing up in the top command makes me wonder if there is another instance of apache installed on my system. What do you think?
The locate command is not realtime. To update it, run the command "updatedb". On a sipconfig 2 system there are 2 apache servers. Try to run: /etc/init.d/ispconfig_server stop does the apache still shows up?
Thanks for your help, Till, I appreciate it. As of now, the server is back to normal, so I will have to wait til it starts acting up again. As soon as it does, I will post an answer to that question. After running updatedb, the Afriend.exe file does not show up anymore. I did not mention that after deleting this file I restarted the server and it has been running normal now for about 20 minutes.
OK, after about an hour of running normal. It started again. I shut down networking so that I can still work in the network from other computers. Shut down apache, and ispconfig_server then run top. Here is the first line in the top output. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME COMMAND 8394 apache 25 0 1636 468 404 R 99.8 0.1 6:02.60 std This is with both httpd and ispconfig_server shut down... Is there a way to find this ghost of apache?
This is not the apache server.This is another application running as user apache. Please post the output of: ps aux | grep apache while this is running.
Here is the output of ps aux | grep apache Code: apache 8394 96.0 0.0 1636 468 ? R 15:42 14:25 ./std 193.226.84.226 55901 apache 8954 0.0 0.1 1936 688 ? Ss 15:55 0:00 httpd root 8987 0.0 0.1 3892 652 tty1 R+ 15:57 0:00 grep apache
run: locste std to check if there is a shell script of that name in one of the website directories or /tmp
/usr/include/boost/algorhythm/string/std /usr/include/boost/assign/std /usr/share/swig/1.3.21.std/ thats about all I see actually have the name std. The rest of the output has std in it, but not as the full name.
This problem has been fixed. I found a cronjob pointing to a file in a tmp folder by running crontab -u apache -e I deleted the cronjob and then deleted the /tmp file, then I added the apache user to the cron.deny file just in case. Restarted the server, everything has been normal for 12 hours now. Thanks Till for your willingness to help.
