hi there, until 5 mins ago I thought I had setup my server secure and nice, I use mod_security. dos_evasive and other general settings to prevent attacks, but severall minutes ago I almost broke down my own server checking one of my sites for broken links. I used xenu sleuth, set it to 100 parallel threads and told it to go max. 10 links deep. I expected to get a lot of errors because the dos_evasive module should block me after so many events for 10 mins, but no my serevr went crazy: check it out here: http://www.web-designerz.de/serverstats/ traffic broke down, my apache processes maxed out at 80 or so (using a lot of swap memory), my load went to 40-60 and the cpu freaked out. any ideas what to check? my guess is that my mysql database had too many connections and broke down - just a guess though... ###edit### I forgot to mention I only got a 512/256 line at home, so how could I possibly almost kill my server?
well, no nothing in there, as much as the system was concerned nothing unusual was happening, still I could get my server down with my 512/256 line from home? the system continued serving or at least trying to serve my link test program files,... apache's processes maxed out, it swapped 1GB out, all other traffic went down ... load was around 40-60, what more coudl I tell you? BUT: mod_security and dos_evasive still work, if you try to do severall refreshes one after another, very wuick, you'll get a 403 error and are blocked for severall minutes.... I am just wondering what the bottleneck was apache2 or mysql? and why did my link test tool did not get blocked? anyone intersted can run a test with xenu sleuth, set it to 100 threads and check links 10 deep...
ok, I installed monit and muni according to the howto flaoting around here I'll let them run for a few days, to understand how its running when not stressend, then I guess I have to stop monit, so it doesn't interfere and start my "attack" again to see what happen, right? I'll pm you the links after I did so that you can check what happened too. thx for the help
hello falko I have some sever problems right now. I do nothing, didn't change anything inside apache, what happens is this: normal traffic flow, then all of a sudden, apache starts children until it maxes out, starts swapping and server is slooowwww due to this... because it is so slow, there is hardly any more traffic flowing right now. I'll pm you the link to the serverstats... p.s. where you see sudden changes in apache children its where monit restarted apache. I stopped monit now so I can see what happens.
Please have a look at your access.log with the comaind "tail -f ..." Which requests where send to your server when the problem starts? This beahaviour happens often when a not well programmed search engine spider hits your server and requests a large number of pages in a very short time.
I'll try that next, meanwhile: why the hell is bind swapping that much and why does it need 28 processes??? and besides that I have 46 apache2 processes, each using 24M and 23M swap..
I think I figured it out for the time being: I had redesigned a wpmu site and had it display a sitewide feed on the mainpage, when I used tail -f to see what was going on, I saw that feed was being requested hundreds of times, must be soem errors, there, I deactivated it and now everythings seems calm...
another strange happening, have a look at this top-screen: if I do a: tail -f /var/log/httpd/ispconfig_access_log I see: one IP being VERY active, after banning him all is good again, but how can I digg deeper, see what the hell was causing this load?
well this thing happened again, arouns 2:00 last night, my server started spawning max. children, opened max. mysql connections and strangled itsself to death... I'll try and attach some statistics from munin and hotsanic. I'll also have a look what was happening when this all started and get back here with that info.
Maybe it was a bot or something like that. Check Apache's access log and try to find out if the requests came from on IP.
it did not seem to be one ip, nevertheless I am back with some nice stats check the attached file funny, isn't it this time there were serverall ips involved too, and as I am checking my log fiels manually, I can't spot a pattern.... do you know of a nice tool, maybe local installed (windows?) which can analyze the logfile if I downlaod it?
