Hi everyone I'm running a server with ubuntu 9.10 set up according to the perfect server guide for ISPConfig 3. There's very little running on the server besides a couple of smaller websites. The box is fairly overpowered and is running just fine usually... But every few days or so something happens that causes the system load to rise indefinitely, up to the point where the system gets completely unresponsive (I've seen it as high as 60.0). A look at 'top' doesn't reveal any suspicious process, nothing is using a lot of CPU or memory. The amount of processes in the "waiting" state rises significantly though. My first guess was disk IO, but "iotop" doesn't reveal anything special either. I have also installed munin, but so far I couldn't make any use of it's output. The spikes are clearly visible in the graphs, but they aren't higher than at other points of the day when the hardware has zero problems handling it. When I'm quick enough a simple restart of mysqld solves the problem temporarily, the load goes back to normal instantly, but since so many services depend on mysql that's not necessarily the source of the problem. Some help in hunting down this issue would be very welcome! Thanks for any tips
Are you pulling data from an NFS mount or anything external called/used? the high wait and low cpu usage is mostlikely something that's waiting for some external process.
Not that I can tell so far. It seems to happen very randomly and "only" once every 2-5 days. Still enough to drive you mad though
Nope, this is mostly a webserver using locally attached disks. I have now disabled two additional software components, maybe this will help... If not I'll probably just move the websites to a different server and scratch that box. But thanks anyway for your help!
Probably not a help but I had a similar thing happening on a newly built server. I tracked it to the software RAID 1 disks re-synchronizing.
I have seen this situation before where a server can have a high load for seamingly no reason, and it has in my case been related to disk io. One thing that helped in my case and might help you too given that mysql is somehow implicated, is to make sure your MySQL query cache is well configured, and also the various key caches. Generally sticking a load of RAM in will massively help here, because mysql can put the results of commonly used queries into ram, which dramatically reduces disk I/O. Not sure this will help, but you never know
