There are spikes in disk latency and disk usage for /dev/sdb. Maybe there's something wrong with the drive?
thanks, I saw that. but postfix was busy sending out a huge newsletter and the HD always gets that busy during that period without crashing everything. since I moved to this new server just a few months ago, I assumed the HDs were ok... I'll run the Strato HW test again tonight, makes the server unreachable for two hours but at least I'll know if there is something wrong... btw. that server runs on a software raid1 that came pre-installed on the minimal debian OS they offer. anyone has any links for raid optimizing? any parameters to tune? I haven't touched it yet...
didn't have time to check the HDs yet, but here we go again: |-amavisd-new---2*[amavisd-new] |-apache2-+-50*[apache2] | |-apache2---60*[php-cgi] swapping like crazy. top with M pressed: http://screencast.com/t/P4m7xKvs8 everything is soooo slow :-( ###edit### managed to dump some info here: http://pastebin.com/qErr4PjE - does that help?
using prefork. config is here: http://howtoforge.com/forums/showpost.php?p=252569&postcount=14 did restart apache and it became a bit better and more responsive... still need to figure out whats going wrong here.
Backup your apache2.conf, make this changes: MaxKeepAliveRequests 100 KeepAlivetimeout 15 <IfModule mpm_prefork_module> StartServers 2 MinSpareServers 2 MaxSpareServers 3 MaxClients 45 MaxRequestsPerChild 500 </IfModule> I am not an expert, but as I can see apache2 is eating all your memory... I don't know your entire configuration and if this could or not work for you. But I think that you must limit the resources destinated to apache, in order to evade swap (and lack of memory... see what happen in four or five days, your memory will not be enough if you let it as it is right now). Try to see if it works better or not than before. Take note of memory consumtion and memory consumption incrementation curve (more slow? more quickly?). If you note any change, it is just a setting problem with this directives. If this not work, you always can rollback the conf file. Restart apache2 this night, it was recently restarted, so... it must be running ok right now. Regards
not sure if your settings will help. I see you want me to start fewer children and limit the maxrequestsperchild but it doesn't look like some script has a memory leak as I don't have a few apache children with high memory consumption, I have heaps of them... default KeepAlivetimeout is 5 - why set it higher? that will tie my ressources down even more :-( still, giving some of your changes a try. thanks
What I want is to isolate you problem (not to find the exactly settings). If you can manage to do that, next you know where to work (if you have changes after this, you can see if you have problems only with apache, or with other process, or whatever) When I am blind about a problem, always worked for me to discard little pieces one at a time. I give special atention to your case, I will use debian with apache2 in production server in a few days, finding a solution for you will be great for me too. I will keep working on your problem. Right know, in my test server, with just 2 test sites with almost no traffic, my ram consumption has growth from 340mb to 1.4gb max in 2 days (with no swaping because I have 4gb ram). I think that apache2 debian package is not prepared out of the box to host vhosts... we will see for your health and mine Regards
down to a crawl again :-( check it live here: http://zice.ro) the current relevant apache settings are: Timeout 120 MaxKeepAliveRequests 100 KeepAliveTimeout 3 <IfModule mpm_prefork_module> StartServers 2 MinSpareServers 2 MaxSpareServers 5 MaxClients 45 MaxRequestsPerChild 2000 </IfModule> I am unsure about one thing though: why are these apache settigns relevant at all? After all all my vhosts are using fastCGI + suexec with its own children spawning settings so these shouldn't impact anything! |-apache2-+-45*[apache2] | |-apache2---67*[php-cgi] | `-sh---vlogger why 45 apache processes? does this help? http://pastebin.com/WzJsZxxj its the memory dump from yesterday and today... countless apache2 and php-CGI processes...
Ok, did it eat your memory more quickly than before???? or it looks like always (same speed, same memory consumption, etc). Was there any change after changing the setting? (possitive or negative).
it feels the same. lagging like crazy. although, according to top there was still some ram left swapping still hit more than 2GB :-( http://zice.ro/munin/serverkompetenz.net/h1870666.serverkompetenz.net/index.html that is where you can verify it.
Just for testing purposes... decrease MaxClient to 25 (and MaxRequestPerChild to 100). See memory consumtion after, see if something changes. MaxClient sets how many connections at same time can apache serve. And MaxRequestsperChild defines after how many requests each of those threads will be recycled. If you have apache set to MaxRequestsPerChild 100 and one thread serves 100 requests, then it will be destroyed, and new apache thread will replace it. Each time apache serves a request, memory consumption grows, and at the same time apache keep parts of the request in memory to get better performance... You server is working ok, this is a configuration issue at worst (see swap and ram as a total... memory consumption will increase while you have free memory, trying to satisfay the freedom that you gave in settings for each app...).
I know what these values do and how they work. of course I can limit myself to very few apache processes, but do you explain those 45 apache2 and 67 php-cgi processes below? the issue is that there is hardly any traffic so how does this result? PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 11682 web1 20 0 421m 167m 106m S 0 4.2 0:53.67 php-cgi and |-apache2-+-45*[apache2] | |-apache2---67*[php-cgi] that's 67 processes each with a huge memory consumption, or do I misunderstand above values? I'm sure something (a wordpress plugin or some other script) is running rogues here - how would I track what web site is spawning most connections or maybe even which script does it?
If this doesn't change the mem behavior, we could think that apache2 is not the main problem (or not at all). I intended to decrease the apache process loaded in memory (and make them to die and reborn quickly) to see if apache was using cache (and your OS swaping as it increased). This is more a "try and error" than other thing (to isolate the problem). Next step I think could be to play with fast cgi setting for your sites. I think that in this forum you can find people with huge more knowledge than me to guide you, but if you want, we can try... (always work with backup) In my opinion, the correct aproach right now would be to identified the site that eats the large memory portion, and work with its conf vhost file (/etc/apache2/sites-available/site.com...) This are the default options of fast-cgi in ispconfig for each site. So, I think that the best place to work is here... if you can manage to tune up one site, next you can modify the entire fastcgi configuration for all your vsites... (correct me please I am wrong...) <IfModule mod_fcgid.c> IdleTimeout 3600 ProcessLifeTime 7200 # MaxProcessCount 1000 DefaultMinClassProcessCount 0 DefaultMaxClassProcessCount 100 IPCConnectTimeout 8 IPCCommTimeout 360 BusyTimeout 300 </IfModule> I will try this weekend to load my servers with lot of sites trying to re-create your problem and test myself... If I can get to do that, I will try this settings (please, be my guess to try them xD) <IfModule mod_fcgid.c> IdleTimeout 600 ProcessLifeTime 7200 # MaxProcessCount 1000 DefaultMinClassProcessCount 0 DefaultMaxClassProcessCount 100 IPCConnectTimeout 8 IPCCommTimeout 360 BusyTimeout 300 IdleScanInterval 60 BusyScanInterval 30 ErrorScanInterval 8 ZombieScanInterval 8 </IfModule> I think that IdleTimeout 300/600 should help you a lot. In theory, after 5' or 10' an idle fcgi process should be killed... (freeing memory). Falko, Till, could you explain a little more about the default configuration please? if good to work on them? I think that the problem here is that a lot of fast cgi process are being created, eating memory continuosly without killing them if them are idle (1 hour idle as default). Edit: maybe could be a good idea to add too: MaxRequestsPerProcess 500 Edit2: read my post on your squirrelmail thread Edit3: If you have just 1 user accessing a site, it is enought for apache for loading it in memory. You don't need a lot of people accessing a site...
I think that web9 is the chosen one (or pick two or three clients with lots of proccesses and high memory usage) Modify the setting, reload apache (or restart? help here please? ) and CONTROL EXCLUSIVELY what happens with this clients (not the general behavior of the server, only the behavior of the php-cgi proccess of the choosen clients). be carefull with this, you will be playing with your clients! - another way could be to make a copy of the most ram eater clients, and play with the copies in a more safer environment. What you do is your decition.-
Well, if you decrease the life time of an idle process, it will be destroyed soon... si, the OS will not swap it (because it was not used for a while...). I was researching about how kernel controls memory... I think that the other part of the solution is to change the swap behavior of the operative system. http://lwn.net/Articles/83588/ (google for more articles, there are lots of web with info about this) To change to a "try to not swap until strictly necessary" (wait until distress value get really high) echo 0 > /proc/sys/vm/swappiness If you want to make this changes persistent at every reboot, add this: echo "vm/swappiness=0" >> /etc/sysctl.conf You can play with value between 0 and 100 with swappines (60 default, you MUST decrease it right now... ). In any way, you are changing HOW and WHEN the kernell will swap, you are NOT disabling swaping. 0-10 is recommended for production servers. Another thing, is ALWAYS recommenden that swap space be separated of your user file space. Regards Edit: just a correction in some words... sorry for my poor english xD
thanks for the input. will read up on this, experiment with the swappiness value and look for a client to test with. web9 isn't good as the guy is moving away in a few days besides, just wanted to mention, this server is just hosting my and some friend's sites so no real customers on it so I can experiment a bit... ##edit### isn't this deprecated? MaxRequestsPerProcess I only found FCGI_MaxRequestsPerProcess ....
