Thrashing issues with mpm-itk

Discussion in 'Server Operation' started by mzandstra, Jun 7, 2013.

  1. mzandstra

    mzandstra New Member

    Hi,

    I'm using a 2 VPSes (1 apache, 1 mysql) to run ~30 php/mysql sites. It's a shared-user environment, so I went with mpm-itk. My problem now is that apache seems to be ignoring my ServerLimit and MaxClients settings and ending up in swap, which causes all the sites to crash.

    Relevant chunk of httpd.conf:

    Code:
    <IfModule mpm_itk_module>
            ServerLimit             30
            StartServers            5
            MinSpareServers          3
            MaxSpareServers         7
            MaxClients              30
            MaxRequestsPerChild   1000
            AssignUserId apache apache
    </IfModule>
    I've confirmed that mpm-itk is being used, and this is CentOS 5.5 with 1 GB RAM. I wrote a shell script / cron job to check apache's memory usage every 5 minutes, here's some of the output:

    Code:
    06-07-2013 14:55:02 - 76 MB Free with 52 httpd processes of average size 20895
    06-07-2013 15:00:01 - 319 MB Free with 62 httpd processes of average size 18776
    06-07-2013 15:05:02 - 137 MB Free with 58 httpd processes of average size 20388
    06-07-2013 15:10:02 - 94 MB Free with 55 httpd processes of average size 22005
    06-07-2013 15:15:03 - 91 MB Free with 55 httpd processes of average size 20211
    06-07-2013 15:20:02 - 121 MB Free with 62 httpd processes of average size 18422
    06-07-2013 15:25:01 - 859 MB Free with 10 httpd processes of average size 12281
    06-07-2013 15:30:01 - 845 MB Free with 10 httpd processes of average size 11435
    06-07-2013 15:35:01 - 852 MB Free with 10 httpd processes of average size 10672
    As you can see, apache is going way over the limit of 30 threads I set. Anyone know why this is? I installed mpm-itk from the CentOS repo, not as a patch.

    Thanks,
    -michael
     
  2. mzandstra

    mzandstra New Member

    *Shameless Bump*

    *Tap* *Tap* Is this thing on?
     
  3. mzandstra

    mzandstra New Member

    Would Apache reading the mpm-itk config first be the issue? I mean, perhaps the prefork settings under
    Code:
    <IfModule prefork.c>
    StartServers       8
    MinSpareServers    5
    MaxSpareServers   20
    ServerLimit      18
    MaxClients       18
    MaxRequestsPerChild  4000
    </IfModule>
    are overwriting my itk settings because they are read later?

    Server crashed again this morning... You can see a large chunk of missing output here from when the server was down:

    Code:
    06-21-2013 12:46:16 - 65 MB Free with 76 httpd processes of average size 18155
    06-21-2013 13:00:56 - 60 MB Free with 104 httpd processes of average size 13890
    06-21-2013 13:00:56 - 60 MB Free with 103 httpd processes of average size 13885
    06-21-2013 13:50:07 - 77 MB Free with 101 httpd processes of average size 11852
    06-21-2013 13:50:07 - 77 MB Free with 102 httpd processes of average size 11736
    06-21-2013 13:01:09 - 74 MB Free with 93 httpd processes of average size 12537
    06-21-2013 13:50:10 - 621 MB Free with 105 httpd processes of average size 9546
    
    My other thought is that perhaps httpd isn't properly killing the itk processes after 4000 requests like it should be... Thoughts?
     
    Last edited: Jun 21, 2013
  4. mzandstra

    mzandstra New Member

    I was able to get a piece of the main httpd error log right as this issue was occuring

    Code:
    [Tue Jun 25 13:43:08 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
    [Tue Jun 25 13:43:10 2013] [error] child died with signal 9
    [Tue Jun 25 13:43:47 2013] [error] child died with signal 9
    [Tue Jun 25 13:43:49 2013] [error] child died with signal 9
    [Tue Jun 25 13:43:55 2013] [error] child died with signal 9
    [Tue Jun 25 13:43:59 2013] [error] child died with signal 9
    [Tue Jun 25 13:44:03 2013] [error] child died with signal 9
    [Tue Jun 25 13:44:05 2013] [error] child died with signal 9
    [Tue Jun 25 13:44:07 2013] [error] child died with signal 9
    [Tue Jun 25 13:44:08 2013] [error] child died with signal 9
    [Tue Jun 25 13:44:09 2013] [error] child died with signal 9
    [Tue Jun 25 13:44:10 2013] [error] child died with signal 9
    [Tue Jun 25 13:44:12 2013] [error] child died with signal 9
    [Tue Jun 25 13:44:13 2013] [error] child died with signal 9
    [Tue Jun 25 13:44:13 2013] [error] child died with signal 9
    [Tue Jun 25 13:44:15 2013] [error] child died with signal 9
    [Tue Jun 25 13:45:18 2013] [error] child died with signal 9
    [Tue Jun 25 13:45:19 2013] [error] child died with signal 9
    [Tue Jun 25 13:45:29 2013] [error] child died with signal 9
    From my memory monitor:

    Code:
    06-25-2013 13:44:02 - 63 MB Free with 42 httpd processes of average size 24894
    My question is, why did httpd wait for those 42 processes to spawn before killing them off? It should never have let that many spawn in the first place.
     
  5. mzandstra

    mzandstra New Member

    Still waiting for a response here. Has nobody had this problem before? It's causing major issues for me.
     

Share This Page