CENTOS 5 Ping Problem

Discussion in 'Server Operation' started by gAnDo, Mar 20, 2008.

  1. gAnDo

    gAnDo New Member

    Hi.
    Can anybody explain why this is happening?
    Code:
    64 bytes from py-in-f99.google.com (64.233.167.99): icmp_seq=1 ttl=243 time=150 ms
    64 bytes from py-in-f99.google.com (64.233.167.99): icmp_seq=2 ttl=243 time=151 ms
    64 bytes from py-in-f99.google.com (64.233.167.99): icmp_seq=3 ttl=243 time=151 ms
    64 bytes from py-in-f99.google.com (64.233.167.99): icmp_seq=4 ttl=243 time=153 ms
    ping: sendmsg: Operation not permitted
    ping: sendmsg: Operation not permitted
    64 bytes from py-in-f99.google.com (64.233.167.99): icmp_seq=7 ttl=243 time=153 ms
    64 bytes from py-in-f99.google.com (64.233.167.99): icmp_seq=8 ttl=243 time=149 ms
    ping: sendmsg: Operation not permitted
    ping: sendmsg: Operation not permitted
    64 bytes from py-in-f99.google.com (64.233.167.99): icmp_seq=11 ttl=243 time=152 ms
    64 bytes from py-in-f99.google.com (64.233.167.99): icmp_seq=12 ttl=243 time=151 ms
    ping: sendmsg: Operation not permitted
    ping: sendmsg: Operation not permitted
    ping: sendmsg: Operation not permitted
    64 bytes from py-in-f99.google.com (64.233.167.99): icmp_seq=16 ttl=243 time=152 ms
    ping: sendmsg: Operation not permitted
    64 bytes from py-in-f99.google.com (64.233.167.99): icmp_seq=18 ttl=243 time=151 ms
    ping: sendmsg: Operation not permitted
    ping: sendmsg: Operation not permitted
    This happens for any site, not just google.

    The server runs mysql, php & apache and is quite busy.

    I think this may be causing connection errors for users also (including myself) so am quite keen to get this sorted, although the load is under 1 so I am suprised by the un-responsiveness of the pages served, and the pings going out!

    Thanks.


    edit: Ok so I stopped the httpd service and it pings ok now.
    So this error must occour becuase Apache is overloading the server?

    Can anybody give me any advice as how to stop this happening? Maybe increase the amount of active connections?
    Ta.
     
    Last edited: Mar 20, 2008
  2. falko

    falko Super Moderator Howtoforge Staff

    Are there any errors in Apache's error log?
     
  3. topdog

    topdog Active Member

    That is typically a networking problem, that is when the kernel refuses to transmit the packet, am suspecting congestion on the wire.
     
  4. gAnDo

    gAnDo New Member

    None, well apart from some 'file does not exist' errors caused by people not updating their bookmarks.

    Sounds fair, if this is the case what would my options be?
    And how can I test this is the case?

    edit: Do you mean my hosts bandwidth? Or my server?

    Thanks again for the help guys.
     
  5. falko

    falko Super Moderator Howtoforge Staff

  6. gAnDo

    gAnDo New Member

    Thanks, I use APF. I will read through the links and post back :)
     
  7. gAnDo

    gAnDo New Member

    One of the links referring to memchached I dont use (one of these days Il get my head round it and use it tho!) eaccelarator does me for now.

    Looking at this link:
    http://www.netfilter.org/documentation/FAQ/netfilter-faq-3.html#ss3.7

    It says to increase the number of connectinos.... but my ip_conntrack_max is already set to 34576.
    Is it ok to up this?

    Cheers again ;)
     
  8. falko

    falko Super Moderator Howtoforge Staff

    Have you tried to disable APF to find out if it really is the firewall that's causing the problem?
     
  9. gAnDo

    gAnDo New Member

    Tried disabling APF with no success, still the same issue.
    Stopped mysqld service (didnt think would make a difference but thought I better try) and again, no difference.

    Stopped httpd service, and voila, no more ping problems. So it def seems like its the number of connections coming from Apache that are causing the issue.

    Im going to look into apache optimization for now but if you have any quick tips I would apreciate it!

    Also see my top output
    Code:
    top - 14:06:23 up 38 days, 13:18,  1 user,  load average: 1.48, 1.89, 1.46
    Tasks: 154 total,   1 running, 153 sleeping,   0 stopped,   0 zombie
    Cpu(s): 31.9%us, 10.3%sy,  0.0%ni, 49.5%id,  3.0%wa,  0.0%hi,  5.3%si,  0.0%st
    Mem:   1994020k total,  1562692k used,   431328k free,   110104k buffers
    Swap:  2031608k total,       96k used,  2031512k free,  1040928k cached
    
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    10388 mysql     15   0  563m  91m 4076 S 12.3  4.7   0:58.58 mysqld
    10419 root      25   0  291m 9728 5288 S  1.0  0.5   0:00.58 httpd
    11534 apache    15   0  292m 8756 3180 S  1.0  0.4   0:00.04 httpd
    11543 apache    15   0  292m 8692 3180 S  1.0  0.4   0:00.04 httpd
    11118 apache    15   0  294m  12m 5808 S  0.7  0.7   0:00.51 httpd
    11126 apache    16   0  294m  13m 6332 S  0.7  0.7   0:00.56 httpd
    11129 apache    15   0  292m 9024 3400 S  0.7  0.5   0:00.45 httpd
    11172 apache    15   0  293m  11m 5416 S  0.7  0.6   0:00.24 httpd
    11176 apache    15   0  293m 9140 3396 D  0.7  0.5   0:00.23 httpd
    11185 apache    15   0  292m 9212 3528 S  0.7  0.5   0:00.25 httpd
    11193 apache    15   0  294m  13m 6380 S  0.7  0.7   0:00.29 httpd
    11197 apache    15   0  294m  12m 5716 S  0.7  0.7   0:00.29 httpd
    11198 apache    15   0  294m  13m 6332 S  0.7  0.7   0:00.30 httpd
    11217 apache    15   0  292m 8908 3260 S  0.7  0.4   0:00.24 httpd
    11221 apache    15   0  292m 8904 3312 S  0.7  0.4   0:00.24 httpd
    11527 apache    15   0  292m 8764 3180 S  0.7  0.4   0:00.06 httpd
    11530 apache    15   0  292m 8736 3180 S  0.7  0.4   0:00.04 httpd
    11533 apache    15   0  292m 8720 3172 S  0.7  0.4   0:00.05 httpd
    11536 apache    15   0  292m 8712 3164 S  0.7  0.4   0:00.05 httpd
    11538 apache    15   0  292m 8784 3164 S  0.7  0.4   0:00.03 httpd
    11541 apache    15   0  292m 8720 3164 S  0.7  0.4   0:00.05 httpd
    11544 apache    15   0  292m 8740 3188 S  0.7  0.4   0:00.02 httpd
    11547 apache    16   0  292m 8708 3164 D  0.7  0.4   0:00.02 httpd
    11548 apache    15   0  292m 8700 3164 S  0.7  0.4   0:00.02 httpd
    11553 apache    15   0  292m 8664 3164 S  0.7  0.4   0:00.02 httpd
    11092 root      15   0 12716 1104  788 R  0.3  0.1   0:00.54 top
    11158 apache    15   0  294m  13m 6380 S  0.3  0.7   0:00.50 httpd
    11186 apache    15   0  292m 8896 3232 S  0.3  0.4   0:00.25 httpd
    11187 apache    15   0  292m 9488 3700 S  0.3  0.5   0:00.26 httpd
    11205 apache    15   0  292m 8896 3220 S  0.3  0.4   0:00.23 httpd
    11229 apache    15   0  294m  13m 5896 S  0.3  0.7   0:00.28 httpd
    11528 apache    16   0  292m 8784 3196 S  0.3  0.4   0:00.03 httpd
    11531 apache    15   0  292m 8712 3212 S  0.3  0.4   0:00.02 httpd
    11535 apache    15   0  292m 8740 3176 S  0.3  0.4   0:00.04 httpd
    11537 apache    15   0  294m  12m 5260 S  0.3  0.6   0:00.06 httpd
    11540 apache    15   0  292m 8732 3180 S  0.3  0.4   0:00.03 httpd
    The number of httpd connections can vary dramatically from 1 to all that you can see above, which shoots up the amount of processor being used.
    Should this be happening?
     
  10. gAnDo

    gAnDo New Member

    Ok so I used this guide:
    http://www.howtoforge.com/configuring_apache_for_maximum_performance

    And it seems to have helped server speed but I am still getting the 'operation not permitted' errors when pinging out.

    I changed the following
    Code:
    KeepAlive Off
    KeepAliveTimeout 15
    <IfModule prefork.c>
    StartServers       8
    MinSpareServers    5
    MaxSpareServers   20
    ServerLimit      256
    MaxClients       256
    MaxRequestsPerChild  4000
    </IfModule>
    to
    Code:
    KeepAlive On
    KeepAliveTimeout 5
    <IfModule prefork.c>
    StartServers       20
    MinSpareServers    30
    MaxSpareServers   50
    ServerLimit      256
    MaxClients       256
    MaxRequestsPerChild  3000
    </IfModule>
    Do you think I should increase the ServerLimit & MaxClients variables too?

    Next step I guess is to work out what modules I need to keep enabled (all default modules are still being loaded). More reading for me I guess!
     
  11. falko

    falko Super Moderator Howtoforge Staff

    Please try this:
    MaxKeepAliveRequests 1000000
    KeepAliveTimeout 4
    MaxRequestsPerChild 1000000
     
  12. gAnDo

    gAnDo New Member

    Tried that but still had the same problem.
    I also noticed increased load and memory usage and had to revert back to the original settings.

    Could there be some max connections limit in CENTOS?

    If found these links which although are way over the limit I need they might give me some idea. I will read through them and see if I can find any tips.

    http://www.stdlib.net/~colmmacc/Apachecon-EU2005/scaling-apache-presentation.pdf
    http://www.stdlib.net/~colmmacc/Apachecon-EU2005/scaling-apache-handout.pdf

    Cheers ;)
     

Share This Page