packet loss

Discussion in 'Server Operation' started by Chad, Jun 11, 2008.

  1. Chad

    Chad New Member

    root@server [~]# tail -f /var/log/messages
    Jun 10 14:14:49 server kernel: printk: 56 messages suppressed.
    Jun 10 14:14:49 server kernel: ip_conntrack: table full, dropping packet.
    Jun 10 14:14:54 server kernel: printk: 59 messages suppressed.
    Jun 10 14:14:54 server kernel: ip_conntrack: table full, dropping packet.
    Jun 10 14:14:59 server kernel: printk: 85 messages suppressed.
    Jun 10 14:14:59 server kernel: ip_conntrack: table full, dropping packet.
    Jun 10 14:15:04 server kernel: printk: 90 messages suppressed.
    Jun 10 14:15:04 server kernel: ip_conntrack: table full, dropping packet.
    Jun 10 14:15:09 server kernel: printk: 58 messages suppressed.
    Jun 10 14:15:09 server kernel: ip_conntrack: table full, dropping packet.
    Jun 10 14:15:14 server kernel: printk: 70 messages suppressed.
    Jun 10 14:15:14 server kernel: ip_conntrack: table full, dropping packet.
    Jun 10 14:15:19 server kernel: printk: 193 messages suppressed.
    Jun 10 14:15:19 server kernel: ip_conntrack: table full, dropping packet.

    Anyone know what this is about?

    Using Centos / Cpanel

    Linux server.domain.com 2.6.9-67.0.15.ELsmp #1 SMP Thu May 8 10:52:19 EDT 2008 i686 i686 i386 GNU/Linux

    root@server [~]# sysctl net.ipv4.netfilter.ip_conntrack_max
    net.ipv4.netfilter.ip_conntrack_max = 65536

    I ran this in the meantime.

    sysctl -w net.ipv4.netfilter.ip_conntrack_max=72000

    The rate is continually rising

    root@server [~]# wc -l /proc/net/ip_conntrack
    65882 /proc/net/ip_conntrack
     

    Attached Files:

    • 1.jpg
      File size:
      23.1 KB
      Views:
      2,123
  2. Chad

    Chad New Member

    Still having problems


    Code:
    root@server [~]# ps -auxf |grep httpd
    Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.3/FAQ
    root     21816  0.0  0.0  4756  724 pts/1    S+   14:05   0:00                      \_ grep httpd
    root     19702  0.0  0.3 46800 14580 ?       SNs  13:53   0:00 /usr/local/apache/bin/httpd -k start -DSSL
    root     19709  0.0  0.1 22804 7784 ?        SN   13:53   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   20418  0.6  0.4 48184 18908 ?       SN   14:01   0:01  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   20420  0.2  0.4 48292 17792 ?       SN   14:01   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   20427  0.3  0.4 48296 18192 ?       SN   14:01   0:01  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21663  0.6  0.5 48728 21880 ?       SN   14:02   0:01  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21667  0.2  0.4 48216 19304 ?       SN   14:02   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21668  0.1  0.3 47828 15048 ?       SN   14:02   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21669  0.4  0.4 48208 17756 ?       SN   14:02   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21674  0.1  0.3 47948 15904 ?       SN   14:03   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21686  0.4  0.5 48548 21208 ?       SN   14:03   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21688  0.1  0.3 47824 15028 ?       SN   14:03   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21689  0.4  0.5 48604 21996 ?       SN   14:03   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21701  0.2  0.3 47824 15036 ?       SN   14:03   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21702  0.3  0.4 48092 17732 ?       SN   14:03   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21703  0.2  0.3 47824 15520 ?       SN   14:03   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21704  0.2  0.3 47908 15340 ?       SN   14:03   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21705  0.5  0.4 48052 18624 ?       SN   14:03   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21707  0.1  0.3 47824 15036 ?       SN   14:03   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21724  0.6  0.3 47824 15400 ?       SN   14:04   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21732  0.1  0.3 47824 15396 ?       SN   14:04   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21733  0.1  0.3 47832 15380 ?       SN   14:04   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21736  0.1  0.3 47824 15028 ?       SN   14:04   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21738  0.1  0.3 47824 15380 ?       SN   14:04   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21739  0.5  0.4 48320 19424 ?       SN   14:04   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21740  1.2  0.4 48532 20252 ?       SN   14:04   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21741  0.2  0.5 51992 22108 ?       SN   14:04   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21742  0.6  0.3 47832 15384 ?       SN   14:04   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21743  0.1  0.3 47824 15020 ?       SN   14:04   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21744  1.1  0.3 47824 15352 ?       SN   14:04   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21805  0.8  0.4 48312 17692 ?       SN   14:05   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   21814  0.2  0.3 47824 14932 ?       SN   14:05   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    Code:
    root@server [~]# netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr
         14 84.254.189.64
          8 76.229.163.66
          5 74.6.18.251
          4 80.248.176.136
          4 24.106.187.61
          4 216.215.213.86
          4 213.42.21.61
          4 208.27.123.54
          4 
          3 89.231.204.143
          3 68.155.200.108
          3 200.201.164.26
          3 192.156.52.34
          3 166.102.162.250
          2 89.111.228.127
          2 88.89.134.212
          2 87.14.100.98
          2 79.72.133.105
          2 74.6.22.125
          2 72.189.166.163
          2 71.72.140.14
          2 68.16.225.194
          2 67.195.37.94
          2 65.80.23.104
          2 63.64.53.2
          2 63.240.134.230
          2 205.178.190.97
          2 202.1.192.10
          1 servers)
          1 Address
          1 98.211.4.230
          1 98.20.163.65
          1 89.108.2.162
    root@server [~]# wc -l /proc/net/ip_conntrack
    wc: /proc/net/ip_conntrack: No space left on device
    1878 /proc/net/ip_conntrack


    Code:
    root@server [~]# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda3             273G   60G  200G  23% /
    /dev/sda1              99M   51M   43M  55% /boot
    none                  2.0G     0  2.0G   0% /dev/shm
    /dev/sdb1             276G   59G  203G  23% /backup
    /usr/tmpDSK           2.5G   43M  2.3G   2% /tmp
    /tmp                  2.5G   43M  2.3G   2% /var/tmp
    Code:
    root@server [~]# tail -f /var/log/messages
    Jun 11 14:09:35 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:12:3f:20:e0:e8:00:14:f2:c7:f1:80:08:00 SRC=97.102.167.110 DST=147.202.66.19 LEN=80 TOS=0x00 PREC=0x00 TTL=114 ID=2369 PROTO=UDP SPT=500 DPT=500 LEN=60 
    Jun 11 14:09:39 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:12:3f:20:e0:e8:00:14:f2:c7:f1:80:08:00 SRC=97.102.167.110 DST=147.202.66.19 LEN=80 TOS=0x00 PREC=0x00 TTL=114 ID=2418 PROTO=UDP SPT=500 DPT=500 LEN=60 
    Jun 11 14:09:40 server kernel: printk: 150 messages suppressed.
    Jun 11 14:09:40 server kernel: ip_conntrack: table full, dropping packet.
    Jun 11 14:09:45 server kernel: printk: 151 messages suppressed.
    Jun 11 14:09:45 server kernel: ip_conntrack: table full, dropping packet.
    Jun 11 14:09:50 server kernel: printk: 119 messages suppressed.
    Jun 11 14:09:50 server kernel: ip_conntrack: table full, dropping packet.
    Jun 11 14:09:55 server kernel: printk: 163 messages suppressed.
    Jun 11 14:09:55 server kernel: ip_conntrack: table full, dropping packet.
    Jun 11 14:10:00 server kernel: printk: 124 messages suppressed.
    Jun 11 14:10:00 server kernel: ip_conntrack: table full, dropping packet.
    Jun 11 14:10:05 server kernel: printk: 168 messages suppressed.
    Jun 11 14:10:05 server kernel: ip_conntrack: table full, dropping packet.
    
    root@server [~]# uname -a
    Linux server.domain.com 2.6.9-67.0.15.ELsmp #1 SMP Thu May 8 10:52:19 EDT 2008 i686 i686 i386 GNU/Linux


    root@server [~]# wc -l /proc/net/ip_conntrack
    wc: /proc/net/ip_conntrack: No space left on device
    1883 /proc/net/ip_conntrack

    root@server [~]# sysctl net.ipv4.netfilter.ip_conntrack_max
    net.ipv4.netfilter.ip_conntrack_max = 75000
     
  3. falko

    falko Super Moderator Howtoforge Staff

Share This Page