Hi, We are using HAproxy for http traffic and around 60-80 servers in rotations We are getting network errors on backend hosts. We are not getting any logs in dmesg and messages. Please refer the netstat and ifconfig outputs as follow on backend servers.. Please let us know if we need to change any kernel settings. We have upgraded network drivers but issue still there. Operating system- CentOS release 5.4 (Final) Kernel - 2.6.18-274 ==================================================== ifconfig RX packets:97198166 errors:3104 dropped:11 overruns:0 frame:3104 TX packets:27200159 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX packets:110829843 errors:5 dropped:14 overruns:0 frame:5 TX packets:132306261 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 ========================================= [root@host1]# netstat -s Ip: 54308864 total packets received 688 with invalid addresses 0 forwarded 0 incoming packets discarded 53901555 incoming packets delivered 64669990 requests sent out 6 dropped because of missing route Icmp: 9997 ICMP messages received 3 input ICMP message failed. ICMP input histogram: destination unreachable: 163 echo requests: 9828 echo replies: 6 10554 ICMP messages sent 0 ICMP messages failed ICMP output histogram: destination unreachable: 720 echo request: 6 echo replies: 9828 IcmpMsg: InType0: 6 InType3: 163 InType8: 9828 OutType0: 9828 OutType3: 720 OutType8: 6 Tcp: 6581507 active connections openings 1607052 passive connection openings 3526 failed connection attempts 15216 connection resets received 2 connections established 52338714 segments received 62884099 segments send out 10711 segments retransmited 0 bad segments received. 6295 resets sent Udp: 1550750 packets received 698 packets to unknown port received. 0 packet receive errors 1764626 packets sent TcpExt: 169 resets received for embryonic SYN_RECV sockets 188 TCP sockets finished time wait in fast timer 28 time wait sockets recycled by time stamp 7987454 TCP sockets finished time wait in slow timer 1 packets rejects in established connections because of timestamp 471256 delayed acks sent 1277 delayed acks further delayed because of locked socket Quick ack mode was activated 2550 times 1164815 packets directly queued to recvmsg prequeue. 2419208 packets directly received from backlog 1339602178 packets directly received from prequeue 16421204 packets header predicted 411964 packets header predicted and directly queued to user 12740664 acknowledgments not containing data received 7724081 predicted acknowledgments 341 times recovered from packet loss due to SACK data Detected reordering 1 times using FACK Detected reordering 5 times using SACK Detected reordering 2 times using time stamp 4 congestion windows fully recovered 53 congestion windows partially recovered using Hoe heuristic 756 congestion windows recovered after partial ack 1186 TCP data loss events TCPLostRetransmit: 2 643 timeouts after SACK recovery 98 timeouts in loss state 1091 fast retransmits 108 forward retransmits 1719 retransmits in slow start 6094 other TCP timeouts 87 sack retransmits failed 2 times receiver scheduled too late for direct processing 2550 DSACKs sent for old packets 496 DSACKs received 365 connections reset due to unexpected data 213 connections aborted due to timeout IpExt: InBcastPkts: 1396 ++++++++++++++++++++++++++ Proto Recv-Q Send-Q State PID/Program name tcp 0 4365 ESTABLISHED 11352/httpd tcp 0 4365 ESTABLISHED 19753/httpd tcp 0 4365 ESTABLISHED - tcp 0 4374 FIN_WAIT1 14063/httpd tcp 0 4435 ESTABLISHED 6652/httpd tcp 0 4505 FIN_WAIT1 3232/httpd tcp 0 4531 FIN_WAIT1 1917/httpd tcp 0 4532 FIN_WAIT1 15552/httpd tcp 0 4611 FIN_WAIT1 16370/httpd tcp 0 4788 ESTABLISHED 5170/httpd tcp 0 4818 FIN_WAIT1 15557/httpd tcp 0 5026 ESTABLISHED 5494/httpd tcp 0 5130 ESTABLISHED - tcp 0 5497 FIN_WAIT1 12466/httpd tcp 0 5645 ESTABLISHED 22611/httpd tcp 0 5773 ESTABLISHED 28445/httpd tcp 0 5841 ESTABLISHED 1653/httpd tcp 0 6010 ESTABLISHED 22245/httpd tcp 0 6228 ESTABLISHED 23534/httpd tcp 0 6392 FIN_WAIT1 24441/httpd tcp 0 6429 ESTABLISHED 1917/httpd tcp 0 6457 FIN_WAIT1 23745/httpd tcp 0 6922 FIN_WAIT1 6649/httpd tcp 0 6988 FIN_WAIT1 14158/httpd tcp 0 7256 ESTABLISHED 29844/httpd tcp 0 7560 FIN_WAIT1 26647/httpd tcp 0 7619 FIN_WAIT1 12465/httpd tcp 0 7742 ESTABLISHED 26388/httpd tcp 0 7922 ESTABLISHED 14985/httpd tcp 0 7922 ESTABLISHED 14158/httpd tcp 0 8529 ESTABLISHED 12462/httpd tcp 0 8569 FIN_WAIT1 23170/httpd tcp 0 8630 ESTABLISHED 20565/httpd tcp 0 8694 ESTABLISHED 27032/httpd tcp 0 8705 ESTABLISHED 2373/httpd tcp 0 8772 ESTABLISHED - tcp 0 8779 ESTABLISHED 6649/httpd tcp 0 8902 ESTABLISHED 14334/httpd tcp 0 9028 FIN_WAIT1 12118/httpd tcp 0 9183 FIN_WAIT1 14985/httpd tcp 0 9982 FIN_WAIT1 16363/httpd tcp 0 10566 FIN_WAIT1 25911/httpd tcp 0 16432 ESTABLISHED 31379/sshd: prod [p tcp 0 16432 ESTABLISHED 22215/sshd: prod [p tcp 0 25125 ESTABLISHED 30203/bash tcp 0 25346 FIN_WAIT1 - ======================================================== Present Kernel settings- net.ipv4.tcp_syncookies = 0 net.ipv4.tcp_keepalive_time = 600 net.ipv4.tcp_synack_retries = 3 net.ipv4.tcp_syn_retries = 3 net.ipv4.conf.all.log_martians = 1 net.ipv4.inet_peer_gc_mintime = 5 net.ipv4.tcp_ecn = 0 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_sack = 1 net.ipv4.tcp_fack = 1 net.ipv4.tcp_dsack = 1 net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 0 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_orphan_retries = 1 net.ipv4.tcp_fin_timeout = 20 net.ipv4.tcp_no_metrics_save = 1 net.ipv4.tcp_moderate_rcvbuf = 1 net.ipv4.tcp_rfc1337 = 1 -thanks in Adv
