Hi, I have been running 4 servers on VMs (2 DNS, 1 web, 1 mail server), which I have been running untouched and very reliable for months now with ISPconfig perfect setups on Centos 5.4 (all of them). Recently, I have been experiencing very slow web surfing which I share BW with my servers. I checked to find on my vm console that all my servers have been using all of my Upload BandWidth (barely any download BW). I don't know what is being uploaded and to whom, but all 4 servers are using equal amounts of BW i.e. 25% each. The duration of max BW can last for a couple of hours at Max then momentary lays idle for a few minutes and max out again. CPU usage on all servers (all single core) is under 25%. My yum update times out during these max BW periods. Any one has experienced this? BTW if I switch off my DNSs the other servers BW goes idle. Questions 1. Have my severs been compromised? ->how to check? 2. Or is this some external BW attack? -> spiders? 3. Or its normal - just set somthing wrongly? My linux skills are at novice level so please keep advice simple.
Possible culprit for abnormally high upload usage Hi, Further my last post, I notice from the 'top' command that I notice that a process command call 'named' has been running when upload BW is being taken once the process stopped the BW when back down to normal. Under the User column (and the command column) the process is also label 'named'. Now this process seem to start and stop on its own accord for periods of 10-15mins max then 2-3 pause then starts itself again. Have my servers been hacked? can I kill this process? if so how?
OK I killed the named process which solves the problem but do I need this process hi, I have used the 'service named stop' command, which resolves the upload BW problem. I have also used 'chkconfig named off' command to prevent autostart of the service upon reboot. May I ask what is the 'named' service for or does, is there any consequence to this action. Anyone know why could the named service be taking BW, who would it check with on the internet. Is this a known bug? is there a workaround. I don't want to unknowingly lose functionality due to my above actions which solves one problem but may create another. Advice anyone?
Named is BIND, DNS software. If you start named and then start to monitor log files, it should show startup of named services. There you should see possibles errors that causes your server high load.
My message log files Hi here are the relevent portion of my log files in which I found these two servers having virtually synchronised network usage profile. I switch my others off the this period so I know its only these two communicating. BW usage was busy from 16:54 till 17:23 pause idle and busy again 17:35 till 18:30 ns1.mydomain.co.uk : xxx.xxx.xxx.200 -> Primary DNS post.mydomain.co.uk : xxx.xxx.xxx.207 -> Mail server ns1 message logs (boot info removed) ----------------------------------------------------------------------- Jun 15 16:54:20 ns1 named[2219]: shutting down: flushing changes Jun 15 16:54:20 ns1 named[2219]: stopping command channel on 127.0.0.1#953 Jun 15 16:54:20 ns1 named[2219]: stopping command channel on ::1#953 Jun 15 16:54:20 ns1 named[2219]: no longer listening on 127.0.0.1#53 Jun 15 16:54:20 ns1 named[2219]: no longer listening on xxx.xxx.xxx.200#53 Jun 15 16:54:20 ns1 named[2219]: exiting Jun 15 16:54:22 ns1 named[3327]: starting BIND 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 -u named -t /var/named/chroot Jun 15 16:54:22 ns1 named[3327]: adjusted limit on open files from 1024 to 1048576 Jun 15 16:54:22 ns1 named[3327]: found 1 CPU, using 1 worker thread Jun 15 16:54:22 ns1 named[3327]: using up to 4096 sockets Jun 15 16:54:22 ns1 named[3327]: loading configuration from '/etc/named.conf' Jun 15 16:54:22 ns1 named[3327]: using default UDP/IPv4 port range: [1024, 65535] Jun 15 16:54:22 ns1 named[3327]: using default UDP/IPv6 port range: [1024, 65535] Jun 15 16:54:22 ns1 named[3327]: listening on IPv4 interface lo, 127.0.0.1#53 Jun 15 16:54:22 ns1 named[3327]: listening on IPv4 interface eth0, xxx.xxx.xxx.200#53 Jun 15 16:54:22 ns1 named[3327]: command channel listening on 127.0.0.1#953 Jun 15 16:54:22 ns1 named[3327]: command channel listening on ::1#953 Jun 15 16:54:22 ns1 named[3327]: the working directory is not writable Jun 15 16:54:22 ns1 named[3327]: zone 0.0.127.in-addr.arpa/IN: loaded serial 1997022700 Jun 15 16:54:22 ns1 named[3327]: zone xxx.xxx.xxx.in-addr.arpa/IN: loaded serial 2010061503 Jun 15 16:54:22 ns1 named[3327]: zone mydomain.co.uk/IN: loaded serial 2010061503 Jun 15 16:54:22 ns1 named[3327]: running Jun 15 16:54:22 ns1 named[3327]: zone xxx.xxx.xxx.in-addr.arpa/IN: sending notifies (serial 2010061503) Jun 15 16:54:22 ns1 named[3327]: zone mydomain.co.uk/IN: sending notifies (serial 2010061503) Jun 15 16:54:22 ns1 proftpd[2729]: ns1.mydomain.co.uk - ProFTPD killed (signal 15) Jun 15 16:54:22 ns1 proftpd[2729]: ns1.mydomain.co.uk - ProFTPD 1.3.1 standalone mode SHUTDOWN Jun 15 16:54:22 ns1 named[3327]: network unreachable resolving 'isc.org/ANY/IN': 2001:500:e::1#53 Jun 15 16:54:22 ns1 named[3327]: network unreachable resolving 'isc.org/ANY/IN': 2001:500:b::1#53 Jun 15 16:54:22 ns1 named[3327]: network unreachable resolving 'isc.org/ANY/IN': 2001:500:f::1#53 Jun 15 16:54:22 ns1 named[3327]: network unreachable resolving 'isc.org/ANY/IN': 2001:500:c::1#53 Jun 15 16:54:23 ns1 proftpd[3352]: ns1.mydomain.co.uk - ProFTPD 1.3.1 (stable) (built Thu Jan 17 01:18:40 GMT 2008) standalone mode STARTUP Jun 15 16:54:23 ns1 named[3327]: network unreachable resolving 'ns.isc.afilias-nst.info/A/IN': 2001:500:8::79#53 Jun 15 16:54:23 ns1 smartd[3372]: smartd version 5.38 [i686-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen Jun 15 16:54:23 ns1 smartd[3372]: Home page is http://smartmontools.sourceforge.net/ Jun 15 16:54:23 ns1 smartd[3372]: Opened configuration file /etc/smartd.conf Jun 15 16:54:23 ns1 smartd[3372]: Configuration file /etc/smartd.conf parsed. Jun 15 16:54:23 ns1 smartd[3372]: Device: /dev/sda, opened Jun 15 16:54:23 ns1 smartd[3372]: Device: /dev/sda, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sda' to turn on SMART features Jun 15 16:54:23 ns1 smartd[3372]: Unable to register SCSI device /dev/sda at line 32 of file /etc/smartd.conf Jun 15 16:54:23 ns1 smartd[3372]: Device /dev/sda not available Jun 15 16:54:23 ns1 smartd[3372]: Monitoring 0 ATA and 0 SCSI devices Jun 15 16:54:23 ns1 smartd[3374]: smartd has fork()ed into background mode. New PID=3374. Jun 15 16:54:32 ns1 named[3327]: client xxx.xxx.xxx.207#45453: transfer of 'mydomain.co.uk/IN': AXFR started Jun 15 16:54:32 ns1 named[3327]: client xxx.xxx.xxx.207#45453: transfer of 'mydomain.co.uk/IN': AXFR ended Jun 15 16:54:56 ns1 ntpd[2590]: frequency initialized 47.565 PPM from /var/lib/ntp/drift Jun 15 16:55:24 ns1 named[3327]: client xxx.xxx.xxx.207#34628: transfer of 'mydomain.co.uk/IN': AXFR started Jun 15 16:55:24 ns1 named[3327]: client xxx.xxx.xxx.207#34628: transfer of 'mydomain.co.uk/IN': AXFR ended Jun 15 16:57:19 ns1 named[3327]: client xxx.xxx.xxx.207#45505: transfer of 'mydomain.co.uk/IN': AXFR started Jun 15 16:57:19 ns1 named[3327]: client xxx.xxx.xxx.207#45505: transfer of 'mydomain.co.uk/IN': AXFR ended Jun 15 16:58:15 ns1 ntpd[2590]: synchronized to LOCAL(0), stratum 10 Jun 15 16:58:15 ns1 ntpd[2590]: kernel time sync enabled 0001 Jun 15 17:00:02 ns1 proftpd[3550]: ns1.mydomain.co.uk (ns1.mydomain.co.uk[127.0.0.1]) - FTP session opened. Jun 15 17:00:02 ns1 proftpd[3550]: ns1.mydomain.co.uk (ns1.mydomain.co.uk[127.0.0.1]) - FTP session closed. Jun 15 17:01:19 ns1 named[3327]: client xxx.xxx.xxx.207#37495: transfer of 'mydomain.co.uk/IN': AXFR started Jun 15 17:01:19 ns1 named[3327]: client xxx.xxx.xxx.207#37495: transfer of 'mydomain.co.uk/IN': AXFR ended Jun 15 17:02:34 ns1 ntpd[2590]: synchronized to 77.78.110.71, stratum 2 Jun 15 17:08:22 ns1 named[3327]: client xxx.xxx.xxx.207#36186: transfer of 'mydomain.co.uk/IN': AXFR started Jun 15 17:08:22 ns1 named[3327]: client xxx.xxx.xxx.207#36186: transfer of 'mydomain.co.uk/IN': AXFR ended Jun 15 17:21:42 ns1 proftpd[3936]: ns1.mydomain.co.uk (ns1.mydomain.co.uk[127.0.0.1]) - FTP session opened. Jun 15 17:21:42 ns1 proftpd[3936]: ns1.mydomain.co.uk (ns1.mydomain.co.uk[127.0.0.1]) - FTP session closed. Jun 15 17:21:54 ns1 ntpd[2590]: time reset +0.228169 s Jun 15 17:23:05 ns1 named[3327]: client xxx.xxx.xxx.207#55279: transfer of 'mydomain.co.uk/IN': AXFR started Jun 15 17:23:05 ns1 named[3327]: client xxx.xxx.xxx.207#55279: transfer of 'mydomain.co.uk/IN': AXFR ended Jun 15 17:25:10 ns1 ntpd[2590]: synchronized to LOCAL(0), stratum 10 Jun 15 17:27:21 ns1 ntpd[2590]: synchronized to 77.78.110.71, stratum 2 Jun 15 17:30:01 ns1 proftpd[4141]: ns1.mydomain.co.uk (ns1.mydomain.co.uk[127.0.0.1]) - FTP session opened. Jun 15 17:30:01 ns1 proftpd[4141]: ns1.mydomain.co.uk (ns1.mydomain.co.uk[127.0.0.1]) - FTP session closed. Jun 15 17:49:19 ns1 named[3327]: client xxx.xxx.xxx.207#56062: transfer of 'mydomain.co.uk/IN': AXFR started Jun 15 17:49:19 ns1 named[3327]: client xxx.xxx.xxx.207#56062: transfer of 'mydomain.co.uk/IN': AXFR ended Jun 15 18:00:01 ns1 proftpd[4583]: ns1.mydomain.co.uk (ns1.mydomain.co.uk[127.0.0.1]) - FTP session opened. Jun 15 18:00:01 ns1 proftpd[4583]: ns1.mydomain.co.uk (ns1.mydomain.co.uk[127.0.0.1]) - FTP session closed. -------------------------------------------------------------------- post.mydomain.co.uk message log (boot info removed) ------------------------------------------------------------------------- Jun 15 16:54:32 post named[3285]: starting BIND 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 -u named -t /var/named/chroot Jun 15 16:54:32 post named[3285]: adjusted limit on open files from 1024 to 1048576 Jun 15 16:54:32 post named[3285]: found 1 CPU, using 1 worker thread Jun 15 16:54:32 post named[3285]: using up to 4096 sockets Jun 15 16:54:32 post named[3285]: loading configuration from '/etc/named.conf' Jun 15 16:54:32 post named[3285]: using default UDP/IPv4 port range: [1024, 65535] Jun 15 16:54:32 post named[3285]: using default UDP/IPv6 port range: [1024, 65535] Jun 15 16:54:32 post named[3285]: listening on IPv4 interface lo, 127.0.0.1#53 Jun 15 16:54:32 post named[3285]: listening on IPv4 interface eth0, xxx.xxx.xxx.207#53 Jun 15 16:54:32 post named[3285]: command channel listening on 127.0.0.1#953 Jun 15 16:54:32 post named[3285]: command channel listening on ::1#953 Jun 15 16:54:32 post named[3285]: the working directory is not writable Jun 15 16:54:32 post named[3285]: zone 0.0.127.in-addr.arpa/IN: loaded serial 1997022700 Jun 15 16:54:32 post named[3285]: zone mydomain.co.uk/IN: loaded serial 2008070603 Jun 15 16:54:32 post named[3285]: running Jun 15 16:54:32 post named[3285]: zone mydomain.co.uk/IN: expired Jun 15 16:54:32 post named[3285]: zone mydomain.co.uk/IN: Transfer started. Jun 15 16:54:32 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: connected using xxx.xxx.xxx.207#45453 Jun 15 16:54:32 post named[3285]: dumping master file: tmp-ninH7UJ0uS: open: permission denied Jun 15 16:54:32 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: failed while receiving responses: permission denied Jun 15 16:54:32 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: end of transfer Jun 15 16:54:32 post proftpd[2681]: post.mydomain.co.uk - ProFTPD killed (signal 15) Jun 15 16:54:32 post proftpd[2681]: post.mydomain.co.uk - ProFTPD 1.3.1 standalone mode SHUTDOWN Jun 15 16:54:32 post proftpd[3310]: post.mydomain.co.uk - ProFTPD 1.3.1 (stable) (built Mon Jan 14 18:23:40 GMT 2008) standalone mode STARTUP Jun 15 16:54:33 post smartd[3330]: smartd version 5.38 [i686-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen Jun 15 16:54:33 post smartd[3330]: Home page is http://smartmontools.sourceforge.net/ Jun 15 16:54:33 post smartd[3330]: Opened configuration file /etc/smartd.conf Jun 15 16:54:33 post smartd[3330]: Configuration file /etc/smartd.conf parsed. Jun 15 16:54:33 post smartd[3330]: Device: /dev/sda, opened Jun 15 16:54:33 post smartd[3330]: Device: /dev/sda, IE (SMART) not enabled, skip device Try 'smartctl -s on /dev/sda' to turn on SMART features Jun 15 16:54:33 post smartd[3330]: Unable to register SCSI device /dev/sda at line 32 of file /etc/smartd.conf Jun 15 16:54:33 post smartd[3330]: Device /dev/sda not available Jun 15 16:54:33 post smartd[3330]: Monitoring 0 ATA and 0 SCSI devices Jun 15 16:54:33 post smartd[3332]: smartd has fork()ed into background mode. New PID=3332. Jun 15 16:54:35 post named[3285]: network unreachable resolving 'ns.isc.afilias-nst.info/AAAA/IN': 2001:500:1b::1#53 Jun 15 16:54:35 post named[3285]: network unreachable resolving 'ns.isc.afilias-nst.info/AAAA/IN': 2001:500:41::1#53 Jun 15 16:54:35 post named[3285]: network unreachable resolving 'ns.isc.afilias-nst.info/A/IN': 2001:500:1b::1#53 Jun 15 16:54:35 post named[3285]: network unreachable resolving 'ns.isc.afilias-nst.info/A/IN': 2001:500:41::1#53 Jun 15 16:54:37 post named[3285]: network unreachable resolving 'ns.isc.afilias-nst.info/AAAA/IN': 2001:500:1c::1#53 Jun 15 16:54:37 post named[3285]: network unreachable resolving 'ns.isc.afilias-nst.info/AAAA/IN': 2001:500:19::1#53 Jun 15 16:54:37 post named[3285]: network unreachable resolving 'ns.isc.afilias-nst.info/AAAA/IN': 2001:500:49::1#53 Jun 15 16:54:41 post named[3285]: network unreachable resolving 'ns.isc.afilias-nst.info/AAAA/IN': 2001:500:a::79#53 Jun 15 16:54:41 post named[3285]: network unreachable resolving 'ns.isc.afilias-nst.info/AAAA/IN': 2001:500:6::79#53 Jun 15 16:54:41 post named[3285]: network unreachable resolving 'ns.isc.afilias-nst.info/AAAA/IN': 2001:500:8::79#53 Jun 15 16:54:41 post named[3285]: network unreachable resolving 'ns.isc.afilias-nst.info/AAAA/IN': 2001:500:7::79#53 Jun 15 16:55:24 post named[3285]: zone mydomain.co.uk/IN: Transfer started. Jun 15 16:55:24 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: connected using xxx.xxx.xxx.207#34628 Jun 15 16:55:24 post named[3285]: dumping master file: tmp-nnS7r32Djy: open: permission denied Jun 15 16:55:24 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: failed while receiving responses: permission denied Jun 15 16:55:24 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: end of transfer Jun 15 16:57:19 post named[3285]: zone mydomain.co.uk/IN: Transfer started. Jun 15 16:57:19 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: connected using xxx.xxx.xxx.207#45505 Jun 15 16:57:19 post named[3285]: dumping master file: tmp-DjRwhhhsaG: open: permission denied Jun 15 16:57:19 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: failed while receiving responses: permission denied Jun 15 16:57:19 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: end of transfer Jun 15 16:57:42 post ntpd[2552]: synchronized to LOCAL(0), stratum 10 Jun 15 16:57:42 post ntpd[2552]: kernel time sync enabled 0001 Jun 15 17:00:01 post proftpd[3517]: post.mydomain.co.uk (post.mydomain.co.uk[127.0.0.1]) - FTP session opened. Jun 15 17:00:01 post proftpd[3517]: post.mydomain.co.uk (post.mydomain.co.uk[127.0.0.1]) - FTP session closed. Jun 15 17:01:19 post named[3285]: zone mydomain.co.uk/IN: Transfer started. Jun 15 17:01:19 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: connected using xxx.xxx.xxx.207#37495 Jun 15 17:01:19 post named[3285]: dumping master file: tmp-4fbWQFDG4O: open: permission denied Jun 15 17:01:19 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: failed while receiving responses: permission denied Jun 15 17:01:19 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: end of transfer Jun 15 17:07:17 post ntpd[2552]: synchronized to 91.121.79.101, stratum 3 Jun 15 17:08:22 post named[3285]: zone mydomain.co.uk/IN: Transfer started. Jun 15 17:08:22 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: connected using xxx.xxx.xxx.207#36186 Jun 15 17:08:22 post named[3285]: dumping master file: tmp-PrHIjW6ysZ: open: permission denied Jun 15 17:08:22 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: failed while receiving responses: permission denied Jun 15 17:08:22 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: end of transfer Jun 15 17:15:48 post ntpd[2552]: synchronized to 77.78.110.71, stratum 2 Jun 15 17:20:34 post proftpd[4237]: post.mydomain.co.uk (post.mydomain.co.uk[127.0.0.1]) - FTP session opened. Jun 15 17:20:34 post proftpd[4237]: post.mydomain.co.uk (post.mydomain.co.uk[127.0.0.1]) - FTP session closed. Jun 15 17:20:46 post proftpd[4254]: post.mydomain.co.uk (post.mydomain.co.uk[127.0.0.1]) - FTP session opened. Jun 15 17:20:46 post proftpd[4254]: post.mydomain.co.uk (post.mydomain.co.uk[127.0.0.1]) - FTP session closed. Jun 15 17:20:47 post named[3285]: client 66.90.97.17#25345: error sending response: host unreachable Jun 15 17:20:47 post last message repeated 8 times Jun 15 17:23:05 post named[3285]: zone mydomain.co.uk/IN: Transfer started. Jun 15 17:23:05 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: connected using xxx.xxx.xxx.207#55279 Jun 15 17:23:05 post named[3285]: dumping master file: tmp-By62Si8gKk: open: permission denied Jun 15 17:23:05 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: failed while receiving responses: permission denied Jun 15 17:23:05 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: end of transfer Jun 15 17:25:25 post ntpd[2552]: time reset +0.345180 s Jun 15 17:28:56 post ntpd[2552]: synchronized to LOCAL(0), stratum 10 Jun 15 17:30:00 post ntpd[2552]: synchronized to 77.78.110.71, stratum 2 Jun 15 17:30:01 post proftpd[4712]: post.mydomain.co.uk (post.mydomain.co.uk[127.0.0.1]) - FTP session opened. Jun 15 17:30:01 post proftpd[4712]: post.mydomain.co.uk (post.mydomain.co.uk[127.0.0.1]) - FTP session closed. Jun 15 17:49:19 post named[3285]: zone mydomain.co.uk/IN: Transfer started. Jun 15 17:49:19 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: connected using xxx.xxx.xxx.207#56062 Jun 15 17:49:19 post named[3285]: dumping master file: tmp-YoCd8pZy3C: open: permission denied Jun 15 17:49:19 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: failed while receiving responses: permission denied Jun 15 17:49:19 post named[3285]: transfer of 'mydomain.co.uk/IN' from xxx.xxx.xxx.200#53: end of transfer Jun 15 18:00:01 post proftpd[5418]: post.mydomain.co.uk (post.mydomain.co.uk[127.0.0.1]) - FTP session opened. Jun 15 18:00:01 post proftpd[5418]: post.mydomain.co.uk (post.mydomain.co.uk[127.0.0.1]) - FTP session closed. ------------------------------------------------------------------------- Hope you can help guys. please advise if you need more info.