Good day everyone, I own a ISPConfig3 multi server setup, and since i bought a server from OVH.com i've received three "Abuse" from Hetzner, they told me, this is the third time it happen from a netscan, from a third VPS, i've checked all logs but don't find anything inside, did a rootkit check, checked that root login is disabled and changed all password from SSH. What should i do ? I do own a virtualization server in Switzerland and don't get any abuse report from them ! Is there system scr*wed ? EDIT : i've just received my 4th alert from hetzner : Code: ########################################################################## # Netscan detected from host 178.32.***.*** # ########################################################################## time protocol src_ip src_port dest_ip dest_port --------------------------------------------------------------------------- Mon May 27 00:11:18 2013 TCP 178.32.***.*** 80 => 78.46.119.133 1234
Hello - your post is interesting as I am affected from the other side! See the destination address - that resolves to a server on the Hetzner network in germany. I have various servers all over their network and am currently being plagued with rogue traffic all from OVH 178.32.0.0/15 subnet. I don't know if its some kind of attack directed at Hetzner or whether its outgoing traffic in general but I do know that OVH have a major problem right now. I also know I am less than satisfied with the lack of response from OVH when I highlighted the potential problem to them this morning - seemed they couldn't care less. Since roughly 201305270100Z I have had literally hundreds of hosts from the above range performing portscans on all of my equipment. Here's an example (MAC address remove and IP's changed to protect the innocent) Code: May 27 22:10:38 server1 kernel: RULE 14 -- DENY IN=eth0 OUT= MAC= SRC=178.32.x.x DST=46.4.x.x LEN=44 TOS=0x00 PREC=0x00 TTL=58 ID=0 DF PROTO=TCP SPT=80 DPT=63571 WINDOW=14600 RES=0x00 ACK SYN URGP=0 Every single dropped packet from OVH network has TCP SPT 80, ie http traffic. I think somebody has managed to find an exploit on http services, eg webscript, sql injection, rogue php script or similar. Check all your websites for rogue scripts, unfamiliar files, unfamiliar process running under http user. Use iptraf or tcpdump to monitor network traffic, use rkhunter or similar rootkit detection tools to see if you can narrow it down. Watch outgoing bandwidth then stop http service - you might find it decreases. If you have any particular portal running it might be useful to check on that portals homepage or forum see if you have latest patches etc, or whether somebody has found a new exploit. It is rather confusing however to see so many hosts on one concentrated network compromised all at the same time. Finally if you have any direct line into somebody who will listen at OVH then I have a 200mb firewall log that will detail potentially compromised hosts. Since then however, I have changed my firewall to silently discard the whole subnet whilst this attack is ongoing. I wish you luck in finding the source of your woes!
Hello, I've not received any other abuse from hetzner right now, i've enabled the ISPConfig firewall on all servers, the things, i never received any of them with my primary multi server setup in Switzerland... (no firewall installed at all, but enabled now) I didn't know that we can run a netscan from a DNS server/Mail server / SQL Server without a firewall, SSH logs don't show anything abnormal
Hello Nioubee, I am still plagued with rogue traffic coming from OVH network but that is a different story. Trying to get OVH to acknowledge it is futile. This is occuring only a few weeks after a large-scale Bitcoin hack on servers hosted by them. Never mind - see the log you were sent - suggests to me that its apache/ngingx that generated that traffic. Did you look at the sites on your server? Are there any suspicious files on there, any recently changed files? Any spurious activity to/from your server? Perhaps a "tcpdump port 80" or similar might reveal something. On the firewall side, maybe if its relevant to you consider outgoing traffic rulesets as well as incoming. Checkout http://www.fwbuilder.org/ for a wonderful GUI tool for implementing firewall rulesets.
Hello monkfish, Thanks for the software, i will look at it. There is no web server installed on the slaves servers being used for the netscan My ISP - CH : ISPConfig Master only Web enabled SQL Server 1 Mail Server 1 DNS Server 1 DNS Server 2 OVH - FR : Web server 2 (currently not reported by hetzner) SQL Server 2 DNS Server 3 DNS Server 4 Only SQL and DNS Server #3 & #4 hosted by OVH was used for the netscan. Like said above, these VPS does not have any web server installed on them. Each VPS have their own public IP addresses.
Sorry, I don't follow Can you clarify, the server you stated above that was reported as performing malicious activity... That is one of your OVH ones? Are you saying you don't think you have a web server running on it? In which case I'd suggest you check that server as there is some process kicking out traffic from tcp port 80 which is what that network report is submitted for. Also, when you say netscan - is it your own machines you are portscanning or other peoples?
This is my OVH machines yes, but i don't portscan anyone. i don't own any server at Hetzner. I installed tcpdump and let it run for 24 hours in a screen an no result. Finally, i am not portscanning anyone. I think i will cancel my rented server, i received the 5th abuse message, but this time this is my broadcast address
That's absolutely interesting! Please don't get me wrong - I am not suggesting you are doing anything wrong. I do understand that you don't have a server at Hetzner (actually if you are looking to move from OVH then I'd say take a look at Hetzner - their service has been brilliant ) It actually ties direct in with my suspicions there is some kind of directed attack towards the hetzner network. Lookup those addresses in the last text file you've posted and most of them resolve to hetzner hosts. What you haven't answered is whether or not you have any kind of webservice or daemon running on port 80 on that particular machine. If not, perhaps traffic is being spoofed from elsewhere. How about an iptables rule to block outgoing tcp port 80. Does it go away? Whether its spoofed or not you are caught in the middle here with an unresponsive ISP who is pointing the finger at you by sending you a text file implying your server is one of the offenders. Do you know at what point in their network they are monitoring this? Are they certain its traffic coming from your server and not spoofed from elsewhere? What happens if you log all outbound traffic from your server? Anything showing? I'd be asking OVH to prove your machine is in fact generating that traffic. Are the binaries on your server intact - ie have not been tampered with in anyway - is "netstat -tanpu" giving you proper output? What about "lsof" - does that show up any spurious items? What about iptraf? Does that show you anything?
This will be what i gonna do if these problem of spoofed IP address don't stop. There is no daemon or web services on theses machines, i took a moment on each VPS to confirm it. These Abuse Messages is not sent by OVH but by Hetzner, i rents some IP address to the RIPE but by OVH. Is Munin data logs can be used as a proof ? See attached files About IPTraf, i am getting only known IP addresses, MySQL Port, SSH port (my remote), My Master ISPConfig Databse Server, BIND traffic, nothing else
Hello, Thanks for that. Nothing jumping out in your txt files as I'm sure you're aware. I think then the traffic is being spoofed somewhere else on the OVH network and you're caught in the middle. What does OVH say about that? Are they seeing any spurious traffic on their network? Why would their network configuration allow traffic seemingly from your broadcast address? Here's how a part of it looks from my side, a tcpdump: Code: 11:40:27.902411 IP 178.32.170.x.http > 46.4.46.x.15530: Flags [S.], seq 145700464, ack 1510529928, win 5840, options [mss 1460], length 0 11:40:28.015501 IP 178.32.81.x.http > 46.4.46.x.9361: Flags [S.], seq 2624546511, ack 1633638097, win 5840, options [mss 1460], length 0 11:40:28.330865 IP 178.32.81.x.http > 46.4.46.x.46018: Flags [S.], seq 1488576285, ack 1342855121, win 5840, options [mss 1460], length 0 11:40:28.689594 IP 178.32.81.x.http > 46.4.46.x.33576: Flags [S.], seq 3790022167, ack 3603716049, win 5840, options [mss 1460], length 0 11:40:28.746009 IP 178.32.96.x.http > 46.4.46.x.20201: Flags [S.], seq 3740042693, ack 2416947335, win 14600, options [mss 1460], length 0 11:40:28.756843 IP 178.32.81.x.http > 46.4.46.x.30776: Flags [S.], seq 3602612149, ack 2180077521, win 5840, options [mss 1460], length 0 I'd definitely be asking OVH for their assistance on this - they have some track record of cutting people's service off over matters like this - booting servers into some kind of resuce/ftp mode only and leaving them there. You can prove to them that its not your machine so they should work with you accordingly to discover the source of this problem. I wouldn't be surprised if its connected with that bitcoin hack a few weeks ago and some kind of retaliation to try and sully their reputation.
