Hi, I need help to resolve my server. It has been compromised. The only hint that i got from my friend is mail.domain.com has been compromised. Any idea how to prevent hacker from coming again?
And what does your friend make believe that this is the case? In almost all cases where a third party thinks a system has been compromised, then that's actually not the case. The much more likely case is that your system has not been compromised by someone e.g. send spam trough your system because he got access to a password of one of your mail accounts without the system being compromised. Or that someone hacked a website and not the system, but hacking a website does not usually compromise the system. And please choose a title that matches and decsribes your question. I've changed your title now.
@till's stated facts are all right on the money. Though it can be more than enough reason for a lot of ISP's to say it's a compromised system (while infact it's only a compromised user or website) and then to block all access or, if it's a vps, isolate it. You can't do much with just a believe of a fiend who's not giving you any further details. So first thing is to ask your friend what his believe is based on. Only then you can do some proper investigation. Until then you only have your logs and finding anything in there is like looking for a needle in a haystack.