is my server hacked ? urgent

Discussion in 'General' started by piyush, Jul 22, 2011.

  1. piyush

    piyush New Member

    Hello All,


    Recently I noticed that cpu is fully used by http.pl, httpd.pl, https.pl process.

    This is result of top command

    :confused:PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    2473 www-data 20 0 36800 6948 1332 R 54 0.7 8:45.96 https.pl
    2348 www-data 20 0 38332 7464 1332 R 52 0.7 8:55.28 http.pl
    2475 www-data 20 0 36688 6884 1332 R 45 0.7 8:29.28 httpd.pl
    2474 www-data 20 0 36952 6948 1332 R 35 0.7 8:37.41 httpd.pl

    if I run top -bcis then all http?.pl display as mail.

    I try to kill those process with kill 2473 but nothing happen to that process with many attempt the process is still running as 2473 ID


    Finally I disconnected my sever from net. I have no idea what should be next.

    Any suggestion highly appreciated.
     
  2. piyush

    piyush New Member

    I think I am dead.

    No one have as such experience of http.pl (mail) process consuming full cpu ?

    The strange thing is I search my all pc and can't find any file named http.pl or any command name mail.


    I think I should buy another hosting and transfer files to there.

    Tomorrow I have to fly to china so no time to try.

    Thanks ................
     
  3. till

    till Super Moderator Staff Member ISPConfig Developer

    Please check your system with rkhunter to see if or which rootkits are installed. as the scripts run all as www-data user, most likely just one website is affected and not the whole server. So it might be possible to fix the problem by just cleaning one website.
     
  4. erosbk

    erosbk New Member

    Install htop to see path of running process.
     
  5. piyush

    piyush New Member

    Hi Erosbk,

    Thanks a lot for suggestion

    I have installed htop and used it. that process is just appeared as mail not any other path.
     
  6. piyush

    piyush New Member


    Hi Till,

    Thanks for suggestion

    I have never used rkhunter going to take look in it.

    how ever currently I have not published and third party website. and none of my website have so much trafic. there are approx 10 website total.
     
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    Which php mode do you use in your websites? Is suexec enabled in the websites?
     
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    login on the shell as root user, then run:

    rkhunter --update

    and then

    rkhunter -c
     
  9. piyush

    piyush New Member

    Most website using fast-cgi. there is no option for suexec.
     
  10. piyush

    piyush New Member

    Here is the result of rkhunter -c

    [20:12:27] Running Rootkit Hunter version 1.3.6 on server1
    [20:12:28]
    [20:12:28] Info: Start date is Fri Jul 22 20:12:27 CST 2011
    [20:12:28]
    [20:12:28] Checking configuration file and command-line options...
    [20:12:28] Info: Detected operating system is 'Linux'
    [20:12:28] Info: Found O/S name: Ubuntu 11.04
    [20:12:28] Info: Command line is /usr/bin/rkhunter -c
    [20:12:28] Info: Environment shell is /bin/bash; rkhunter is using bash
    [20:12:28] Info: Using configuration file '/etc/rkhunter.conf'
    [20:12:28] Info: Installation directory is '/usr'
    [20:12:28] Info: Using language 'en'
    [20:12:28] Info: Using '/var/lib/rkhunter/db' as the database directory
    [20:12:29] Info: Using '/usr/share/rkhunter/scripts' as the support script directory
    [20:12:29] Info: Using '/usr/local/sbin /usr/local/bin /usr/sbin /usr/bin /sbin /bin /usr/X11R6/bin /bin /usr/bin /sbin /usr/sbin /usr/local/bin /usr/local/sbin /usr/libexec /usr/local/libexec' as the command directories
    [20:12:29] Info: Using '/' as the root directory by default
    [20:12:29] Info: Using '/var/lib/rkhunter/tmp' as the temporary directory
    [20:12:29] Info: No mail-on-warning address configured
    [20:12:29] Info: X will be automatically detected
    [20:12:29] Info: Found the 'basename' command: /usr/bin/basename
    [20:12:29] Info: Found the 'diff' command: /usr/bin/diff
    [20:12:29] Info: Found the 'dirname' command: /usr/bin/dirname
    [20:12:30] Info: Found the 'file' command: /usr/bin/file
    [20:12:30] Info: Found the 'find' command: /usr/bin/find
    [20:12:30] Info: Found the 'ifconfig' command: /sbin/ifconfig
    [20:12:30] Info: Found the 'ip' command: /sbin/ip
    [20:12:30] Info: Found the 'ldd' command: /usr/bin/ldd
    [20:12:30] Info: Found the 'lsattr' command: /usr/bin/lsattr
    [20:12:30] Info: Found the 'lsmod' command: /sbin/lsmod
    [20:12:30] Info: Found the 'lsof' command: /usr/bin/lsof
    [20:12:30] Info: Found the 'mktemp' command: /bin/mktemp
    [20:12:31] Info: Found the 'netstat' command: /bin/netstat
    [20:12:31] Info: Found the 'perl' command: /usr/bin/perl
    [20:12:31] Info: Found the 'pgrep' command: /usr/bin/pgrep
    [20:12:31] Info: Found the 'ps' command: /bin/ps
    [20:12:31] Info: Found the 'pwd' command: /bin/pwd
    [20:12:31] Info: Found the 'readlink' command: /bin/readlink
    [20:12:31] Info: Found the 'sort' command: /usr/bin/sort
    [20:12:31] Info: Found the 'stat' command: /usr/bin/stat
    [20:12:31] Info: Found the 'strings' command: /usr/bin/strings
    [20:12:32] Info: Found the 'uniq' command: /usr/bin/uniq
    [20:12:32] Info: System is not using prelinking
    [20:12:32] Info: Using the '/usr/bin/sha1sum' command for the file hash checks
    [20:12:32] Info: Stored hash values used hash function '/usr/bin/sha1sum'
    [20:12:32] Info: Stored hash values did not use a package manager
    [20:12:32] Info: The hash function field index is set to 1
    [20:12:32] Info: No package manager specified: using hash function '/usr/bin/sha1sum'
    [20:12:32] Info: Previous file attributes were stored
    [20:12:32] Info: Enabled tests are: all
    [20:12:33] Info: Disabled tests are: suspscan hidden_procs deleted_files packet_cap_apps apps
    [20:12:33] Info: Found ksym file '/proc/kallsyms'
    [20:12:33] Info: Using 'date' to process epoch second times.
    [20:12:33]
    [20:12:33] Checking if the O/S has changed since last time...
    [20:12:33] Info: Nothing seems to have changed
    [20:12:33] Info: Locking is not being used
    [20:12:34]
    [20:12:34] Starting system checks...
    [20:12:34]
    [20:12:34] Checking system commands...
    [20:12:34] Info: Starting test name 'system_commands'
    [20:12:34]
    [20:12:34] Performing 'strings' command checks
    [20:12:34] Info: Starting test name 'strings'
    [20:12:34] Scanning for string /usr/sbin/ntpsx [ OK ]
    [20:12:35] Scanning for string /usr/sbin/.../bkit-ava [ OK ]
    [20:12:35] Scanning for string /usr/sbin/.../bkit-d [ OK ]
    [20:12:35] Scanning for string /usr/sbin/.../bkit-shd [ OK ]
    [20:12:35] Scanning for string /usr/sbin/.../bkit-f [ OK ]
    [20:12:35] Scanning for string /usr/include/.../proc.h [ OK ]
    [20:12:36] Scanning for string /usr/include/.../.bash_history [ OK ]
    [20:12:36] Scanning for string /usr/include/.../bkit-get [ OK ]
    [20:12:36] Scanning for string /usr/include/.../bkit-dl [ OK ]
    [20:12:36] Scanning for string /usr/include/.../bkit-screen [ OK ]
    [20:12:36] Scanning for string /usr/include/.../bkit-sleep [ OK ]
    [20:12:37] Scanning for string /usr/lib/.../bkit-adore.o [ OK ]
    [20:12:37] Scanning for string /usr/lib/.../ls [ OK ]
    [20:12:37] Scanning for string /usr/lib/.../netstat [ OK ]
    [20:12:37] Scanning for string /usr/lib/.../lsof [ OK ]
    [20:12:37] Scanning for string /usr/lib/.../bkit-ssh/bkit-shdcfg [ OK ]
    [20:12:38] Scanning for string /usr/lib/.../bkit-ssh/bkit-shhk [ OK ]
    [20:12:38] Scanning for string /usr/lib/.../bkit-ssh/bkit-pw [ OK ]
    [20:12:38] Scanning for string /usr/lib/.../bkit-ssh/bkit-shrs [ OK ]
    [20:12:38] Scanning for string /usr/lib/.../bkit-ssh/bkit-mots [ OK ]
    [20:12:38] Scanning for string /usr/lib/.../uconf.inv [ OK ]
    [20:12:39] Scanning for string /usr/lib/.../psr [ OK ]
    [20:12:39] Scanning for string /usr/lib/.../find [ OK ]
    [20:12:39] Scanning for string /usr/lib/.../pstree [ OK ]
    [20:12:39] Scanning for string /usr/lib/.../slocate [ OK ]
    [20:12:39] Scanning for string /usr/lib/.../du [ OK ]
    [20:12:40] Scanning for string /usr/lib/.../top [ OK ]
    [20:12:40] Scanning for string /usr/sbin/... [ OK ]
    [20:12:40] Scanning for string /usr/include/... [ OK ]
    [20:12:40] Scanning for string /usr/include/.../.tmp [ OK ]
    [20:12:40] Scanning for string /usr/lib/... [ OK ]
    [20:12:41] Scanning for string /usr/lib/.../.ssh [ OK ]
    [20:12:41] Scanning for string /usr/lib/.../bkit-ssh [ OK ]
    [20:12:41] Scanning for string /usr/lib/.bkit- [ OK ]
    [20:12:41] Scanning for string /tmp/.bkp [ OK ]
    [20:12:41] Scanning for string /tmp/.cinik [ OK ]
    [20:12:42] Scanning for string /tmp/.font-unix/.cinik [ OK ]
    [20:12:42] Scanning for string /lib/.sso [ OK ]
    [20:12:42] Scanning for string /lib/.so [ OK ]
    [20:12:42] Scanning for string /var/run/...dica/clean [ OK ]
    [20:12:42] Scanning for string /var/run/...dica/dxr [ OK ]
    [20:12:42] Scanning for string /var/run/...dica/read [ OK ]
    [20:12:43] Scanning for string /var/run/...dica/write [ OK ]
    [20:12:43] Scanning for string /var/run/...dica/lf [ OK ]
    [20:12:43] Scanning for string /var/run/...dica/xl [ OK ]
    [20:12:43] Scanning for string /var/run/...dica/xdr [ OK ]
    [20:12:43] Scanning for string /var/run/...dica/psg [ OK ]
    [20:12:44] Scanning for string /var/run/...dica/secure [ OK ]
    [20:12:44] Scanning for string /var/run/...dica/rdx [ OK ]
    [20:12:44] Scanning for string /var/run/...dica/va [ OK ]
    [20:12:44] Scanning for string /var/run/...dica/cl.sh [ OK ]
    [20:12:44] Scanning for string /var/run/...dica/last.log [ OK ]
    [20:12:45] Scanning for string /usr/bin/.etc [ OK ]
    [20:12:45] Scanning for string /etc/sshd_config [ OK ]
    [20:12:45] Scanning for string /etc/ssh_host_key [ OK ]
    [20:12:45] Scanning for string /etc/ssh_random_seed [ OK ]
    [20:12:45] Scanning for string /dev/ptyp [ OK ]
    [20:12:46] Scanning for string /dev/ptyq [ OK ]
    [20:12:46] Scanning for string /dev/ptyr [ OK ]
    [20:12:46] Scanning for string /dev/ptys [ OK ]
    [20:12:46] Scanning for string /dev/ptyt [ OK ]
    [20:12:46] Scanning for string /dev/fd/.88/freshb-bsd [ OK ]
    [20:12:47] Scanning for string /dev/fd/.88/fresht [ OK ]
    [20:12:47] Scanning for string /dev/fd/.88/zxsniff [ OK ]
    [20:12:47] Scanning for string /dev/fd/.88/zxsniff.log [ OK ]
    [20:12:47] Scanning for string /dev/fd/.99/.ttyf00 [ OK ]
    [20:12:47] Scanning for string /dev/fd/.99/.ttyp00 [ OK ]
    [20:12:48] Scanning for string /dev/fd/.99/.ttyq00 [ OK ]
    [20:12:48] Scanning for string /dev/fd/.99/.ttys00 [ OK ]
    [20:12:48] Scanning for string /dev/fd/.99/.pwsx00 [ OK ]
    [20:12:48] Scanning for string /etc/.acid [ OK ]
    [20:12:48] Scanning for string /usr/lib/.fx/sched_host.2 [ OK ]
    [20:12:49] Scanning for string /usr/lib/.fx/random_d.2 [ OK ]
    [20:12:49] Scanning for string /usr/lib/.fx/set_pid.2 [ OK ]
    [20:12:49] Scanning for string /usr/lib/.fx/setrgrp.2 [ OK ]
     
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    All recent ISPConfig 3 versions have a suexec option in the website settings. Which exact ISPConfig version do you use?

    Regarding rkhunter, please post just the result summary that you receive after all rkhunter checks have been done.
     
  12. piyush

    piyush New Member

    I just see in mail queue there are many emails in queue one of them is as below

    -Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
    41EF0201777 9313 Thu Jul 21 13:57:23 MAILER-DAEMON
    (host mx3.efwmx.net[64.94.160.236] refused to talk to me: 550 550 This system is configured to reject mail from 220.135.105.28 [220.135.105.28] (Host blacklisted - Found on Realtime Black List server 'zen.spamhaus.org'))
    [email protected]

    so it means some one is trying to send email to [email protected]

    Which is not I and none of us. that means some one is using this server for bulk email.

    How can we identify and stop strange mailing like this ?
     
  13. piyush

    piyush New Member

    Hi Till,

    Thanks a lot for giving your precious time.

    I am using ISPConfig Version: 3.0.3.3

    There is option as suphp but not suexec. is that same ?

    Here is summery of RK Hunter as below

    System checks summary
    =====================

    File properties checks...
    Files checked: 130
    Suspect files: 1

    Rootkit checks...
    Rootkits checked : 245
    Possible rootkits: 0

    Applications checks...
    All checks skipped

    The system checks took: 18 minutes and 7 seconds

    All results have been written to the log file (/var/log/rkhunter.log)

    One or more warnings have been found while checking the system.
    Please check the log file (/var/log/rkhunter.log)
     
  14. erosbk

    erosbk New Member

    Watch for Till instructions, this is probably a problem with your apache2 or one of your sites...

    Regarding open relay and postfix, test your server here:
    http://www.mxtoolbox.com/diagnostic.aspx

    If using postfix (it is not an open relay by default as I think), please run:

    cat /var/log/mail.log | grep "smtp" | tail

    and

    postconf |grep 'mynetworks ='

    Post both results here.

    Edit: for suexec enabled, you need to have it installed "apache2-suexec" and enabled with "a2enmod suexec" before installing ispconfig. Which perfect guide did you follow?
     
    Last edited: Jul 22, 2011
  15. piyush

    piyush New Member

    Here is test result with http://www.mxtoolbox.com/diagnostic.aspx

    220 server1.sarakuchh.com ESMTP Postfix (Ubuntu)

    OK - 220.135.105.28 resolves to 220-135-105-28.hinet-ip.hinet.net
    Warning - Reverse DNS does not match SMTP Banner
    0 seconds - Good on Connection time
    Not an open relay.
    6.583 seconds - Warning on Transaction time


    Here is result of cat /var/log/mail.log|grep "smtp" | tail

    Jul 22 22:05:02 server1 postfix/smtpd[9459]: lost connection after CONNECT from localhost[127.0.0.1]
    Jul 22 22:05:02 server1 postfix/smtpd[9459]: disconnect from localhost[127.0.0.1]
    Jul 22 22:05:04 server1 postfix/smtpd[9433]: warning: 64.20.227.133: address not listed for hostname recover.mxtoolbox.com
    Jul 22 22:05:04 server1 postfix/smtpd[9433]: connect from unknown[64.20.227.133]
    Jul 22 22:05:04 server1 postfix/smtpd[9433]: NOQUEUE: reject: RCPT from unknown[64.20.227.133]: 554 5.7.1 <[email protected]>: Relay access denied; from=<[email protected]> to=<[email protected]> proto=SMTP helo=<please-read-policy.mxtoolbox.com>
    Jul 22 22:05:05 server1 postfix/smtpd[9433]: disconnect from unknown[64.20.227.133]
    Jul 22 22:05:11 server1 postfix/smtpd[9459]: warning: 122.180.61.226: hostname NSG-Corporate-226.61.180.122.airtel.in verification failed: Name or service not known
    Jul 22 22:05:11 server1 postfix/smtpd[9459]: connect from unknown[122.180.61.226]
    Jul 22 22:05:14 server1 postfix/smtpd[9459]: NOQUEUE: reject: RCPT from unknown[122.180.61.226]: 550 5.1.1 <[email protected]>: Recipient address rejected: User unknown in virtual mailbox table; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<mail.naukrinews.com>
    Jul 22 22:05:17 server1 postfix/smtpd[9459]: disconnect from unknown[122.180.61.226]


    and here is result of postconf|grep 'mynetworks ='
    mynetworks = 127.0.0.0/8 [::1]/128

    I am not remember that I install apache2-suexec or not but I did follow the guide.
     
  16. piyush

    piyush New Member

    Hi Erosbk,

    Thanks for your suggestion.

    If any doubts I can reinstall server from scratch with following tutorial.

    What's your suggestion ?
     
  17. erosbk

    erosbk New Member

    Ok, it looks like you are not an open relay.

    Try this: a2enmod suexec

    Post here results to see if you have it installed or not... I think that there is no need to reinstall whole server. Maybe you can install what is not installed, and as a last step, update and reconfigure ispconfig. Till could help you better with this I think...
     
  18. piyush

    piyush New Member

    here is result of a2enmod suexec
    Module suexec already enabled


    It's already installed
     
  19. erosbk

    erosbk New Member

    Enter in ISPConfig as admin, go sites, select one site, and see if you have this options:


    ...
    SSI
    Ruby
    SuEXEC (this is what you need to have selected, does it exist?)
    Own error documents
    Autosubdomain
    SSL
    PHP (combo here to select php mode)
    Activar
     
  20. piyush

    piyush New Member


    Aah it's there. Before I was looking this option inside combo.

    So should I check this option in all sites ?
     

Share This Page