suddenly can not login: Username or Password wrong.1

Check that MySQL / MariaDB is started

 

Did you maybe alter the password of the ispconfig MySQL user?

 

I guess you have to examine the access.log files of your server and try to find out how this happened. Also take care that all programs that you use to access MySQL use SSL encryption e.g. phpmyadmin. Check if mysql is listening on the external network interface or localhost, if you don't need external access and it currently listening externally, change it to localhost. Another possibility is that a desktop system which is used by a system administrator has a trojan. Which CentOS version do you use, if it's CentOS 6, then you should consider upgrading to a current version. Of course, the issue could be a vulnerability in ISPConfig itself, but this is not very likely as we would see hundreds or thousands of similar issues here in the forum then, and that's not the case. So it's more likely that it's an issue specific to your system.

 

Are the tow mentioned databases all databases on this server, or are there other ones that are not affected?

 

Ok, but as far as I can see, you don't have any other databases, which means all databases are affected. The MySQL and information_schema are system databases, which explains why they are not affected as you won't be able to read their message in case they would have encrypted them.

 

The first important thing to note is that all databases on this server were hacked, this indicates that the attacked used the MySQL root user or a user with similar permissions. If outside access to MySQL is closed, then the user must have been able to run shell commands (not necessarily as shell root user, but maybe as user of one of the hosted websites) or that the server has been compromised by a rootkit. Plus the attacker must be able to get the mysql root password somehow, possible reasons might be a insecure password used somewhere else or the password was stored on the server in a file that's world readable or the password was stored in a backup and this backup was accessible to the attacker. It might also be that an attacker from previous attack as left a backdoor which e.g. gets reactivated by a cronjob as it's not the first time that this happened on your system.

Even with all updates installed in our cms, it might be that you run a plugin or theme on one of these cms which is vulnerable e.g. because the author does not provide updates anymore.

The best solution would probably be a reinstall, if possible. then take care to disable shell exec functions in all websites that don't need them, and use different mysql passwords and ensure that config files from websites which contain MySQL passwords are chmod 0600.

 

Also schedule an update to ISPConfig 3.2, there are security fixes that could come into play. Check what other services you might have running on the machine, either listening on the network or even listening on localhost (as you run websites on this server, a hacked site can connect to any service listening on the localhost addr). Keep a copy of your logs files to pore over and try to narrow a timeframe when this happened (though actual access to your systems could have been quite some time prior).

 

It's probably less work that exhaustively searching a compromised system for all places that Bad Things(tm) may be hiding. But, do both - the latter is an excellent learning process, and reinstalling has a higher reassurance of less backdoors. You could even change distributions and get to newer software versions.

It could help if the initial vulnerability(ies) exploited were via the web. Also keep in mind XSS family of vulnerabilities, the access log could show requests made by you yourself (eg. a stored xss attack could have been in place for some time before you did whatever triggered it).

Change all db passwords, especially other users with high levels of permissions like 'ispconfig' and 'debian-sys-maint'. There is also sensitive info in the data they stole, definitely reset all ISPConfig account passwords (including any remote users for the api), database account passwords, change dkim keys, replace any ssl certificates you uploaded under the SSL tab, and you'll have password hashes of many things, etc. ftp accounts, shell and email accounts, website stats. And there's more (rspamd interface password, and surely more) things to uncover as you browse. Change all shell user passwords (both system level accounts and those within ISPConfig), and audit all users in /etc/passwd.

You edit the numerous php.ini files on your system, there is one for each mode of each php version you have. Be sure you use a comprehensive list of functions to disable (there are quite a few), and keep in mind when you edit php.ini for your CLI version(s) that you need many of them for normal use (eg. if you disable exec() for the system default php cli, most of ISPConfig's back-end will cease to function).

Setting open_basedir helps some, too.

Throw in maldet, and be sure you have a good signature set in clamav when it runs. ISPProtect is another option to try, as are chrootkit, lynis, ossec, and others. They may or may not find anything. Some of them can work much better by setting them up on a clean baseline system, so they can detect when things change, rather than having to positively identify the changed thing.

Check cron jobs and init scripts, and of course check for web-accessible backdoors left anywhere. Sometimes even php's .user.ini is used to reinfect things (via auto prepend).

Also remove old site copies (dev/testing sites, backups, etc.).


One last note, you have a database named c2jfile_manager, and web-based file managers are difficult to write securely; be sure to consider that app as candidate for being exploited. PHPmyadmin is more likely as it was only your databases which were affected, but worth checking.

 

No. Those wouldn't be the initial means of compromise, but left there afterwards. Perhaps they could cause later reinfection via some means, or perhaps they are something like Windows executables, that wouldn't even run on your Linux box. Did you examine them at all to see what they may have been? I'm just curious.