How is it possible to create rules for fail2ban and ban that would prevent the malformed requests to port 8080 (ispconfig) and 8081 (phpmyadmin) with nginx webserver? Thanks for the support.
ISPConfig has already a builtin protection against brute force password attacks, so an attacker will get blocked independantly from fail2ban. But it should be possible to create a fail2ban rule wich reads the ispconfig log and blocks on iptables level as well. If you want to protect your server against malformed requests, the fail2ban is not the right softare. you should use something like mod_security fro nginx then: http://www.modsecurity.org/projects/modsecurity/nginx/