Ispconfig 3 - DDOS attack mitigation

Discussion in 'General' started by nicog, Jul 8, 2019.

  1. nicog

    nicog New Member

    Hello,
    is there a way in IspConfig 3 to prevent DOS and DDOS attacks?
     
  2. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    for DOS, as a start you could install and configure libapache2-mod-evasive
    for DDOS, it's pretty much just chuck lots of money and server resources at it. same for everyone on any server with any control panel really. fail2ban will help, but depending on how many different ip's are in the attack, and how often each ip makes a request, it may not help a lot without setting the bar so low you end up banning a lot of legitimate users as well.
     
  3. nicog

    nicog New Member

    first off all thank you for your quick reply.
    I know about mod-svasive, but the problem is that i'm running on the same server and IP my web sites with about 700 users in a day and also there is running stratum pool where I get mostly the DOS or DDOS attacks. so the problem is I can block (D)DOS attacks but don't like to block my website's legitimate users! As I see I must check every config entry for this
     
  4. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    dunno what OS you're using, on ubuntu, I've only had to install the mod and set the config once in /etc/apache2/mods-enabled/evasive.conf, I've not put anything into any site specific config.

    I've left most of the config on it's defaults, ie:
    DOSHashTableSize 3097
    DOSPageCount 2
    DOSSiteCount 50
    DOSPageInterval 5
    DOSSiteInterval 1
    DOSBlockingPeriod 10

    and even on a very busy/popular site, we've had no reports or indications that a single legitimate user has been blocked, and there's been no problem/downtime on that server since I installed it 5 days ago. the site was taken offline 4 times in the preceding 3 days due to DOS attacks.
    looking at the logs, it seems like they'd worked out the fail2ban jail numbers/times and were staying just outside of getting banned by that
     
  5. nicog

    nicog New Member

    I installed it but it is not working against Low Orbito Ion Cannon :( Also it does not block any user on HTTPS at all
     
  6. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    You probably don't have it installed correctly, make sure the module is enabled (a2enmod evasive) and check the settings in the .conf file to make sure they're reasonable. And in an alternative response, I'd ask "based on what?" :) If you are basing the "does not block" on syslog entries to daemon.log, it actually is blocking, but there seems to be a bug where it doesn't log (nor email) about the blocks.

    First off, the specifics of the bug I mentioned is that if you set a DOSLogDir, it creates a file there called dos-xx.xx.xx.xx for each ip blocked, and the first time it blocks it both emails, calls syslog, and performs any DOSSystemCommand commands. It does not clean up old dos-<ip> files however, and when such a file exists already, it does not perform any further email/syslog/system commands. It does still block (you can confirm that by triggering the limits yourself, and watch that the 403 responses do happen in your browser console). And if you manually remove the dos-<ip> file, it will work again correctly one time. So the obvious workaround is to put in place some local file cleanup measures.

    So I just ran through playing with this a bit on a debian9 server, and I could trigger being blocked if I clicked around quickly (eg. click around the ispconfig tabs too quickly), so I eased the settings a bit. Rather than finding or implementing a firewall integration, I simply used fail2ban, which has everything ready to go (ie. hooks to firewall and cleanup at later time). Ie. mod_evasive does it's job and blocks an address (for 10 seconds) and logs the blocked IP address to syslog, so I wrote a little fail2ban jail which monitors daemon.log for these blocks, and propogates them to the firewall. I also used both mod_evasive's DOSSystemCommand as well as a custom fail2ban action script to keep the aforementioned dos-<ip> files cleaned up.

    First, install mod_evasive:
    Code:
    apt-get install libapache2-mod-evasive
    
    a2enmod evasive     # not needed on debian9, but just in case
    
    mkdir /var/log/mod_evasive/
    chown www-data:www-data /var/log/mod_evasive/
    
    Then edit /etc/apache2/mods-enabled/evasive.conf to adjust any default settings; I also whitelisted localhost and the server's own ip address, probably out of paranoia/unfamiliarity rather than necessity:
    Code:
    <IfModule mod_evasive20.c>
        DOSHashTableSize    3079
        DOSPageCount        24
        DOSSiteCount        90
        #DOSPageInterval     1
        #DOSSiteInterval     1
        #DOSBlockingPeriod   10
    
        DOSEmailNotify      "[email protected]"
        DOSSystemCommand    "/bin/sleep 10; /bin/rm /var/log/mod_evasive/dos-%s"
        DOSLogDir           "/var/log/mod_evasive/"
    
        DOSWhitelist        127.0.*.*
        DOSWhitelist        11.22.33.44
    </IfModule>
    
    Now create the fail2ban filter file, /etc/fail2ban/filter.d/apache-mod_evasive.local:
    Code:
    # Fail2Ban configuration file for mod_evasive
    #
    # Author: Jesse Norell
    
    [Definition]
    failregex = mod_evasive.*: Blacklisting address <HOST>: possible DoS attack.
    ignoreregex =
    
    And create the fail2ban action file, /etc/fail2ban/action.d/apache-mod_evasive-cleanup.local:
    Code:
    # Fail2Ban action.d configuration file for mod_evasive cleanup
    #
    # Author: Jesse Norell
    
    [INCLUDES]
    
    [Definition]
    actionstart = /bin/touch /var/log/mod_evasive/dos-test-file && /bin/rm /var/log/mod_evasive/dos-*
    actionunban = /bin/rm -f /var/log/mod_evasive/dos-<ip>
    
    Then add a new jail to /etc/fail2ban/jail.local:
    Code:
    [apache-mod_evasive]
    
    enabled = true
    banaction = %(banaction_allports)s
                apache-mod_evasive-cleanup
    logpath = %(syslog_daemon)s
    
    And that's it .. restart the daemons:
    Code:
    systemctl restart fail2ban
    systemctl restart apache2
    
    One last note, it appears mod_evasive also logs to the apache error.log file, so if you are using the apache-auth jail, it will catch the same ip addresses there (which matters if say you want to add your own ip address to the ignores list). You can configure apache-auth to ignore mod_evasive by creating /etc/fail2ban/filter.d/apache-auth.local:
    Code:
    # ignoreregex for Fail2Ban apache-auth filter
    #
    
    # the apache-auth jail is picking up logs for mod_evasive,
    # so the intended 10 second block (which mod_evasive performs)
    # becomes much longer (starting at 10 minutes with fail2ban)
    
    [Definition]
    
    ignoreregex = \[evasive20:error\]
    
     
    Last edited: Feb 19, 2020
    elmacus, Neptun, tal56 and 1 other person like this.
  7. nicog

    nicog New Member

    Hello Jesse.
    thank you for your great article and "How to". This is really very helpful incl. fail2ban integration.
    My system is Centos 7 linux whith apache 2.4.6 and ISPconfig 3 with multiple sites on 1 IP all with https (port 443 and not 80) and I installed the mod_evasive from the epel repo and also from here https://github.com/shivaas/mod_evasive , which is fork from original one for apache 2.4 but it does not blocks any browser requests when I also go down with all the parameters in config file. the blocking works only when I run included perl script test.pl: it gives first 200 and then 403.
    I found this article https://www.linuxquestions.org/ques...problem-with-mod_evasive-on-https-4175424203/ where it is written that evasive does not works with https.
    But as I see now, if it would be work on my system, it could only block 80 and 443 traffic and not other traffics. So I need something stronger then this module.
    I found this article https://javapipe.com/blog/iptables-ddos-protection/ , which is exactly what I need, but because of ISPconfig on my system IPtables is disabled and now works only Bastile firewall.
    Now my question is, can I use Iptables parallel whith bastile firewall?
     
  8. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    Or I would suggest, change to a different distribution like debian or ubuntu.

    It has been many years since I worked with bastille, I can't answer that from memory. I can fully recommend switching to ufw though, which is likely to integrate with other tools out of the box, in addition to being a more complete/better firewall overall, easier to manage, and supporting ipv6.
     
  9. nicog

    nicog New Member

    OK. I will try next time to install ubuntu and work with this. But now I made changes to to my kernel and iptables like in this guide https://javapipe.com/blog/iptables-ddos-protection/ and I can say that it works: I can now while attacking my own server ping google.com. before this changes the float was so height the I could not ping google.com :) And I can see now in iftop that the rules realy work, because the traffic is not going higher then 51 mbs and before it was floating fully my 1GBs bandwidth!
    I will now try it in the real world and hope that it helps there too!
     
    Neptun and Jesse Norell like this.
  10. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    Neptun likes this.
  11. Neptun

    Neptun Member

    That is a great guide !
    i will test it with my debian 10 nginx server - i hope it will work too

    Do you have here also something similar for nginx ?
     
  12. nicog

    nicog New Member

    In that guide the rules I do not used are:
    iptables -A INPUT -p tcp -m conntrack --ctstate NEW -m limit --limit 60/s --limit-burst 20 -j ACCEPT
    iptables -A INPUT -p tcp -m conntrack --ctstate NEW -j DROP
    Because I do not know exactly if my legit users can be blocked or not with this rules.
    And this rule: iptables -t raw -A PREROUTING -p tcp -m tcp --syn -j CT --notrack blocked my hole 80 and 443 connections, so I do not use it at all.

    On my Centos 7 I can install mod_evasive with yum install mod_evasive. I have httpd (apache 2.4.6) running on my server
     
  13. pzajda

    pzajda Member HowtoForge Supporter

    Sarhan likes this.
  14. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    Just a follow-up on mod_evasive parameters, I've had to increase the limits a little to avoid some legitimate requests. Most recently (yesterday) we had a customer that tripped our previous limits when using roundcube, as all requests are to the same location, /roundcube/ (varying by query parameters, but not the location itself). I have edited the above config in this post as well, but these are the adjustted settings I'm currently using:

    Code:
       DOSPageCount        24
       DOSSiteCount        90
    
     
    till likes this.
  15. Jesse Norell

    Jesse Norell Well-Known Member Staff Member Howtoforge Staff

    A bit more follow-up / improvement, it turns out when mod_evasive temporarily denies requests for clients exceeding the threshholds, those log messages match the default apache-auth fail2ban jail, so if you're using that jail (as we are), what is intended to be a 10 second ignore by mod_evasive quickly turns into a 10 minute block by fail2ban. The fix is easy, just create
    /etc/fail2ban/filter.d/apache-auth.local with:
    Code:
    # ignoreregex for Fail2Ban apache-auth filter
    #
    
    # the apache-auth jail is picking up logs for mod_evasive,
    # so the intended 10 second block (which mod_evasive performs)
    # becomes much longer (starting at 10 minutes with fail2ban)
    
    [Definition]
    
    ignoreregex = \[evasive20:error\]
    
    (I'll edit my original comment above and include that.)

    We're still using the same limits as above (24 / 90). In the back of my head I wonder how that will change, and indeed how effective mod_evasive can be as more sites use http/2.0, which is designed for many simultaneous queries. Probably we'll just need to increase limits some more, but time will tell.
     
  16. johnymas

    johnymas Member

    Hi, Fail2Ban won't start anymore. I always get an error when I check fail2ban status:
    No file(s) found for glob /var/log/daemon.log
    Failed during configuration: Have not found any log file for apache-mod_evasive jail
     
  17. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    What shows
    Code:
    ls -lh /var/log/dae*
    Is fail2ban running inside a jail?
     
  18. johnymas

    johnymas Member

    ls: cannot access '/var/log/dae*': No such file or directory
     
  19. johnymas

    johnymas Member

    I specified the backend in /etc/fail2ban/jail.conf and it works now.
    But I'm not sure this step is ok on Ubuntu 18.04.4 LTS

    I changed
    backend = auto
    to
    backend = systemd

    And restart fail2ban:
    systemctl restart fail2ban
     
    Last edited: May 25, 2020
  20. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    just re-reading this thread after a long time, looking at this, I can understand DOSSiteCount, as that's requests for any objects, but 24 for DosPageCount? isn't allowing 24 requests for the same page (URI) from the same client within one second is excessively high?
    surely not even roundcube would be triggering that.
    surely you'd have to be very unlucky to have a large enough number of users from the same external office/network accessing webmail in that second to trigger it at even half that count?
     

Share This Page