fail2ban apache-badbots regex not working

Discussion in 'Server Operation' started by DrHappyAngry, Apr 12, 2013.

  1. DrHappyAngry

    DrHappyAngry New Member

    So I'm trying to get this working on an nginx proxy, but the regex is just not catching stuff.

    Here's apache-badbots.conf
    Code:
    [Definition]
    
    badbotscustom = EmailCollector|WebEMailExtrac|TrackBack/1\.02|sogou music spider|MJ12bot|yandex\.ru|Baiduspider|peerindex|SearchmetricsBot|SkimBot
    badbots = atSpider/1\.0|autoemailspider|China Local Browse 2\.6|ContentSmartz|DataCha0s/2\.0|DataCha0s/2\.0|DBrowse 1\.4b|DBrowse 1\.4d|Demo Bot DOT 16b|Demo Bot Z 16b|DSurf15a 01|DSurf15a 71|DSurf15a 81|DSurf15a VA|EBrowse 1\.4b|Educate Search VxB|EmailSiphon|EmailWolf 1\.00|ESurf15a 15|ExtractorPro|Franklin Locator 1\.8|FSurf15a 01|Full Web Bot 0416B|Full Web Bot 0516B|Full Web Bot 2816B|Industry Program 1\.0\.x|ISC Systems iRc Search 2\.1|IUPUI Research Bot v 1\.9a|LARBIN-EXPERIMENTAL \(efp@gmx\.net\)|LetsCrawl\.com/1\.0 +http\://letscrawl\.com/|Lincoln State Web Browser|LWP\:\:Simple/5\.803|Mac Finder 1\.0\.xx|MFC Foundation Class Library 4\.0|Microsoft URL Control - 6\.00\.8xxx|Missauga Locate 1\.0\.0|Missigua Locator 1\.9|Missouri College Browse|Mizzu Labs 2\.2|Mo College 1\.9|Mozilla/2\.0 \(compatible; NEWT ActiveX; Win32\)|Mozilla/3\.0 \(compatible; Indy Library\)|Mozilla/4\.0 \(compatible; Advanced Email Extractor v2\.xx\)|Mozilla/4\.0 \(compatible; Iplexx Spider/1\.0 http\://www\.iplexx\.at\)|Mozilla/4\.0 \(compatible; MSIE 5\.0; Windows NT; DigExt; DTS Agent|Mozilla/4\.0 efp@gmx\.net|Mozilla/5\.0 \(Version\: xxxx Type\:xx\)|MVAClient|NASA Search 1\.0|Nsauditor/1\.x|PBrowse 1\.4b|PEval 1\.4b|Poirot|Port Huron Labs|Production Bot 0116B|Production Bot 2016B|Production Bot DOT 3016B|Program Shareware 1\.0\.2|PSurf15a 11|PSurf15a 51|PSurf15a VA|psycheclone|RSurf15a 41|RSurf15a 51|RSurf15a 81|searchbot admin@google\.com|sogou spider|sohu agent|SSurf15a 11 |TSurf15a 11|Under the Rainbow 2\.2|User-Agent\: Mozilla/4\.0 \(compatible; MSIE 6\.0; Windows NT 5\.1\)|WebVulnCrawl\.blogspot\.com/1\.0 libwww-perl/5\.803|Wells Search II|WEP Search 00
    
    failregex = ^<HOST> -.*"(GET|POST).*HTTP.*"(?:%(badbots)s|%(badbotscustom)s)"$
    
    ignoreregex =
    If I try to test it on a log entry line that I copied out of my logs, like so
    fail2ban-regex '93.219.94.15 - - [12/Apr/2013:13:27:48 -0400] "GET /your-first-visit HTTP/1.0" 200 3915 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)" "-"' /etc/fail2ban/filter.d/apache-badbots.conf

    I get this
    Code:
    Running tests
    =============
    
    Use regex file : /etc/fail2ban/filter.d/apache-badbots.conf
    Use single line: 93.219.94.15 - - [12/Apr/2013:13:27:48 -0400] "GET...
    
    
    Results
    =======
    
    Failregex: 0 total
    
    Ignoreregex: 0 total
    
    Summary
    =======
    
    Sorry, no match
    
    Look at the above section 'Running tests' which could contain important
    information.
    Obviously not matching. As far as I can tell the regex looks right to catch that.
     
  2. conductive

    conductive Member HowtoForge Supporter

    Something is up with your connection and fail2ban is shutting you down because of it.
     

Share This Page