Wordfence firewall rules not updating

Discussion in 'Server Operation' started by mrunix, Dec 9, 2021.

  1. mrunix

    mrunix New Member

    Sorry if this is the wrong location for this post as I am new here.

    I have a Debian 10 server running ISPConfig 3.2.7p1. It's hosting several Wordpress sites. For about a month I have been receiving the following errors when trying to update the Wordfence firewall rules: “No rules were updated. Please verify you have permissions to write to the /wp-content/wflogs directory." & "Rule Update Failed: No rules were updated. Please verify your website can reach the Wordfence servers." Checked the diagnostics report for the sites and they are all showing connectivity is fine. Checked file permissions for wflogs and rules.php. Wflogs being 755 and rules.php being 644. After speaking with wordpress support all they could recommend was moving to MySQLi storage engine. From what I can tell there are no issues with the server configuration.

    Any help would be appreciated. Thanks.
     
  2. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    If permissions are OK, what are owner and group owner of the directories and files?
    Website user and client should be owner and group for the web/ directory and all files in the subtree. Have you checked suexec in website settings?
     
  3. mrunix

    mrunix New Member

    We are using suexec so each site has it's own user assigned. I've checked the settings and there are no issues there either.
    Thanks for the response.
     
  4. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Do you get any errors in error.log when trying to update rules? What do you have in php.ini disable_functions?
     
  5. mrunix

    mrunix New Member

    No errors in regards to the wordfence. It's mostly some theme conflicts such as the following PHP Notice: Undefined variable: img_src "File-Path".

    As for the disable_functions, nothing is defined. It only shows the following:
    Code:
    ; This directive allows you to disable certain functions for security reasons.
    ; It receives a comma-delimited list of function names.
    ; http://php.net/disable-functions
    disable_functions =
    
     
  6. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Did this issue start with all your sites at the same time? Or is it just one site?

    What php version and mode is the site running? If php-fpm, is it using chroot php-fpm or running outside chroot?

    What diagnostics / tests offhand?

    Did this fix the issue, or does it persist? Even if you don't want that for your long term setup, it might help track things down, eg. if that works, you know contacting the update servers works.

    You could try setting up a default wordpress site with only the wordfence plugin enabled and see if that works (eliminates potential plugin/theme conflicts).

    Beyond gathering info there, check other logs for errors, eg. the php-fpm daemon log in /var/log/php*.log; if there are no further clues, you can see if there are any debugging steps within wordfence itself that would log what it's doing and what fails - if that's not on option, I'd attach to the php process with strace or ltrace when it's trying to update and see what's happening (which starts getting a bit more technical...).
     
  7. mrunix

    mrunix New Member

    It started with all sites and each of them got the same error.

    The php version is 7.3.31 and it is using php-fpm. I believe it is outside chroot as we are using suexec. If I'm understanding you correctly.

    The diagnostics test used was the built in diagnostic tool in wordfence. It showed proper connectivity to all wordpress services.

    I haven't implemented their fix as I was hoping for an alternative. Since as far as I can tell there isn't anything incorrectly setup on the server or in wordpress. I was reluctant to change the MySQLi Storage Engine since they weren't able to give me a definitive answer. I'll move forward with the fix if that appears to be the best course of action though.

    Thanks for the assistance. I'll give an update if switching resolves the issue.
     
    Last edited: Dec 14, 2021
  8. nhybgtvfr

    nhybgtvfr Well-Known Member HowtoForge Supporter

    have you checked .user.ini in your docroot, and wordfence-waf.php in your wordpress root?

    a change in the php chroot settings may have affected the paths in these files.
    a recent update to wordfence may have also changed them to relative paths when a full path is still required.
     
  9. mrunix

    mrunix New Member

    I went through and a few of the sites to the MySQLi Storage Engine and have not seen any changes. Here are the changes I made just in case I made a mistake.

    In the sites root web directory I edited the wordfence-waf.php file with the following line:
    Code:
    define('WFWAF_STORGAE_ENGINE', 'mysqli');
    The file now reads as follows:
    Code:
    <?php
    // Before removing this file, please verify the PHP ini setting <code>auto_prepend_file</code> does not point to this.
    define('WFWAF_STORGAE_ENGINE', 'mysqli');
    // This file was the current value of auto_prepend_file during the Wordfence WAF installation (Tue, 13 Apr 2021 22:0$
    if (file_exists('/var/www/clients/client4/web12/web/wordfence-waf.php')) {
            include_once '/var/www/clients/client4/web12/web/wordfence-waf.php';
    }
    if (file_exists('/var/www/clients/client4/web12/web/wp-content/plugins/wordfence/waf/bootstrap.php')) {
            define("WFWAF_LOG_PATH", '/var/www/clients/client4/web12/web/wp-content/wflogs/');
            include_once '/var/www/clients/client4/web12/web/wp-content/plugins/wordfence/waf/bootstrap.php';
    }
    ?>
    After saving the changes, I tried updating the rules. However it is still giving me the same error. I followed the steps provided here https://www.wordfence.com/help/firewall/mysqli-storage-engine/ but am I missing something?

    The file paths appear to be unaffected and are showing full paths such as below:
    Code:
    auto_prepend_file = '/var/www/clients/client4/web12/web/wordfence-waf.php'
    Thanks in advance for the help.
     
    Last edited: Dec 14, 2021
  10. mrunix

    mrunix New Member

    Update: Created a fresh wordpress install and we are still receiving the same error. Has only default wp theme and wordfence installed.
     
  11. mrunix

    mrunix New Member

    Update: Fresh VM with a Debian 10 install running ISPConfig is having the issue as well.
     
  12. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    Maybe a problem with your network provider?
     
  13. mrunix

    mrunix New Member

    Whitelisted the wordfence / wordpress services in our router so the network shouldn't be causing any problems.
     
  14. Th0m

    Th0m ISPConfig Developer Staff Member ISPConfig Developer

    It shouldn't, but it might. Who knows where a second firewall could be, or what you might have misconfigured ;) No problems with Wordfence on my Debian servers using ISPConfig 3.

    You could also try deleting the rules.php file in wp-content/wflogs. Let Wordfence automatically regenerate the file (usually within 30 minutes).

    Maybe it's a license issue, where you are using the license for a second (sub)domain?

    Is WordFence up to date?
     
  15. chab43

    chab43 New Member

    The second server we built for testing was on an entirely separate internet connection so firewall misconfiguration or ISP blocking seems unlikely. Furthermore WordFence's own diagnostic tool always says there aren't issues connecting to wordfence servers: https://postimg.cc/sMZ3zxY5

    Test server was a vanilla build, following the Debian 10 Perfect Server guide to a T with no changes to any settings. Brand new WordPress 5.8.2 install with the latest version of WordFence from the plugin store.
    It's interesting because upon plugin install you get a fully populated rules.php file, but all subsequent attempts to update it will fail. Wordfence had a big red banner on our admin dashboard for all the sites saying it hadn't been able to update the rules in a month: https://postimg.cc/D8rWTC72

    It either throws a generic error Saying the update failed and to check our connectivity (which we have done) or it claims it can't write to the wflogs folder. We are using the default ispconfig permission structure, all site resources are owned be web# and are owner r/w so there shouldn't be any permission problems. We even tried the MySQLi storage engine and that fails as well.

    If you move/delete it, it does seem to eventually recreate the file as you suggested. But then why is it failing to update the file again after creation? This all started because a bunch of the sites claimed they hadn't been able to update in a month.

    We are using the free version of wordfence and it is up to date. It's worked for years with no issues and then about two months ago every site just stopped getting rule updates.
     
    Last edited: Dec 15, 2021
  16. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    That is strange, I use Wordfence on all our WordPress sites and haven't had a problem like this; I don't use Wordfence central, though, maybe that is related?

    As mentioned above, I would try attaching to the php process with strace when it does an update and see what you find. The output is long, so save it to a file.
     
    Th0m likes this.
  17. chab43

    chab43 New Member

    We don't use WordFence central. We manage each hosted site individually currently.

    My reading on strace makes it sound resource intensive because its recording each system call and it shouldn't be run on production servers. Is this correct or will it not be a problem?

    I've never used them before but willing to give it a shot if I can get some assistance interpreting the results.

    Do I attach it to the web users FPM process or do I attach to the main PHP process?
     
  18. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    It isn't likely to cause any noticeable performance issues on your server, but I would just run it on the new test server you setup, where there is no live traffic/activity to confuse things.
    You bet. In general you will get wordfence to try to update (and hope that takes a few seconds, not fails immediately), quickly identify the php-fpm process id (eg. use ps or lsof), attach to it with strace (eg. something like strace -ff -o /tmp/tracing.out -s1024 -p {pid} ), then you'll have the output - what you are looking for is generally towards the end (I'd search the ouput for 'No rules were updated.' then look at the few dozen (maybe couple hundred?) lines above that for the target area). You're hoping to find a failing system call eg. connect fail or such).

    There are other problems that can happen which won't show any failure in strace, eg. say your system can't verify the ssl certificate on wordfence's servers, so there may not be anything "failing" at the syscall level. But it should shed some light/understanding.

    Ideally attach to the child process which is trying to download the signature/rule updates, which means you have to initiate an update, then attach to it before it fails and exits (if it exits immediately, that may be nearly impossible without adding a 'sleep 10;' into their update code or similar). On a test system (no other activity), you might attach to the main php-fpm daemon and write all output from the children it creates; on a busy server that would be way too much to sift through.
     
  19. chab43

    chab43 New Member

    Ok great, since I've got a test server on my LAN at home I'll use that as there should be no traffic except for mine. That way I can just attach to the main PHP process and not worry about rushing to find the child (the update fails in less than 10 seconds generally, so not a lot of time). I'll post another update tonight with traces.
     
    Th0m likes this.
  20. chab43

    chab43 New Member

    I ended up attaching to the child process after all. The main file is to big for pastebin and it spawned two sub processes so I just tossed them all in a zip. I started it right before trying a manual update and ended it right after but it still logged a lot.

    I'm looking over it now but if more experienced eyes want to take a crack I'd appreciate it.

    I'm not finding anything when searching for "no rules updated"...

    EDIT: Jesse, at line 11590 you can see the beginning of the connection to one of wordfence's noc servers: 44.238.191.15 (they're all listed here https://www.wordfence.com/help/advanced/)

    I'm trying to dig around near that line to see if i can find the problem.

    EDIT2: Ok I've pastebin'd the section of the main trace where the connection attempts seem to start: https://controlc.com/4601ee86

    EDIT3: I really don't know how to read this trace properly but i do see a lot of resource unavailable and timeouts around the time the update appears to start in the trace log.


    EDIT4: so right after the IP line it looks to be reading a bunch of certdata from /var/www/clients/client1/web1/web/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/cacert.pem

    then on line 11699 it makes a write call and mentions something about noc4.wordfence.com, then several read commands that all seem to fail with resource temporarily unavailable?

    I'm starting to understand bits and pieces of this but I'm not there yet.
     

    Attached Files:

    Last edited: Dec 16, 2021

Share This Page