Just as info, I'm using Wordfence as well on one of my sites on an ISPConfig Debain 10 system and it works flawlessly, so I doubt the problem is ISPConfig related.
Hi Till, yes the consensus seems to be it's something with Wordfence but I'm unable to get Wordfence staff to give us further troubleshooting beyond deleting wflogs and or using the MySQLi storage engine. I'm considering paying for Wordfence premium so we can get access to their second tier support staff. I've been using ispconfig and Wordfence for years and this is the first time ive ever had such a strange issue even with a totally fresh server build. I'm going to take another look at it with fresh eyes this morning but I don't know if I'll be able to discover anything new. EDIT: Ok I've pastebin'd the section of the main trace where the connection attempts seem to start: https://controlc.com/4601ee86
Yeah, I don't know where you saw that message, but it could be generated when you view the status or something. That trace (pid 2724) does show successfully connecting to noc4.wordfence.com and pulling down a lot of data, so we learned that that part is working fine (past connection and ssl issues). It then performs some sql queries which I believe are gathering the data needed to build the actual wflogs/rules.php file, then (with no attempt to write that file), it returns this and exits: Code: {\"learningMode\":false,\"rules\":{\"2147483648\":{\"ruleID\":2147483648,\"type\":null,\"category\":\"auth-bypass\",\"score\":100,\"description\":\"Infinite WP Client - Authentication Bypass < 1.9.4.5\",\"whitelist\":0,\"action\":\"block\"}},\"whitelistedURLParams\":[],\"disabledRules\":[],\"rulesLastUpdated\":1639257481,\"isPaid\":false,\"updated\":false,\"failure\":\"filesystem\",\"nonce\":\"24c3d8f545\"} My guess (without being familiar with wordfence's code) is the updated:false indicates there were no rules updates, or possibly no changes at all (rule updates and blocked hosts/usernames, though I don't know if it would attempt to rewrite rules.php if there were no changes to blocked hosts/usernames), so maybe "do nothing" is the correct action. Then there's the failure:filesystem in the return as well, which I don't know what it indicates, as I don't see anything failing in that strace output. Maybe some other process (at that time, or even at another time and status tracked in the db?) had a filesystem related failure and this is reporting the result of that other process? So what environment is this in? Ie. what filesystem type, vps/container technology, web server? I wouldn't suspect an apparmor type issue, as php can apparently function well enough to install plugins an such, and writing rules.php wouldn't be much different. You have any security scanners in place? (Eg. something that triggered via inotify and scanned files as they were written could come into play, though you would expect such to log it's activity as well.)
FWIW, a quick search finds this error generated in js/admin.####.js as the catch-all failure message (for any reason other than 'ratelimit' or 'unreachable'). It looks like failure:filesystem is returned by ajax_updateWAFRules_callback() when the rules update did get data with a 'signature' but the event's fire() function returned failure. The rules updater will verify the signature if openssl is found - do you have openssl installed? Any identifiable issues with your openssl installation there? If you don't have openssl it verifies the rules data against a hash (effectively a checksum). What do you get from php cli Code: php -r 'echo "openssl_verify " . ( function_exists("openssl_verify") ? "does exist" : "does not exist" ) . "\n";' And more importantly, what do you get if you run that code in a php file from the website (ie. running under php-fpm, not cli, which could be a different php version)? Code: <?php echo "openssl_verify " . ( function_exists("openssl_verify") ? "does exist" : "does not exist" ) . "\n";
As mrunix mentioned at the start of the thread we always get one of the two same error messages. This is helpful. If I present this to wordfence with my support thread over there maybe they can provide an explanation from their dev team about why it returns update:false and never tries to write a file. How would it know if there are rule updates or not if I've deleted the rules.php file? The test server where I made the trace is built in Hyper-V in a Gen 2 virtual machine here on my home PC (Windows 10 Enterprise). The other server this is on is also a Hyper-V box though I'm not sure of the specific hardware. Ext4 is the file system, apache is the web server. Apparmor is loaded but doesnt seem to be monitoring apache or php. I have made no changes to apparmor beyond whatever the defaults are in debian and ispconfig. Code: root@debian:~# aa-status apparmor module is loaded. 18 profiles are loaded. 16 profiles are in enforce mode. /usr/bin/evince /usr/bin/evince-previewer /usr/bin/evince-previewer//sanitized_helper /usr/bin/evince-thumbnailer /usr/bin/evince//sanitized_helper /usr/bin/freshclam /usr/bin/man /usr/sbin/clamd /usr/sbin/ntpd libreoffice-senddoc libreoffice-soffice//gpg libreoffice-xpdfimport man_filter man_groff nvidia_modprobe nvidia_modprobe//kmod 2 profiles are in complain mode. libreoffice-oopslash libreoffice-soffice 3 processes have profiles defined. 3 processes are in enforce mode. /usr/bin/freshclam (727) /usr/sbin/clamd (567) /usr/sbin/ntpd (626) 0 processes are in complain mode. 0 processes are unconfined but have a profile defined. No security scanners in place, just clamav as part of the ispconfig install. This is 100% vanilla debian with zero changes besides setting up ispconfig. I do have openssl installed and as far as I know there are no issues with the install. How would I verify if there were issues? It generated my self-signed certificates for ISPconfig just fine. I get: Code: root@debian:~# php -r 'echo "openssl_verify " . ( function_exists("openssl_verify") ? "does exist" : "does not exist" ) . "\n";' openssl_verify does exist I threw that into a file (test.php) in the wordpress root directory and opened it. It returns "Openssl_verify does exist" I also tried disabling ssl verification (it's an option in the wordfence debug settings) and it didn't seem to make a difference.
I'd add some code in vendor/wordfence/wf-waf/src/lib/waf.php to dump (eg. print_r()) the entire $jsonData array into a file right after it is set on line 1859: Code: $jsonData = wfWAFUtils::json_decode($this->response->getBody(), true); For convenience you might also save $jsonData['data']['signature'] and $jsonData['data']['rules'] into separate files, so you can see what data waf.php is working with. You can write a little php script that would try to verify that data (see verifySignedRequest() in waf.php, and the $publicKey is hardcoded in that file) and see if openssl_verify succeeds. I really have few guesses as to what's going on at this point, or why "it's only on your systems", so just keep tracing through what is going on till you find more clues.
Thank you for the extremely detailed explanation and suggested next steps. We're going to take what we've learned so far and approach wordfence for some further assistance before we start trying to add print statements to their code in an attempt to further debug. TBH neither of us are programmers and I know jack about writing PHP but if the wordfence team continues to be unhelpful I'll try what you suggested and see if I can't narrow it down further and report back.
After tons of back and forth with WordFence they helped us narrow it down to DNS on the server. When I built the server, even though it's not part of the perfect server tutorial, I manually defined the DNS servers I wanted to be used in /etc/resolv.conf as I'd always done this in the past without issues. Apparently this caused some weirdness with DNS resolution and the solution was to set resolv.conf to 127.0.0.53, the listening address for the systemd-resolved daemon and then define our desired DNS servers in etc/systemd/resolved.conf instead. Our wordfence rules are back to updating regularly. My apologies for my assertion that I'd not done anything outside of the ispconfig tutorial. I completely forgot I'd ever changed resolv.conf
