A few nights ago the folder /etc/ssl/certs started to get wiped out every night. Mail can't be sent and other problems occur. mail.log gets these: 2024-10-03T03:06:04.048497+03:00 host1 dovecot: doveadm([email protected]): Error: Disconnected from remote: Received invalid SSL certificate: unable to get local issuer certificate: /C=US/O=Let's Encrypt/CN=R10 (check ssl_client_ca_* settings?) syslog has these: syslog:2024-10-03T06:07:14.961708+03:00 host1 certbot[3244521]: Failed to renew certificate example.com with error: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/ssl/certs/ca-certificates.crt This command solves it for the day: # update-ca-certificates Updating certificates in /etc/ssl/certs... rehash: warning: skipping ispserver.pem, it does not contain exactly one certificate or CRL 141 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... done. All certificates inside have current dates, meaning they were all missing prior to the update command. The other server (master master) has old dates for most and has not had this problem. I have checked the cron for any mention of certs. /etc# grep -R "cert" cron* cron.d/certbot:0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew --no-random-sleep-on-renew The first error seen in the mail.log, past three nights: 2024-10-03 01:41:27 2024-10-02 01:16:40 2024-10-01T01:16:45 How should one proceed in troubleshooting? It's a debian 12 system (updated from earlier versions in July), but this problem only started 4-5 days ago.
It seems it has something to do witth a newly installed ssh jailkit user for a client and unison? I use unison to replicate between two servers in case one VPS is lost. I installed auditd to get to the bottom of this. I audited for ispserver.pem but all certs are affected (wiped). Code: auditctl -w /etc/ssl/certs/ispserver.pem -p w -k ssl_deletion Here is the result: Timestamp: Sun Oct 6 00:55:09 2024 Process: A process named jk_update (likely a script or Python program) deleted the file. Executable: /usr/bin/python3.11 Current Working Directory (CWD): /usr/local/ispconfig/server File Path Deleted: /var/www/clients/client13/web25/lib/.unison.ssl.2a2ef8f072849a7d8cedf789b062f733.unison.tmp/certs/ Syscall: SYSCALL msg=audit(1728165309.273:173993) - syscall 87 (likely unlink or a related system call). UID (User): The action was performed by root (UID: 0). The File Path deleted is a direct link points to the certs folder. ls -lA /var/www/clients/client13/web25/lib/.unison.ssl.2a2ef8f072849a7d8cedf789b062f733.unison.tmp total 0 lrwxrwxrwx 1 root root 14 Sep 27 11:53 certs -> /etc/ssl/certs Do these details help anyone to solve this mystery?
The issue is strange. I haven't heard of that issue yet, but I don't run any unison synced systems at the moment. So we have a file that is a hard link in a folder synce dby unison here and the target of the hardlink is outside of the synced folder and unsion then removes the file when syncing with the other server. Please check the jail of the user; I guess it still has the SSL files, but the synced ones from the other server, probably because they were created after the one on the local system.
Server2 didn't have the folder /var/www/clients/client13/web25/lib/.unison.ssl.2a2ef8f072849a7d8cedf789b062f733.unison.tmp, only Server1. Code: server2: /var/www/clients/client13/web25/lib# ls -la total 32 drwxr-xr-x 6 root root 4096 Oct 6 01:01 . drwxr-xr-x 8 root root 4096 Sep 27 11:53 .. drwxr-xr-x 2 root root 4096 Oct 6 01:01 openssh lrwxrwxrwx 1 root root 19 Oct 6 01:01 sftp-server -> openssh/sftp-server drwxr-xr-x 2 root root 4096 Oct 6 01:01 ssl drwxr-xr-x 16 root root 4096 Sep 27 11:52 terminfo drwxr-xr-x 2 root root 12288 Oct 6 01:01 x86_64-linux-gnu /var/www/clients/client13/web25/lib# ls -la ssl/ total 8 drwxr-xr-x 2 root root 4096 Oct 6 01:01 . drwxr-xr-x 6 root root 4096 Oct 6 01:01 .. lrwxrwxrwx 1 root root 14 Oct 6 01:01 certs -> /etc/ssl/certs What could be the reason the jk_update runs in the night around the same time, is it a maintenance feature triggered from ispconfig? It seems to follow the link and delete the destination content for some reason. The /var/www/clients/client13/web25/lib/.unison.ssl.2a2ef8f072849a7d8cedf789b062f733.unison.tmp-folder seems to be a leftover from unison, I now deleted the folder and I'm curious to see if things change tomorrow night.
There is a nightly cronjob that updates the programs in the jails. You can try to move the file /usr/local/ispconfig/server/lib/classes/cron.d/600-jailkit_maintenance.inc.php on the second server to another location temporarily if removing the /var/www/clients/client13/web25/lib/.unison.ssl.2a2ef8f072849a7d8cedf789b062f733.unison.tmp-folder does not help.
