Just noticed today that scheduled backups stopped working. The last backup was several weeks ago, about the time I upgraded ]# ispconfig_update.sh. To be fair, I'm not sure if this was the cause. Initial Troubleshooting: Running a website manual backup works. Executing from command line seems to work, no errors ]# php /usr/local/ispconfig/server/cron_debug.php --cronjob=500-backup.inc.php ]# df -h shows plenty of disk space. Tried changing backup time from 01:30 to 17:30. Switched it back to 01:30 for the next day, it did not run. Installed BorgBackup even though I do not use it, just in case that was preventing standard backups. All other CronJobs run normally, no other disruptions that I can see. Server Environment: CentOS 7.9, ISPConfig 3.2.8p1 Maybe I told ispconfig_update.sh to reconfigure services last update, and this affected the backups? Thanks for any ideas!
Did you set log level to debug and then run the cron backup to see what happens? https://www.howtoforge.com/community/threads/please-read-before-posting.58408/ https://www.faqforge.com/linux/debugging-ispconfig-3-server-actions-in-case-of-a-failure/
Hi Taleman, Thanks for your reply. I haven't debugged ISPConfig in a while a completely forgot about the cron debugging feature. 1. I set the log level to Debug as instructed. 2. I set the backups to run in a hour. Will see what comes up in the log. Here are the results from Till's common issues script: Code: wget -q -O htf-common-issues.php "http://gitplace.net/pixcept/ispconfig-tools/raw/stable/htf-common-issues.php" && php -q htf-common-issues.php cat htf_report.txt | more Code: ##### SCRIPT FINISHED ##### Results can be found in htf_report.txt To view results use your favourite text editor or type 'cat htf_report.txt | more' on the server console. If you want to see the non-anonymized output start the script with --debug as parameter (php -q htf-common-issues.php --debug). [root@blade home]# cat htf_report.txt | more ##### SERVER ##### IP-address (as per hostname): ***.***.***.*** [WARN] could not determine server's ip address by ifconfig [INFO] OS version is "CentOS Linux release 7.9.2009 (Core)" [INFO] uptime: 14:21:38 up 76 days, 12:42, 1 user, load average: 0.67, 0.55, 0.62 [INFO] memory: total used free shared buff/cache available Mem: 31G 7.5G 1.0G 1.7G 22G 21G Swap: 15G 48M 15G [INFO] systemd failed services status: UNIT LOAD ACTIVE SUB DESCRIPTION ● cdp-agent.service loaded failed failed LSB: Starts R1Soft CDP Agent ● systemd-tmpfiles-clean.service loaded failed failed Cleanup of Temporary Directories LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. 2 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'. [INFO] ISPConfig is installed. ##### ISPCONFIG ##### ISPConfig version is 3.2.8p1 ##### VERSION CHECK ##### [INFO] php (cli) version is 7.4.16 [INFO] php-cgi (used for cgi php in default vhost!) is version 7.4.16 ##### PORT CHECK ##### [WARN] Port 22 (SSH server) seems NOT to be listening ##### MAIL SERVER CHECK ##### [WARN] I found no "submission" entry in your postfix master.cf [INFO] this is not critical, but if you want to offer port 587 for smtp connections you have to enable this. ##### RUNNING SERVER PROCESSES ##### [INFO] I found the following web server(s): Unknown process (httpd) (PID 29277) [INFO] I found the following mail server(s): Postfix (PID 12564) [INFO] I found the following pop3 server(s): Dovecot (PID 12631) [INFO] I found the following imap server(s): Dovecot (PID 12631) [INFO] I found the following ftp server(s): PureFTP (PID 12880) ##### LISTENING PORTS ##### (only () Local (Address) [anywhere]:993 (12631/dovecot) [anywhere]:995 (12631/dovecot) [localhost]:10024 (12592/amavisd) [localhost]:10025 (12564/master) [localhost]:10026 (12592/amavisd) [localhost]:10027 (12564/master) [anywhere]:15531 (1137/sshd) [anywhere]:110 (12631/dovecot) [anywhere]:143 (12631/dovecot) [anywhere]:1167 (4212/cdp) [anywhere]:465 (12564/master) ***.***.***.***:53 (12905/named) ***.***.***.***:53 (12905/named) ***.***.***.***:53 (12905/named) [localhost]:53 (12905/named) [anywhere]:21 (12880/pure-ftpd) [localhost]:953 (12905/named) [anywhere]:25 (12564/master) [localhost]:8891 (1373/opendkim) *:*:*:*::*:993 (12631/dovecot) *:*:*:*::*:995 (12631/dovecot) *:*:*:*::*:10024 (12592/amavisd) *:*:*:*::*:10026 (12592/amavisd) *:*:*:*::*:3306 (12434/mysqld) [localhost]5531 (1137/sshd) [localhost]10 (12631/dovecot) [localhost]43 (12631/dovecot) *:*:*:*::*:8080 (29277/httpd) *:*:*:*::*:80 (29277/httpd) *:*:*:*::*:8081 (29277/httpd) *:*:*:*::*:465 (12564/master) *:*:*:*::*:53 (12905/named) *:*:*:*::*:21 (12880/pure-ftpd) *:*:*:*::*:953 (12905/named) *:*:*:*::*:25 (12564/master) *:*:*:*::*:443 (29277/httpd) ##### IPTABLES ##### Chain INPUT (policy ACCEPT) target prot opt source destination f2b-FTP tcp -- [anywhere]/0 [anywhere]/0 tcp dpt:21 f2b-dovecot tcp -- [anywhere]/0 [anywhere]/0 multiport dports 110,995,1 43,993 f2b-postfix-sasl tcp -- [anywhere]/0 [anywhere]/0 multiport dports 25,4 65,587 Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain f2b-FTP (1 references) target prot opt source destination RETURN all -- [anywhere]/0 [anywhere]/0 Chain f2b-dovecot (1 references) target prot opt source destination RETURN all -- [anywhere]/0 [anywhere]/0 Chain f2b-postfix-sasl (1 references) target prot opt source destination REJECT all -- ***.***.***.*** [anywhere]/0 reject-with icmp-port-unrea chable REJECT all -- ***.***.***.*** [anywhere]/0 reject-with icmp-port-unreach able REJECT all -- ***.***.***.*** [anywhere]/0 reject-with icmp-port-unrea chable REJECT all -- ***.***.***.*** [anywhere]/0 reject-with icmp-port-unrea chable REJECT all -- ***.***.***.*** [anywhere]/0 reject-with icmp-port-unreac hable REJECT all -- ***.***.***.*** [anywhere]/0 reject-with icmp-port-unrea chable RETURN all -- [anywhere]/0 [anywhere]/0 ##### LET'S ENCRYPT ##### Certbot is installed in /usr/bin/letsencrypt
I may be wrong, but I don't think the schedule works like that, there is a "next backup cron job" scheduled to run at the moment, and when that runs (at 1:30am it sounds like) it will set the time for the next job according to the time it is configured to run at.
Ah...you may be right. So I'd have to wait another 24 hours for the next one to trigger? Here are my settings:
No. The version that is shipped with CentOS 7 is PHP 5.4.16. The installed PHP version is also not the most recent version of the PHP 7.4 branch. As long as the PHP version was correctly upgraded/installed I don't see any issues, although it is not recommended to upgrade the base PHP version... The question remains though, how the actual situation looks like on the server. So, how or from which repository (or software collection) was this PHP version installed and what are the ispconfig.log and especially the cron.log files saying when the OP runs the backup via cron_debug.php? @bpmee
Hi @michelangelo 1. I upgraded PHP 1+ years ago because some software (Wordpress, and some plugins) were complaining about PHP 5.x. Both ISPConfig and backups ran fine after the update. The backup problem started in the past few weeks. 2. I believe it was EPEL repo and REMI repos? As I said, no major problems after update. This is a common thing to do as PHP 5 is no longer supported as of 2019. 3. I can send ]# php -i output if you need it. Hi @Taleman - I was able to install BorgBackup successfully on my system. Again, I don't use it. ISPConfig warned it was missing. I thought this might be causing backups stop running. Hi @all I scheduled the backup to run at a different time yesterday (15:30) to test. Looks like it worked. Immediately following that result, I rescheduled it to 05:00 (normal). I'm waiting for that event to trigger tomorrow morning, as it takes 24 hours following what @Jesse Norell and @till advised. Will see if it runs. Question: Any chance the backup script conflicts with something running around 05:00?
This is actually not common what you did since the best-practice way is to set up a newer PHP version for ISPConfig under "Additional PHP Versions" and leave the base PHP version with what it was initally shipped. The distribution maintainer always takes care of the security updates and will apply them as necessary. Upgrading the base PHP version is ofc generally possible but only adviseable if you know what you might cause with such actions.
@michelangelo Thanks for your expertise. I didn't know about that feature. I followed some tutorials I found elsewhere. I was lucky the upgrade didn't cause any additional problems.
It will run tomorrow at the same time it ran today and the day after tomorrow it will run at 05:00. The next start time is set when the cronjob runs, so when you change the time after the current run of a cronjob starts, the change will get applied after the next run. Changing the base PHP version on centOS is less critical than changing it on Debian and ubuntu as CentOS puts every PHP version in the same path, so there will be no issues unless you choose a PHP version > 7.4 on CentOS.
So @bpmee, note that out of a concern for using outdated/insecure php versions, you installed a specific 7.4.x point release which is now a year out of date and has many known security vulnerabilities. You should ensure all your php versions stay updated (I'm not familiar enough with centos to give specific recommendations in doing so).
@till, @jesse Norwell - Thanks very much. I did a ]# yum update php and nothing was marked for update (yet). I periodically check my packages. Luckily no security issues (yet). Will monitor backups over the next day. I now understand that backup time is set with each cron job, firing the backup at the new time after the existing day's job.
Hi, scheduled backups for 05:00 didn't run. I don't see any errors at that time in /cron.log or /ispconfig.log. Any ideas?
@Taleman @Jesse Norell Scheduled backups still aren't running. I don't see any errors when I run the debugging script from the command line: Code: ]# php /usr/local/ispconfig/server/cron_debug.php --cronjob=500-backup.inc.php Can I run ISPConfig's actual backup script from the command line and watch it "live" for any errors? Very strange as the rest of the server is fine. There are no alerts in the Monitoring panel.
SOLVED! - Crontab was missing the ISPConfig Cron task. Code: * * * * * /usr/local/ispconfig/server/cron.sh 2>&1 | while read line; do echo `/bin/date` "$line" >> /var/log/ispconfig/cron.log; done The above must be present in addition to the /server.sh job. I initially missed it because standard debugging didn't return any errors. Also, ISPConfig's Monitor had no errors or warnings. Backups are running nicely again. Thanks @till, @Taleman @Jesse Norell , @michelangelo for your assistance! So everything was fine **except** for the cronjob entry on the root crontab. Finally, I researched potential causes. I don't think it was any ISPConfig update. Rather, I may have removed the cronjob by mistake or some other script trimmed it off. I've been using ISPConfig since 2008. While I'm not an "expert" user, I've successfully managed my own server over the years. To date, no major problems, security issues, or hacks. It's very stable and reliable!
