No cron jobs are run after running the ISPconfig3 Migration Toolkit (ISPcopy)

Discussion in 'Plugins/Modules/Addons' started by axxies, Oct 13, 2023.

Tags:
  1. axxies

    axxies Member

    Hi,

    I purchased the ISPmigration toolkit a week ago and I have used it for migrating an old server to a new (everything but the emails) and another time for migrating the emails to another server (because p25 is closed where I have the main server). Please disregard the mail server. It's not the issue (except for that no mails are going in/out of the main server).

    The thing is that whatever I do, the cron jobs aren't executed on the main server and this was never an issue with the old server which I migrated from. This in turn means that it doesn't work properly. I am for instance using NextCloud on one of the sites, and this requires at least 8.1, but mine is till at 8.0.x because the setting I made in the ISPC3 settings has not been executed.
    There are no error messages that gives any hints in /var/log/syslog , /var/log/ispconfig/ nor /var/log/apache2/ .
    The crontab only contains the two crons for ISPconfig3 (server.sh and cron.sh - both with full paths, and the .acme.sh cron)

    I really have no clue on what's wrong. The cron process IS running. Full cron IS enabled, but who knows, if crons doesn't work, then maybe that is one of the 348 cronjobs in queue right now.

    If I tail the cron.log in /var/log/ispconfig I do see a PHP fatal error. Could that be it?

    <snip>

    PHP Fatal error: Uncaught ValueError: Path cannot be empty in /usr/local/ispconfig/server/lib/classes/system.inc.php:884
    Stack trace:
    #0 /usr/local/ispconfig/server/lib/classes/system.inc.php(884): file_get_contents()
    #1 /usr/local/ispconfig/server/plugins-available/mail_plugin_dkim.inc.php(309): system->file_get_contents()
    #2 /usr/local/ispconfig/server/plugins-available/mail_plugin_dkim.inc.php(383): mail_plugin_dkim->remove_from_amavis()
    #3 /usr/local/ispconfig/server/plugins-available/mail_plugin_dkim.inc.php(394): mail_plugin_dkim->remove_dkim()
    #4 /usr/local/ispconfig/server/lib/classes/plugins.inc.php(120): mail_plugin_dkim->domain_dkim_delete()
    #5 /usr/local/ispconfig/server/mods-available/mail_module.inc.php(136): plugins->raiseEvent()
    #6 /usr/local/ispconfig/server/lib/classes/modules.inc.php(302): mail_module->process()
    #7 /usr/local/ispconfig/server/lib/classes/modules.inc.php(235): modules->raiseTableHook()
    #8 /usr/local/ispconfig/server/server.php(180): modules->processDatalog()
    #9 {main}
    thrown in /usr/local/ispconfig/server/lib/classes/system.inc.php on line 884​
    </snip>

    I use Debian 12.

    Is this by any chance something that has happened before when using the ISPcopy feature of the Migration Toolkit? Is it at all related?
    Any hints on what to do? And yes, I HAVE read the posts here about cron jobs not starting. Not every post because some seemed irrelevant for my case.

    Thanks!




    PS. @till CloudFlare blocked me when I tried to use the Business Support page for the Migration Toolkit. It's quite annoying. You should be aware of that the "business support" doesn't work. It's not my IP, I tried using a VPN service too, same result.
     
    Last edited: Oct 13, 2023
    axxies, Oct 13, 2023
    #1
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    I wonder why you used ISPCopy and not the Migration Tool from the Toolkit? ISPCopy is a special tool that is used to migrate slave nodes of multiserver setups that have to stay connected to their master only, it is not used to migrate standalone systems or do partial migrations. ISPCopy requires that old and new systems are identical in their setup and OS while the Migration Tool supports you to migrate to other OS and setups. That said, you can use ISPCopy for a single server system, but it has way stricter requirements in regard to the target setup, and the likeliness that something breaks due to system incompatibilities is way higher, so the Migration Tool is in almost all cases a better choice.

    Yes, that's possible. Did you run an ISPConfig update and let it reconfigure services as required after running ISPCopy? This is always required.

    That's indeed the reason.

    I'm sorry to hear that you were blocked by Cloudflare, our site is attacked a lot so we can not run it without a security solution like the one provided by CloudFlare. Just tested it, sending messages works fine from here and I receive messages through the form regularly. So it must be your IP or location or VPN that gets blocked by Cloudflare.
     
    Last edited: Oct 13, 2023
    till, Oct 13, 2023
    #2
  3. axxies

    axxies Member

    I used it because the help text indicated that it was the best for creating a clone of the old ISPconfig installation. Previously when I have purchased the Migration Toolkit, it messed up the structure of client IDs.

    There are differences in the old server and the new. The old was a Debian 10 server, while the new is a Debian 12 (and different versions of MySQL and Apache). The hardware is very different and I use LVM for the disks (besides that the disks now are NVMe disks too).


    To my knowledge I have done what was required.

    What can I do about it?

    I tried the VPN after everything else failed.
    Maybe you block out Sweden? I can see why ;)
     
    axxies, Oct 13, 2023
    #3
  4. axxies

    axxies Member

    This is what I read

    upload_2023-10-13_16-25-52.png
     
    axxies, Oct 13, 2023
    #4
  5. till

    till Super Moderator Staff Member ISPConfig Developer

    ClientID's can change, but the Migration Tool tries to keep ID's when possible and also adjusts paths where necessary.

    The problem is that a config from Debian 10 will not always work on Debian 12 without changes, that's why an exact copy of all config files, as ISPCopy is doing, can break the system.

    We have to find out in which way your old and new servers differ. I guess you installed rspamd on the new system while the old one uses amavisd. Please run these two commands on old and new server and post the result:

    ps aux | grep amavis
    ps aux | grep rspamd

    We do not block by country, of course :)
     
    till, Oct 13, 2023
    #5
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    This is correct, but as the readme of the ISPCopy tool outlines in the prerequisites section, you must use the same OS and exact same configuration. That's what is not the case on your system, and that's why you have issues now because coping over config files to a system that does not match the exact same specs will fail. This is the reason why I always recommend using the Migration tool, as it uses a different approach (importing records and data via API) and is therefore able to adapt the config to allow you to move to a more modern OS or an OS with different setup of services.
     
    till, Oct 13, 2023
    #6
  7. axxies

    axxies Member

    Ok, well, THAT was the reason why I chose ISPcopy and that I only saw that the ISPconfig as such should be identical, which they were.

    ok

    Yes, I have problems with rspamd too - now I know why :)


    Oh and by the way, both hosts are VMs on top of Proxmox.

    (I wanted to leave my old hosting provider since 15 years ago since they suddenly changed their terms and didn't fuilfill what they would do. I got pissed off so I got myself some badass HW (not that much badass these days, but far better than their best HW) which I put together and just move it there instead. Just moving the Proxmox disk. Debian 10 was a bit too old though for both the host and the VM)
     
    axxies, Oct 13, 2023
    #7
  8. axxies

    axxies Member


    OK, then I know the root cause of this.

    What would you recommend? That I create a new VM and migrate it to there instead using the Migration Toolkit. There is pleeeenty of disk so that is not a problem. It's more a time issue, but I have wasted a lot of time so far in trying to trace the error.
     
    axxies, Oct 13, 2023
    #8
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    Ok, so the issue is you copied over a amavis setup to a rspamd system, this will break. also all mail directories are in wrong paths now and so on, as the paths are different when using amavis vs. rspamd. The Migration Tool is able to deal with all these differences, while ISPCopy can't as it just copies over existing config files and requires the exact same setup on old and new server for this.
     
    till, Oct 13, 2023
    #9
  10. axxies

    axxies Member

    Ok, in case of the mail directories, I haven't noticed any issues with that. I had to put that on another hosting provider with a tiny VPS and with ISPconfig3. The mail server is elsewhere than the main server which I am having these issues with.
     
    axxies, Oct 13, 2023
    #10
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    Basically, you have three options.

    You can either try to fix the old system, but I can't even give step-by-step instructions for this as I never had that case, might involve some tweaks in the database plus trying to fix the file structure etc.

    Or you set up the new system using amavis plus the same LE client (probably certbot on the old system, new default is acme.sh) with Debian 12. This might work with ISPCopy with some minor manual adjustments afterwards to fix differences between deb 10 and 12 when the whole overall setup is the same. You can instruct the auto-installer via command line options to use Amavis and certbot.

    Or you go the safe route plus get a modernized setup with a new server with rspamd (but you must take care to use the same LE client here too, likely certbot) and use the Migration Tool.
     
    till, Oct 13, 2023
    #11
  12. till

    till Super Moderator Staff Member ISPConfig Developer

    Ok, so this server does not has active mailboxes? In this case, we can maybe fix the issues on the current server.
     
    till, Oct 13, 2023
    #12
  13. axxies

    axxies Member

    Thank you!

    I gave up. I created a new VM and ran the migration toolkit to that one. Now I am fixing other issues after the migration. Certificate stuff for now. I noticed that the migrationtoolkit copied the /var/www/<website>/ssl so I tried to rsync what was in /root/.acme.sh/ (since that wasn't transferred), but that wasn't successful. I guess I have to disable all websites Let's Encrypt certs, save it, wait for the cron to do its job and then enable it again.

    Cron works now at least :)

    [EDIT: I regenerated all certificates, so that thing is solved now. For some reason it had to be done from within ISPC3, when I tried to do an .acme.sh --renewAll it didn't work. ]
     
    Last edited: Oct 14, 2023
    axxies, Oct 14, 2023
    #13
  14. axxies

    axxies Member

    Another thing... I noticed when I migrated (ISP_copy) the email inboxes to the other server that it was not possible to do a resync (from within ISPC3) to get them into ISPC3. Is this intentional or is it supposed to work?
     
    axxies, Oct 14, 2023
    #14
  15. till

    till Super Moderator Staff Member ISPConfig Developer

    This should work. I guess this did not work in your case due to the amavisd vs. Rspamd issue we had in the first migration attempt and the resync can not work when server.sh processing is blocked. As I mentioned, the paths are different, so a resync will likely fail as well as the mailboxes are in different folders.
     
    till, Oct 14, 2023
    #15
  16. axxies

    axxies Member

    No, that is not what happened.
    I started with manually creating the clients and domains on that host that was only intended for email.
    Then I rsynced all mails to the new mail server so the structure of mails, where they belong etc should be identical to the original server (the reason why I have a separate mail server is because port 25 is closed for the connection where I have the main server).

    Then I ran the resync feature inside of ISPC3. Nothing happened. ISPC3 was not aware of the inboxes nor the emails after that.

    I had to delete it all and start all over with an isp_copy of emails only. That worked.

    Rspamd, Amavis etc was never affected and there are no problems with them on the new mail server.

    It just didn't work to do an ISPC3 resync.

    Any ideas? A bug perhaps?
     
    axxies, Oct 14, 2023
    #16
  17. till

    till Super Moderator Staff Member ISPConfig Developer

    Unlikely that there is a bug, resync works exactly as it is intended to do. But without debug output from server.sh script, I can not say what was wrong with the system at that time, the most likely reason is that you copied over filesystem data while the setup on the old and new system was not 100% identical, e.g. when the old system would have used courier and the new system uses dovecot, this will fail too. Or you copied over data that was not in the ISPConfig database in the exact same way. resync will not detect any folders you copy over from a different server, it is not meant to do that. Resync is a function to update config files based on existing ISPConfig database content.

    Besides that, you posted an error in regard to server.sh which halted server.sh and server.sh is what resync is running, so if this was the same server, the resync was halted by the amavis vs. rspamd issue.

    To migrate emails only, you should have used the Migraton Tool as well, not isp_copy. The Migration Tool has command line options to limit the migration to emails only.
     
    till, Oct 14, 2023
    #17
  18. axxies

    axxies Member

    The only things I rsynced (not rEsynced) was the mail directories under /var/vmail , not the files and other directories there. THEN I tried the resync. It was hopeless. I never got that to work.

    No, that was not the mail server. That was the server for everything else BUT email. I understand that it is confusing.

    Yeps, I know that now, but that worked at least.


    Either way, thanks for your prompt answers and willingness to help me!
     
    axxies, Oct 14, 2023
    #18
  19. axxies

    axxies Member

    @till , I have a suggestion for ISPC3... Similar to that websites, databases etc belong to a certain clientID, so do email domains, but that is not reflected on disk. My suggestion is that ISPC3 creates a "clients" directory under "/var/vmail" and then the different clientIDs under there and then the email domains under respective clientID.

    I did this for my mail server (plus created a symlink from /var/vmail that pointed out the mail domains under /var/vmail/clients/client<id>/<domain> so I could make sure to only make backups of my own emails and not my friend's emails. We share the same mail server.

    Generally I think the structure would make sense and be consistent with how it is with everything else.
     
    axxies, Oct 17, 2023
    #19

Share This Page