recovering from disaster....

Discussion in 'ISPConfig 3 Priority Support' started by craig baker, Oct 12, 2025.

  1. till

    till Super Moderator Staff Member ISPConfig Developer

    You should use the same LE client on the old and new server, as mentioned under prerequisites for a server migration: https://www.howtoforge.com/tutorial...-confixx-plesk-to-ispconfig-31-single-server/
    So when your old system is using certbot, your new one should use it as well.

    As I mentioned, you delete one account, so your system will have just a single certbot account; there is no need to alter any file content of certbot config files.
     
  2. craig baker

    craig baker Member HowtoForge Supporter

    I use --use-certbot for that reason in the new build. how do I delete the account? on ns10 (source) both the v01 and v02 api account folders point to the same place (v01)!
    do I just delete the v02 api folder and redo the migration (after wipe and reload deb12/ispconfig)?
    do I change ns9 to ns10 in the meta file since it no longer exists?

    one more thing, can I do a migration completely within my local network to a server that may not have a 'public' ip yet? so both would be 192.168.2.xxx ?
     
    Last edited: Nov 5, 2025
  3. till

    till Super Moderator Staff Member ISPConfig Developer

    Which is perfectly fine, as v01 and vo2 are not accounts, they are API versions. As I explained above, accounts are the folders inside these folders.

    No. As I mentioned, these are versions and not accounts. v01 = API version 1, v02 = API version 2. The accounts are inside these folders. An account gets removed by deleting the account folder of that account and not one of the global version folders.

    I've answered that question already, so here again:

     
  4. craig baker

    craig baker Member HowtoForge Supporter

    I thought I had asked - but when I do a migration, can I migrate to a server that is on a LOCAL ip, but has no public ip yet?
    ie from 192.168.2.x to 192.168.2.y where the source DOES have a public ip (ns10). the target will have a public ip in future, but currently it may just have the local ip.
    does that work? or must both be reachable from the big bad world?

    also - I'm getting back the /var tree from the dead server. nothing I can boot of course.
    but when I have /var/www I can copy the domains and their websites to same corresponding folder on new server.
    when I have /var/lib/mysql I can copy over the databases for these domains to the same database name on new server (which may not exist until I make the copy.
    and /var/mail/vmail will get the emails associated with these domains.
    from what I can see, copying over the database and the /web folder seems to be enough to bring up the website once database is correct.
    mysql should see the database properly when the database mynewdb (from/var/lib/mysql/mynewdb) off the recovered drive is added to /var/lib/mysql/mynewdb folder on new drive. so I need to do anything to register it with mysql? or is the presence of the folder enough (with correct permissions of course).
    same with the emails right? if I copy over the /var/vmails/whoever folder from recovered drive to /var/vmail, and delete the cache, dovecot will recreate it including all the new emails next login wont it?
    maybe nightmare is almost over...
    thanks for your continued and invaluable help till. now about mailman.... (grin)
     
  5. till

    till Super Moderator Staff Member ISPConfig Developer

    Yes.

    This will likely not work if they are InnoDB tables (which is the default format for MySQL and MariaDB for years), as InnoDB stores all data for all databases in a single file, unless you reconfigured it to use separate files per database. The files you see in the database folders are just the form files, not the actual data. If you have old-style MySQL databases, copying will work, though. Best way to restore such a dead database is to try to restore the whole database folder into a separate VM or use Docker and then export the databases from there properly using mysqldump.

    Website files and email files can be copied.
     
  6. craig baker

    craig baker Member HowtoForge Supporter

    ok - so how about I use a brand new server running deb12, NOT with autoinstaller, then install mysql and copy the entire /var/lib/mysql folder from the failed server. you think mysqldump would then work correctly?
    the failed server was running deb12.
     
  7. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    If it's just another Deb12 vm, just go for it. You've got nothing to loose to copy to it right?
     
  8. craig baker

    craig baker Member HowtoForge Supporter

    trying the migration to a local ip, it fails at the 'give URL to login to the /remote/' matters not if I give https or http, API calls fail.
    the local ip address has no SSL yet, obviously.
    I seem to remember there was a way to migrate without the SSL login?


     
  9. craig baker

    craig baker Member HowtoForge Supporter

    ok I tried to find the old post but could not.
    during migrate at the URL connect (which is to https://192.168.2.xx:8080/remote/) I get the API connection failure.
    doing a wget on the address gives:
    wget https://192.168.2.230:8080/remote/
    --2025-11-08 07:01:06-- https://192.168.2.230:8080/remote/
    Connecting to 192.168.2.230:8080... connected.
    ERROR: The certificate of ‘192.168.2.230’ is not trusted.
    ERROR: The certificate of ‘192.168.2.230’ hasn't got a known issuer.
    The certificate's owner does not match hostname ‘192.168.2.230’

    which of course it does not match. I thought we could migrate to a local ip that has no public ip yet? how is this achieved?
    http://192.168.2.230:8080/remote/ also gives same error. what am I missing?
    could not find the old post about how to disable SSL (I think it was a centos to debian migration?)
     
  10. craig baker

    craig baker Member HowtoForge Supporter

    another issue - I just redid the migration from ns10 to ns11 (not the internal ip one above different server)
    and migrate completes but I see:

    [2687/4239] Syncing /var/lib/mailman/spam/ to /var/lib/mailman/spam/
    0 100% 0.00kB/s 0:00:00 (xfr#0, to-chk=0/1)
    [4239/4239]
    [ERROR] API call to server_config_set failed. See log file for details.
    [WARN] Failed setting migration mode inactive on the target web server.
    [ERROR] API call to system_config_set failed. See log file for details.
    [WARN] Failed restoring prefixes on target. You find them under system -> Main config -> Sites.
    =============
    Migration tool run completed.

    tailing migrate.log produces:
    [root@ns10 migration]# tail migrate.log
    2025-11-08 16:42:26 - [WARN] Curl exception: cURL error: [7] Failed to connect to ns11.cdbsystems.com port 8080: Connection refused
    2025-11-08 16:42:26 - [ERROR] JSON API ERROR in API call (system_config_set): NO ACCESS
    2025-11-08 16:42:26 - [ERROR] API call to system_config_set failed.
    2025-11-08 16:42:26 - [ERROR] JSON API ERROR. Arguments sent were: array (
    'section' => 'sites',
    'key' => 'dbname_prefix',
    'value' => 'c[CLIENTID]',
    'session_id' => 'q3cf763fc624750ab1ef480a6e58a68ac23e61d53',
    )
    2025-11-08 16:42:26 - [WARN] Failed restoring prefixes on target. You find them under system -> Main config -> Sites.

    hmmm. loss of communication for some reason???
     
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    That's ok as it was at the end, so the migration was finished successfully, the tool was just not able to turn off migration mode after doing the data sync and restore the original database and user prefix settings. You can turn off migration mode for the server manually in ISPConfig under system > Server config. Also, set the correct prefixes for users adn databases again under System > Interface config. The problem was that the ISPConfig API was not reachable anymore. If the ISPConfig GUI is down, then check the error.log of the web server to find out why. Might be a invalid SSL cert or something similar.
     
  12. craig baker

    craig baker Member HowtoForge Supporter

    I just cant catch a break! after the ./migration to ns11 I see ns11 httpd is not running on 8080 (or anything else).
    I ssh in and see errors that 100-ns11.cdbsystem.com.vhost has missing certificates.
    I deleted the 100-n11 file (and the corresponding ns11) but apache2 still wont run.
    I ispconfig_update.sh --force reconfigure the 8080 ssl and it still wont load with new errors.
    when I ssh in and look at /var/log/apache2 I see:
    [Sun Nov 09 12:29:52.979858 2025] [ssl:emerg] [pid 2164367:tid 2164367] AH02565: Certificate and private key ns11.cdbsystems.com:8080:0 from /usr/local/ispconfig/interface/ssl/ispserver.crt and /usr/local/ispconfig/interface/ssl/ispserver.key do not match

    as I said, I just cant catch a break! and still cant migrate to 192.168.2.230 (which has NO public ip).
     
  13. till

    till Super Moderator Staff Member ISPConfig Developer

    This cert is not altered during migration, it is created when you install ISPConfig. But you can create a new one using ispconfig_update.sh--force
     
  14. craig baker

    craig baker Member HowtoForge Supporter

    as I said I ran ispconfig_update.sh --force and reconfigured the 8080 cert. it gives the error noted from /var/log/apache2/error.log
    [Sun Nov 09 12:29:52.979858 2025] [ssl:emerg] [pid 2164367:tid 2164367] AH02565: Certificate and private key ns11.cdbsystems.com:8080:0 from /usr/local/ispconfig/interface/ssl/ispserver.crt and /usr/local/ispconfig/interface/ssl/ispserver.key do not match

    since I recreated the 8080 cert why is it complaining?
    looking at the /usr/local/ispconfig/interfacessl we find:
    lrwxrwxrwx 1 root root 59 Nov 9 14:45 ispserver.crt -> /etc/letsencrypt/live/ns11.cdbsystems.com_ecc/fullchain.pem
    lrwxrwxrwx 1 root root 57 Nov 9 14:45 ispserver.key -> /etc/letsencrypt/live/ns11.cdbsystems.com_ecc/privkey.pem

    looking further:
    Code:
    root@ns11:/etc/letsencrypt/live# cd ns11.cdbsystems.com_ecc
    root@ns11:/etc/letsencrypt/live/ns11.cdbsystems.com_ecc# ls -al
    total 12
    drwxr-xr-x  2 root root 4096 Oct 12 11:12 .
    drwx------ 70 root root 4096 Oct 25 16:20 ..
    lrwxrwxrwx  1 root root   47 Oct 12 11:12 cert.pem -> ../../archive/ns11.cdbsystems.com_ecc/cert1.pem
    lrwxrwxrwx  1 root root   48 Oct 12 11:12 chain.pem -> ../../archive/ns11.cdbsystems.com_ecc/chain1.pem
    lrwxrwxrwx  1 root root   52 Oct 12 11:12 fullchain.pem -> ../../archive/ns11.cdbsystems.com_ecc/fullchain1.pem
    lrwxrwxrwx  1 root root   50 Oct 12 11:12 privkey.pem -> ../../archive/ns11.cdbsystems.com_ecc/privkey1.pem
    -rw-r--r--  1 root root  692 Oct 12 11:12 README
    root@ns11:/etc/letsencrypt/live/ns11.cdbsystems.com_ecc# ls ../../archive/ns11.cdbsystems.com_ecc -al
    total 24
    drwxr-xr-x  2 root root 4096 Oct 12 11:12 .
    drwx------ 70 root root 4096 Oct 25 16:20 ..
    -rw-r--r--  1 root root 1302 Oct 12 11:12 cert1.pem
    -rw-r--r--  1 root root 1566 Oct 12 11:12 chain1.pem
    -rw-r--r--  1 root root 2868 Oct 12 11:12 fullchain1.pem
    -rw-------  1 root root 3272 Nov  9 12:12 privkey1.pem
    root@ns11:/etc/letsencrypt/live/ns11.cdbsystems.com_ecc#
    
    these keys all have a creation date of october 12. but I just ran
    ispconfig_update.sh --force
    should they not be brand new?
     
  15. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    They are still valid, and ISPConfig force update only forcing ISPConfig update, not forcing LE certs update / renewal.
     
  16. craig baker

    craig baker Member HowtoForge Supporter

    but apache2 cannot start as the ispconfig key/cert do not match. and I did a ispconfig_update.sh --force and told it to redo the LE certs for ispconfig?
     
  17. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    You know you can simply remove your ispconfig.vhost and let force update recreate it correctly right? While doing that, make sure you do not have conf-custom copy that use older conf, and let it uses the default.
     
  18. craig baker

    craig baker Member HowtoForge Supporter

    alas removing my ispconfig.vhost, ispconfig_update.sh --force did NOT recreate it. nothing improved
    so now I have no ispconfig.vhost :)

    from /var/log/apache2/error.log:

    [Wed Nov 12 11:39:54.381364 2025] [ssl:emerg] [pid 2895189:tid 2895189] AH02565: Certificate and private key ns11.cdbsystems.com:8081:0 from /usr/local/ispconfig/interface/ssl/ispserver.crt and /usr/local/ispconfig/interface/ssl/ispserver.key do not match
     
    Last edited: Nov 12, 2025 at 5:52 PM
  19. remkoh

    remkoh Active Member HowtoForge Supporter

    Those can be sent to the trashbin too then. Mismatches files are useless and need to be recreated (by the update script).
    And if those files are mismatched you probably have problems with other services too besides your webserver.
    Postfix, dovecot, pure-ftpd, apache2/nginx all use those same files. And maybe more services.
     

Share This Page