ISPconfig migration tool?

Discussion in 'Plugins/Modules/Addons' started by jboud, Jan 17, 2017.

  1. jboud

    jboud Member

    Hey there,
    I saw this tool by pure accident. I think it suits my needs for this week as one of my servers is compromized and I wasnt to migrate around 50 websites to 2 other servers I own. I want to ask a few questions, I can understand the obvious backup it does, but I want to verify/clarify a few things.

    Backs up the whole account of a user, meaning: Control panel account, websites with their statistics, (DNSs?), Databases and dbusers, mail servers and email accounts (and their /var/vmail/maildir emails?).
    Can I select which control panel accounts it will backup and prime for migration? Maybe I want to select 2 now, 5 tomorrow, etc? Also the target ISPconfig3 can accept multiple occurences of restores from different panels, meaning it's not a one use thingy on the target? E.g. I restored 2 clients from server 1 today and 5 clients from server 2 tomorrow.

    Or if there is a detailed info on what it does or doesn't, please point me in the right direction.
    PS this will only be used for<=>to ISPconfig 3 panels.

    Hyped for it.
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    The tool is not backing up data (it is not a backup software), the tool migrates data in a way that it imports data of one ispconfig instance into another instance and the website and mail data gets copied from the old server to the new server, so the old and new server have to be online at the same time. The tool migrates all clients, websites, mail accounts etc. in one migration run, you can not exclude clients or domains from the migration. You can migrate several servers into one target server (join servers) with the tool and the sourca eas well as the target can be multiserver systems.
     
  3. jboud

    jboud Member

    Aaah, i get it now. It's a "server as a whole" migration tool. This still might be convinient in my case. I can exclude some clients by deleting them prior to the migration.
    A few more questions.
    1) Does the target server need to be blank?
    2) The target server is older or newer version of ispconfig 3 version than the source, does it matter?
    3) Does it migrate the /var/vmail/maildir contents of the email accounts too?
    4) How usable are the websites after migration? I mean most of the time the websites are some sort of dynamic php, joomla, wordpress etc. Their config file asks a minimum of the user/name/pass of the database. Is this taken under consideration so the backup/restore tries to keep the same user/name/pass in mysql or I'll have to dig in manually after the migration?
    5) Does it take care of DNS records also? If it does, does it do it blindly? like if the IP of the original server is included in some records does it take that into consideration and change each IP address that is from the old to the new server?
    I know I'm getting greedy, if 1,2,3 works the rest I can do manually but still valid questions. :D
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    First a short explanation on how the tools works, the tool is a two step approach:

    1) you run the migrate script on the old server. It will connect to the remote api of the ispconfig installation on the new server and imports the configuration (clients, resellers, websites, mail domains, mailboxes etc). During this first step, the tool writes a shell script that contains the commands to copy over the data.
    2) in the second step, you copy over the command script that was created in step one to the new server and run it there. It will copy the mail, website and database data to the new server. this command script can be run again to resync data.

    The benefit of this approach is that you can resync the actual website and mail data by running the command script multiple times without having to import the config again. So a typical migration is like this:

    1) import the configuration
    2) copy over the data
    3) test the websites, mail accounts etc. on the new server.
    4) when you are satisfied with the result, run the data copy script again to get the latest data before you switch over the live traffoc t the new server.


    The target server must have ispconfig 3 installed. This ispconfig installation can be empty or it can contain data, this does not matter as the tool supports both.

    Same or newer is ok, older might cause problems.

    yes

    The tool migrates the website databases incl. the mysql users and passwords and it also changes paths in config files when a change is required, so websites should be working fine after migration. Most cases of failing websites happen due to newer php versions, e.g. when the old server runs php 5.4 and the new one runs php 7, then sites that dont support php 7 must fail. in such a case, just install an older php version as additional php on the new server and select that older version for the site after you migrated it.

    The tool migrates the dns records too, but it will not change IP addresses automatically as many admins run the new server on a temporary IP and then switch over the original IP after migration. But you can do a mass change of IP addresses directly in the dns_soa and dns_rr tables of the ispconfig database with phpmyadmin and then use Tools > Resync in ISPConfig to force ispconfig to write your manual dns changes to the bind config files.
     
    jboud likes this.
  5. jboud

    jboud Member

    So satisflying answers. I'll buy this today, work until the end of the week and avoid myself a stroke or heart attack by the client pressure :D Thanks.
     
    till likes this.
  6. jboud

    jboud Member

    End of the week... nope, I'm done.

    Ok, all worked almost perfect. I did encounter 3 errors in total, during the creation of the clients. It was too late so I didn't waste time on checking why, just moved those 3 clients manually, then deleted them from the source server. I didn't trust my phpmyadmin skills late at night, so i went and changed manually the DNS settings. Migration took around 1/2 hour where the servers where like my slaves, so while I was watching the progress on one screen I relaxed by playing Overwatch on another! I need some sleep but I dont need to waste any more labor hours on it anymore :D
    The whole interraction took me around 1 hour, coz I had to move those 3 clients manually and change DNS manually, but that's on me. The automation took less, like 30mins tops, where you can see the data flying from one server to the other. I also had to go on my domains provider to change name servers, took around 45mins.

    I'm writting this as a feedback to all potential buyers of this addon, it works, it's easy and saves you loads of time, also if it works it works, no typos, no human error. Worth every penny.
    Thank you.
     
    Taleman likes this.
  7. till

    till Super Moderator Staff Member ISPConfig Developer

    Thank you very much for your feedback!

    I'll add a short explanation on the client import errors with some hints how to debug and solve them.

    You can find the reason for the errors in the migrate.log file, the file is in the migration folder or legacy subfolder (depending on the php version), the migrate.log file shows all migration steps that the tool is doing in detail and also failures incl. the original error message that was thrown by the target ISPConfig system and a data dump of the failed record . ISPConfig 3.1 has some additional and stricter settings when clients get added, e.g. the VAT-ID is validated by ISPConfig 3.1 against the European VAT database and it throws an error when the validation fails, so an invalid VAT ID can cause the import of a client to fail. This can be corrected then on the old system by changing or removing the VAT record (or other data that caused the failure) and then running migrate.php again.
     
  8. jboud

    jboud Member

    VAT ID might be the issue, coz all I do I usually do the same way for all my clients, no templates, no funny things. But yea I fill in the details depending on what client information I have at the moment I create the account.
    Thanks for the reply.
     
  9. webguyz

    webguyz Active Member HowtoForge Supporter

    Till,
    Speaking of migration scripts, is there any progress on a single system migration script that will allow moving users,mail, and websites around on a single instance of ISPConfig? Would be really handy for a large multi-server system.

    Thanks!
     
    chaosad likes this.
  10. craigfanman

    craigfanman Member

    Hi. I am also desperate for a version of this script that can do it site by site. Is there any plans for this to be added or shall I just write it myself? Don't want to waste time if its going to come out soon.

    Or if I buy the addon will I be able to modify the code to do this? Or is it all locked behind ioncube loader? cheers
     
    Last edited: Feb 2, 2017
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    @webguyz We are working on a tool already to move web sites and mail accounts within an ispconfig multiserver setup.

    @craigfanman Do you mean multiservers as well?
     
  12. craigfanman

    craigfanman Member

    Hiya, yeah I have a multiserver setup, 1 master and about 10 slaves. So I want to be able to click a button and move a site from Server A to Server B. Currently we just do this manually which takes a lot of time, creating accounts and moving data.
     
  13. till

    till Super Moderator Staff Member ISPConfig Developer

    Yes, that's the tool we are already working on.
     
    craigfanman and ispcomm like this.
  14. jboud

    jboud Member

    Oh cool, hyped about that tool!
     
  15. ispcomm

    ispcomm Member

    This is a long needed addition.
    I'm happy I found this comment before staring the same work. I'm wondering if you have an ETA and if it would be possible to participate in the development (either coding or testing).
    Thank you.
    ispcomm
     
  16. till

    till Super Moderator Staff Member ISPConfig Developer

    Currently, I'm working on the technical basics which require some additions in ispconfig itself like extending the action framework to allow chained actions (action B on server 2 requires that Action A on server 1 succeeded) and a function that allows the creation of temporary system ftp and ssh users, these additions will be part of ISPConfig 3.2 in April and will build the foundation to implement the ispconfig addon to move data around and I'm sure that they will be useful for other features in the next ispconfig releases as well. The next step then is the addon itself, I will make some test versions available, hopefully in June.
     
  17. ispcomm

    ispcomm Member

    I understand what you mean. Implementing a generic framework for every object will take time. You'll probably need a "rollback" mechanism to undo partially executed "batches" after they fail for whatever reason and this may not be simple.

    My idea was much simpler (although limited to sites and mail domains). What I had in mind was to make an "archive" of a site, including accounts, databases, passwords and files and "zip" everything in a single file that could be copied to another server and "imported" there, either via gui, or more feasibly via shell like this: export sitexyz <outfile> -> rsync to the other server -> import <infile>. This could be easily scripted from the master server with the only requirement of a password-less ssh login from the master to all slaves.

    I started a proof of concept php script a few years back but then never finished it. In my prototype, a single tar file contained a one text (json) file describing all accounts (ftp, database, passwords etc) and one file per "object" where "object" : <database>|<site archive>|<logs>|...

    Sorry for steering the thread.

    I'll be looking forward for your announcements and will try to help as much as i can.
     
  18. till

    till Super Moderator Staff Member ISPConfig Developer

    The functionality that you describe is on my todo list for the migration tool as well but I would see that more as a single site migration tool or website backup tool that includes meta data that I planned for nonconnected servers (multiple standalone systems). I'll keep you updated.
     
    Jesse Norell and chaosad like this.
  19. ispcomm

    ispcomm Member

    Till, I'd like to have an update on the status of this "migration" project. Is there anything I can do to make it happen ? Perhaps coding too. You know I've done some things in ispconfig in the past. Perhaps we can join forces.
     
  20. till

    till Super Moderator Staff Member ISPConfig Developer

    I'm working on the tool. Thank you for offering your help but I expect that I will be able to code it completely.
     

Share This Page