How to migrate a website area from a server to another

Hi!

I'm basically reviving this thread here: https://www.howtoforge.com/community/threads/single-web-site-migration-tool.77022/

As I seem unable to find more information regarding operations such a moving a website from one host to another with ispconfig, possibly carrying over the data together with the configuration.

From what I could see the only possibilities that I could come up (that I haven't yet actually tried) involve at least:

1. creating a new db on the new host, same user/db name
2. manually creating user/group like it had on the previous host
3. copying over the files with the same structure it had on the previous host, preserving user/group
4. messing with the ispconfig master db to move the website to the new host
5. do something to trigger ispconfig to rebuild its things on the new host (?)
6. change the FTP settings to switch to the new host

Am I looking at the problem from the right angle, or am I missing something huge somewhere?

I saw @ispcomm mention on that thread he wrote their own tool to accomplish this job, so I suspect that indeed ispconfig doesn't have such tool built-in and I'd need to create something myself (if I can't find anything already available around, I tried PMing @ispcomm a couple of weeks ago but I haven't got an answer yet) to do the above steps?

Thank you for any insight anybody could give me!

 

within the same multi-server installation.

And, amongst the thing that I'd like to see preserved is the full path to the website, since some CMSs embed the path in their configuration and I really don't want to mess with customers' websites for this. (sorry, forgot to mention in the OP)

 

create the website on the new server
create a new database on the new server with the same db-user
ex- and import the database
use rsync to transfer the site (use a shell-user)
adjust your website-config

 

are you sure that this will work? the restore does not create a database and the database-id is "linked" to the backup.

 

i've done it before, moving a number of sites from one member server of a multiserver setup to another member server.
@Jesse Norell has got it right.
in mysql, using cli or phpmyadmin, change the server_id value for the site/user/db in (depending on what ftp/db/ssh accounts exist) ftp_user, shell_user, web_backup, web_database, web_domain, web_folder, web_folder_user, and webdav_user (assuming webdav users, or protected folders/users are configured for those last 3 columns, and assuming the db is on the same server as the website.)
if mail is on the same server as well, then the same change is needed for the various mail and spamfilter tables if your moving that service as well.

resync the relevant services for the server the site is being moved to.

everything required should be created on the target server, then it's just a case of copying over the db and website files from the source server using whatever method you find easiest.
at this point, i would suggest creating an entry for the domain in your local hosts file, pointing to the new server, and test everythings ok.
once happy, update the domains dns to point the necessary records to the new hosting servers ip.

i would advise not making any changes to the website itself (or inform the site owner not to), for at least 72 hours, leaving the site still running on the old server as well, in case visitors are still getting the old ip's, after this time, you can start removing the site/database from the old server.

then it's just a case of cleaning up those records in the dbispconfig database on the old hosting server (assuming it's not also the master)
can be done by manually removing them, or as @Jesse Norell suggests, using the sys_datalog table (never done it using this method, so suggest being very careful what you put in this table). i do know that just running a resync of those services on the old hosting server does not remove them from the database, or at least it didn't for me, but that was back around ispconfig 3.0.4....


*you may also have to play around a bit with copying over letsencrypt certs / config or briefly delete letsencrypt and then re-apply it after changing the dns records. letsencrypt wasn't an issue/option when i last moved sites this way.

 

I think, this depends on the amount of sites running on the destination-server. It's not very funny, when you resync 100+ websites.

Using your way (changing the server_id) you have to remove everything on the old server by hand.

I'm pretty sure, that you can not sys_datalog when there are no records (anymore) in the database. A resync will just resync the configs and will never deleted any records.

Personally, i see no problem with creating the website on the new server (you can use the same name) and a new database (you need a different name but you can use the same db-user.
Most CMS do not care about the full-path (iirc, joomla does) so can use sed to replace the db-user in the cms-config (and maybe the directory). I use this to automaticly import web-sites from a remote server. You must the directory with joomla and nextcloud. For magento, shopware5, shopware6 (maybe adjust COMPOSER_HOME) and wordpress it's enough to update the db-connection.

This way allows you, to keep both sites for a longer time. I you are find with new server, just change the dns. You can revert back at time and this leads to a much cleaner setup when you delete the site and database from the old server.[/QUOTE][/QUOTE]

 

you're right, it's not fun when you have to resync a lot of websites. if doing that though, i would also suggest turning off the config checks and restarts first. testing every vhost config, and then restarting apache after every vhost change *really* slows down the resync, and of course, a hundred plus apache restarts will inevitably lead to calls from site owners saying their site is down.


joomla et al might not care about the filepath, and if you're lucky wordpress won't be either, but a lot of times wordpress is a problem. wordfence for one, puts the filepath into several files, loads of plugins put the full filepath in the database. i've moved sites where there's literally 100000+ full filepaths in the wordpress database that require changing. if moving the wordpress site from one ispconfig member server to another, at least with the changing server_id method, the full filepath never changes. the whole site can be moved without worrying about having to make any changes at all.
if moving a wordpress site from some other provider / control panel, i would use the duplicator wordpress plugin by snap-creek. makes it very easy, and handles any filepath / domain-name/url changes automatically.

 

nope. see i.e. $log_path and $tmp_path

from wordfence: "If you are moving your site to a new server we recommend that you first remove extended protection. When your site has been migrated, you will then optimize the Firewall again."

 

So, I gave it a shot with a test website. I didn't try yet to move the db, but that looks trivial in comparison.

We use php-fastcgi, so that sadly is not our case

So, in our setup we only have a couple of shell_user entries, and they are not affected by this move at the moment, one less thing to care about.
I'm also postponing trying to move the db, which looks trivial so I'll try that later.

Regarding the actual website: I tried a single `update web_domain set server_id = 16 where domain_id=2544;` and then made a silly change on the ispconfig web interface to trigger it (in this case, changed the PHP version; after all that was also invalidated by the change above).

This did create the users using the same username and group, different UID/GID as you figured, but indeed I don't think (hope!) that's a problem. it also created the basic directory structure, however:

• it didn't create the "client0" group and the `/var/www/clients/client0` directory (so there were errors such as `ln: failed to create symbolic link '/var/www/clients/client0/XXXXX': No such file or directory` in the ispconfig log; I did that manually
• this being a branch new host, apache wasn't really fully configured (missed a couple of a2enmod I had to take care of)
• more importantly: the permissions weren't fixed, despite the user/groups having been created correctly:

Code:

root@ssweb2 ~www/clients/client1320/web2544 # la
total 40
drwxr-xr-x 10 root    root       4096 Oct 28 13:30 .
drwxr-xr-x  3 root    root       4096 Oct 28 13:30 ..
drwxr-xr-x  2 root    root       4096 Oct 28 13:30 cgi-bin
drwxr-xr-x  2 root    root       4096 Oct 28 13:30 log
drwx--x---  2 web2544 client1320 4096 Oct 28 13:30 private
drwx------  2 web2544 client1320 4096 Oct 28 13:30 .ssh
drwxr-xr-x  2 root    root       4096 Oct 28 13:30 ssl
drwxr-xr-x  2 root    root       4096 Oct 28 13:30 tmp
drwxr-xr-x  4 root    root       4096 Oct 28 13:30 web
drwxr-xr-x  2 root    root       4096 Oct 28 13:30 webdav
root@ssweb2 ~www/clients/client1320/web2544 #

Now, I suppose I can go ahead and chown it myself, however it leaves me a feeling any script I'll write to automate the change will be troublesome and full of double checks.

Is there somewhere some documentation about how sys_datalog works (that doesn't involve reading through ispconfig's code )?
I also see that there is no direct reference/access to sys_datalog in the documented APIs.

probably
though replicating this whole setup would be a huge pain, so actually right now I'm daring to use a single testing web_domain for this, mostly hoping that ispconfig won't go and mess with the other unrelated things while I play with this website... /o\

This setup is technically a bunch of VMs all hosted in 3 physical servers in the same rack, so moving files around is trivial. TBH I was hoping to leverage rsync so that it would retain permissions/mtimes/etc, which is of course also possible by tarballing with the proper options but well.

I also just realize that I need to verify how these websites are connecting to their databases... I hope nobody thought it was a good idea to use anything that is not "localhost" :\

Since I'm doing this as a paid job I'll have to verify with the business if/how they'd be open to sharing whatever I create (which is likely to be a bunch of scripts to be run across origin server, target server, ispconfig master).
But otherwise, I'd happy to dump such things in the open myself

 

i don't think just making a change like the php version triggers the creation of everything that's needed.
bits like the /var/www/clients/client0 folder, or the symlink creation may only be done by a new site creation, or a resync. just a rebuild of the vhost conf file or the php conf may not trigger the folder/symlink creation. similarly, some things like the file/folder permissions, may only be on new site creation / resync, or if 'set folder permission on update' is set in the server config.

i think running an actual website resync on that server will successfully create everything whilst avoiding all the issues you've encountered by just making a config change to the website.
once this parts done, you should be able to go ahead and rsync everything from the old server to the new server for that domain.

 

That would make sense indeed.

I actually forgot about the existence of that "resync" page, thank you for reminding me of it. I just tried, but even that didn't create the client0 folder, nor fixed the directories owner/group of /var/www/clientXXX/siteYYY/* (I tried even after removing the /var/www/clientXXX directory and let the resync recreate it, which it did, everything except private/ and .ssh/ fully owned by root).

 

yep. very nice tool.. used it for that many times. good for scripting wordpress installs from scratch as well. and for automating scheduled wordpress updates. (hate that wordpress are trying to have everything auto-update at random times. i don't want sites randomly breaking in the middle of the night because some silly little plugin decided to auto-update)

 

Right, but there are two problems with doing that:

• touching customer's data more than absolutely necessary is *bad*
• even if we knew that that particular CMS stores the path in that specific way, it would likely not cover other CMS, or customer-developed sites that do whatever they want

So we can't just go and change things like that willy-nilly. However, I do plan on perhaps moving things to fpm or such in a future stage (or at least do some research and considerations on how that affects our setups) when this particular move is not urgent anymore.

 

i would start thinking that maybe something isn't fully configured, or some parts misconfigured. if a file/folder isn't there, a resync should recreate it as if creating the site from new. might be worth turning up the debugging level on that server and running another resync, see if any problems are getting logged.

 

it would only be done like this, if it's known that such changes are required, eg moving a site around your own servers so it has different filepaths, or a site is migrated from hosting elsewhere, or changing domain names, or changing from http to https, everything except for the first issue would be known to, and should have been requested by the customer. so it wouldn't be willy-nilly, or more than absolutely necessary. and the issues described, and the use of wp-cli would be specific to wordpress sites anyway.

also if you're planning on switching sites from fastcgi to fpm, you can just do that in the ispconfig interface. nothing needs to be done on wordpress sites for such a change.