Testing migration / rehost to new servers

I absolutely must move all ISPConfig resources to new servers. That means new hardware, a current OS, the latest ISPConfig, and migrating existing configs and data. With the Migration Toolkit I'm sure I can manage all of this. The detail that always confounds me is how to test the new environment while the existing environment is still running.

The plan so far is to migrate MX, DBs, Sites, etc., and migrate DNS, but not to activate DNS until everything else is tested. By changing individual DNS zone records in the existing installation, to shift from production to the new servers, I can test individual features for individual sites without doing everything all at once. To test DNS, I'll temporarily change a few sites at the registrar. Then after a final data migration, I'll re-assign the public IP addresses to the new servers and "Bob's yer uncle!" (it all should work) .

I just don't want to do a migration, have a failure, and have everything down while we work through details. That's irresponsible - though sometimes we get caught and that's what happens. But to minimize the chaos, we need a checklist for migrating features, testing, and then going live.

Ideally I'd like to establish a checklist that anyone here can follow, and enhance it over time as-required. But I gotta believe that's already been done.

So to summarize: How do YOU test a new system before moving it to production? Do you have scripts that toggle configs? Do you have checklists? Are these details already published somewhere?

Thanks as always.

 

just google:
"etc"+"hosts"+"your operating system on your desktop"
maybe add "howto"

Also depending on the importance:
mysql / mariadb replication
and
php session redis

 

Not friggin cool at all @ztk-me . Worst response here in the 4 years that I've been here. WTF dude?
@ahrasis I'm surprised at you too.
If it's not a legitimate question, tell me how I'm reading too much into the plan.
Otherwise, if you don't have anything constructive to contribute, just STFU.
Can you tell I'm pissed?

 

Thanks for the suggestion @till - the issue is not simply accessing the site.

Changing my local hosts reference to the site doesn't help with testing the new system's mail server, inbound or outbound. The SPF TXT record in the live system either needs to include the IP for the new server. The MX priority needs to be be set in the live server to route to the new server and fall back to the existing server on failure. The secondary MX needs to be tested here as well.

Primary DNS and secondary need to be tested. That doesn't happen with a client-side change in /etc/hosts.

I will be getting a license for the Migration Tools to move from 3.2.2 to 3.2.11. Significant changes were made to configuration handling for Dovecot. I need to "sew together" custom Dovecot+Postfix config file changes with the new config model. That all needs to be tested thoroughly. After the initial migration I also need to change our file naming to ISPConfig defaults. For example, rather than /clientZ folder names I initially configured to use /cloudZ names, and there were a couple other minor changes that I need to reverse to make this a more normal ISPConfig site.

The secondary DNS and MX also need to be completely reconfigured because the Rspamd implementation in the current MX2 isn't working well with the MX1.

This is going to take a lot of time. I need to write scripts to automate manual changes. There will be several re-installations to ensure everything goes smoothly. I probably need to do this in phases : 1-to-1 migration first, reconfigurations later. I need to know what others here do during a full ISPC re-hosting.

Telling me to google for BS isn't helpful.
I hope we can restart this more productively.

 

Is this a hatred post? What did I do?

 

I don't have an ounce of hatred for anyone in my body. ZTK posted a highly insulting and useless response, and you liked it. If I'm taking that wrong, I apologize to you. As I said that situation would surprise me.

Oh yeah, since your name is above my strong reaction it looks like that was directed at you. It was directed at ZTK.

 

We can, but that's no way to deal with people who are trying to help. It's disrespectful and inappropriate.

Back to the topic. You expect someone here in the forum to do the work for you and being respectful if people try to point you into the right direction. There is no tutorial for doing big migrations of highly custom hosting solutions with production data. You have to test, check and plan this yourself as no one here knows your system and setup as much as you. Sure you can pay someone to do it for you but thats beside the point.

As i did some huge service migration from old systems to new ones recently i can asure you there is no straight forward way and you cannot asure that everything will be working 100% and no issues come up. That's just not possible.
And so due to the fact that such migrations highly depend on the specific setups and requirements there is no "Checkllist" that anyone but yourself can create for your process.

 

It is not an acceptable behaviour at all.

Indeed.

 

I am really sorry you got this impression.
By no means I did not want to be rude.
Look, I could have asked you what operating system you are using or post all sort of possible answers.
But I know for fact, that "etc" "hosts" "howto" "youros" does spit out good video and text howtos better and more complete I'd be abloe to put in one post.
That was the only reason. I'm happy to help you with any further questions. Really, I love to asist and share knowledge.
Again, I did not mean it the way you understood this posting.
This is entirely my fault, I could have said that in the first place.
I thought you needed the hosts change first, and we could have gone from there.
Still we can, if you accept my apology.

 

OK, I took your "google for" response as a snide RTFM. You can understand my response to that. But that's not what you intended, so that's the end of that. Thank you very much.

Let's bring this back to basics. In a typical ISPConfig environment we need to test a migration to new servers before we go production. We all agree on that. Every topology is different. Of course. Every OS has nuances. Certainly.

This is an inquiry in an ISPConfig v3 forum. I'm talking about ISPConfig. I'm not asking about /etc/hosts. My OS isn't relevant. I'm talking about modules in this platform. There are common things that everyone has to do here for DNS, mail, sites, databases, users, etc.. Read my text, it's obvious what I'm talking about and there was no need for anyone to comment on things I didn't say.

My questions in the OP were clear: "How do YOU test a new system before moving it to production? Do you have scripts that toggle configs? Do you have checklists?"

I didn't ask for a step by step "Perfect" guide, or a tutorial for someone who someone who doesn't know where the power switch is. Check my sig - I've been doing I.T. for longer than many of you have been alive. I read the code. I do have processes for system migrations. I'm asking colleagues here what they do in this complex environment.

Thank you for any constructive, collaborative responses - again, even ... "it aint that complicated." But I've cited examples of areas where I expect complications. Am I wrong? Why? What do YOU do to test these features in a new system while you still have live systems running? I don' think this is an unreasonable inquiry.

Best to all.

 

for DNS it is fairly easy for me as my DNS servers are dedicated and replicated.
Likeweise I can test a new setup before switching easy. To verify my DNS-Server is giving me the expected answers, I can do the following to specify a DNS server to query: dig www.domainorg @myNewDnsIp
a good read would be https://www.cyberciti.biz/faq/unix-linux-test-and-validate-dnssec-using-dig-command-line/

MX can be switched either by using a temp backup MX or a shared Filesystem, both have cavieats to learn about, like file locking is different and some services needs some configuration for that. Or just move over, SMTP has a tempfail and retry mechanism, just ensure your old server does not fetch mails after you moved files. Sending and Receiving mails can be testet locally. If this works fine and the server sends signed mails and everything, remote usually works, too. Also one can use a temp Mailserver with ... well name resolution changed to the new DNS-Server. There is no short command for this unfortunally.

Websites. Also highly depends, do your customer / scripts write on the local filesystem? For example PHP uses session-files by default, this can be switched to something better suited, redis or mysql backend. In Regards of log files, I do not know what I'd nowdays as I do not write log files to webservers, they get sent to elasticsearch via udp.
But most certainly you want to compare the configuration you may have made or differs from the previous version/maintainers default.
Using phpinfo(); on a site would reveal the locations of the ini files for that websapce.

Take care for your database setup, some defaults changed, especially charsets can be missed easily or the default sql mode could start to complain about missing default values and so on as the setting may gotten more strict https://mariadb.com/kb/en/sql-mode/

In a scenario where I have two servers only, I'd do a replication, meaning I have the old server configured as primary and the new one as secondary, allowing me to have almost no missed traffic.

Having a Loadbalancer like HAProxy configured on a different host where you route your traffic first is also nice. Because you can define your server as backend, and if for any reason and testing something goes terribly wrong, you only have to switch the backend on the haproxy and do not need to wait for dns propagation.

If you need suggestions for more services or how to do things in details, happy to answer as best as I can.

However all of this is very non-ISPConfig way of doing things.

 

Thank you very much for your time on that. For this ISPConfig installation it looks like I need to proceed with the original plan, which so far looks like this:

1. This is just a normal (Perfect Server / ISPConfig) installation. Duplicate the topology of the existing ISPConfig servers.
2. Get this working like it's a new company.
3. Get server snapshots.
   
4. Use the Migration Tool to port configs and data from live to test. Migration must be to same topology.
   
5. Work out the differences, move old /etc/dovecot and other configs to new locations for functional equivalence.
   
6. Test all services with a single low-profile domain with NS set at the registrar to the new servers. Note all required changes/patches to configs, code, system packages, apps, databases.
   
7. Restore from snapshot. Rerun Migration Tool. Reapply changes with scripts. Re-test. Repeat as required.
8. Get server snapshots.
   
9. Switch a live site NS at registrar to new server and run with that for a while.
10. When satisfactory, copy backups from live site on new server.
11. Repeat step 7, migrating live data and applying all known patches.
    
12. Restore live site used for testing.
    
13. Change the public floating IPs to point to the new servers and reset test domains NS at registrar.

Plan for a periodic migration just to keep the environment clean and current, and to keep this plan fresh in mind. Focus on "cattle vs pets" for rapid replacements. That is, system configs shouldn't be so tough to migrate - a migration should be as simple as running (for example) scripts with OpenStack and Ansible. Updates to ISPConfig won't usually involve the significant Dovecot config changes that were required in 3.2. But changes are expected and details in step 5-6 will just change with each update.

Some of that is certainly common DevOps but the ISPConfig nuances require special consideration. That is my focus for this inquiry.

Suggestions? Thanks.

 

I see, in general it is looking fine, there is no such thing as too much testing.

You want to route an existing site and if it works you want to copy backups over? You mean the old backups of that site?
I assume yes, but it is a confusing point

 

The intent there is that a live site running on a new server will gather new data which needs to be preserved in a server refresh. That site's live data on the new server isn't on the production server that everyone else is running on. So when the production server is migrated, this test site's live data needs to be brought up to current.

Any better?

(We're back on the same page - thank you VERY much - I really don't like social issues in technical forums. (Um, or anywhere.) )

 

I see, so there won't be any issue with that live site. I feared you'd sacrifice some minutes/hours and overwrite the data with old backup.

Yes, I am aware I have a specific, direct way and sometimes fail to reflect how my postings could possibly be interpreted, bear with me on that, I tend to be a little impulsive too. Though I try to stay nice and actually usually am. And yes, I am a human too and may miss the point sometimes