Hi, does anyone have experience of running the migration tool on large servers? I've got one with 10k clients, and its taking a long time. I mean its been going for 72 hours and its done 3k so far. So this is looking at like a 10 day process, is this broken!? The servers are not slow or anything, no hardware is being taxed. The weird thing to me is each client is taking longer, at the beginning it took about 20 seconds each, now its taking 120 seconds to create each client!? Any ideas appreciated thanks
Hi, the migtool uses the ISPConfig API to create clients, websites etc. on the target. If the process is getting slower than this must be something on the target, I think. Could you please look into the target's access log of the ISPConfig instance and also look at the top processes and database usage, just to find a clue what's going on.
And also check the apache or nginx error.log if you find any errors about vlogger. If vlogger is missing or any dependencies of vlogger, then apache can become slower and slower with each request due to failed log writing. I second Croydons thoughts that this is probably an issue of the target server configuration and not the migtool.
ok thanks. the target is 8 core, 16gb ram, not doing anything else apart from this. I've been trying to get the target to dump the POST data to see whats going on but havent managed to do it. on the target access logs look like this, about 20 posts per second: 81.201.139.249 - - [02/Jun/2017:10:53:46 +0100] "POST /remote/index.php HTTP/1.1" 200 15128 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15203 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15171 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15125 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15174 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15198 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15154 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15163 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15135 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15095 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15173 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15187 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15190 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15147 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15189 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15139 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15169 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15173 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15121 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15159 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15151 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15149 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15191 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15210 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:47 +0100] "POST /remote/index.php HTTP/1.1" 200 15137 "-" "PHP-SOAP/5.3.26" 81.201.139.249 - - [02/Jun/2017:10:53:48 +0100] "POST /remote/index.php HTTP/1.1" 200 15186 "-" "PHP-SOAP/5.3.26" load: load average: 1.35, 1.24, 1.18 iotop shows basically nothing writing at like 10k/s apache error log shows this every 5 mins: [Fri Jun 02 10:55:02.609340 2017] [autoindex:error] [pid 2763] [client ::1:55530] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html,index.php) found, and server-generated directory index forbidden by Options directive the servers are with different hosting so potentially bandwidth/latency issues but i wouldnt have thought so client creation is now taking 130 seconds compared to 120 seconds earlier
The error.log entry is ok and can be ignored, that's the ispconfig web server uptime check. According to the access.log, the migration tool creates about 20 items per second on the server. I guess a typical client on your server does not has hundreds of items (like websites, mailboxes etc.)? May you please post the output of: which vlogger
hiya ok sure: [root@ispconfig ~]# which vlogger /usr/bin/which: no vlogger in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin) is this bad. I followed the usual centos perfect server guide. This is going from centos6 to centos 7 dont know if that makes a difference?
I can't say for sure if this is the issue and if vlogger is in the path on centos, just tested that it is in the path on Debian and Ubuntu. Please check if there is a vlogger package with yum and if yes, install it. ISPConfig ships with it's own vlogger version, but installing the system wide vlogger too is a way to ensure that no dependencies are missing.
hiya ok thanks for the ideas, I tried yum search for vlogger but nothing Warning: No matches found for: vlogger
If you followed the whole perfect server guide, then everything should be installed fo vlogger and we can search for other possible problems. Or did you left out any chapters? What you can try is that you look at the mysql log for errors and check the mysql status in e.g. phpmyadmin or even enable mysql query log to see if the sql queries are flowing fast. One other thing that you can try is to comment out the server.sh cronjob on the target server in the root crontab to see if this speeds up the import. When server.sh is commented out, then the new clients, websites etc. will get added to the ISPConfig database only but not written to disk once a minute. when the import is finished, then enable the server.sh again and it will write all changes to disk then. The benefit might be that you avoid apache restarts during import that way.
hiya yeh I followed this guide https://www.howtoforge.com/tutorial...l-php-pureftpd-postfix-dovecot-and-ispconfig/ it doesnt mention anything about installing vlogger, just shows in the ispconfig install it should do 'Configuring vlogger' Anyway my script has just crashed. The first time I ran it it kept timing out after 30 mins. so on target i edited /usr/local/ispconfig/interface/lib/classes/remoting.inc.php and changed 1800 to 18000 assuming this would be plenty. I now realise this is 50 hours and my script has been running for guess what, 50 hours will try again and see how it goes
Ok, so vlogger should not be an issue then. And about 20 api connects per second are fine as well. Never seen a migration taking so long. Please check the migrate.log file for errors as Croydon pointed out and you might want to check the mysql logs as well plus maybe disable the server.sh on the target temporarily.
ok cool I have got mysql logging turned on on the target. It seems to me like theres a very high amount of queries going on. Im not exactly sure but I have an idea why it is getting progressively slower. Each client add runs queries for every id below it. like this: 14805 Connect ispconfig@localhost as anonymous on 14805 Query USE `dbispconfig` 14805 Query SET NAMES utf8 14805 Query SET character_set_results = 'utf8', character_set_client = 'utf8', character_set_connection = 'utf8', character_set_database = 'utf8', character_set_server = 'utf8' 14805 Query SELECT * FROM remote_session WHERE remote_session = '75145032c97cf132ab063f3302caf18d' AND tstamp >= UNIX_TIMESTAMP() 14805 Query SELECT * FROM `client` WHERE `customer_no` = '_626' AND 1 14805 Query SELECT CONCAT(`assigned_template_id`, ':', `client_template_id`) as `item` FROM `client_template_assigned` WHERE `client_id` = '635' so when I was adding client id 10, it did this 10 times. Now I'm up to client id 900, so for each add it runs this query for 1,2,3 etc up to 900. and gets 1 higher each time. Is it supposed to be doing this? This means its doing literally thousands of SELECTS for each client add... any ideas appreciated, thanks for your help with this
Thank you for your findings, we'll check that in ISPConfig API code and the migration tool to find the source of this behavior.
Hiya, any news on this? My script has been running for 6 days now and up to 5k out of 10k clients, taking 4 minutes per client now. In good news I have run a seperate migration of a standalone server that was just 10 clients/10 sites and that went perfectly
Hello, we are still investigating this. It seems that it has to do with some plugin in ISPConfig that deals with client templates.
Oh! Could you please check if the customer_no is filled for your clients on the (ispc3) source server? Seems it is empty and causing that problem.
Hiya I saw the announcement about the new release thats really fantastic thank you! Havent had a chance to test it yet, still working on my last migration but will use this for the next one.
