PureFTPD slow when uploading multiple files

Hi All !

Thanks for helping us everyday !

I've just install an Ubuntu 18.04 server following the Perfect Server instructions.

We're using PureFTPd Server for FTP connections.

I'm hosting the server on DigitalOcean Droplet, and when I run a speedtest I get:

root@cloud01:~# ./speedtest-cli
Retrieving speedtest.net configuration...
Testing from Digital Ocean (111.11.11.11)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Ridge Wireless (Cupertino, CA) [5.08 km]: 2.717 ms
Testing download speed............................................. ...................................
Download: 853.46 Mbit/s
Testing upload speed............................................. .................................................. .
Upload: 683.06 Mbit/s

The problem is:

OLD Scenario - Shared Server in a Brazilian ISP
When I send a 112MB file to the server (PureFTP too) it takes about 00:00:31 (average: 3.6MB/s)

NEW Scenario - VPS Droplet in DigitalOcean SFO2
When I send a 112MB file to the server (PureFTP) it takes about 00:00:34 (average 3.2MB/s)

Its not a big difference and it is ok for me.

BUT, When I send 100 files of 12KB:

OLD Scenario takes 00:00:06 (about 360KB/s)
NEW Scenario takes 00:01:45 (about 11KB/s)

The same happens when I'm deleting files, on the new server it takes too longer than in the old server.

========================

LOGs

New_Scenario: attached
Old_Scenario: attached

CONF:

NewServer Conf: attached

========================

So we know that the problem remains in working with a lots of file, and not with the size of the file. So, we don't think it's a bandwidht problem but in the communication between client and server before each file transfer.

Any clue will be appreciate !

Thanks in advance !

 

FTP is slow when sending lots of small files. Rsync The same files would be faster.
My guess is settings are different between old and new FTP.

 

Hi Taleman, thanks for your answer. The main point is exactly that, I'm hoping someone could tell me what kind of configuration could be changed to fix this problem.

I understand FTP with multiple files is slowest when comparing with one file, but in this case it's working fine on the old scenario (I don't have access to the setup of Old Scenario) and it's having problem on the new scenario.

I can't use Rsync or other solution in this case.

Thanks, hope someone can point me the way to troubleshooting and fix it !

 

I'm searching the web for three days and can't find anything to fix it. I don't think the problem is in DigitalOcean because speedtest shows a good performance:

Download: 853.46 Mbit/s
Upload: 683.06 Mbit/s

Also, when transfering one single file of 112MB the difference is 3 seconds between the old scenario and the new one and the connection speed reach 3.5Mbps instead of 100Kbps when uploading multiple files.

So I really think the problem is with some configuration when uploading multiple files. I'll keep trying to find a solution. tks.

 

What is ping in old and new setup?
What is default for bandwith settings? Or does some verbose log show used value?
Restore from backup old config file and compare to new.

 

Hi Taleman,
I think my english isn't good enough, sorry for that.
I don't have access to the ISP Server Setup, I'm moving from ISP (old scenario) to a VPS that I can manage (New Scenario). The logs were attached to the main post message.
The ping (latency) is higher to VPS (USA) than to the Brazilian ISP, but as I tried do show in the replies above is that if the problem was with bandwidth or communication, the time to send 112MB will take too long, but it not happens.
What file or information do you need ?

SPEEDTEST from VPS:
Download: 853.46 Mbit/s
Upload: 683.06 Mbit/s

UPLOAD 112 MB (one file):
ISP: 31 seconds
VPS: 34 seconds
It's ok, the difference could be because of the latency, it's ok 3 seconds more. The throughput is almost de same, and about 3.5MB/s

UPLOAD 100 Files 12KBytes (total of 10MB of data)
ISP: 6 to 10 seconds (150 to 360KB/s) it varies depends of the hour of the day because it's a shared server
VPS: 01:45 = 105 seconds (about 11KB/s)

So when sending 1 file the throughput is almost 3.5MB/s and with a lot of file it's about 11KB/s.

 

The reason is already understood (single large file vs hundred of small files, not settings) and the solution is rather to compress, upload and extract instead. I think rsync like ftp (though faster then ftp) also transfer slower in similar case (bulk vs single file).

 

Thanks Till, I'll test if the latency is the problem, I'll create a VM in another ISP but from USA and test the upload with low latency.
Thanks Ahrasis, the problem, if confirmed after the test, is not with multiple files, but related with the latency in multiple files scenario, because multiple files, in a server without the high latency, files are uploading very fast.

Latency from BR to ISP (New Scenario) - 300ms
Latency from BR to Brazilian ISP (Old Scenario) - 40ms.

Let's see what the tests show us.

Thanks in advance.

 

While I read "latency will accumulate" in @till reply above means more files higher latency thus slower, you understanding may be true also. Do post your finding later @Wellington Watanabe Filho.

 

Thanks Jesse, I saw those instructions on some foruns and I've checked up and it's ok. In the new versions of PureFTP this feature is enabled as default

Thanks Ahrasis, maybe I did not understand the context because my english.

I'll do the tests and come back with the results.

Thanks in advance