PureFTPD slow when uploading multiple files

Discussion in 'Installation/Configuration' started by Wellington Watanabe Filho, Jan 17, 2019.

  1. Hi All !

    Thanks for helping us everyday !

    I've just install an Ubuntu 18.04 server following the Perfect Server instructions.

    We're using PureFTPd Server for FTP connections.

    I'm hosting the server on DigitalOcean Droplet, and when I run a speedtest I get:

    root@cloud01:~# ./speedtest-cli
    Retrieving speedtest.net configuration...
    Testing from Digital Ocean (111.11.11.11)...
    Retrieving speedtest.net server list...
    Selecting best server based on ping...
    Hosted by Ridge Wireless (Cupertino, CA) [5.08 km]: 2.717 ms
    Testing download speed............................................. ...................................
    Download: 853.46 Mbit/s
    Testing upload speed............................................. .................................................. .
    Upload: 683.06 Mbit/s

    The problem is:

    OLD Scenario - Shared Server in a Brazilian ISP
    When I send a 112MB file to the server (PureFTP too) it takes about 00:00:31 (average: 3.6MB/s)

    NEW Scenario - VPS Droplet in DigitalOcean SFO2
    When I send a 112MB file to the server (PureFTP) it takes about 00:00:34 (average 3.2MB/s)

    Its not a big difference and it is ok for me.

    BUT, When I send 100 files of 12KB:

    OLD Scenario takes 00:00:06 (about 360KB/s)
    NEW Scenario takes 00:01:45 (about 11KB/s)

    The same happens when I'm deleting files, on the new server it takes too longer than in the old server.

    ========================

    LOGs

    New_Scenario: attached
    Old_Scenario: attached

    CONF:

    NewServer Conf: attached

    ========================

    So we know that the problem remains in working with a lots of file, and not with the size of the file. So, we don't think it's a bandwidht problem but in the communication between client and server before each file transfer.

    Any clue will be appreciate !

    Thanks in advance !
     

    Attached Files:

  2. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    FTP is slow when sending lots of small files. Rsync The same files would be faster.
    My guess is settings are different between old and new FTP.
     
  3. Hi Taleman, thanks for your answer. The main point is exactly that, I'm hoping someone could tell me what kind of configuration could be changed to fix this problem.

    I understand FTP with multiple files is slowest when comparing with one file, but in this case it's working fine on the old scenario (I don't have access to the setup of Old Scenario) and it's having problem on the new scenario.

    I can't use Rsync or other solution in this case.

    Thanks, hope someone can point me the way to troubleshooting and fix it !
     
  4. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    Just a guess, maybe it's resource limits in your new VPS, or connection rate limiting at the network level? I don't use digitalocean, and maybe someone who does will post with some experience, but I'd try searching specifically for that environment. (you could also try setting up network connection tests, maybe strace your ftp server and/or run some utils like atsar/iostat/dstat/ifstat to try to find more info, but searching for others with that same results may end up with a quicker answer)

    https://www.google.com/search?q=digitalocean+vps+ftp+upload+slow
     
  5. I'm searching the web for three days and can't find anything to fix it. I don't think the problem is in DigitalOcean because speedtest shows a good performance:

    Download: 853.46 Mbit/s
    Upload: 683.06 Mbit/s

    Also, when transfering one single file of 112MB the difference is 3 seconds between the old scenario and the new one and the connection speed reach 3.5Mbps instead of 100Kbps when uploading multiple files.

    So I really think the problem is with some configuration when uploading multiple files. I'll keep trying to find a solution. tks.
     
  6. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    What is ping in old and new setup?
    What is default for bandwith settings? Or does some verbose log show used value?
    Restore from backup old config file and compare to new.
     
  7. Hi Taleman,
    I think my english isn't good enough, sorry for that.
    I don't have access to the ISP Server Setup, I'm moving from ISP (old scenario) to a VPS that I can manage (New Scenario). The logs were attached to the main post message.
    The ping (latency) is higher to VPS (USA) than to the Brazilian ISP, but as I tried do show in the replies above is that if the problem was with bandwidth or communication, the time to send 112MB will take too long, but it not happens.
    What file or information do you need ?

    SPEEDTEST from VPS:
    Download: 853.46 Mbit/s
    Upload: 683.06 Mbit/s

    UPLOAD 112 MB (one file):
    ISP: 31 seconds
    VPS: 34 seconds
    It's ok, the difference could be because of the latency, it's ok 3 seconds more. The throughput is almost de same, and about 3.5MB/s

    UPLOAD 100 Files 12KBytes (total of 10MB of data)
    ISP: 6 to 10 seconds (150 to 360KB/s) it varies depends of the hour of the day because it's a shared server
    VPS: 01:45 = 105 seconds (about 11KB/s)

    So when sending 1 file the throughput is almost 3.5MB/s and with a lot of file it's about 11KB/s.
     
    Last edited: Jan 17, 2019
  8. till

    till Super Moderator Staff Member ISPConfig Developer

    When you send a lot of small files then the longer latency might be the cause as commands will go force and back for each file so the latency will accumulate.
     
    ahrasis likes this.
  9. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    The reason is already understood (single large file vs hundred of small files, not settings) and the solution is rather to compress, upload and extract instead. I think rsync like ftp (though faster then ftp) also transfer slower in similar case (bulk vs single file).
     
  10. Thanks Till, I'll test if the latency is the problem, I'll create a VM in another ISP but from USA and test the upload with low latency.
    Thanks Ahrasis, the problem, if confirmed after the test, is not with multiple files, but related with the latency in multiple files scenario, because multiple files, in a server without the high latency, files are uploading very fast.

    Latency from BR to ISP (New Scenario) - 300ms
    Latency from BR to Brazilian ISP (Old Scenario) - 40ms.

    Let's see what the tests show us.

    Thanks in advance.
     
  11. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    This is just a guess, but maybe you have a name resolution problem for your client? Try:
    Code:
    echo 'yes' > /etc/pure-ftpd/conf/DontResolve
    and restart pure-ftpd, and test again.
     
  12. ahrasis

    ahrasis Well-Known Member HowtoForge Supporter

    While I read "latency will accumulate" in @till reply above means more files higher latency thus slower, you understanding may be true also. Do post your finding later @Wellington Watanabe Filho.
     
  13. Thanks Jesse, I saw those instructions on some foruns and I've checked up and it's ok. In the new versions of PureFTP this feature is enabled as default

    Thanks Ahrasis, maybe I did not understand the context because my english.

    I'll do the tests and come back with the results.

    Thanks in advance
     

Share This Page