Pure-FTPD windows to linux charset problems

Discussion in 'Installation/Configuration' started by Aris Dizdarevic, Jan 10, 2019.

  1. Aris Dizdarevic

    Aris Dizdarevic New Member

    Hi all, I'm new at this forum and I tried a lot of sudgestions but i can not get the solution to my problem. I hope someone from ispConfig community can help me...

    I installed ispConfig 3.1 on centOS and configured and everything works, except problem when I upload files text (.txt, .html...) from windows computer over FTP (Pure-FTPD) special characters inside the file (for example šČĆŽĐŠćććčžš changes to ▒▒ƎЊ▒▒▒▒▒).

    I tried and nothing seems to help:
    - tried FTP-client to use Binary instead of Ascii, tried to force it to use UTF-8 - nothing helped, used FTP over SSL...
    - tried different distributions ispConfig with centos 7, ispConfig with debian (all installations were done in virtualized environment vmWare ESXi and one localy on vmware player) - nothing helped
    - I even tried packet analyzing (wireshark in windows and tcpdump in linux) and content of FTP_DATA was the same in sent and recieved packet
    - I tried saving file in windows as ASCII (put to server NOT o.k.), Unicode (put to server NOT o.k.) - UTF-8 (the only option that preserves characters as it is - but I can not tell every customer to convert ANSI -> UTF-8 before FTP uploading)
    in our country windows uses "WINDOWS-1250" charset, in linux charset is "sl_SI.UTF-8".

    I noticed whenever we put file over FTP (for example test.txt) if i check file-charset (file -i test.txt) I get charset=unknown-8bit?

    If i use command iconv -f WINDOWS-1250 -t UTF-8 test.txt > test2.txt i get correct output and test2.txt charset is utf-8 and content is o.k.

    I believe this problem also occurs with other European languages (almost all have some special characters and most of us use windows os) so anyone who solved this problem please let me know the solution...
     
  2. Taleman

    Taleman Well-Known Member HowtoForge Supporter

    I have not had problems with non-ASCII characters. What is locale on that CentOS host? If both ends have UTF-8 text files should transfer just OK.
     
  3. Aris Dizdarevic

    Aris Dizdarevic New Member

    Taleman, thank you for comment.

    On Windows:
    sl_SI, ASCII (windows-1250)

    On CentOS locale is:
    System Locale: LANG=sl_SI.UTF-8, VC Keymap: si, X11 Layout: si

    On Debian locale is:
    System Locale: LANG=sl_SI.UTF-8, VC Keymap: n/a, X11 Layout: si, X11 Model: pc105

    You are right, if on windows machine I manually open text file and convert to UTF-8 and then upload the file is fine, but this way all the users would have to do that manually for all text, html files - so imagine website with 50 html pages... Windows default when editing/creating txt, html files is ASCII (windows-1250).
    I need some kind of solution for pure-ftpd or linux os to correctly set charset so i don't need to convert it manually.

    I even tried with to start pure-ftpd with --uploadscript directive, so it would execute upon txt, html file upload to auto-convert the file, but it seems EPEL repo of pure-ftpd wasn't compiled with --with-uploadscript directive or I'm doing something wrong - I can't get it to work... (Reference from systutorials)
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    pure-ftpd should be able to do charset conversions:

    But maybe the binary from Epel is compiled without that feature?
     
  5. Aris Dizdarevic

    Aris Dizdarevic New Member

    Till, thanks for answer, as I see you are one of the "mastermind" of ispConfig and thank you for all the help...

    Still I couldn't get it to work...

    Instead I wrote my own bash-script for checking files that are uploaded over FTP; if their charset is unknown-8bit convert them to UTF-8 so they are editable the correct way (in linux and in windows).

    I believe some other people have similar issues as I had, so I postet the code on github so anyone can use it:
    h??s://github.com/arisdizdarevic/windows-linux-ftp-charset-update

    I hope the link will be seen, and this bash script could also be used for antivirus scanning of FTp uploaded files (upon little modification) ;)
     
    till likes this.

Share This Page