Issue with character sets (debian 3.1)

Discussion in 'Installation/Configuration' started by adam, Sep 10, 2006.

  1. adam

    adam New Member

    I've got a really weird problem. I run an oscommerce estore that's in swedish and exported an sql file through phpmyadmin from my webhost. The character set of the sql file is utf-8. The tables I exported the sql are all latin1 and when I open up the file on my ubuntu system all looks just fine.

    When i transfer the sql to my debian server and open it, some of the swedish letters are just screwed up and some are just fine. The locale on the server was set up as sv_ISO8859-1.

    Here's the real weird thing. When i import this sql file to the mysql database it displays fine on the website.

    e.g Väskor/Plånböcker

    Now when I click a link on the site that pretty much refreshes the page it becomes

    V�skor/Pl�nb�cker->

    When I click a link again it turns back to normal (Väskor/Plånböcker), and then click again it goes to this V�skor/Pl�nb�cker->

    Sometimes, I can click several links and it's fine and then the next click it changes to V�skor/Pl�nb�cker-> again.

    The server locales at the moment are set at UTF-8 and this is the problem I get. If i put the server locale to is8859-1 i get exactly the same issue with different gibberish where ÖÅÄ are supposed to be.

    Can anyone tell me what's going on?:confused:
     
  2. falko

    falko Super Moderator Howtoforge Staff

    Do both servers use the same MySQL version?
     
  3. adam

    adam New Member

    My server is running MySQL 4.1.11 while my old host is running 4.1.20.

    I've used apt-get mysql-server-4.1 mysql-common-4.1 to install mine.

    I tried to downgrade to 4.0 but it won't run, i get a mysqld.sock error.

    My /etc/locale.gen reads

    sv_SE UTF-8
    sv_SE ISO-8859-1

    In phpmyadmin on my server it's set to

    MySQL charset: UTF-8 Unicode (utf8)
    MySQL connection collation: utf8_general_ci

    the database was created like this ...

    "create database tdfdbosc character set latin1 collate latin1_swedish_ci;"

    On my host, the character set of the tables is reported as latin1.

    When I import the sql file into my database through phpmyadmin it looks totally fine. But when i look online i get this switching problem. This is the first time i've ever had a problem with this. I've had character set problems int he past but always managed to fix them by fiddling with the different char set options in the client and host tools.

    If it is mysql being a bit older, whats the easiest way to isntall the new one? just downlaod the sources and recompile?
     
  4. adam

    adam New Member

    i've enclosed two images.
     

    Attached Files:

  5. falko

    falko Super Moderator Howtoforge Staff

    I don't think it's a problem with your MySQL versions.
    Can you try to export and import the databases with ISO8859-1?
     
  6. adam

    adam New Member

    sorry to waste your time on this , falko. I found my problem just this second!

    The problem was the cache in /tmp. I cleared the directory and the problem disappeared. I've moved the oscommerce cache setting to where i store my sessions and hopefully that will fix.

    I was getting really worried the mysql was serving wrong information.
     
  7. pinz0

    pinz0 New Member


    hi adam!

    i have the same problem:
    i migrated a website + myswl-db from a debian 3.1 server to my ubuntu 6.06 server. everything worls fine, but i also get these strange characters.

    wow did you solve this problem?
    could you drop me some helpful ideas?

    bests,
     
  8. adam

    adam New Member

    download your database to your pc and open the sql file. if you see strange characters in the file you want to change the character encoding that the file is being displayed withe and see if you can "fix" it on the fly. it's usually either utf-8 or iso-8859. I installed kate on my ubuntu system has it's got better options for this, you can simply go into one of its menus and choose which encoding to display. The trick is to know what format your document is in that you exported. Ubuntu is utf-8 by default.

    if you use a tool like phpmyadmin or mysql admin you can specifiy the encoding of your document. note, this has nothing to do with the destination encoding on your mysql tables, which is quite often latin1, iso-8859.

    once you've imported your sql file correctly, the next thing you want to check is the source file of your html page after rendering.

    content="text/html; charset=iso-8859-1">

    if you look for a line like this it will tell your browser what to use. you can also toggle this manually in firefox in the view menu and then character encoding. If this is missing it's advisable to put it in so you're browser will sort it out automatically.
     
  9. pinz0

    pinz0 New Member

    thanks for yr reply!
    unfortunately things haven't changed yet.
    might it be a problem with the global charset-settings on my ubuntu-machine?
    the thing is, that every website on my machine won't display website correctly.

    take a look at the default installation of a phpBB-Forum on my server.
    http://www.allocat.net/forum

    ???? :(
     
  10. adam

    adam New Member

    hmmm when i load your site it's being detected as UTF-8. When i go to to view->character encoding and select western 8859 in firefox the site displays perfectly.

    When i look at your source code for that page , the encoding is not being set (it tells me the site is being viewed as a utf8 site), so it's probably using the encoding of the server which is probably utf8.

    your actual data seems fine.

    compare my source header with yours

    yours
    Code:
    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
    <html dir="ltr">
    <head>
    <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
    <meta http-equiv="Content-Style-Type" content="text/css">
    
    mine
    Code:
    !doctype html public "-//W3C//DTD HTML 4.01 Transitional//EN">
    <html dir="LTR" lang="se">
    <head>
    <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
    
    The difference seems to be that i have lang="se", you probably want lang="de". maybe check the php board settings in the admin panel.

    you could try editing /etc/locale.gen and tell the server to use iso-8859-1 instead of utf 8 and see what happens.
     
    Last edited: Sep 13, 2006
  11. pinz0

    pinz0 New Member


    i changed the code to lang="de" (...) , but this wasn't helpul at all..
    i think, i have to change the /etc/locale.gen
    the thing is, there is no locale.gen in /etc.

    ??? ahhh! :eek:
     
  12. adam

    adam New Member

    dpkg-reconfigure locales


    does this command work?


    otherwise you might want to do

    apt-get install locales

    then sort it out.
     
  13. pinz0

    pinz0 New Member

    yes, but with this results:

    Code:
    perl: warning: Setting locale failed.
    perl: warning: Please check that your locale settings:
            LANGUAGE = "de_AT:de_DE:de:en_GB:en",
            LC_ALL = (unset),
            LANG = "de_AT"
        are supported and installed on your system.
    perl: warning: Falling back to the standard locale ("C").
    locale: Cannot set LC_CTYPE to default locale: No such file or directory
    locale: Cannot set LC_MESSAGES to default locale: No such file or directory
    locale: Cannot set LC_ALL to default locale: No such file or directory
    Generating locales...
      de_AT.UTF-8... up-to-date
      de_DE.ISO-8859-15@euro... up-to-date
      en_US.UTF-8... up-to-date
    Generation complete.
    Current default timezone: 'Europe/Vienna'.
    Local time is now:      Thu Sep 14 00:09:24 CEST 2006.
    Universal Time is now:  Wed Sep 13 22:09:24 UTC 2006.
    Run 'tzconfig' if you wish to change it.
    
     
  14. adam

    adam New Member

    whats in

    /etc/environment
     
  15. pinz0

    pinz0 New Member


    Code:
    root@lemmi:/etc# more environment
    PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11:/usr/games"
    LANG="de_AT.UTF-8"
    LANGUAGE="de_AT:de_DE:de:en_GB:en"
    ### BEGIN DEBCONF SECTION FOR localeconf
    # Do not edit within this region if you want your changes to be preserved
    # by debconf.  Instead, make changes before the "### BEGIN DEBCONF SECTION
    # FOR localeconf" line, and/or after the "### END DEBCONF SECTION FOR
    # localeconf" line.
    LANG=de_AT
    ### END DEBCONF SECTION FOR localeconf
    
     
  16. adam

    adam New Member

    maybe try changing utf-8 to ISO-8859-1?
    I must say though, mine is much simpler than yours.
     
  17. pinz0

    pinz0 New Member

    hm, not working ... :(
     
  18. adam

    adam New Member

    sorry i dont know what else to suggest.
     
  19. pinz0

    pinz0 New Member


    but thanks for everything! :)
     
  20. falko

    falko Super Moderator Howtoforge Staff

    Run
    Code:
    apt-get install localeconf

    Select locales to install (e.g. en_US ISO-8859-1) and select the standard locale (e.g. en_US).

    You will be asked a few questions:

    Manage locale configuration files with debconf? <-- Yes
    Environment settings that should override the default locale: <-- do not select anything
    Replace existing locale configuration files? <-- Yes
    Default system locale: <-- e.g. en_US ISO-8859-1
     

Share This Page