NO -- First let's decide what the full situation is.
What version are you using? If MySQL 5.7, consider going to utf8mb4 so that you can handle Emoji and all of Chinese. If 5.5 or 5.6, that is also possible, but you might run into some problems.
http://mysql.rjweb.org/doc.php/charcoll#fixes_for_various_cases
Case 1: The columns are currently CHARACTER SET latin1 and contain only latin1-encoded text. Then do this for each table:
ALTER TABLE t CONVERT TO CHARACTER SET utf8;
Case 2: The columns are currently CHARACTER SET latin1 but you have utf8-encoded characters in them. This leads to Mojibake or the silent "double encoding". The fix needs a pair of alters for each column:
Case 3 (double encoding): Then, and only then, this is called for:
UPDATE tbl SET col = CONVERT(BINARY(CONVERT(col USING latin1)) USING utf8mb4);
More discussion
CHARACTER SET latin1, but have utf8 bytes in it; leave bytes alone while fixing charset: First, lets assume you have this declaration for tbl.col:
col VARCHAR(111) CHARACTER SET latin1 NOT NULL
Then to convert the column without changing the bytes:
ALTER TABLE tbl MODIFY COLUMN col VARBINARY(111) NOT NULL;
ALTER TABLE tbl MODIFY COLUMN col VARCHAR(111) CHARACTER SET utf8mb4 NOT NULL;
Note: If you start with TEXT, use BLOB as the intermediate definition. (This is the "2-step ALTER, as discussed elsewhere.) (Be sure to keep the other specifications the same - VARCHAR, NOT NULL, etc.)
Which case??
In order to determine which case you have, please provide a small sample of the current data via:
SELECT HEX(col), col FROM t WHERE ...
Example: If the col has é, and the HEX is E9 -- that's latin1. If the Hex is C3A9, it's utf8 improperly stored into latin1. Hex of C383C2A9 would indicate "double encoding".
Generate ALTERs
A tip on how to generate the ALTERs can be found here . (It is not exactly what you need, but close.)
NO -- First let's decide what the full situation is.
What version are you using? If MySQL 5.7, consider going to utf8mb4 so that you can handle Emoji and all of Chinese. If 5.5 or 5.6, that is also possible, but you might run into some problems.
http://mysql.rjweb.org/doc.php/charcoll#fixes_for_various_cases
Case 1: The columns are currently CHARACTER SET latin1 and contain only latin1-encoded text. Then do this for each table:
ALTER TABLE t CONVERT TO CHARACTER SET utf8;
Case 2: The columns are currently CHARACTER SET latin1 but you have utf8-encoded characters in them. This leads to Mojibake or the silent "double encoding". The fix needs a pair of alters for each column:
Case 3 (double encoding): Then, and only then, this is called for:
UPDATE tbl SET col = CONVERT(BINARY(CONVERT(col USING latin1)) USING utf8mb4);
More discussion
CHARACTER SET latin1, but have utf8 bytes in it; leave bytes alone while fixing charset: First, lets assume you have this declaration for tbl.col:
col VARCHAR(111) CHARACTER SET latin1 NOT NULL
Then to convert the column without changing the bytes:
ALTER TABLE tbl MODIFY COLUMN col VARBINARY(111) NOT NULL;
ALTER TABLE tbl MODIFY COLUMN col VARCHAR(111) CHARACTER SET utf8mb4 NOT NULL;
Note: If you start with TEXT, use BLOB as the intermediate definition. (This is the "2-step ALTER, as discussed elsewhere.) (Be sure to keep the other specifications the same - VARCHAR, NOT NULL, etc.)
Which case??
In order to determine which case you have, please provide a small sample of the current data via:
SELECT HEX(col), col FROM t WHERE ...
Example: If the col has é, and the HEX is E9 -- that's latin1. If the Hex is C3A9, it's utf8 improperly stored into latin1. Hex of C383C2A9 would indicate "double encoding".
Generate ALTERs
A tip on how to generate the ALTERs can be found here . (It is not exactly what you need, but close.)
May case : The table is CHARACTER SET utf8mb4 but some columns had lain1 text (after an upgrade from MySQL 5.7 to 8.0)
Solution that worked for me :
BACKUP YOUR BD FIRST
To preview the result before execution :
SELECT column_name, CONVERT(CAST(CONVERT(column_name USING LATIN1) AS BINARY) USING UTF8MB4) AS converted_column_name FROM table_name
To convert :
UPDATE table_name set column_name = CONVERT(CAST(CONVERT(column_name USING LATIN1) AS BINARY) USING UTF8MB4) WHERE CONVERT(CAST(CONVERT(column_name USING LATIN1) AS BINARY) USING UTF8MB4) IS NOT NULL;
Do that for each column.
Why the WHERE clause...? Because converting a text that has already been converted in UTF8 will set your column_name to NULL
Convert UTF-8 to ISO-8859-1 (Latin-1) - LaTeX.org
php - Convert latin1 to UTF8 - Stack Overflow
character encoding - Python: Converting from ISO-8859-1/latin1 to UTF-8 - Stack Overflow
python - Python3: Convert Latin-1 to UTF-8 - Stack Overflow
The following MySQL function will return the correct utf8 string after double-encoding:
CONVERT(CAST(CONVERT(field USING latin1) AS BINARY) USING utf8)
It can be used with an UPDATE statement to correct the fields:
UPDATE tablename SET field = CONVERT(CAST(CONVERT(field USING latin1) AS BINARY) USING utf8);
if you need to convert the whole database , you can back it as databaseback.sql file then form your command line
iconv -f latin1 -t utf-8 < databaseback.sql > databaseback.utf8.sql
you can use the http://www.php.net/manual/en/function.iconv.php
to convert each row in php in case you don't have command line access
and lastly don't forget to convert the collation of each field in phpmyadmin , then you can resotre the utf8 back easily
update
if you got iconv is not recognized , it means that you don't have iconv installed
much more easier solution is : Migrating MySQL Data to Unicode
http://daveyshafik.com/archives/166-migrating-mysql-data-to-unicode.html
This is a common problem, so here's a relatively thorough illustration.
For non-unicode strings (i.e. those without u prefix like u'\xc4pple'), one must decode from the native encoding (iso8859-1/latin1, unless modified with the enigmatic sys.setdefaultencoding function) to unicode, then encode to a character set that can display the characters you wish, in this case I'd recommend UTF-8.
First, here is a handy utility function that'll help illuminate the patterns of Python 2.7 string and unicode:
>>> def tell_me_about(s): return (type(s), s)
A plain string
>>> v = "\xC4pple" # iso-8859-1 aka latin1 encoded string
>>> tell_me_about(v)
(<type 'str'>, '\xc4pple')
>>> v
'\xc4pple' # representation in memory
>>> print v
?pple # map the iso-8859-1 in-memory to iso-8859-1 chars
# note that '\xc4' has no representation in iso-8859-1,
# so is printed as "?".
Decoding a iso8859-1 string - convert plain string to unicode
>>> uv = v.decode("iso-8859-1")
>>> uv
u'\xc4pple' # decoding iso-8859-1 becomes unicode, in memory
>>> tell_me_about(uv)
(<type 'unicode'>, u'\xc4pple')
>>> print v.decode("iso-8859-1")
Äpple # convert unicode to the default character set
# (utf-8, based on sys.stdout.encoding)
>>> v.decode('iso-8859-1') == u'\xc4pple'
True # one could have just used a unicode representation
# from the start
A little more illustration — with “Ä”
>>> u"Ä" == u"\xc4"
True # the native unicode char and escaped versions are the same
>>> "Ä" == u"\xc4"
False # the native unicode char is '\xc3\x84' in latin1
>>> "Ä".decode('utf8') == u"\xc4"
True # one can decode the string to get unicode
>>> "Ä" == "\xc4"
False # the native character and the escaped string are
# of course not equal ('\xc3\x84' != '\xc4').
Encoding to UTF
>>> u8 = v.decode("iso-8859-1").encode("utf-8")
>>> u8
'\xc3\x84pple' # convert iso-8859-1 to unicode to utf-8
>>> tell_me_about(u8)
(<type 'str'>, '\xc3\x84pple')
>>> u16 = v.decode('iso-8859-1').encode('utf-16')
>>> tell_me_about(u16)
(<type 'str'>, '\xff\xfe\xc4\x00p\x00p\x00l\x00e\x00')
>>> tell_me_about(u8.decode('utf8'))
(<type 'unicode'>, u'\xc4pple')
>>> tell_me_about(u16.decode('utf16'))
(<type 'unicode'>, u'\xc4pple')
Relationship between unicode and UTF and latin1
>>> print u8
Äpple # printing utf-8 - because of the encoding we now know
# how to print the characters
>>> print u8.decode('utf-8') # printing unicode
Äpple
>>> print u16 # printing 'bytes' of u16
���pple
>>> print u16.decode('utf16')
Äpple # printing unicode
>>> v == u8
False # v is a iso8859-1 string; u8 is a utf-8 string
>>> v.decode('iso8859-1') == u8
False # v.decode(...) returns unicode
>>> u8.decode('utf-8') == v.decode('latin1') == u16.decode('utf-16')
True # all decode to the same unicode memory representation
# (latin1 is iso-8859-1)
Unicode Exceptions
>>> u8.encode('iso8859-1')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0:
ordinal not in range(128)
>>> u16.encode('iso8859-1')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0:
ordinal not in range(128)
>>> v.encode('iso8859-1')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 0:
ordinal not in range(128)
One would get around these by converting from the specific encoding (latin-1, utf8, utf16) to unicode e.g. u8.decode('utf8').encode('latin1').
So perhaps one could draw the following principles and generalizations:
- a type
stris a set of bytes, which may have one of a number of encodings such as Latin-1, UTF-8, and UTF-16 - a type
unicodeis a set of bytes that can be converted to any number of encodings, most commonly UTF-8 and latin-1 (iso8859-1) - the
printcommand has its own logic for encoding, set tosys.stdout.encodingand defaulting to UTF-8 - One must decode a
strto unicode before converting to another encoding.
Of course, all of this changes in Python 3.x.
Hope that is illuminating.
Further reading
- Characters vs. Bytes, by Tim Bray.
And the very illustrative rants by Armin Ronacher:
- The Updated Guide to Unicode on Python (July 2, 2013)
- More About Unicode in Python 2 and 3 (January 5, 2014)
- UCS vs UTF-8 as Internal String Encoding (January 9, 2014)
- Everything you did not want to know about Unicode in Python 3 (May 12, 2014)
Try decoding it first, then encoding:
apple.decode('iso-8859-1').encode('utf8')
I have found a half-part way in this. This is not what you want / need, but might help others in the right direction...
# First read the file
txt = open("file_name", "r", encoding="latin-1") # r = read, w = write & a = append
items = txt.readlines()
txt.close()
# and write the changes to file
output = open("file_name", "w", encoding="utf-8")
for string_fin in items:
if "é" in string_fin:
string_fin = string_fin.replace("é", "é")
if "ë" in string_fin:
string_fin = string_fin.replace("ë", "ë")
# this works if not to much needs changing...
output.write(string_fin)
output.close();
*note for detection
For python 3.6:
your_str = your_str.encode('utf-8').decode('latin-1')
I managed to solve it by running updates on text fields like this:
UPDATE table SET title = CONVERT(CONVERT(CONVERT(title USING latin1) USING binary) USING UTF8)
The situation isn't as bad as you think it is, unless you already have lots of non-Roman characters (that is, characters that aren't representable in Latin-1) in your database already. Latin-1 is a proper subset of utf8. Your web app works in utf8 and your tables' contents are in utf8 as well. So there's no need to convert the tables.
So, try changing the SET NAMES latin1 to SET NAMES utf8. It will probably solve your problem, by allowing your php program's connection to work with the same character set as the code on either end of the connection.
Read this. http://dev.mysql.com/doc/refman/5.7/en/charset-connection.html
Unicode is certainly difficult, and the UTF-8 encoding has a couple of inconvenient properties. However, UTF-8 has become the de-facto standard encoding on the web, surpassing ASCII, Latin-1, UCS-2 and UTF-16. Just use UTF-8 everywhere.
The most important reason why you should support Unicode is that you shouldn't make unnecessary assumptions about user input. I have no idea what your domain is, but things like Hebrew usernames, a blog post about China, a comment with Emoji, or simply well styled text – like “this” – should be possible… Oh, those were typographically correct quotation marks (“” rather than ""), en-wide dashes, and an ellipsis, which are characters that are common in English text, but not supported by ASCII or Latin-1. So not supporting other scripts isn't just a big f*ck you to other cultures, but sticking to Latin-1 doesn't even allow you to write proper English.
The notion that Unicode only allows “bad characters” is wrong. Yes, text is really complicated, and Unicode won't hide that from you. Your boss may be thinking about composed characters, where one base codepoint such as a is modified by subsequent codepoints that e.g. represent diacritics to form one visual character such as á. This doesn't really get into your way when trying to do searches if you do some kind of normalization. For example, you could store all text in the NFC form which collapses such compositions into their precomposed form if one is available. When doing searching, you could also strip all composing characters from the text, but this may substantially change their meaning in some languages.
Unicode also adds a lot of unprintable characters – but even ASCII has loads of them. Will you handle a NUL in the middle of a string? How about 0x1C, a “File Separator”? I've never seen half of those. Latin-1 adds a soft hyphen that indicates word break opportunities, but is otherwise invisible. Does that also break your full-text search? In other words, even ASCII and Latin-1 allow you to completely break your input if you assume it's all just printable text!
I think beyond the technical question, your boss may not have the time to keep up to date on current standards.
Since his stance is not completely out to lunch, just out-dated, respect his position when discussing this matter (and you need to remember to discuss, not argue), and try to work through concerns he has with regards to UTF-8. I suspect the underlying issue is not a technical issue and may require some level of soft-skill negotiation.