NO -- First let's decide what the full situation is.

What version are you using? If MySQL 5.7, consider going to utf8mb4 so that you can handle Emoji and all of Chinese. If 5.5 or 5.6, that is also possible, but you might run into some problems.

http://mysql.rjweb.org/doc.php/charcoll#fixes_for_various_cases

Case 1: The columns are currently CHARACTER SET latin1 and contain only latin1-encoded text. Then do this for each table:

ALTER TABLE t CONVERT TO CHARACTER SET utf8;

Case 2: The columns are currently CHARACTER SET latin1 but you have utf8-encoded characters in them. This leads to Mojibake or the silent "double encoding". The fix needs a pair of alters for each column:

Case 3 (double encoding): Then, and only then, this is called for:

UPDATE tbl SET col = CONVERT(BINARY(CONVERT(col USING latin1)) USING utf8mb4);

More discussion

CHARACTER SET latin1, but have utf8 bytes in it; leave bytes alone while fixing charset: First, lets assume you have this declaration for tbl.col:

col VARCHAR(111) CHARACTER SET latin1 NOT NULL

Then to convert the column without changing the bytes:

ALTER TABLE tbl MODIFY COLUMN col VARBINARY(111) NOT NULL;
ALTER TABLE tbl MODIFY COLUMN col VARCHAR(111) CHARACTER SET utf8mb4 NOT NULL;

Note: If you start with TEXT, use BLOB as the intermediate definition. (This is the "2-step ALTER, as discussed elsewhere.) (Be sure to keep the other specifications the same - VARCHAR, NOT NULL, etc.)

Which case??

In order to determine which case you have, please provide a small sample of the current data via:

 SELECT HEX(col), col FROM t WHERE ...

Example: If the col has é, and the HEX is E9 -- that's latin1. If the Hex is C3A9, it's utf8 improperly stored into latin1. Hex of C383C2A9 would indicate "double encoding".

Generate ALTERs

A tip on how to generate the ALTERs can be found here . (It is not exactly what you need, but close.)

Answer from Rick James on Stack Overflow
🌐
Milosophical Me
milosophical.me › blog › 2018 › latin1-to-utf8.html
Converting latin-1 To utf-8 with Python | Milosophical Me
July 21, 2018 - It's not. While Unicode itself does contain a Latin-1 Supplement, all of the upper 128 character bit-patterns of ISO-8859-1 have different meaning in utf-8, and there are many legitimate ISO-8859-1 characters that are illegal utf-8 encoding byte sequences. This often confuses me because I'm ...
Top answer
1 of 3
6

NO -- First let's decide what the full situation is.

What version are you using? If MySQL 5.7, consider going to utf8mb4 so that you can handle Emoji and all of Chinese. If 5.5 or 5.6, that is also possible, but you might run into some problems.

http://mysql.rjweb.org/doc.php/charcoll#fixes_for_various_cases

Case 1: The columns are currently CHARACTER SET latin1 and contain only latin1-encoded text. Then do this for each table:

ALTER TABLE t CONVERT TO CHARACTER SET utf8;

Case 2: The columns are currently CHARACTER SET latin1 but you have utf8-encoded characters in them. This leads to Mojibake or the silent "double encoding". The fix needs a pair of alters for each column:

Case 3 (double encoding): Then, and only then, this is called for:

UPDATE tbl SET col = CONVERT(BINARY(CONVERT(col USING latin1)) USING utf8mb4);

More discussion

CHARACTER SET latin1, but have utf8 bytes in it; leave bytes alone while fixing charset: First, lets assume you have this declaration for tbl.col:

col VARCHAR(111) CHARACTER SET latin1 NOT NULL

Then to convert the column without changing the bytes:

ALTER TABLE tbl MODIFY COLUMN col VARBINARY(111) NOT NULL;
ALTER TABLE tbl MODIFY COLUMN col VARCHAR(111) CHARACTER SET utf8mb4 NOT NULL;

Note: If you start with TEXT, use BLOB as the intermediate definition. (This is the "2-step ALTER, as discussed elsewhere.) (Be sure to keep the other specifications the same - VARCHAR, NOT NULL, etc.)

Which case??

In order to determine which case you have, please provide a small sample of the current data via:

 SELECT HEX(col), col FROM t WHERE ...

Example: If the col has é, and the HEX is E9 -- that's latin1. If the Hex is C3A9, it's utf8 improperly stored into latin1. Hex of C383C2A9 would indicate "double encoding".

Generate ALTERs

A tip on how to generate the ALTERs can be found here . (It is not exactly what you need, but close.)

2 of 3
2

May case : The table is CHARACTER SET utf8mb4 but some columns had lain1 text (after an upgrade from MySQL 5.7 to 8.0)

Solution that worked for me :

BACKUP YOUR BD FIRST

To preview the result before execution :

SELECT column_name, CONVERT(CAST(CONVERT(column_name USING LATIN1) AS BINARY) USING UTF8MB4) AS converted_column_name FROM table_name

To convert :

UPDATE table_name set column_name = CONVERT(CAST(CONVERT(column_name USING LATIN1) AS BINARY) USING UTF8MB4) WHERE CONVERT(CAST(CONVERT(column_name USING LATIN1) AS BINARY) USING UTF8MB4) IS NOT NULL;

Do that for each column.


Why the WHERE clause...? Because converting a text that has already been converted in UTF8 will set your column_name to NULL

Discussions

Convert UTF-8 to ISO-8859-1 (Latin-1) - LaTeX.org
Hey there, Following a new installation, I accidentally managed to create a few files in UTF-8, rather than ISO-8859-1 (also known as Latin-1 I think?) which I normally use. Now I can't by any means seem to convert these files to ISO-8859-1 encoding, no matter what I do. More on latex.org
🌐 latex.org
January 22, 2014
php - Convert latin1 to UTF8 - Stack Overflow
I have a DB - with the table articles. I want to convert the title, and content field to utf8 now - all data looks like this: פורטל רעל × ×¤×ª×— רשמית! I want it to become normal hebrew More on stackoverflow.com
🌐 stackoverflow.com
character encoding - Python: Converting from ISO-8859-1/latin1 to UTF-8 - Stack Overflow
For non-unicode strings (i.e. those ... encoding (iso8859-1/latin1, unless modified with the enigmatic sys.setdefaultencoding function) to unicode, then encode to a character set that can display the characters you wish, in this case I'd recommend UTF-8.... More on stackoverflow.com
🌐 stackoverflow.com
python - Python3: Convert Latin-1 to UTF-8 - Stack Overflow
The files I work with are encoded in Latin-1 (I could not open them in UTF-8 obviously). More on stackoverflow.com
🌐 stackoverflow.com
Author: nicjansma
🌐
LaTeX.org
latex.org › board index › fonts & character sets
Convert UTF-8 to ISO-8859-1 (Latin-1) - LaTeX.org
January 22, 2014 - When changing a file encoding, your first step should always be a backup. You can mess things up quite easily. The encoding of the future (i.e. future standard) is utf8, thats why all modern editors have it as its standard. So, you might consider to convert your files from latin-1 to utf-8.
Top answer
1 of 5
181

This is a common problem, so here's a relatively thorough illustration.

For non-unicode strings (i.e. those without u prefix like u'\xc4pple'), one must decode from the native encoding (iso8859-1/latin1, unless modified with the enigmatic sys.setdefaultencoding function) to unicode, then encode to a character set that can display the characters you wish, in this case I'd recommend UTF-8.

First, here is a handy utility function that'll help illuminate the patterns of Python 2.7 string and unicode:

>>> def tell_me_about(s): return (type(s), s)

A plain string

>>> v = "\xC4pple" # iso-8859-1 aka latin1 encoded string

>>> tell_me_about(v)
(<type 'str'>, '\xc4pple')

>>> v
'\xc4pple'        # representation in memory

>>> print v
?pple             # map the iso-8859-1 in-memory to iso-8859-1 chars
                  # note that '\xc4' has no representation in iso-8859-1, 
                  # so is printed as "?".

Decoding a iso8859-1 string - convert plain string to unicode

>>> uv = v.decode("iso-8859-1")
>>> uv
u'\xc4pple'       # decoding iso-8859-1 becomes unicode, in memory

>>> tell_me_about(uv)
(<type 'unicode'>, u'\xc4pple')

>>> print v.decode("iso-8859-1")
Äpple             # convert unicode to the default character set
                  # (utf-8, based on sys.stdout.encoding)

>>> v.decode('iso-8859-1') == u'\xc4pple'
True              # one could have just used a unicode representation 
                  # from the start

A little more illustration — with “Ä”

>>> u"Ä" == u"\xc4"
True              # the native unicode char and escaped versions are the same

>>> "Ä" == u"\xc4"  
False             # the native unicode char is '\xc3\x84' in latin1

>>> "Ä".decode('utf8') == u"\xc4"
True              # one can decode the string to get unicode

>>> "Ä" == "\xc4"
False             # the native character and the escaped string are
                  # of course not equal ('\xc3\x84' != '\xc4').

Encoding to UTF

>>> u8 = v.decode("iso-8859-1").encode("utf-8")
>>> u8
'\xc3\x84pple'    # convert iso-8859-1 to unicode to utf-8

>>> tell_me_about(u8)
(<type 'str'>, '\xc3\x84pple')

>>> u16 = v.decode('iso-8859-1').encode('utf-16')
>>> tell_me_about(u16)
(<type 'str'>, '\xff\xfe\xc4\x00p\x00p\x00l\x00e\x00')

>>> tell_me_about(u8.decode('utf8'))
(<type 'unicode'>, u'\xc4pple')

>>> tell_me_about(u16.decode('utf16'))
(<type 'unicode'>, u'\xc4pple')

Relationship between unicode and UTF and latin1

>>> print u8
Äpple             # printing utf-8 - because of the encoding we now know
                  # how to print the characters

>>> print u8.decode('utf-8') # printing unicode
Äpple

>>> print u16     # printing 'bytes' of u16
���pple

>>> print u16.decode('utf16')
Äpple             # printing unicode

>>> v == u8
False             # v is a iso8859-1 string; u8 is a utf-8 string

>>> v.decode('iso8859-1') == u8
False             # v.decode(...) returns unicode

>>> u8.decode('utf-8') == v.decode('latin1') == u16.decode('utf-16')
True              # all decode to the same unicode memory representation
                  # (latin1 is iso-8859-1)

Unicode Exceptions

 >>> u8.encode('iso8859-1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0:
  ordinal not in range(128)

>>> u16.encode('iso8859-1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0:
  ordinal not in range(128)

>>> v.encode('iso8859-1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 0:
  ordinal not in range(128)

One would get around these by converting from the specific encoding (latin-1, utf8, utf16) to unicode e.g. u8.decode('utf8').encode('latin1').

So perhaps one could draw the following principles and generalizations:

  • a type str is a set of bytes, which may have one of a number of encodings such as Latin-1, UTF-8, and UTF-16
  • a type unicode is a set of bytes that can be converted to any number of encodings, most commonly UTF-8 and latin-1 (iso8859-1)
  • the print command has its own logic for encoding, set to sys.stdout.encoding and defaulting to UTF-8
  • One must decode a str to unicode before converting to another encoding.

Of course, all of this changes in Python 3.x.

Hope that is illuminating.

Further reading

  • Characters vs. Bytes, by Tim Bray.

And the very illustrative rants by Armin Ronacher:

  • The Updated Guide to Unicode on Python (July 2, 2013)
  • More About Unicode in Python 2 and 3 (January 5, 2014)
  • UCS vs UTF-8 as Internal String Encoding (January 9, 2014)
  • Everything you did not want to know about Unicode in Python 3 (May 12, 2014)
2 of 5
138

Try decoding it first, then encoding:

apple.decode('iso-8859-1').encode('utf8')
Find elsewhere
🌐
Quora
quora.com › How-should-I-convert-a-latin1-encoded-file-into-an-UTF-8-file
How should I convert a latin1 encoded file into an UTF-8 file? - Quora
Compare normalized text: diff <(iconv -f latin1 -t utf-8 input.txt) output.txt ... Latin1 is a single-byte superset of the first 256 Unicode code points; every latin1 byte maps to a Unicode code point, so conversion is lossless unless the file was mislabelled (actually another legacy encoding ...
🌐
Holodyn
billing.holodyn.com › knowledgebase › 61 › Converting-a-Database-from-latin1-to-UTF-8.html
Converting a Database from latin1 to UTF-8 - Knowledgebase - The Holodyn Corporation
Edit the SQL file using a UTF8 capable editor, like Notepad+. ** You can skip this step and run the Step 6 PHP script instead · Find and replace "CHARSET=latin1" with "CHARSET=utf8" Find and remove "set latin1 collate latin1_general_ci" from table field lines Save the file
🌐
GitHub
github.com › Fdawgs › fix-latin1-to-utf8
GitHub - Fdawgs/fix-latin1-to-utf8: Node.js module to fix mojibake when converting Latin-1 encoded text to UTF-8 · GitHub
When converting Latin-1 (or Windows-1252) encoded text to UTF-8, some characters may be incorrectly converted (mojibake). This module fixes those errors. ... Please refer to the JSDoc comments in the source code or the generated type definitions ...
Author: Fdawgs
🌐
Charset
charset.org › charset tools › utf-8 to latin converter (and vice versa)
UTF-8 to Latin Converter (and vice versa) - Charset Tools
Convert text from UTF-8 to Latin (ISO-8859-1) or from Latin to UTF-8. Use the online converter or programming routines in PHP, JavaScript and more.
🌐
Hey
world.hey.com › mtruty › beyond-character-chaos-converting-latin-1-to-utf-8-867e4a07
Beyond Character Chaos: Converting Latin-1 to UTF-8
September 20, 2023 - The Objective: Convert billions of rows of Latin-1 encoded data to UTF-8 with minimal data loss. The Solution: A WordPress Codex technique proposed a two-step conversion using a binary intermediary. The Result: Mission accomplished!
🌐
Stack Overflow
stackoverflow.com › questions › 38939565 › convert-string-from-latin-1-to-utf-8-and-back-to-latin-1
python - Convert string from Latin-1 to UTF-8 and back to Latin-1 - Stack Overflow
You're using Python 3; text is in Unicode already; then you encode it into latin 1, getting Ã\x96land in latin 1; which you decode as UTF-8, which is Öland now as Unicode text...
🌐
Google Groups
groups.google.com › g › irod-chat › c › pGvhVkkyDh0
encoding UTF-8 versus Latin1
From iROD-Chat:9519, I gathered that >> by default iRODS uses the LATIN1 character set, AKA ISO-8859-1. I >> changed the Jargon to use ISO-8859-1, and it can now list the file >> correctly as �7�9AQV��ׄ|�^3�. >> >> I'm going to recommend that we recompile our iRODS to use UTF-8 as >> its character set, but you might want to change Jargon to use >> ISO-8859-1 by default.
Top answer
1 of 6
136

Unicode is certainly difficult, and the UTF-8 encoding has a couple of inconvenient properties. However, UTF-8 has become the de-facto standard encoding on the web, surpassing ASCII, Latin-1, UCS-2 and UTF-16. Just use UTF-8 everywhere.

The most important reason why you should support Unicode is that you shouldn't make unnecessary assumptions about user input. I have no idea what your domain is, but things like Hebrew usernames, a blog post about China, a comment with Emoji, or simply well styled text – like “this” – should be possible… Oh, those were typographically correct quotation marks (“” rather than ""), en-wide dashes, and an ellipsis, which are characters that are common in English text, but not supported by ASCII or Latin-1. So not supporting other scripts isn't just a big f*ck you to other cultures, but sticking to Latin-1 doesn't even allow you to write proper English.

The notion that Unicode only allows “bad characters” is wrong. Yes, text is really complicated, and Unicode won't hide that from you. Your boss may be thinking about composed characters, where one base codepoint such as a is modified by subsequent codepoints that e.g. represent diacritics to form one visual character such as á. This doesn't really get into your way when trying to do searches if you do some kind of normalization. For example, you could store all text in the NFC form which collapses such compositions into their precomposed form if one is available. When doing searching, you could also strip all composing characters from the text, but this may substantially change their meaning in some languages.

Unicode also adds a lot of unprintable characters – but even ASCII has loads of them. Will you handle a NUL in the middle of a string? How about 0x1C, a “File Separator”? I've never seen half of those. Latin-1 adds a soft hyphen that indicates word break opportunities, but is otherwise invisible. Does that also break your full-text search? In other words, even ASCII and Latin-1 allow you to completely break your input if you assume it's all just printable text!

2 of 6
64

I think beyond the technical question, your boss may not have the time to keep up to date on current standards.

Since his stance is not completely out to lunch, just out-dated, respect his position when discussing this matter (and you need to remember to discuss, not argue), and try to work through concerns he has with regards to UTF-8. I suspect the underlying issue is not a technical issue and may require some level of soft-skill negotiation.