You don't need to encode data that is already encoded. When you try to do that, Python will first try to decode it to unicode before it can encode it back to UTF-8. That is what is failing here:
>>> data = u'\u00c3' # Unicode data
>>> data = data.encode('utf8') # encoded to UTF-8
>>> data
'\xc3\x83'
>>> data.encode('utf8') # Try to *re*-encode it
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)
Just write your data directly to the file, there is no need to encode already-encoded data.
If you instead build up unicode values instead, you would indeed have to encode those to be writable to a file. You'd want to use codecs.open() instead, which returns a file object that will encode unicode values to UTF-8 for you.
You also really don't want to write out the UTF-8 BOM, unless you have to support Microsoft tools that cannot read UTF-8 otherwise (such as MS Notepad).
For your MySQL insert problem, you need to do two things:
Add
charset='utf8'to yourMySQLdb.connect()call.Use
unicodeobjects, notstrobjects when querying or inserting, but use sql parameters so the MySQL connector can do the right thing for you:artiste = artiste.decode('utf8') # it is already UTF8, decode to unicode c.execute('SELECT COUNT(id) AS nbr FROM artistes WHERE nom=%s', (artiste,)) # ... c.execute('INSERT INTO artistes(nom,status,path) VALUES(%s, 99, %s)', (artiste, artiste + u'/'))
It may actually work better if you used codecs.open() to decode the contents automatically instead:
import codecs
sql = mdb.connect('localhost','admin','ugo&(-@F','music_vibration', charset='utf8')
with codecs.open('config/index/'+index, 'r', 'utf8') as findex:
for line in findex:
if u'#artiste' not in line:
continue
artiste=line.split(u'[:::]')[1].strip()
cursor = sql.cursor()
cursor.execute('SELECT COUNT(id) AS nbr FROM artistes WHERE nom=%s', (artiste,))
if not cursor.fetchone()[0]:
cursor = sql.cursor()
cursor.execute('INSERT INTO artistes(nom,status,path) VALUES(%s, 99, %s)', (artiste, artiste + u'/'))
artists_inserted += 1
You may want to brush up on Unicode and UTF-8 and encodings. I can recommend the following articles:
The Python Unicode HOWTO
Pragmatic Unicode by Ned Batchelder
The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky
unicode - python encoding utf-8 - Stack Overflow
Working with UTF-8 encoding in Python source - Stack Overflow
UTF-8 encoding
Must not the utf-8 encoding, in this case, be on the first line and the comment on the second line?
You can actually put both on the first line. The python encoding behavior is defined in PEP0263 which states that you can put a "magic" encoding comment in line 1 or line 2. What counts as an encoding specification is very loose, technically anything that matches the regular expression coding[:=]\s*([-\w.]+). So all of the following comments on line 1 (or 2) will change the default encoding of Python 2 to utf-8:
# -*- coding: utf-8 -*- # !?! coding: utf-8 ?!? # coding=utf-8 #### the quick brown fox jumped coding=utf-8 over the lazy dog
You can mix in non-ascii characters on that same line provided the interpreter gets a match for the coding expression that is valid for those characters:
# åäö coding=utf-8 åäö
But if you have non-ascii characters on line 1, you can't leave the encoding specification until line 2. The following
# åäö # coding=utf-8 åäö
Results in:
SyntaxError: Non-ASCII character '\xc3'
Of course, as others point out, this is completely trivial in python 3 where utf-8 is the default encoding. You just put those characters in a comment in the first line.
More on reddit.comDebate: Enforcing Python source encoding as UTF-8
You don't need to encode data that is already encoded. When you try to do that, Python will first try to decode it to unicode before it can encode it back to UTF-8. That is what is failing here:
>>> data = u'\u00c3' # Unicode data
>>> data = data.encode('utf8') # encoded to UTF-8
>>> data
'\xc3\x83'
>>> data.encode('utf8') # Try to *re*-encode it
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)
Just write your data directly to the file, there is no need to encode already-encoded data.
If you instead build up unicode values instead, you would indeed have to encode those to be writable to a file. You'd want to use codecs.open() instead, which returns a file object that will encode unicode values to UTF-8 for you.
You also really don't want to write out the UTF-8 BOM, unless you have to support Microsoft tools that cannot read UTF-8 otherwise (such as MS Notepad).
For your MySQL insert problem, you need to do two things:
Add
charset='utf8'to yourMySQLdb.connect()call.Use
unicodeobjects, notstrobjects when querying or inserting, but use sql parameters so the MySQL connector can do the right thing for you:artiste = artiste.decode('utf8') # it is already UTF8, decode to unicode c.execute('SELECT COUNT(id) AS nbr FROM artistes WHERE nom=%s', (artiste,)) # ... c.execute('INSERT INTO artistes(nom,status,path) VALUES(%s, 99, %s)', (artiste, artiste + u'/'))
It may actually work better if you used codecs.open() to decode the contents automatically instead:
import codecs
sql = mdb.connect('localhost','admin','ugo&(-@F','music_vibration', charset='utf8')
with codecs.open('config/index/'+index, 'r', 'utf8') as findex:
for line in findex:
if u'#artiste' not in line:
continue
artiste=line.split(u'[:::]')[1].strip()
cursor = sql.cursor()
cursor.execute('SELECT COUNT(id) AS nbr FROM artistes WHERE nom=%s', (artiste,))
if not cursor.fetchone()[0]:
cursor = sql.cursor()
cursor.execute('INSERT INTO artistes(nom,status,path) VALUES(%s, 99, %s)', (artiste, artiste + u'/'))
artists_inserted += 1
You may want to brush up on Unicode and UTF-8 and encodings. I can recommend the following articles:
The Python Unicode HOWTO
Pragmatic Unicode by Ned Batchelder
The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky
Unfortunately, the string.encode() method is not always reliable. Check out this thread for more information: What is the fool proof way to convert some string (utf-8 or else) to a simple ASCII string in python
In Python 3, UTF-8 is the default source encoding (see PEP 3120), so Unicode characters can be used anywhere.
In Python 2, you can declare in the source code header:
# -*- coding: utf-8 -*-
....
This is described in PEP 0263.
Then you can use UTF-8 in strings:
# -*- coding: utf-8 -*-
u = 'idzie wąż wąską dróżką'
uu = u.decode('utf8')
s = uu.encode('cp1250')
print(s)
Do not forget to verify if your text editor encodes properly your code in UTF-8.
Otherwise, you may have invisible characters that are not interpreted as UTF-8.