In Python 2

>>> plain_string = "Hi!"
>>> unicode_string = u"Hi!"
>>> type(plain_string), type(unicode_string)
(<type 'str'>, <type 'unicode'>)

^ This is the difference between a byte string (plain_string) and a unicode string.

>>> s = "Hello!"
>>> u = unicode(s, "utf-8")

^ Converting to unicode and specifying the encoding.

In Python 3

All strings are unicode. The unicode function does not exist anymore. See answer from @Noumenon

č is not represented by u'\u00c4\u008d'. Those two hex values are the UTF-8-encoded values, so should be written in a byte string as '\xc4\x8d'. Example:

>>> s = '\xc4\x8d'
>>> s.decode('utf8')
u'\u010d'
>>> print(s.decode('utf8'))
č

Caveat: Your terminal must be configured with an encoding that supports the character to print correctly, or you will see a UnicodeEncodeError.

If for some reason you have a mis-decoded Unicode string, you can take advantage of the fact that the first 256 code points of Unicode correlate to the latin1 encoding and fix it:

>>> s = u'\u00c4\u008d'
>>> s.encode('latin1')
'\xc4\x8d'
>>> s.encode('latin1').decode('utf8')
u'\u010d'
>>> print(s.encode('latin1').decode('utf8'))
č

If you have a mis-decoded Unicode string, you should show the file you have or the code that read it and solve that problem.

Always decode on input, and encode on output. (There ought to be handy mnemonic for this: perhaps "take your code [coat] off when you come indoors".)

Decode on input: You say that the database encoding is "UTF_8_bin". Are you using MySQL-Python? If so, then you can set the use_unicode option when you connect to the database. Then all strings are fetched from the database in Unicode, so you don't have worry about decoding them.

Encode on output: You can find out the current character encoding (or "code page" as they call it in Windows) with the chcp command. Let's suppose it's code page 1252. Then you can write

print text.encode('windows-1252')

to produce something that you can read from the Windows command line.

If you're writing the strings back to another MySQL database using MySQL-Python, you shouldn't need to do anything special: MySQL-Python claims that "you can always write Unicode strings" (regardless of whether you specified use_unicode when you opened the connection).

Just call encode() on your unicode string, then hexlify() it.

s = u'\u0905 \u0905 \u0918 \ua5c4'
print s
अ अ घ ꗄ
s_utf8 = s.encode('utf8')
print s_utf8
अ अ घ ꗄ
>>> s_utf8
'\xe0\xa4\x85 \xe0\xa4\x85 \xe0\xa4\x98 \xea\x97\x84'
>>> from binascii import hexlify
>>> hexlify(s_utf8)
>>> 'e0a48520e0a48520e0a49820ea9784'

If you want to preserve the spaces, you could do this:

>>> ' '.join(hexlify(c) for c in s_utf8.split())
'e0a485 e0a485 e0a498 ea9784'
>>> b' '.join(hexlify(c) for c in s_utf8.split())    # Python 3
b'e0a485 e0a485 e0a498 ea9784'

Or in one go:

>>> s
u'\u0905 \u0905 \u0918 \ua5c4'
>>> ' '.join(hexlify(c.encode('utf8')) for c in s.split())
'e0a485 e0a485 e0a498 ea9784'
>>> b' '.join(hexlify(c.encode('utf8')) for c in s.split())    # Python 3
b'e0a485 e0a485 e0a498 ea9784'

You can call upper() on the resultant strings to produce 'E0A485 E0A485 E0A498 EA9784':

If you have a unicode value with UTF-8 bytes, encode to Latin-1 to preserve the 'bytes':

content = content.encode('latin1')

because the Unicode codepoints U+0000 to U+00FF all map one-on-one with the latin-1 encoding; this encoding thus interprets your data as literal bytes.

For your example this gives me:

>>> content = u'\xe5\xb1\x82\xe5\x8f\xa0\xe6\xa0\xb7\xe5\xbc\x8f\xe8\xa1\xa8'
>>> content.encode('latin1')
'\xe5\xb1\x82\xe5\x8f\xa0\xe6\xa0\xb7\xe5\xbc\x8f\xe8\xa1\xa8'
>>> content.encode('latin1').decode('utf8')
u'\u5c42\u53e0\u6837\u5f0f\u8868'
>>> print content.encode('latin1').decode('utf8')
层叠样式表

PyQuery uses either requests or urllib to retrieve the HTML, and in the case of requests, uses the .text attribute of the response. This auto-decodes the response data based on the encoding set in a Content-Type header alone, or if that information is not available, uses latin-1 for this (for text responses, but HTML is a text response). You can override this by passing in an encoding argument:

dom = PyQuery('http://zh.wikipedia.org/w/index.php', encoding='utf8',
              {'title': 'CSS', 'printable': 'yes', 'variant': 'zh-cn'})

at which point you'd not have to re-encode at all.

• The L postfixes signify long integers. They are the same thing as (short) integers really; there really is no need to convert these. It is only their repr() output that includes the L; print the value directly or write it to a file and the L postfix is not included.

• Unicode values can be encoded to UTF-8 with the unicode.encode() method:

  encoded = unicodestr.encode('utf8')
  

Your beef is with the list representation here; you logged all rows, and Python containers represent their contents by calling repr() on each value. These representations are great for debugging as their types are made obvious.

It depends on what you do with these values next. It is generally a good idea to use Unicode throughout your code, and only encode at the last moment (when writing to a file, or printing or sending over the network). A good many methods handle this for you. Printing will encode to your terminal codec automatically, for example. When adding to an XML file, most XML libraries handle Unicode for you. Etc.

You don't need to encode data that is already encoded. When you try to do that, Python will first try to decode it to unicode before it can encode it back to UTF-8. That is what is failing here:

>>> data = u'\u00c3'            # Unicode data
>>> data = data.encode('utf8')  # encoded to UTF-8
>>> data
'\xc3\x83'
>>> data.encode('utf8')         # Try to *re*-encode it
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

Just write your data directly to the file, there is no need to encode already-encoded data.

If you instead build up unicode values instead, you would indeed have to encode those to be writable to a file. You'd want to use codecs.open() instead, which returns a file object that will encode unicode values to UTF-8 for you.

You also really don't want to write out the UTF-8 BOM, unless you have to support Microsoft tools that cannot read UTF-8 otherwise (such as MS Notepad).

For your MySQL insert problem, you need to do two things:

• Add charset='utf8' to your MySQLdb.connect() call.

• Use unicode objects, not str objects when querying or inserting, but use sql parameters so the MySQL connector can do the right thing for you:

  artiste = artiste.decode('utf8')  # it is already UTF8, decode to unicode
  
  c.execute('SELECT COUNT(id) AS nbr FROM artistes WHERE nom=%s', (artiste,))
  
  # ...
  
  c.execute('INSERT INTO artistes(nom,status,path) VALUES(%s, 99, %s)', (artiste, artiste + u'/'))
  

It may actually work better if you used codecs.open() to decode the contents automatically instead:

import codecs

sql = mdb.connect('localhost','admin','ugo&(-@F','music_vibration', charset='utf8')

with codecs.open('config/index/'+index, 'r', 'utf8') as findex:
    for line in findex:
        if u'#artiste' not in line:
            continue

        artiste=line.split(u'[:::]')[1].strip()

    cursor = sql.cursor()
    cursor.execute('SELECT COUNT(id) AS nbr FROM artistes WHERE nom=%s', (artiste,))
    if not cursor.fetchone()[0]:
        cursor = sql.cursor()
        cursor.execute('INSERT INTO artistes(nom,status,path) VALUES(%s, 99, %s)', (artiste, artiste + u'/'))
        artists_inserted += 1

You may want to brush up on Unicode and UTF-8 and encodings. I can recommend the following articles:

• The Python Unicode HOWTO

• Pragmatic Unicode by Ned Batchelder

• The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky

What am I doing wrong?

Not printing it.

When you just evaluate a string in Python REPL, you will get its repr. This is '\xd0\x90\xd0\xb1\xd0\xb0\xd0\xba\xd0\xb0\xd0\xbd'. When you print it, you will get Абакан.

print(string.encode('utf-8'))

Python 3.x:

In Python 3.x, str is the class for Unicode text, and bytes is for containing octets.

If by "octets" you really mean strings in the form '0xc5' (rather than '\xc5') you can convert to bytes like this:

>>> bytes(int(x,0) for x in ['0xc5', '0x81'])
b'\xc5\x81'

You can then convert to str (ie: Unicode) using the str constructor...

>>> str(b'\xc5\x81', 'utf-8')
'Ł'

...or by calling .decode('utf-8') on the bytes object:

>>> b'\xc5\x81'.decode('utf-8')
'Ł'
>>> hex(ord('Ł'))
'0x141'

Pre-3.x:

Prior to 3.x, the str type was a byte array, and unicode was for Unicode text.

Again, if by "octets" you really mean strings in the form '0xc5' (rather than '\xc5') you can convert them like this:

>>> ''.join(chr(int(x,0)) for x in ['0xc5', '0x81'])
'\xc5\x81'

You can then convert to unicode using the constructor...

>>> unicode('\xc5\x81', 'utf-8')
u'\u0141'

...or by calling .decode('utf-8') on the str:

>>> '\xc5\x81'.decode('utf-8')
u'\u0141'