In Python 2

>>> plain_string = "Hi!"
>>> unicode_string = u"Hi!"
>>> type(plain_string), type(unicode_string)
(<type 'str'>, <type 'unicode'>)

^ This is the difference between a byte string (plain_string) and a unicode string.

>>> s = "Hello!"
>>> u = unicode(s, "utf-8")

^ Converting to unicode and specifying the encoding.

In Python 3

All strings are unicode. The unicode function does not exist anymore. See answer from @Noumenon

č is not represented by u'\u00c4\u008d'. Those two hex values are the UTF-8-encoded values, so should be written in a byte string as '\xc4\x8d'. Example:

>>> s = '\xc4\x8d'
>>> s.decode('utf8')
u'\u010d'
>>> print(s.decode('utf8'))
č

Caveat: Your terminal must be configured with an encoding that supports the character to print correctly, or you will see a UnicodeEncodeError.

If for some reason you have a mis-decoded Unicode string, you can take advantage of the fact that the first 256 code points of Unicode correlate to the latin1 encoding and fix it:

>>> s = u'\u00c4\u008d'
>>> s.encode('latin1')
'\xc4\x8d'
>>> s.encode('latin1').decode('utf8')
u'\u010d'
>>> print(s.encode('latin1').decode('utf8'))
č

If you have a mis-decoded Unicode string, you should show the file you have or the code that read it and solve that problem.

If you have a unicode value with UTF-8 bytes, encode to Latin-1 to preserve the 'bytes':

content = content.encode('latin1')

because the Unicode codepoints U+0000 to U+00FF all map one-on-one with the latin-1 encoding; this encoding thus interprets your data as literal bytes.

For your example this gives me:

>>> content = u'\xe5\xb1\x82\xe5\x8f\xa0\xe6\xa0\xb7\xe5\xbc\x8f\xe8\xa1\xa8'
>>> content.encode('latin1')
'\xe5\xb1\x82\xe5\x8f\xa0\xe6\xa0\xb7\xe5\xbc\x8f\xe8\xa1\xa8'
>>> content.encode('latin1').decode('utf8')
u'\u5c42\u53e0\u6837\u5f0f\u8868'
>>> print content.encode('latin1').decode('utf8')
层叠样式表

PyQuery uses either requests or urllib to retrieve the HTML, and in the case of requests, uses the .text attribute of the response. This auto-decodes the response data based on the encoding set in a Content-Type header alone, or if that information is not available, uses latin-1 for this (for text responses, but HTML is a text response). You can override this by passing in an encoding argument:

dom = PyQuery('http://zh.wikipedia.org/w/index.php', encoding='utf8',
              {'title': 'CSS', 'printable': 'yes', 'variant': 'zh-cn'})

at which point you'd not have to re-encode at all.

To convert a string of that content, encode as ASCII first to create a byte string, then decode with the 'unicode-escape' codec:

s = r'"\u0627\u0644\u0625\u062f\u0627"'
print(s)
print(s.encode('ascii').decode('unicode-escape'))

Output:

"\u0627\u0644\u0625\u062f\u0627"
"الإدا"

Writing and reading a file that way:

with open('file.txt', 'w', encoding='unicode-escape') as f:
    f.write('"\u0627\u0644\u0625\u062f\u0627"')

with open('file.txt', 'r', encoding='unicode-escape') as f:
    print(f.read())

Content of file:

"\u0627\u0644\u0625\u062f\u0627"

Output:

"الإدا"

Solutions to support surrogate escapes. They need to be converted to actual Unicode code points and the surrogatepass error handler allows that, but requires another encode/decode cycle.

s = r'"\ud83c\uddfa\ud83c\uddf8"'
print(s)
print(s.encode('ascii').decode('unicode-escape').encode('utf-16le', errors='surrogatepass').decode('utf-16le'))

Output:

"🇺🇸"

with open('file.txt', 'w', encoding='unicode-escape') as f:
    f.write('"\ud83c\uddfa\ud83c\uddf8"')

with open('file.txt', encoding='unicode-escape') as f:
    data = f.read().encode('utf-16le', errors='surrogatepass').decode('utf-16le')
    print(data)
    print(ascii(data)) # To see the Unicode codepoints

Output:

"🇺🇸"
'"\U0001f1fa\U0001f1f8"'

See unicodedata.normalize

title = u"Klüft skräms inför på fédéral électoral große"
import unicodedata
unicodedata.normalize('NFKD', title).encode('ascii', 'ignore')
'Kluft skrams infor pa federal electoral groe'

In Python3 all strings are unicode, so the problem you're having is likely due to your locale settings not being correct. The Python3 interpreter looks to use the locale environment variables and if it cannot find them it emulates basic ASCII

From locale.py:

except ImportError:

    # Locale emulation

    CHAR_MAX = 127
    LC_ALL = 6
    LC_COLLATE = 3
    LC_CTYPE = 0
    LC_MESSAGES = 5
    LC_MONETARY = 4
    LC_NUMERIC = 1
    LC_TIME = 2
    Error = ValueError

Double check the locale on your shell from which you are executing. Here are a few work arounds you can try to see if they get you working before you go through the task of getting your env setup correctly.

1) Validate UTF-8 locale or language files are installed (see link above)

2) Try adding this to the top of your script

#!/usr/bin/env LC_ALL=en_US.UTF-8 /usr/local/bin/python3
print('カタカナ')

or

#!/usr/bin/env LANG=en_US.UTF-8 /usr/local/bin/python3
print('カタカナ')

Or export shell variables before executing the Python interpreter

export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
python3
>>> print('カタカナ')

Sorry I cannot be more specific, as these settings are platform and OS specific. You can forcefully attempt to set the locale in Python directly using the locale module, but I don't recommend that, and it won't help if they are not installed.

Hope that helps.