In Python 2

>>> plain_string = "Hi!"
>>> unicode_string = u"Hi!"
>>> type(plain_string), type(unicode_string)
(<type 'str'>, <type 'unicode'>)

^ This is the difference between a byte string (plain_string) and a unicode string.

>>> s = "Hello!"
>>> u = unicode(s, "utf-8")

^ Converting to unicode and specifying the encoding.

In Python 3

All strings are unicode. The unicode function does not exist anymore. See answer from @Noumenon

Literal strings are unicode by default in Python3.

Assuming that text is a bytes object, just use text.decode('utf-8')

unicode of Python2 is equivalent to str in Python3, so you can also write:

str(text, 'utf-8')

if you prefer.

Here's a class that overrides the representation of the string it wraps:

>>> class OctUTF8:
...   def __init__(self,s):
...     self.s = s.encode()
...   def __repr__(self):
...     return "b'" + ''.join(f'\\{n:03o}' for n in self.s) + "'"
...
>>> s='õ'
>>> OctUTF8(s)
b'\303\265'

This representation can be evaluated as a byte string and decoded back to the original:

>>> eval(repr(OctUTF8(s))).decode()
'õ'

To convert a string of that content, encode as ASCII first to create a byte string, then decode with the 'unicode-escape' codec:

s = r'"\u0627\u0644\u0625\u062f\u0627"'
print(s)
print(s.encode('ascii').decode('unicode-escape'))

Output:

"\u0627\u0644\u0625\u062f\u0627"
"الإدا"

Writing and reading a file that way:

with open('file.txt', 'w', encoding='unicode-escape') as f:
    f.write('"\u0627\u0644\u0625\u062f\u0627"')

with open('file.txt', 'r', encoding='unicode-escape') as f:
    print(f.read())

Content of file:

"\u0627\u0644\u0625\u062f\u0627"

Output:

"الإدا"

Solutions to support surrogate escapes. They need to be converted to actual Unicode code points and the surrogatepass error handler allows that, but requires another encode/decode cycle.

s = r'"\ud83c\uddfa\ud83c\uddf8"'
print(s)
print(s.encode('ascii').decode('unicode-escape').encode('utf-16le', errors='surrogatepass').decode('utf-16le'))

Output:

"🇺🇸"

with open('file.txt', 'w', encoding='unicode-escape') as f:
    f.write('"\ud83c\uddfa\ud83c\uddf8"')

with open('file.txt', encoding='unicode-escape') as f:
    data = f.read().encode('utf-16le', errors='surrogatepass').decode('utf-16le')
    print(data)
    print(ascii(data)) # To see the Unicode codepoints

Output:

"🇺🇸"
'"\U0001f1fa\U0001f1f8"'

The u'' syntax only works for string literals, e.g. defining values in source code. Using the syntax results in a unicode object being created, but that's not the only way to create such an object.

You cannot make a unicode value from a byte string by adding u in front of it. But if you called str.decode() with the right encoding, you get a unicode value. Vice-versa, you can encode unicode objects to byte strings with unicode.encode().

Note that when displaying a unicode object, Python represents it by using the Unicode string literal syntax again (so u'...'), to ease debugging. You can paste the representation back in to a Python interpreter and get an object with the same value.

Your a value is defined using a byte string literal, so you only need to decode:

a = 'Entre\xc3\xa9'
b = a.decode('utf8')

Your first example created a Mojibake, a Unicode string containing Latin-1 codepoints that actually represent UTF-8 bytes. This is why you had to encode to Latin-1 first (to undo the Mojibake), then decode from UTF-8.

You may want to read up on Python and Unicode in the Unicode HOWTO. Other articles of interest are:

• The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky

• Pragmatic Unicode by Ned Batchelder