You have to decide what code-points a non-ASCII byte refers to. For instance, what code-point does the byte 0xA1 refer to?

For instance, you could use any of the iso-8859-X encodings:

bytes = chr(161)
utf8 = bytes.decode('iso-8859-1').encode('utf-8')
# compare with: utf8 = bytes.decode('iso-8859-2').encode('utf-8')

Note that the choice of encoding makes a difference - under iso-8859-1 the byte 0xA1 is encoded as u'\xc2\xa1' but under iso-8859-2 it's encoded as u'\xc4\x84'.

In Python 2

>>> plain_string = "Hi!"
>>> unicode_string = u"Hi!"
>>> type(plain_string), type(unicode_string)
(<type 'str'>, <type 'unicode'>)

^ This is the difference between a byte string (plain_string) and a unicode string.

>>> s = "Hello!"
>>> u = unicode(s, "utf-8")

^ Converting to unicode and specifying the encoding.

In Python 3

All strings are unicode. The unicode function does not exist anymore. See answer from @Noumenon

If utf-8 => bytestring conversion is what do you want then you may use str.encode, but first you need to properly mark the type of source string in your example - prefix with u for unicode:

# coding: utf-8
import random

    def random_utf8_seq(length):
        # from http://www.w3.org/2001/06/utf-8-test/postscript-utf-8.html
        test_charset = u" !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ ¡¢£¤¥¦§¨©ª«¬­ ®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿĂ㥹ĆćČčĎďĐđĘęĚěĹ弾ŁłŃńŇňŐőŒœŔŕŘřŚśŞşŠšŢţŤťŮůŰűŸŹźŻżŽžƒˆˇ˘˙˛˜˝–—‘’‚“”„†‡•…‰‹›€™"

        utf8_seq = u''

        for i in range(length):
            utf8_seq += random.choice(test_charset)

        print utf8_seq.encode('utf-8')
        return utf8_seq.encode('utf-8')

    print( type(random_utf8_seq(200)) )

-- output -- ­

õ3×sÔP{Ć.s(Ë°˙ě÷xÓ@bűV—û´ő¢uZÓČn˜0|_"Ðyø`êš·ÏÝhunÍÅ=ä?
óP{tlÇűpb¸7s´ňƒG—čøň\zčłŢXÂYqLĆúěă(ÿî ¥PyÐÔŇnל¦Ì˝+•ì›
ŻÛ°Ñ^ÝC÷ŢŐIñJĹţÒył­"MťÆ‹ČČ4þ!»šåŮ@Öhň-
ÈLGĄ¢ß˛Đ¯.ªÆź˘Ř^ĽÛŹËaĂŕ¹#¢éüÜńlÊqš=VřU…‚–MŽÎÉèoÙŹŠ¨Ð
<type 'str'>

You should specify encoding as argument to open that is

import codecs
with codecs.open("test.txt", encoding="utf-8") as reader:
    for line in reader:
        print(line)

>>> x = "\x61\x62\x63"
>>> x
'abc'
>>> bytes(x, 'utf-8')
b'abc'
>>> str(bytes(x, 'utf-8'))[2:-1]
'abc'

Decode the bytes object to produce a string:

>>> b"abcde".decode("utf-8")
'abcde'

The above example assumes that the bytes object is in UTF-8, because it is a common encoding. However, you should use the encoding your data is actually in!

Just some_string or str(some_string) will do in python2, since some_string is already an ascii string, and both of those actions convert it to type str. In python 3 the str type is the same as the unicode type in python 2.

Please read this answer, I think it answers your questions nicely.

In Python 2, str and bytes are the same type:

bytes is str True In Python 3, the str type is Python 2's unicode type, which is the default encoding of all strings.

In other words, bytes(some_string, 'UTF-8') in python 2 is just str(some_string), because str is a byte string.