As the other poster mentioned, you might try:

df = pd.read_csv('1459966468_324.csv', encoding='utf8')

However this could still leave you looking at 'object' when you print the dtypes. To confirm they are utf8, try this line after reading the CSV:

df.apply(lambda x: pd.lib.infer_dtype(x.values))

Example output:

args            unicode
date         datetime64
host            unicode
kwargs          unicode
operation       unicode

It depends on how you're outputting the data. If you're simply using csv files, which you then import to KDB, then you can specify that easily:

df.to_csv('df_output.csv', encoding='utf-8')

Or, you can set the encoding when you import the data to Pandas originally, using the same syntax.

If you're connecting directly to KDB using SQLAlchemy or something similar, you should try specifying this in the connection itself - see this question: Another UnicodeEncodeError when using pandas method to_sql with MySQL

If you're not using csv, and you want to encode your string index, this is what worked for me:

df.index = df.index.str.encode('utf-8')

Df.x.str.encode('utf-8')

Will fix your problems.

http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.str.encode.html

Your "bad" output is UTF-8 displayed as CP1252.

On Windows, many editors assume the default ANSI encoding (CP1252 on US Windows) instead of UTF-8 if there is no byte order mark (BOM) character at the start of the file. While a BOM is meaningless to the UTF-8 encoding, its UTF-8-encoded presence serves as a signature for some programs. For example, Microsoft Office's Excel requires it even on non-Windows OSes. Try:

df.to_csv('file.csv',encoding='utf-8-sig')

That encoder will add the BOM.

It works:

data['name'] = data['name'].apply(
    lambda x:x[2:-1].encode().decode("unicode_escape").encode('raw_unicode_escape').decode()
)

Step by step

In:

x = r"b'\xd9\x85\xd9\x86\xd8\xaa\xd8\xb2\xd9\x87\xd8\xb1\xd8\xa7\xd8\xa8'"
print(f"x {type(x)}\n\t= {x}\n")

x = x[2:-1]
print(f"x[2:-1] {type(x)}\n\t= {x}\n")

x = x.encode()
print(f"x[2:-1].encode() {type(x)}\n\t= {x}\n")

x = x.decode("unicode_escape").encode('raw_unicode_escape')
print(f"x[2:-1].encode().decode('unicode_escape').encode('raw_unicode_escape') {type(x)}\n\t= {x}\n")

x = x.decode()
print(f"x[2:-1].encode().decode('unicode_escape').encode('raw_unicode_escape').decode() {type(x)}\n\t= {x}\n")

Out:

x <class 'str'>
    = b'\xd9\x85\xd9\x86\xd8\xaa\xd8\xb2\xd9\x87\xd8\xb1\xd8\xa7\xd8\xa8'

x[2:-1] <class 'str'>
    = \xd9\x85\xd9\x86\xd8\xaa\xd8\xb2\xd9\x87\xd8\xb1\xd8\xa7\xd8\xa8

x[2:-1].encode() <class 'bytes'>
    = b'\\xd9\\x85\\xd9\\x86\\xd8\\xaa\\xd8\\xb2\\xd9\\x87\\xd8\\xb1\\xd8\\xa7\\xd8\\xa8'

x[2:-1].encode().decode('unicode_escape').encode('raw_unicode_escape') <class 'bytes'>
    = b'\xd9\x85\xd9\x86\xd8\xaa\xd8\xb2\xd9\x87\xd8\xb1\xd8\xa7\xd8\xa8'

x[2:-1].encode().decode('unicode_escape').encode('raw_unicode_escape').decode() <class 'str'>
    = منتزهراب