Your "bad" output is UTF-8 displayed as CP1252.
On Windows, many editors assume the default ANSI encoding (CP1252 on US Windows) instead of UTF-8 if there is no byte order mark (BOM) character at the start of the file. While a BOM is meaningless to the UTF-8 encoding, its UTF-8-encoded presence serves as a signature for some programs. For example, Microsoft Office's Excel requires it even on non-Windows OSes. Try:
df.to_csv('file.csv',encoding='utf-8-sig')
That encoder will add the BOM.
Answer from Mark Tolonen on Stack Overflow Top answer 1 of 2
387
Your "bad" output is UTF-8 displayed as CP1252.
On Windows, many editors assume the default ANSI encoding (CP1252 on US Windows) instead of UTF-8 if there is no byte order mark (BOM) character at the start of the file. While a BOM is meaningless to the UTF-8 encoding, its UTF-8-encoded presence serves as a signature for some programs. For example, Microsoft Office's Excel requires it even on non-Windows OSes. Try:
df.to_csv('file.csv',encoding='utf-8-sig')
That encoder will add the BOM.
2 of 2
2
encoding='utf-8-sig does not work for me. Excel reads the special characters fine now, but the Tab separators are gone! However, encoding='utf-16 does work correctly: special characters OK and Tab separators work. This is the solution for me.
Reddit
reddit.com › r/learnpython › pandas dataframe.to_csv() struggling with encoding
r/learnpython on Reddit: Pandas DataFrame.to_csv() struggling with encoding
January 14, 2021 -
Heya,
I got a DataFrame filled with strings in the first column and the rest consisting of integers (except for the headers).
Now when I export this dataframe to a csv file, and the strings contain German Umlauts (ä,ö,ü or something like ß), the exported csv file has weird looking strings at these indices.
Like "für" became "für".
As far as I know the default encoding for to_csv() is utf-8, which means it should work fine? But I also tried the parameter encoding='utf-8', same results.
What am I doing wrong here?
Top answer 1 of 2
1
How are you actually opening the generated CSV file?
2 of 2
1
Sounds like you're using an application like Excel to view it that doesn't default to UTF-8. Check https://superuser.com/questions/911369/excel-change-default-encoding-file-origin-of-text-import-wizard-to-utf-8-650 for various options here. Although my personal workaround is to use LibreOffice Calc.
to_csv() enconding option "utf-8-BOM"
Hey, could there be a way to save csvs as "utf-8-BOM" encoded? Because Excel needs the BOM to open csvs correctly. Everytime I save a csv with Pandas I have to open it with Notepad++ and ... More on github.com
3
November 5, 2021Python 3 writing to_csv file ignores encoding argument.
# is missing the UTF8 BOM (encoded with default encoding UTF8) with open('path_to_f', 'w') as f: df.to_csv(f, encoding='utf-8-sig') # is not missing the UTF8 BOM (encoded wi... More on github.com
32
May 3, 2016Trying to Convert Text File to CSV using Python Pandas but having 'utf-8' codec problems
That means your "Output Configured 3.txt" file was saved with a different encoding, not the default UTF8. You need to tell python what encoding was used. account = pd.read_csv("Output Configured 3.txt", encoding='cp1252', delimiter = '/') If you don't know what encoding was used to save the file, you can try a couple common ones, like latin-1, utf16, cp1251, cp1252, big5. https://docs.python.org/3/library/codecs.html#standard-encodings Or you can tell python to just ignore or replace any trouble spots. account = pd.read_csv("Output Configured 3.txt", encoding_errors='ignore', delimiter = '/') https://docs.python.org/3/library/codecs.html#error-handlers More on reddit.com
2
0
July 7, 2022Bug: On Python 3 to_csv() encoding defaults to ascii if the dataframe contains special characters.
I can only read test2.csv when I explicitely state encoding='ansi', so read_csv() definitely expects utf-8. More on github.com
11
July 27, 2017Sentry
sentry.io › sentry answers › python › write a python pandas dataframe to a csv file
Write DataFrame to CSV in Python Pandas With to_csv | Sentry
July 3, 2026 - import pandas as pd data = { "Name": ["Alice", "Bob", "André"], "Age": [30, 25, 35], "City": ["New York", "Paris", "London"], } df = pd.DataFrame(data) df.to_csv("people.csv", index=False, encoding="utf-8", sep=";")
Saturn Cloud
saturncloud.io › blog › a-list-of-pandas-readcsv-encoding-options
A List of Pandas readcsv Encoding Options | Saturn Cloud Blog
May 1, 2026 - It supports all characters in the Unicode standard and is compatible with ASCII. To specify UTF-8 encoding in Pandas, use the encoding='utf-8' parameter. import pandas as pd df = pd.read_csv('file.csv', encoding='utf-8')
Net Informations
net-informations.com › ds › pd › tocsv.htm
How to Export Pandas DataFrame to a CSV File
The default encoding is usually 'utf-8', but you can explicitly set it using the encoding parameter. df.to_csv('D:\panda.csv',sep='\t',encoding='utf-8')
GitHub
github.com › pandas-dev › pandas › issues › 44323
to_csv() enconding option "utf-8-BOM" · Issue #44323 · pandas-dev/pandas
November 5, 2021 - Because Excel needs the BOM to open csvs correctly. Everytime I save a csv with Pandas I have to open it with Notepad++ and change the encoding from utf-8 to uf8-BOM so that I can open it with Excel.
Author: pandas-dev
Medium
medium.com › @anala007 › dealing-with-the-unicodedecodeerror-in-pandas-when-reading-csv-files-edc4987bf68b
Dealing with the UnicodeDecodeError in Pandas When Reading CSV Files | by Arun | Medium
June 12, 2023 - import pandas as pd df = pd.read_csv('file.csv', encoding='utf-8', errors='replace')
Pandas
pandas.pydata.org › docs › reference › api › pandas.DataFrame.to_csv.html
pandas.DataFrame.to_csv — pandas 3.0.6 documentation
A string representing the encoding to use in the output file, defaults to ‘utf-8’. encoding is not supported if path_or_buf is a non-binary file object.
Varunpramanik
varunpramanik.com › chronicles › 2019 › 12 › 12 › pandas-to_csv-encoding-error-solution
Pandas to_csv Encoding Error Solution – Chronicles by Varun Pramanik
December 12, 2019 - new_df = original_df.applymap(lambda x: str(x).encode("utf-8", errors="ignore").decode("utf-8", errors="ignore")) I entirely expect this approach is imperfect and non-optimal, but it works. I’d be happy to hear suggestions. Relevant reading: pandas.DataFrame.applymap ·
Plain English
python.plainenglish.io › easy-ways-to-handle-unicodedecodeerrors-when-reading-csv-files-in-pandas-84c7aad1d5ac
Easy Ways To Handle UnicodeDecodeErrors When Reading CSV Files in Pandas | by Gabriel Ejiro | Python in Plain English
November 10, 2023 - # Latin1 import pandas as pd filepath = r"C:\path\to\your\file.csv" df = pd.read_csv(filepath, encoding='latin1') # utf-8 import pandas as pd filepath = r"C:\path\to\your\file.csv" df = pd.read_csv(filepath, encoding='ISO-8859-1')
GitHub
github.com › pandas-dev › pandas › issues › 13068
Python 3 writing to_csv file ignores encoding argument. · Issue #13068 · pandas-dev/pandas
May 3, 2016 - with open('path_to_f', 'wb') as f: df.to_csv(f, encoding='utf-8-sig') To write the file correctly. #!/usr/bin/env python3 import pandas as pd df = pd.DataFrame() with open('file_one', 'w') as f: df.to_csv(f, encoding='utf-8-sig') assert open('file_one', 'rb').read() == b'""\n' # is not missing the UTF8 BOM (encoded with passed encoding utf-8-sig) df.to_csv('file_two', encoding='utf-8-sig') assert open('file_two', 'rb').read() == b'\xef\xbb\xbf""\n' Reactions are currently unavailable ·
Author: pandas-dev
DEV Community
dev.to › _aadidev › 3-ways-to-handle-non-utf-8-characters-in-pandas-242
3 Ways to Handle non UTF-8 Characters in Pandas - DEV Community
January 20, 2022 - Pandas, by default, assumes utf-8 encoding every time you do pandas.read_csv, and it can feel like staring into a crystal ball trying to figure out the correct encoding.
Reddit
reddit.com › r/learnpython › trying to convert text file to csv using python pandas but having 'utf-8' codec problems
r/learnpython on Reddit: Trying to Convert Text File to CSV using Python Pandas but having 'utf-8' codec problems
July 7, 2022 -
I get the error 'utf-8' codec can't decode byte 0xe9 in position 5070: invalid continuation byte. Nothing exports.
my code in question
# importing pandas library
import pandas as pd
# reading the given csv file
# and creating dataframe
account = pd.read_csv("Output Configured 3.txt",
delimiter = '/')
# store dataframe into csv file
account.to_csv('Output Configured 3.csv',
index = None)
What do I need to do to have it actually implement?GitHub
github.com › pandas-dev › pandas › issues › 17097
Bug: On Python 3 to_csv() encoding defaults to ascii if the dataframe contains special characters. · Issue #17097 · pandas-dev/pandas
July 27, 2017 - The problem is only solved by explicitely specifying encoding='utf8' in to_csv().
Author: pandas-dev
GitHub
github.com › ChrisMuir › Zillow › issues › 12
Specify encoding in pandas.DataFrame.to_csv() · Issue #12 · ChrisMuir/Zillow
March 15, 2018 - Per the pandas docs, the default ... this edit would mean that anyone using Python 2 will need to explicitly specify encoding = "UTF-8" when reading any of the csv files back into Python....
Author: ChrisMuir
YouTube
youtube.com › how to fix your computer
PYTHON : Pandas df.to_csv("file.csv" encode="utf-8") still gives trash characters for minus sign - YouTube
PYTHON : Pandas df.to_csv("file.csv" encode="utf-8") still gives trash characters for minus sign [ Gift : Animated Search Engine : https://www.hows.tech/p/re...
Published: December 5, 2021
Views: 981
Top answer 1 of 5
46
As the other poster mentioned, you might try:
df = pd.read_csv('1459966468_324.csv', encoding='utf8')
However this could still leave you looking at 'object' when you print the dtypes. To confirm they are utf8, try this line after reading the CSV:
df.apply(lambda x: pd.lib.infer_dtype(x.values))
Example output:
args unicode
date datetime64
host unicode
kwargs unicode
operation unicode
2 of 5
5
Use the encoding keyword with the appropriate parameter:
df = pd.read_csv('1459966468_324.csv', encoding='utf8')
Kanaries
docs.kanaries.net › topics › Pandas › pandas-dataframe-to-csv
Pandas DataFrame to CSV: Complete Guide to to_csv() – Kanaries
February 10, 2026 - DataFrame.to_csv() is the standard way to export Pandas data to CSV files. For clean exports, always pass index=False. Use encoding='utf-8-sig' when Excel compatibility matters.
Quora
quora.com › How-do-I-fix-a-Unicode-error-while-reading-a-CSV-file-with-a-pandas-library-in-Python-3-6
How to fix a Unicode error while reading a CSV file with a pandas library in Python 3.6 - Quora
Answer (1 of 6): import pandas as pd dataset=pd.read_csv(“Your_filename.csv”, encoding=”ISO-8859–1”) This will solve the UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 16: invalid start byte Happy Learning !!!
GitHub
github.com › pandas-dev › pandas › issues › 23854
df.to_csv() ignores encoding when given a file object or any other filelike object. · Issue #23854 · pandas-dev/pandas
November 22, 2018 - DocsIO CSVread_csv, to_csvread_csv, to_csv · Milestone · 1.2 · eode · opened · on Nov 22, 2018 · Issue body actions · import pandas as pd import io # !! NOTE # This example uses `io.BytesIO`, however this also applies to file buffers that are # returned by `io.open` (the `open` function) when opened in binary mode. buf = io.BytesIO('a, b, 🐟\n1, 2, 3\n4, 5, 6'.encode('utf-8')) df = pd.read_csv(buf) # reads in fine using default encoding (utf-8) buf = io.BytesIO() df.to_csv(buf, encoding='utf-8') # this should work, but doesn't.
Author: pandas-dev