It's definitely faster. You can even pass in lists or dicts to control what changes to what. Answer from RhinoRhys on reddit.com
🌐
Pandas
pandas.pydata.org › pandas-docs › version › 0.17.0 › text.html
Working with Text Data — pandas 0.17.0 documentation
# Consider the following badly formatted financial data In [25]: dollars = pd.Series(['12', '-$10', '$10,000']) # This does what you'd naively expect: In [26]: dollars.str.replace('$', '') Out[26]: 0 12 1 -10 2 10,000 dtype: object # But this doesn't: In [27]: dollars.str.replace('-$', '-') Out[27]: 0 12 1 -$10 2 $10,000 dtype: object # We need to escape the special character (for >1 len patterns) In [28]: dollars.str.replace(r'-\$', '-') Out[28]: 0 12 1 -10 2 $10,000 dtype: object ·
🌐
Reddit
reddit.com › r/learnpython › how to efficiently remove a list of special characters from a pandas dataframe?
r/learnpython on Reddit: How to efficiently remove a list of special characters from a pandas dataframe?
September 29, 2023 -

I have a program that loops through each column, and a nested loop that goes through the list of bad characters and converts each column to a str using astype() and then replace(). I also use re.escape() inside my replace method.

I noticed on large data frames of 100k or 500k rows it takes a long time.

I wonder if applying replace() on the entire dataframe is more reliable and faster? Like in the below example:

df = df.replace()

My concern is if replace() will guarantee removing the characters regardless of the data type in each column? I need assurance that the special characters are truly removed from the dataframe. My method right now is reliable but I’ve noticed it’s slow with large data frames so I would like someone with experience to help me refactor if possible.

Discussions

Python pandas doesn't recognize special characters - Stack Overflow
Regarding the semantic difference between count on a normal string and on a Series, count on a python str just does a character count, but str.count takes a regex pattern. The ^ and + are special characters which need to be escaped with a backslash if you are searching for those characters More on stackoverflow.com
🌐 stackoverflow.com
Escape special characters in Dataframe Pandas Python - Stack Overflow
Input: a dataframe with special characters: "’" in some of the values. Output: search for this string "’" and replace it with "'" Example: I execute this code but it More on stackoverflow.com
🌐 stackoverflow.com
python - Escaped quotes in pandas read_csv - Stack Overflow
Communities for your favorite technologies. Explore all Collectives · Stack Overflow for Teams is now called Stack Internal. Bring the best of human thought and AI automation together at your work More on stackoverflow.com
🌐 stackoverflow.com
Saving CSV with backslashed-escaping is not idempotent.
@pdbaines and I noticed this bug. I want Pandas to write a CSV file so that all field data is backslash escaped if the character has a special interpretation (e.g. quotes or backslashes themselves). If a quote is backslashed, it is treat... More on github.com
🌐 github.com
19
August 31, 2016
🌐
Like Geeks
likegeeks.com › home › python › pandas › handling columns with special characters in pandas query
Handling Columns with Special Characters in Pandas query
December 3, 2023 - To escape a special character within a column name, use double backticks (` ` ) to enclose it. This tells Pandas to treat the special character as a part of the column name.
🌐
GeeksforGeeks
geeksforgeeks.org › pandas-remove-special-characters-from-column-names
Pandas - Remove special characters from column names - GeeksforGeeks
September 5, 2020 - Now we will use a list with replace function for removing multiple special characters from our column names.
Find elsewhere
🌐
GitHub
github.com › pandas-dev › pandas › issues › 14122
Saving CSV with backslashed-escaping is not idempotent. · Issue #14122 · pandas-dev/pandas
August 31, 2016 - df2 = pd.read_csv("out.csv", quoting=csv.QUOTE_NONNUMERIC, encoding="utf-8", escapechar='\\', doublequote=False)
Author   pandas-dev
🌐
Statology
statology.org › home › pandas: how to remove special characters from column
Pandas: How to Remove Special Characters from Column
October 10, 2022 - Hi Ritesh…To remove the second occurrence of special characters from a column in a Pandas DataFrame, you can use regular expressions (regex) and apply it to the column of interest.
🌐
GitHub
github.com › pandas-dev › pandas › issues › 18516
Strange matching behaviour when using str.contains with escape characters · Issue #18516 · pandas-dev/pandas
November 27, 2017 - Minimal example of a dataframe with a string with an escaped backslash. import io import pandas as pd s1 = """X "\\x2f" """ df = pd.read_csv(io.StringIO(s1)) Printing out the dataframe shows one backslash as expected: Out[13]: X 0 \x2f T...
Author   pandas-dev
🌐
Linux find Examples
queirozf.com › entries › pandas-dataframes-csv-quoting-and-escaping-strategies
Pandas Dataframes: CSV Quoting and Escaping Strategies
April 12, 2020 - import csv import pandas as pd df = # build dataframe here df.to_csv( "/path/to/output/file.csv", quoting=csv.QUOTE_NONNUMERIC, escapechar="\\", doublequote=False, index=False)
🌐
Bobby Hadz
bobbyhadz.com › blog › pandas-remove-special-characters-from-column
Pandas: Remove special characters from Column Values/Names | bobbyhadz
April 12, 2024 - The \s character matches Unicode whitespace characters like [ \t\n\r\f\v]. All spaces in the column values are kept in the result. I've also written a detailed guide on how to remove special characters except spaces in Python.
Top answer
1 of 4
19

The docs on pandas.DataFrame.replace says you have to provide a nested dictionary: the first level is the column name for which you have to provide a second dictionary with substitution pairs.

So, this should work:

>>> df=pd.DataFrame({'a': ['NÍCOLAS','asdč'], 'b': [3,4]})
>>> df
         a  b
0  NÍCOLAS  3
1     asdč  4

>>> df.replace({'a': {'č': 'c', 'Í': 'I'}}, regex=True)
         a  b
0  NICOLAS  3
1     asdc  4

Edit. Seems pandas also accepts non-nested translation dictionary. In that case, the problem is probably with character encoding, particularly if you use Python 2. Assuming your CSV load function decoded the file characters properly (as true Unicode code-points), then you should take care your translation/substitution dictionary is also defined with Unicode characters, like this:

dictionary = {u'í': 'i', u'á': 'a'}

If you have a definition like this (and using Python 2):

dictionary = {'í': 'i', 'á': 'a'}

then the actual keys in that dictionary are multibyte strings. Which bytes (characters) they are depends on the actual source file character encoding used, but presuming you use UTF-8, you'll get:

dictionary = {'\xc3\xa1': 'a', '\xc3\xad': 'i'}

And that would explain why pandas fails to replace those chars. So, be sure to use Unicode literals in Python 2: u'this is unicode string'.

On the other hand, in Python 3, all strings are Unicode strings, and you don't have to use the u prefix (in fact unicode type from Python 2 is renamed to str in Python 3, and the old str from Python 2 is now bytes in Python 3).

2 of 4
9

replace works out of the box without specifying a specific column in Python 3.

Load Data:

df=pd.read_csv('test.csv', sep=',', low_memory=False, encoding='iso8859_15')
df

Result:

col1    col2
0   he  hello
1   Nícolas shárk
2   welcome yes

Create Dictionary:

dictionary = {'í':'i', 'á':'a'}

Replace:

df.replace(dictionary, regex=True, inplace=True)

Result:

 col1   col2
0   he  hello
1   Nicolas shark
2   welcome yes
🌐
Towards Data Science
towardsdatascience.com › home › latest › simplify your dataset cleaning with pandas
Simplify your Dataset Cleaning with Pandas | Towards Data Science
January 16, 2025 - We’re done with this column, we removed the special characters. Note that I didn’t include the currencies characters and the dot "." in the special characters list above. The reason is that some results titles contain the price of the flights tickets they are selling (e.g.
🌐
W3Schools
w3schools.com › python › gloss_python_escape_characters.asp
Python Escape Characters
An escape character is a backslash \ followed by the character you want to insert.