The docs on pandas.DataFrame.replace says you have to provide a nested dictionary: the first level is the column name for which you have to provide a second dictionary with substitution pairs.

So, this should work:

>>> df=pd.DataFrame({'a': ['NÍCOLAS','asdč'], 'b': [3,4]})
>>> df
         a  b
0  NÍCOLAS  3
1     asdč  4

>>> df.replace({'a': {'č': 'c', 'Í': 'I'}}, regex=True)
         a  b
0  NICOLAS  3
1     asdc  4

Edit. Seems pandas also accepts non-nested translation dictionary. In that case, the problem is probably with character encoding, particularly if you use Python 2. Assuming your CSV load function decoded the file characters properly (as true Unicode code-points), then you should take care your translation/substitution dictionary is also defined with Unicode characters, like this:

dictionary = {u'í': 'i', u'á': 'a'}

If you have a definition like this (and using Python 2):

dictionary = {'í': 'i', 'á': 'a'}

then the actual keys in that dictionary are multibyte strings. Which bytes (characters) they are depends on the actual source file character encoding used, but presuming you use UTF-8, you'll get:

dictionary = {'\xc3\xa1': 'a', '\xc3\xad': 'i'}

And that would explain why pandas fails to replace those chars. So, be sure to use Unicode literals in Python 2: u'this is unicode string'.

On the other hand, in Python 3, all strings are Unicode strings, and you don't have to use the u prefix (in fact unicode type from Python 2 is renamed to str in Python 3, and the old str from Python 2 is now bytes in Python 3).

Answer from randomir on Stack Overflow
Top answer
1 of 4
19

The docs on pandas.DataFrame.replace says you have to provide a nested dictionary: the first level is the column name for which you have to provide a second dictionary with substitution pairs.

So, this should work:

>>> df=pd.DataFrame({'a': ['NÍCOLAS','asdč'], 'b': [3,4]})
>>> df
         a  b
0  NÍCOLAS  3
1     asdč  4

>>> df.replace({'a': {'č': 'c', 'Í': 'I'}}, regex=True)
         a  b
0  NICOLAS  3
1     asdc  4

Edit. Seems pandas also accepts non-nested translation dictionary. In that case, the problem is probably with character encoding, particularly if you use Python 2. Assuming your CSV load function decoded the file characters properly (as true Unicode code-points), then you should take care your translation/substitution dictionary is also defined with Unicode characters, like this:

dictionary = {u'í': 'i', u'á': 'a'}

If you have a definition like this (and using Python 2):

dictionary = {'í': 'i', 'á': 'a'}

then the actual keys in that dictionary are multibyte strings. Which bytes (characters) they are depends on the actual source file character encoding used, but presuming you use UTF-8, you'll get:

dictionary = {'\xc3\xa1': 'a', '\xc3\xad': 'i'}

And that would explain why pandas fails to replace those chars. So, be sure to use Unicode literals in Python 2: u'this is unicode string'.

On the other hand, in Python 3, all strings are Unicode strings, and you don't have to use the u prefix (in fact unicode type from Python 2 is renamed to str in Python 3, and the old str from Python 2 is now bytes in Python 3).

2 of 4
9

replace works out of the box without specifying a specific column in Python 3.

Load Data:

df=pd.read_csv('test.csv', sep=',', low_memory=False, encoding='iso8859_15')
df

Result:

col1    col2
0   he  hello
1   Nícolas shárk
2   welcome yes

Create Dictionary:

dictionary = {'í':'i', 'á':'a'}

Replace:

df.replace(dictionary, regex=True, inplace=True)

Result:

 col1   col2
0   he  hello
1   Nicolas shark
2   welcome yes
🌐
Pandas
pandas.pydata.org › pandas-docs › version › 0.17.0 › text.html
Working with Text Data — pandas 0.17.0 documentation
# Consider the following badly formatted financial data In [25]: dollars = pd.Series(['12', '-$10', '$10,000']) # This does what you'd naively expect: In [26]: dollars.str.replace('$', '') Out[26]: 0 12 1 -10 2 10,000 dtype: object # But this doesn't: In [27]: dollars.str.replace('-$', '-') Out[27]: 0 12 1 -$10 2 $10,000 dtype: object # We need to escape the special character (for >1 len patterns) In [28]: dollars.str.replace(r'-\$', '-') Out[28]: 0 12 1 -10 2 $10,000 dtype: object ·
Discussions

Python Pandas Replace Special Character - Stack Overflow
This question is about pandas. str.replace is a Series method. Although, I suspect you may be dead on with the alternative code point... ... Good point. Thanks for pointing that out; I hadn't realized. I assumed .str returned the string. ... Another update does not work. The replace value has to be like u'\xc9' ... You can use replace function with special character ... More on stackoverflow.com
🌐 stackoverflow.com
python - Replace special characters in pandas dataframe from a string of special characters - Stack Overflow
The columns contain some special characters (/ and @) that I need to replace with a blank space. More on stackoverflow.com
🌐 stackoverflow.com
How to efficiently remove a list of special characters from a pandas dataframe?
It's definitely faster. You can even pass in lists or dicts to control what changes to what. More on reddit.com
🌐 r/learnpython
4
9
September 29, 2023
python - Replacing string with special characters in Pandas column - Stack Overflow
I have a large pandas dataframe where one of the columns has weird formatting. I am tring to replace the string, but I keep getting error messages saying: 'error: unterminated character set at posi... More on stackoverflow.com
🌐 stackoverflow.com
🌐
GeeksforGeeks
geeksforgeeks.org › pandas-remove-special-characters-from-column-names
Pandas - Remove special characters from column names - GeeksforGeeks
September 5, 2020 - # import pandas import pandas as pd # create data frame Data = {'Name#': ['Mukul', 'Rohan', 'Mayank', 'Shubham', 'Aakash'], 'Location@' : ['Saharanpur', 'Meerut', 'Agra', 'Saharanpur', 'Meerut'], 'Pay&' : [25000,30000,35000,40000,45000]} df=pd.DataFrame(Data) # print original data frame print(df) # remove special character df.columns=df.columns.str.replace('[#,@,&]','') # print file after removing special character print("\n\n" , df)
🌐
Bobby Hadz
bobbyhadz.com › blog › pandas-remove-special-characters-from-column
Pandas: Remove special characters from Column Values/Names | bobbyhadz
April 12, 2024 - The method will replace all special characters with an empty string to remove them. ... Copied!import pandas as pd df = pd.DataFrame({ '$name$': ['Ali#ce', 'Bobby@', 'Ca$r%l', 'D^a&n'], '!experience@': [11, 14, 16, 18], '^salary*': [175.1, 180.2, 190.3, 210.4], }) df['$name$'] = df['$name$'].str.replace(r'\W', '', regex=True) # $name$ !experience@ ^salary* # 0 Alice 11 175.1 # 1 Bobby 14 180.2 # 2 Carl 16 190.3 # 3 Dan 18 210.4 print(df)
🌐
Statology
statology.org › home › pandas: how to remove special characters from column
Pandas: How to Remove Special Characters from Column
October 10, 2022 - Hi Ritesh…To remove the second occurrence of special characters from a column in a Pandas DataFrame, you can use regular expressions (regex) and apply it to the column of interest.
Top answer
1 of 2
8

I'm assuming you're using Python 2.x here and this is likely a Unicode problem. Don't worry, you're not alone--unicode is really tough in general and especially in Python 2, which is why it's been made standard in Python 3.

If all you're concerned about is the ñ, you should decode in UTF-8, and then just replace the one character.

That would look something like the following:

DF['name'] = DF['name'].str.decode('utf-8').replace(u'\xf1', 'n')

As an example:

>>> "sureño".decode("utf-8").replace(u"\xf1", "n")
u'sureno'

If your string is already Unicode, then you can (and actually have to) skip the decode step:

>>> u"sureño".replace(u"\xf1", "n")
u'sureno'

Note here that u'\xf1' uses the hex escape for the character in question.

Update

I was informed in the comments that <>.str.replace is a pandas series method, which I hadn't realized. The answer to this possibly might be something like the following:

DF['name'] = map(lambda x: x.decode('utf-8').replace(u'\xf1', 'n'), DF['name'].str)

or something along those lines, if that pandas object is iterable.

Another update

It actually just occurred to me that your issue may be as simple as the following:

DF['NAME']=DF['NAME'].str.replace(u"ñ","n")

Note how I've added the u in front of the string to make it unicode.

2 of 2
0

You can use replace function with special character to be replaced with a different value of your choice in the following way.

if your dataframe is df and you have to do it in all the columns that are string. in case of mine I am doing it for "\n"

df= df.applymap(lambda x: x.replace("\n"," "))
🌐
Saturn Cloud
saturncloud.io › blog › how-to-remove-special-characters-in-pandas-dataframe
How to Remove Special Characters in Pandas Dataframe | Saturn Cloud Blog
May 1, 2026 - This expression uses the re.sub() function from the regular expressions module to replace all characters that match the pattern r'[^\w\s]' with an empty string (''). This pattern matches any character that is not an alphanumeric character (\w) ...
Find elsewhere
🌐
GeeksforGeeks
geeksforgeeks.org › replace-characters-in-strings-in-pandas-dataframe
Replace Characters in Strings in Pandas DataFrame - GeeksforGeeks
December 29, 2022 - Example 1: The following program is to replace a character in strings for the entire dataframe. ... # import pandas import pandas as pd data = {'Student_Full_Name': ['Mukul_Jatav', 'Rahul_Shukla', 'Robin_Singh', 'Mayank_Sharma', 'Akash_Verma'], 'Father_Full_name': ['Mukesh_Jatav', 'Siddhart_Shukla', 'Rohit_Singh', 'Sunil_Sharma', 'Rajesh_Verma'] } # create an dataframe df = pd.DataFrame(data, columns=['Student_Full_Name', 'Father_Full_name']) # print dataframe print(" original dataframe \n", df) # replace '_' with '-' df = df.replace('_', '+', regex=True) # print dataframe print(" After replace character \n", df)
🌐
Reddit
reddit.com › r/learnpython › how to efficiently remove a list of special characters from a pandas dataframe?
r/learnpython on Reddit: How to efficiently remove a list of special characters from a pandas dataframe?
September 29, 2023 -

I have a program that loops through each column, and a nested loop that goes through the list of bad characters and converts each column to a str using astype() and then replace(). I also use re.escape() inside my replace method.

I noticed on large data frames of 100k or 500k rows it takes a long time.

I wonder if applying replace() on the entire dataframe is more reliable and faster? Like in the below example:

df = df.replace()

My concern is if replace() will guarantee removing the characters regardless of the data type in each column? I need assurance that the special characters are truly removed from the dataframe. My method right now is reliable but I’ve noticed it’s slow with large data frames so I would like someone with experience to help me refactor if possible.

🌐
Towards Data Science
towardsdatascience.com › home › latest › simplify your dataset cleaning with pandas
Simplify your Dataset Cleaning with Pandas | Towards Data Science
January 16, 2025 - If you want to be safe, you can use a complete list of special characters and remove them using a loop: ... spec_chars = ["!",'"',"#","%","&amp;","'","(",")", "*","+",",","-",".","/",":",";","<", "=",">","?","@","[","","]","^","_", "`","{","|","}","~","–"] ... Now you shouldn't have any of those characters in your title column. Because we replaced the special characters with a plus, we might end up with double whitespaces in some titles.
🌐
Sling Academy
slingacademy.com › article › pandas-remove-special-characters-and-whitespace-from-column-names
Pandas: Remove special characters and whitespace from column names - Sling Academy
This code cleans each column name by stripping leading and trailing spaces, replacing interior spaces with underscores, and removing special characters, all in a neat, reusable function. When dealing with larger datasets or more complex scenarios, you might want to automate the cleaning process further. One way is to leverage the power of list comprehensions combined with regex: import pandas as pd import re df = pd.DataFrame({"name with space": range(5), "@special*char#column": range(5)}) df.columns = [re.sub(r"[^\w\s]", "", col).replace(" ", "_") for col in df.columns] print(df.head())
🌐
CSDN
devpress.csdn.net › python › 63045ef9c67703293080bf99.html
Replacing special characters in pandas dataframe_python_Mangs-Python
August 23, 2022 - I have a few columns which contain names and places in Brazil, so some of them contain special characters such as "í" or "Ô". I have the key to replace them in a dictionary {'í':'i', 'á':'a', ...} I tried replacing it a couple of ways (below), but none of them worked. df.replace(dictionary, regex=True, inplace=True) ###BOTH WITH AND WITHOUT REGEX AND REPLACE ... None of them had the expected output, which would be for strings such as "NÍCOLAS" to become "NICOLAS". ... The docs on pandas.DataFrame.replace says you have to provide a nested dictionary: the first level is the column name for which you have to provide a second dictionary with substitution pairs.