The docs on pandas.DataFrame.replace says you have to provide a nested dictionary: the first level is the column name for which you have to provide a second dictionary with substitution pairs.

So, this should work:

>>> df=pd.DataFrame({'a': ['NÍCOLAS','asdč'], 'b': [3,4]})
>>> df
         a  b
0  NÍCOLAS  3
1     asdč  4

>>> df.replace({'a': {'č': 'c', 'Í': 'I'}}, regex=True)
         a  b
0  NICOLAS  3
1     asdc  4

Edit. Seems pandas also accepts non-nested translation dictionary. In that case, the problem is probably with character encoding, particularly if you use Python 2. Assuming your CSV load function decoded the file characters properly (as true Unicode code-points), then you should take care your translation/substitution dictionary is also defined with Unicode characters, like this:

dictionary = {u'í': 'i', u'á': 'a'}

If you have a definition like this (and using Python 2):

dictionary = {'í': 'i', 'á': 'a'}

then the actual keys in that dictionary are multibyte strings. Which bytes (characters) they are depends on the actual source file character encoding used, but presuming you use UTF-8, you'll get:

dictionary = {'\xc3\xa1': 'a', '\xc3\xad': 'i'}

And that would explain why pandas fails to replace those chars. So, be sure to use Unicode literals in Python 2: u'this is unicode string'.

On the other hand, in Python 3, all strings are Unicode strings, and you don't have to use the u prefix (in fact unicode type from Python 2 is renamed to str in Python 3, and the old str from Python 2 is now bytes in Python 3).

Answer from randomir on Stack Overflow
Top answer
1 of 4
19

The docs on pandas.DataFrame.replace says you have to provide a nested dictionary: the first level is the column name for which you have to provide a second dictionary with substitution pairs.

So, this should work:

>>> df=pd.DataFrame({'a': ['NÍCOLAS','asdč'], 'b': [3,4]})
>>> df
         a  b
0  NÍCOLAS  3
1     asdč  4

>>> df.replace({'a': {'č': 'c', 'Í': 'I'}}, regex=True)
         a  b
0  NICOLAS  3
1     asdc  4

Edit. Seems pandas also accepts non-nested translation dictionary. In that case, the problem is probably with character encoding, particularly if you use Python 2. Assuming your CSV load function decoded the file characters properly (as true Unicode code-points), then you should take care your translation/substitution dictionary is also defined with Unicode characters, like this:

dictionary = {u'í': 'i', u'á': 'a'}

If you have a definition like this (and using Python 2):

dictionary = {'í': 'i', 'á': 'a'}

then the actual keys in that dictionary are multibyte strings. Which bytes (characters) they are depends on the actual source file character encoding used, but presuming you use UTF-8, you'll get:

dictionary = {'\xc3\xa1': 'a', '\xc3\xad': 'i'}

And that would explain why pandas fails to replace those chars. So, be sure to use Unicode literals in Python 2: u'this is unicode string'.

On the other hand, in Python 3, all strings are Unicode strings, and you don't have to use the u prefix (in fact unicode type from Python 2 is renamed to str in Python 3, and the old str from Python 2 is now bytes in Python 3).

2 of 4
9

replace works out of the box without specifying a specific column in Python 3.

Load Data:

df=pd.read_csv('test.csv', sep=',', low_memory=False, encoding='iso8859_15')
df

Result:

col1    col2
0   he  hello
1   Nícolas shárk
2   welcome yes

Create Dictionary:

dictionary = {'í':'i', 'á':'a'}

Replace:

df.replace(dictionary, regex=True, inplace=True)

Result:

 col1   col2
0   he  hello
1   Nicolas shark
2   welcome yes
🌐
Reddit
reddit.com › r/learnpython › trying to remove special characters and suffixes from dataframe
r/learnpython on Reddit: Trying to remove special characters and suffixes from dataframe
March 4, 2023 -

I'm trying to strip all columns in my dataframe of all special characters and name suffixes like Jr and II.

The replacements I am attempting to loop through aren't working. What am I doing wrong here?

import pandas as pd

dffg = pd.read_csv("fg2.csv")
dfstuff = pd.read_csv("stuffplus.csv")
dfadp = pd.read_csv("adp.csv")
dfzpit = pd.read_csv("ZPitchers.csv")
dfhpit = pd.read_csv("ZHitters.csv")

dflist = [dfhpit, dfzpit, dffg, dfadp, dfstuff]

for index in range(len(dflist)):
    dflist[index] = dflist[index].replace(r'[^\w\s]|_\*', '', regex=True).replace(' Jr', '').replace(' II', '')

func = lambda x: ''.join([i[:3] for i in x.strip().split(' ')])
dffg['Key'] = dffg.Name.apply(func)
dfstuff['Key'] = dfstuff.player_name.apply(func)
dfadp['Key'] = dfadp.Player.apply(func)
dfzpit['Key'] = dfzpit.Name.apply(func)
dfhpit['Key'] = dfhpit.Name.apply(func)

dffg.columns = dffg.columns.str.strip()
dfstuff.columns = dfstuff.columns.str.strip()
dfadp.columns = dfadp.columns.str.strip()

dfadp = dfadp.drop(['ESPN','CBS','RTS','NFBC','FT'], axis=1)

df1 = dfadp.merge(dffg, on=["Key"], how="left").merge(dfstuff[['Key', 'STUFFplus', 'LOCATIONplus', 'PITCHINGplus']], on=["Key"], how="left").merge(dfzpit[['Key', 'Total Z-Score']], on=["Key"], how="left").merge(dfhpit[['Key', 'Total Z-Score']], on=["Key"], how="left")
df1 = df1.drop(['Team_y', 'playerid','Name'], axis=1)
df1['Rank'] = df1['Rank'].astype(float)
df1 = df1.fillna('')
df2 = df1.sort_values(by=['Rank'])

df2.to_csv('output.csv')
🌐
Pythonhelpdesk
pythonhelpdesk.com › 2024 › 02 › 25 › data-frame-replace-special-characters-using-str-replace
Data Frame: Replace Special Characters Using `str.replace()` – Python HelpDesk
Create a sample DataFrame with text columns containing special characters. Utilize .str.replace() with a regex pattern (r'[@#$]’) to remove occurrences of ‘@’, ‘#’, and ‘$’.
🌐
Reddit
reddit.com › r/learnpython › how to efficiently remove a list of special characters from a pandas dataframe?
r/learnpython on Reddit: How to efficiently remove a list of special characters from a pandas dataframe?
September 29, 2023 -

I have a program that loops through each column, and a nested loop that goes through the list of bad characters and converts each column to a str using astype() and then replace(). I also use re.escape() inside my replace method.

I noticed on large data frames of 100k or 500k rows it takes a long time.

I wonder if applying replace() on the entire dataframe is more reliable and faster? Like in the below example:

df = df.replace()

My concern is if replace() will guarantee removing the characters regardless of the data type in each column? I need assurance that the special characters are truly removed from the dataframe. My method right now is reliable but I’ve noticed it’s slow with large data frames so I would like someone with experience to help me refactor if possible.

🌐
CSDN
devpress.csdn.net › python › 63045ef9c67703293080bf99.html
Replacing special characters in pandas dataframe_python_Mangs-Python
August 23, 2022 - >>> df=pd.DataFrame({'a': ['NÍCOLAS','asdč'], 'b': [3,4]}) >>> df a b 0 NÍCOLAS 3 1 asdč 4 >>> df.replace({'a': {'č': 'c', 'Í': 'I'}}, regex=True) a b 0 NICOLAS 3 1 asdc 4 · Edit. Seems pandas also accepts non-nested translation dictionary. In that case, the problem is probably with character encoding, particularly if you use Python 2.
Find elsewhere
Top answer
1 of 2
8

I'm assuming you're using Python 2.x here and this is likely a Unicode problem. Don't worry, you're not alone--unicode is really tough in general and especially in Python 2, which is why it's been made standard in Python 3.

If all you're concerned about is the ñ, you should decode in UTF-8, and then just replace the one character.

That would look something like the following:

DF['name'] = DF['name'].str.decode('utf-8').replace(u'\xf1', 'n')

As an example:

>>> "sureño".decode("utf-8").replace(u"\xf1", "n")
u'sureno'

If your string is already Unicode, then you can (and actually have to) skip the decode step:

>>> u"sureño".replace(u"\xf1", "n")
u'sureno'

Note here that u'\xf1' uses the hex escape for the character in question.

Update

I was informed in the comments that <>.str.replace is a pandas series method, which I hadn't realized. The answer to this possibly might be something like the following:

DF['name'] = map(lambda x: x.decode('utf-8').replace(u'\xf1', 'n'), DF['name'].str)

or something along those lines, if that pandas object is iterable.

Another update

It actually just occurred to me that your issue may be as simple as the following:

DF['NAME']=DF['NAME'].str.replace(u"ñ","n")

Note how I've added the u in front of the string to make it unicode.

2 of 2
0

You can use replace function with special character to be replaced with a different value of your choice in the following way.

if your dataframe is df and you have to do it in all the columns that are string. in case of mine I am doing it for "\n"

df= df.applymap(lambda x: x.replace("\n"," "))
🌐
Erikrood
erikrood.com › Python_References › find_replace_col.html
Finding and replacing characters in Pandas columns
raw_data = {'name': ['Willard Morris', 'Al Jennings', 'Omar Mullins', 'Spencer McDaniel'], 'age': [20, 19, 22, 21], 'favorite_color': ['blue', 'red', 'yellow', "green"], 'grade': [88, 92, 95, 70]} df = pd.DataFrame(raw_data, index = ['Willard Morris', 'Al Jennings', 'Omar Mullins', 'Spencer McDaniel']) df · df.columns = [x.strip().replace('_', '_TEST_') for x in df.columns] df.head()
🌐
Quora
quora.com › How-do-you-remove-special-characters-from-a-data-frame-in-Python
How to remove special characters from a data frame in Python - Quora
Answer (1 of 2): You can use a regex expression or a package (helpful for emojis or unknown special characters). If you’re using NLTK, some tokenizers and lemmatizers will remove those characters automatically.
🌐
Pandas
pandas.pydata.org › pandas-docs › version › 0.17.0 › text.html
Working with Text Data — pandas 0.17.0 documentation
The string methods on Index are especially useful for cleaning up or transforming DataFrame columns.
🌐
Bobby Hadz
bobbyhadz.com › blog › pandas-remove-special-characters-from-column
Pandas: Remove special characters from Column Values/Names | bobbyhadz
April 12, 2024 - The method will replace all special characters with an empty string to remove them. ... Copied!import pandas as pd df = pd.DataFrame({ '$name$': ['Ali#ce', 'Bobby@', 'Ca$r%l', 'D^a&n'], '!experience@': [11, 14, 16, 18], '^salary*': [175.1, 180.2, 190.3, 210.4], }) df['$name$'] = df['$name$'].str.replace(r'\W', '', regex=True) # $name$ !experience@ ^salary* # 0 Alice 11 175.1 # 1 Bobby 14 180.2 # 2 Carl 16 190.3 # 3 Dan 18 210.4 print(df)
🌐
Towards Data Science
towardsdatascience.com › home › latest › simplify your dataset cleaning with pandas
Simplify your Dataset Cleaning with Pandas | Towards Data Science
January 16, 2025 - That’s up to you, but be consistent in your script, if you need to join two DataFrames, based on an Id column, for instance, convert them first in the same format (string or integer but you have to choose one). Let’s go back to our dataset. In the second section, we created a new column containing the prices in dollars. At this point, we know there are in dollars, right? Because this information about the currency is in the column name. That being said, we can get rid of the dollar characters:
🌐
YouTube
youtube.com › watch
How to Replace Special Symbols in a Dataframe Column with Pandas - YouTube
Learn how to efficiently replace special symbols in a pandas DataFrame column in Python, transforming your data as needed.---This video is based on the quest...
Published   March 30, 2025
Views   1
🌐
Stack Exchange
datascience.stackexchange.com › questions › 102803 › cleaning-rows-of-special-characters-and-creating-dataframe-columns
python - Cleaning rows of special characters and creating dataframe columns - Data Science Stack Exchange
October 5, 2021 - Below is my Dataframe format consisting of 2 columns (one is index and other is the data to be cleaned) 0 ans({'A,B,C,Bad_QoS,Sort'}) 1 ans({'A,B,D,QoS_Miss1,Sort'}) I want to remove special characters · create a data frame for all comma separated items. I have managed to first remove ans from all rows using: ds_[col2] = ds_.replace('ans', '', regex=True) > 0 ({'A,B,C,Bad_QoS,Sort'}) > 1 ({'A,B,D,QoS_Miss1,Sort'}) Then I try to apply replace regex, see below: ds_['col2'] = ds_['col2'].str.replace( r' \(\{\' | \'\}\) ', '', regex=True) ds_['col2'] I get no errors, but no changes.