The docs on pandas.DataFrame.replace says you have to provide a nested dictionary: the first level is the column name for which you have to provide a second dictionary with substitution pairs.
So, this should work:
>>> df=pd.DataFrame({'a': ['NÍCOLAS','asdč'], 'b': [3,4]})
>>> df
a b
0 NÍCOLAS 3
1 asdč 4
>>> df.replace({'a': {'č': 'c', 'Í': 'I'}}, regex=True)
a b
0 NICOLAS 3
1 asdc 4
Edit. Seems pandas also accepts non-nested translation dictionary. In that case, the problem is probably with character encoding, particularly if you use Python 2. Assuming your CSV load function decoded the file characters properly (as true Unicode code-points), then you should take care your translation/substitution dictionary is also defined with Unicode characters, like this:
dictionary = {u'í': 'i', u'á': 'a'}
If you have a definition like this (and using Python 2):
dictionary = {'í': 'i', 'á': 'a'}
then the actual keys in that dictionary are multibyte strings. Which bytes (characters) they are depends on the actual source file character encoding used, but presuming you use UTF-8, you'll get:
dictionary = {'\xc3\xa1': 'a', '\xc3\xad': 'i'}
And that would explain why pandas fails to replace those chars. So, be sure to use Unicode literals in Python 2: u'this is unicode string'.
On the other hand, in Python 3, all strings are Unicode strings, and you don't have to use the u prefix (in fact unicode type from Python 2 is renamed to str in Python 3, and the old str from Python 2 is now bytes in Python 3).
The docs on pandas.DataFrame.replace says you have to provide a nested dictionary: the first level is the column name for which you have to provide a second dictionary with substitution pairs.
So, this should work:
>>> df=pd.DataFrame({'a': ['NÍCOLAS','asdč'], 'b': [3,4]})
>>> df
a b
0 NÍCOLAS 3
1 asdč 4
>>> df.replace({'a': {'č': 'c', 'Í': 'I'}}, regex=True)
a b
0 NICOLAS 3
1 asdc 4
Edit. Seems pandas also accepts non-nested translation dictionary. In that case, the problem is probably with character encoding, particularly if you use Python 2. Assuming your CSV load function decoded the file characters properly (as true Unicode code-points), then you should take care your translation/substitution dictionary is also defined with Unicode characters, like this:
dictionary = {u'í': 'i', u'á': 'a'}
If you have a definition like this (and using Python 2):
dictionary = {'í': 'i', 'á': 'a'}
then the actual keys in that dictionary are multibyte strings. Which bytes (characters) they are depends on the actual source file character encoding used, but presuming you use UTF-8, you'll get:
dictionary = {'\xc3\xa1': 'a', '\xc3\xad': 'i'}
And that would explain why pandas fails to replace those chars. So, be sure to use Unicode literals in Python 2: u'this is unicode string'.
On the other hand, in Python 3, all strings are Unicode strings, and you don't have to use the u prefix (in fact unicode type from Python 2 is renamed to str in Python 3, and the old str from Python 2 is now bytes in Python 3).
replace works out of the box without specifying a specific column in Python 3.
Load Data:
df=pd.read_csv('test.csv', sep=',', low_memory=False, encoding='iso8859_15')
df
Result:
col1 col2
0 he hello
1 Nícolas shárk
2 welcome yes
Create Dictionary:
dictionary = {'í':'i', 'á':'a'}
Replace:
df.replace(dictionary, regex=True, inplace=True)
Result:
col1 col2
0 he hello
1 Nicolas shark
2 welcome yes
You can use apply with str.replace:
import re
chars = ''.join(map(re.escape, listOfSpecialChars))
df2 = df.apply(lambda c: c.str.replace(f'[{chars}]', '', regex=True))
Alternatively, stack/unstack:
df2 = df.stack().str.replace(f'[{chars}]', '', regex=True).unstack()
output:
col1 col2
0 1 A
1 3 B
2 4 C
## Removes everything except letters, numbers, dash, and underscore.
df['data'] = df['data'].str.replace(r'[^A-Za-z0-9\-\_]+', '')
I'm trying to strip all columns in my dataframe of all special characters and name suffixes like Jr and II.
The replacements I am attempting to loop through aren't working. What am I doing wrong here?
import pandas as pd
dffg = pd.read_csv("fg2.csv")
dfstuff = pd.read_csv("stuffplus.csv")
dfadp = pd.read_csv("adp.csv")
dfzpit = pd.read_csv("ZPitchers.csv")
dfhpit = pd.read_csv("ZHitters.csv")
dflist = [dfhpit, dfzpit, dffg, dfadp, dfstuff]
for index in range(len(dflist)):
dflist[index] = dflist[index].replace(r'[^\w\s]|_\*', '', regex=True).replace(' Jr', '').replace(' II', '')
func = lambda x: ''.join([i[:3] for i in x.strip().split(' ')])
dffg['Key'] = dffg.Name.apply(func)
dfstuff['Key'] = dfstuff.player_name.apply(func)
dfadp['Key'] = dfadp.Player.apply(func)
dfzpit['Key'] = dfzpit.Name.apply(func)
dfhpit['Key'] = dfhpit.Name.apply(func)
dffg.columns = dffg.columns.str.strip()
dfstuff.columns = dfstuff.columns.str.strip()
dfadp.columns = dfadp.columns.str.strip()
dfadp = dfadp.drop(['ESPN','CBS','RTS','NFBC','FT'], axis=1)
df1 = dfadp.merge(dffg, on=["Key"], how="left").merge(dfstuff[['Key', 'STUFFplus', 'LOCATIONplus', 'PITCHINGplus']], on=["Key"], how="left").merge(dfzpit[['Key', 'Total Z-Score']], on=["Key"], how="left").merge(dfhpit[['Key', 'Total Z-Score']], on=["Key"], how="left")
df1 = df1.drop(['Team_y', 'playerid','Name'], axis=1)
df1['Rank'] = df1['Rank'].astype(float)
df1 = df1.fillna('')
df2 = df1.sort_values(by=['Rank'])
df2.to_csv('output.csv')I have a program that loops through each column, and a nested loop that goes through the list of bad characters and converts each column to a str using astype() and then replace(). I also use re.escape() inside my replace method.
I noticed on large data frames of 100k or 500k rows it takes a long time.
I wonder if applying replace() on the entire dataframe is more reliable and faster? Like in the below example:
df = df.replace()
My concern is if replace() will guarantee removing the characters regardless of the data type in each column? I need assurance that the special characters are truly removed from the dataframe. My method right now is reliable but I’ve noticed it’s slow with large data frames so I would like someone with experience to help me refactor if possible.
Something like this?
df.column_name = df.column_name.str.replace(r'["\']', '')
Edit:
Use regex, thanks to @mozway
Another option:
df = pd.DataFrame({"est_soilty_Gh''": [1,2,4],
"upd_siffer_Kh'g": [0,0.2,0.5],
"est_soilty_M'''": [2,3,4]})
est_soilty_Gh'' upd_siffer_Kh'g est_soilty_M'''
0 1 0.0 2
1 2 0.2 3
2 4 0.5 4
df.columns = df.columns.str.replace(r"'", '')
print(df)
est_soilty_Gh upd_siffer_Khg est_soilty_M
0 1 0.0 2
1 2 0.2 3
2 4 0.5 4
You need to escape the $ character using \ in the old_values list:
old_values = ["\${z_mask0}", "\${z_mask1}", "\${z_mask2}"]
The above should be enough. Here is all the code:
old_values = ["\${z_mask0}", "\${z_mask1}", "\${z_mask2}"]
new_values = ["${z_00}", "${z_01}", "${z_02}"]
df = df.replace(old_values, new_values, regex=True)
print(df)
Output:
col1
0 series ${z_00}
1 series ${z_01}
2 series ${z_02}
Have you tried using raw strings for the old_values? There are some RegEx characters in there that may be interfering with your result ("{", "}", and "$"). Try this instead:
old_values = [r"${z_mask0}", r"${z_mask1}", r"${z_mask2}"]
Note the r before each string
I'm assuming you're using Python 2.x here and this is likely a Unicode problem. Don't worry, you're not alone--unicode is really tough in general and especially in Python 2, which is why it's been made standard in Python 3.
If all you're concerned about is the ñ, you should decode in UTF-8, and then just replace the one character.
That would look something like the following:
DF['name'] = DF['name'].str.decode('utf-8').replace(u'\xf1', 'n')
As an example:
>>> "sureño".decode("utf-8").replace(u"\xf1", "n")
u'sureno'
If your string is already Unicode, then you can (and actually have to) skip the decode step:
>>> u"sureño".replace(u"\xf1", "n")
u'sureno'
Note here that u'\xf1' uses the hex escape for the character in question.
Update
I was informed in the comments that <>.str.replace is a pandas series method, which I hadn't realized. The answer to this possibly might be something like the following:
DF['name'] = map(lambda x: x.decode('utf-8').replace(u'\xf1', 'n'), DF['name'].str)
or something along those lines, if that pandas object is iterable.
Another update
It actually just occurred to me that your issue may be as simple as the following:
DF['NAME']=DF['NAME'].str.replace(u"ñ","n")
Note how I've added the u in front of the string to make it unicode.
You can use replace function with special character to be replaced with a different value of your choice in the following way.
if your dataframe is df and you have to do it in all the columns that are string. in case of mine I am doing it for "\n"
df= df.applymap(lambda x: x.replace("\n"," "))