I have a program that loops through each column, and a nested loop that goes through the list of bad characters and converts each column to a str using astype() and then replace(). I also use re.escape() inside my replace method.
I noticed on large data frames of 100k or 500k rows it takes a long time.
I wonder if applying replace() on the entire dataframe is more reliable and faster? Like in the below example:
df = df.replace()
My concern is if replace() will guarantee removing the characters regardless of the data type in each column? I need assurance that the special characters are truly removed from the dataframe. My method right now is reliable but I’ve noticed it’s slow with large data frames so I would like someone with experience to help me refactor if possible.
Python pandas doesn't recognize special characters - Stack Overflow
Escape special characters in Dataframe Pandas Python - Stack Overflow
python - Escaped quotes in pandas read_csv - Stack Overflow
Saving CSV with backslashed-escaping is not idempotent.
You need to escape the plus sign:
In[10]:
df = pd.DataFrame({'a':['dsa^', '^++', '+++','asdasads']})
df
Out[10]:
a
0 dsa^
1 ^++
2 +++
3 asdasads
In[11]:
df['a'].str.count("\+")
Out[11]:
0 0
1 2
2 3
3 0
Name: a, dtype: int64
Also when you do df['a'].str.count('^') this just returns 1 for all rows:
In[12]:
df['a'].str.count('^')
Out[12]:
0 1
1 1
2 1
3 1
Name: a, dtype: int64
Again you need to escape the pattern:
In[16]:
df['a'].str.count('\^')
Out[16]:
0 1
1 1
2 0
3 0
Name: a, dtype: int64
EDIT
Regarding the semantic difference between count on a normal string and on a Series, count on a python str just does a character count, but str.count takes a regex pattern. The ^ and + are special characters which need to be escaped with a backslash if you are searching for those characters
in str.count() for special characters you need to use backslash for regex patters. (it is explained detaily from @EdChum above).
On the other hand in str.contains() we don't need to use backslash for regex patters. Only need to add regex=False parameter like df['a'].str.contains("+", regex=False)) to search and find the string which include special characters.
Thanks for posting the link to the Google Sheet. I downloaded it and loaded it via pandas:
df = pd.read_excel(r'~\relatorio_vendas_CRM.xlsx', encoding = 'utf-8')
df.columns = df.columns.str.replace('°', '')
df.columns = df.columns.str.replace('º', '')
Note that the two replace statements are replacing different characters, although they look very similar.
Help from: Why do I get a SyntaxError for a Unicode escape in my file path?
I was able to copy the values into another column. You could try that
df['N Pedido'] = df['N° Pedido']
df.drop('N° Pedido',inplace=True,axis=1)
I solved this by using Pandas' regex replacer:
df = df.replace('\\\\', '\\\\\\\\', regex=True)
We need four slashes per final slash, because we are doing two layers of escaping. One for literal Python strings, and one to escape them in the regular expression. This will find-replace any \s in any column in the data frame, anywhere they appear in the string.
It is mind-boggling to me that this is still the default behavior.
Huh. This seems like an open issue with round-tripping data from pandas to csv. See this issue: https://github.com/pandas-dev/pandas/issues/14122, and especially pandas creator Wes McKinney's post:
This behavior is present in the csv module https://gist.github.com/wesm/7763d396ae25c9fd5b27588da27015e4 . From first principles seems like the offending backslash should be escaped. If I manually edit the file to be
"a" "Hello! Please \"help\" me. I cannot quote a csv.\\"then read_csv returns the original input
I fiddled with R and it doesn't seem to do much better
> df <- data.frame(a=c("Hello! Please \"help\" me. I cannot quote a csv.\\"))> write.table(df, sep=',', qmethod='e', row.names=F) "a" "Hello! Please \"help\" me. I cannot quote a csv.\"Another example of CSV not being a high fidelity data interchange tool =|
I'm as baffled as you that this doesn't work, but seems like the official position is... df[col]=df[col].str.replace({"\\": "\\\\"})?
The docs on pandas.DataFrame.replace says you have to provide a nested dictionary: the first level is the column name for which you have to provide a second dictionary with substitution pairs.
So, this should work:
>>> df=pd.DataFrame({'a': ['NÍCOLAS','asdč'], 'b': [3,4]})
>>> df
a b
0 NÍCOLAS 3
1 asdč 4
>>> df.replace({'a': {'č': 'c', 'Í': 'I'}}, regex=True)
a b
0 NICOLAS 3
1 asdc 4
Edit. Seems pandas also accepts non-nested translation dictionary. In that case, the problem is probably with character encoding, particularly if you use Python 2. Assuming your CSV load function decoded the file characters properly (as true Unicode code-points), then you should take care your translation/substitution dictionary is also defined with Unicode characters, like this:
dictionary = {u'í': 'i', u'á': 'a'}
If you have a definition like this (and using Python 2):
dictionary = {'í': 'i', 'á': 'a'}
then the actual keys in that dictionary are multibyte strings. Which bytes (characters) they are depends on the actual source file character encoding used, but presuming you use UTF-8, you'll get:
dictionary = {'\xc3\xa1': 'a', '\xc3\xad': 'i'}
And that would explain why pandas fails to replace those chars. So, be sure to use Unicode literals in Python 2: u'this is unicode string'.
On the other hand, in Python 3, all strings are Unicode strings, and you don't have to use the u prefix (in fact unicode type from Python 2 is renamed to str in Python 3, and the old str from Python 2 is now bytes in Python 3).
replace works out of the box without specifying a specific column in Python 3.
Load Data:
df=pd.read_csv('test.csv', sep=',', low_memory=False, encoding='iso8859_15')
df
Result:
col1 col2
0 he hello
1 Nícolas shárk
2 welcome yes
Create Dictionary:
dictionary = {'í':'i', 'á':'a'}
Replace:
df.replace(dictionary, regex=True, inplace=True)
Result:
col1 col2
0 he hello
1 Nicolas shark
2 welcome yes
Something like this?
df.column_name = df.column_name.str.replace(r'["\']', '')
Edit:
Use regex, thanks to @mozway
Another option:
df = pd.DataFrame({"est_soilty_Gh''": [1,2,4],
"upd_siffer_Kh'g": [0,0.2,0.5],
"est_soilty_M'''": [2,3,4]})
est_soilty_Gh'' upd_siffer_Kh'g est_soilty_M'''
0 1 0.0 2
1 2 0.2 3
2 4 0.5 4
df.columns = df.columns.str.replace(r"'", '')
print(df)
est_soilty_Gh upd_siffer_Khg est_soilty_M
0 1 0.0 2
1 2 0.2 3
2 4 0.5 4