Use the Unidecode library for this: https://pypi.org/project/Unidecode/, that guarantees a ascii string in return.
Answer from DrummerMann on Stack OverflowUse the Unidecode library for this: https://pypi.org/project/Unidecode/, that guarantees a ascii string in return.
For a web query you probably need to use urlencode
urllib.parse.urlencode(query, doseq=False, safe='', encoding=None, errors=None, quote_via=quote_plus)
or for general character translations the string maketrans method
Python 3.9.5 (default, Nov 18 2021, 16:00:48)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> txt = "Tutu Au Mic' – dumbéa"
>>> mytable = txt.maketrans("é", "e")
>>> print(txt.translate(mytable))
Tutu Au Mic' – dumbea
>>>
python - Replace special characters with ASCII equivalent - Stack Overflow
Special Characters in str.replace()
How can I convert encoding of special characters in python? - Stack Overflow
How to convert special characters to normal text when exporting to file in Python? - Stack Overflow
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import unicodedata
text = u'Cześć'
print unicodedata.normalize('NFD', text).encode('ascii', 'ignore')
The package unidecode worked best for me:
from unidecode import unidecode
text = "Björn, Łukasz and Σωκράτης."
print(unidecode(text))
# ==> Bjorn, Lukasz and Sokrates.
You might need to install the package:
pip install unidecode
The above solution is easier and more robust than encoding (and decoding) the output of unicodedata.normalize(), as suggested by other answers.
# This doesn't work as expected:
ret = unicodedata.normalize('NFKD', text).encode('ascii', 'ignore')
print(ret)
# ==> b'Bjorn, ukasz and .'
# Besides not supporting all characters, the returned value is a
# bytes object in python3. To yield a str type:
ret = ret.decode("utf8") # (not required in python2)
I have the following code.
The aim is to remove all special characters from the column of a DataFrame, although it does not matter if all special characters are removed from the DataFrame.
The code i have used is:
words = combined_body_title.title_body.str.split().explode().str.replace("[?.',)(/:!]","", regex=True)
This works until i put in quotation markets or brackets.
I have read the documentation, from that i think i am using it wrong. I should not be trying to change more than one character within the str.replace, but for some reason it still works just not for brackets and quotation marks.
If you could suggest an alternative solution or help me fix this one i would really appreciate it!
Maybe I don't understand something, but:
>>> print u'\xb0C'.encode("UTF-8")
°C
If by "normal string" you mean ASCII encoded string, then you can't do exactly what you want. The degree symbol is not part of the ASCII character set, so the best you can hope to do is either drop it or convert it to a best approximation character from the ASCII character set. You could choose a different encoding, however you have to be sure that whatever systems you are interacting with will work with the encoding you choose. UTF-8 is usually a safe bet, and can encode pretty much any character you'll ever likely run in to.