#!/usr/bin/env python
# -*- coding: utf-8 -*-
import unicodedata
text = u'Cześć'
print unicodedata.normalize('NFD', text).encode('ascii', 'ignore')
Answer from nosklo on Stack Overflow#!/usr/bin/env python
# -*- coding: utf-8 -*-
import unicodedata
text = u'Cześć'
print unicodedata.normalize('NFD', text).encode('ascii', 'ignore')
The package unidecode worked best for me:
from unidecode import unidecode
text = "Björn, Łukasz and Σωκράτης."
print(unidecode(text))
# ==> Bjorn, Lukasz and Sokrates.
You might need to install the package:
pip install unidecode
The above solution is easier and more robust than encoding (and decoding) the output of unicodedata.normalize(), as suggested by other answers.
# This doesn't work as expected:
ret = unicodedata.normalize('NFKD', text).encode('ascii', 'ignore')
print(ret)
# ==> b'Bjorn, ukasz and .'
# Besides not supporting all characters, the returned value is a
# bytes object in python3. To yield a str type:
ret = ret.decode("utf8") # (not required in python2)
python - Turn special characters into ascii-like characters or someting else without losing readability - Stack Overflow
Replace special characters in a file with their unicode code (Python) - Stack Overflow
string - Replace special characters in python - Stack Overflow
python - encoding issue. Replace special character - Stack Overflow
How do I remove non-ASCII characters in Python?
How do I remove only selected Unicode symbols?
Why should I avoid removing all Unicode?
For converting to ASCII you might want to try ASCII, Dammit or this recipe, which boils down to:
>>> title = u"Klüft skräms inför på fédéral électoral große"
>>> import unicodedata
>>> unicodedata.normalize('NFKD', title).encode('ascii','ignore')
'Kluft skrams infor pa federal electoral groe'
- Use the
fileinputmodule to loop over standard input or a list of files, - decode the lines you read from UTF-8 to unicode objects
- then map any unicode characters you desire with the
translatemethod
translit.py would look like this:
#!/usr/bin/env python2.6
# -*- coding: utf-8 -*-
import fileinput
table = {
0xe4: u'ae',
ord(u'ö'): u'oe',
ord(u'ü'): u'ue',
ord(u'ß'): None,
}
for line in fileinput.input():
s = line.decode('utf8')
print s.translate(table),
And you could use it like this:
$ cat utf8.txt
sömé täßt
sömé täßt
sömé täßt
$ ./translit.py utf8.txt
soemé taet
soemé taet
soemé taet
- Update:
In case you are using python 3 strings are by default unicode and you dont' need to encode it if it contains non-ASCII characters or even a non-Latin characters. So the solution will look as follow:
line = 'Verhältnismäßigkeit, Möglichkeit'
table = {
ord('ä'): 'ae',
ord('ö'): 'oe',
ord('ü'): 'ue',
ord('ß'): 'ss',
}
line.translate(table)
>>> 'Verhaeltnismaessigkeit, Moeglichkeit'
The data for the .ics file should not be decoded, but passed directly to .from_ical. Use res.content instead. Then Calendar generates the data decoded correctly as UTF-8 (probably part of the .ICS spec) and print can print Unicode strings correctly. For the JSON, write with utf8 encoding and ensure_ascii=False as @JosefZ recommended to see it correctly as well:
import requests
import json
from icalendar import Calendar
url = 'http://www.formula1.com/calendar/Formula_1_Official_Calendar.ics'
res = requests.get(url)
calendar = Calendar.from_ical(res.content)
events = [
{
'id': event['UID'].split('@')[-1].strip(),
'startTime': event['DTSTART'].dt.strftime('%Y-%m-%dT%H:%M:%S.%f')[:-3],
'summary': event['SUMMARY']
} for event in calendar.walk('VEVENT') if str(event['UID']).split('@')[0].startswith('Race')]
for event in events:
print(event['summary'])
with open('events.json', 'w', encoding='utf8') as f:
json.dump(events, f, ensure_ascii=False, indent=2)
print Output:
FORMULA 1 GULF AIR BAHRAIN GRAND PRIX 2021 - Race
FORMULA 1 PIRELLI GRAN PREMIO DEL MADE IN ITALY E DELL'EMILIA ROMAGNA 2021 - Race
FORMULA 1 HEINEKEN GRANDE PRÉMIO DE PORTUGAL 2021 - Race
FORMULA 1 ARAMCO GRAN PREMIO DE ESPAÑA 2021 - Race
FORMULA 1 GRAND PRIX DE MONACO 2021 - Race
FORMULA 1 AZERBAIJAN GRAND PRIX 2021 - Race
FORMULA 1 HEINEKEN GRAND PRIX DU CANADA 2021 - Race
FORMULA 1 EMIRATES GRAND PRIX DE FRANCE 2021 - Race
FORMULA 1 MYWORLD GROSSER PREIS VON ÖSTERREICH 2021 - Race
FORMULA 1 PIRELLI BRITISH GRAND PRIX 2021 - Race
FORMULA 1 MAGYAR NAGYDÍJ 2021 - Race
FORMULA 1 ROLEX BELGIAN GRAND PRIX 2021 - Race
FORMULA 1 HEINEKEN DUTCH GRAND PRIX 2021 - Race
FORMULA 1 HEINEKEN GRAN PREMIO D’ITALIA 2021 - Race
FORMULA 1 VTB RUSSIAN GRAND PRIX 2021 - Race
FORMULA 1 SINGAPORE AIRLINES SINGAPORE GRAND PRIX 2021 - Race
FORMULA 1 JAPANESE GRAND PRIX 2021 - Race
FORMULA 1 ARAMCO UNITED STATES GRAND PRIX 2021 - Race
FORMULA 1 GRAN PREMIO DE LA CIUDAD DE MÉXICO 2021 - Race
FORMULA 1 HEINEKEN GRANDE PRÊMIO DE SÃO PAULO 2021 - Race
FORMULA 1 ROLEX AUSTRALIAN GRAND PRIX 2021 - Race
FORMULA 1 SAUDI ARABIAN GRAND PRIX 2021 - Race
FORMULA 1 ETIHAD AIRWAYS ABU DHABI GRAND PRIX 2021 - Race
events.json:
[
{
"id": "1064",
"startTime": "2021-03-28T16:00:00.000",
"summary": "FORMULA 1 GULF AIR BAHRAIN GRAND PRIX 2021 - Race"
},
{
"id": "1065",
"startTime": "2021-04-18T14:00:00.000",
"summary": "FORMULA 1 PIRELLI GRAN PREMIO DEL MADE IN ITALY E DELL'EMILIA ROMAGNA 2021 - Race"
},
{
"id": "1066",
"startTime": "2021-05-02T15:00:00.000",
"summary": "FORMULA 1 HEINEKEN GRANDE PRÉMIO DE PORTUGAL 2021 - Race"
},
{
"id": "1086",
"startTime": "2021-05-09T14:00:00.000",
"summary": "FORMULA 1 ARAMCO GRAN PREMIO DE ESPAÑA 2021 - Race"
},
{
"id": "1067",
"startTime": "2021-05-23T14:00:00.000",
"summary": "FORMULA 1 GRAND PRIX DE MONACO 2021 - Race"
},
{
"id": "1068",
"startTime": "2021-06-06T13:00:00.000",
"summary": "FORMULA 1 AZERBAIJAN GRAND PRIX 2021 - Race"
},
{
"id": "1069",
"startTime": "2021-06-13T19:00:00.000",
"summary": "FORMULA 1 HEINEKEN GRAND PRIX DU CANADA 2021 - Race"
},
{
"id": "1070",
"startTime": "2021-06-27T14:00:00.000",
"summary": "FORMULA 1 EMIRATES GRAND PRIX DE FRANCE 2021 - Race"
},
{
"id": "1071",
"startTime": "2021-07-04T14:00:00.000",
"summary": "FORMULA 1 MYWORLD GROSSER PREIS VON ÖSTERREICH 2021 - Race"
},
{
"id": "1072",
"startTime": "2021-07-18T15:00:00.000",
"summary": "FORMULA 1 PIRELLI BRITISH GRAND PRIX 2021 - Race"
},
{
"id": "1073",
"startTime": "2021-08-01T14:00:00.000",
"summary": "FORMULA 1 MAGYAR NAGYDÍJ 2021 - Race"
},
{
"id": "1074",
"startTime": "2021-08-29T14:00:00.000",
"summary": "FORMULA 1 ROLEX BELGIAN GRAND PRIX 2021 - Race"
},
{
"id": "1075",
"startTime": "2021-09-05T14:00:00.000",
"summary": "FORMULA 1 HEINEKEN DUTCH GRAND PRIX 2021 - Race"
},
{
"id": "1076",
"startTime": "2021-09-12T14:00:00.000",
"summary": "FORMULA 1 HEINEKEN GRAN PREMIO D’ITALIA 2021 - Race"
},
{
"id": "1077",
"startTime": "2021-09-26T13:00:00.000",
"summary": "FORMULA 1 VTB RUSSIAN GRAND PRIX 2021 - Race"
},
{
"id": "1078",
"startTime": "2021-10-03T13:00:00.000",
"summary": "FORMULA 1 SINGAPORE AIRLINES SINGAPORE GRAND PRIX 2021 - Race"
},
{
"id": "1079",
"startTime": "2021-10-10T06:00:00.000",
"summary": "FORMULA 1 JAPANESE GRAND PRIX 2021 - Race"
},
{
"id": "1080",
"startTime": "2021-10-24T20:00:00.000",
"summary": "FORMULA 1 ARAMCO UNITED STATES GRAND PRIX 2021 - Race"
},
{
"id": "1081",
"startTime": "2021-10-31T19:00:00.000",
"summary": "FORMULA 1 GRAN PREMIO DE LA CIUDAD DE MÉXICO 2021 - Race"
},
{
"id": "1082",
"startTime": "2021-11-07T17:00:00.000",
"summary": "FORMULA 1 HEINEKEN GRANDE PRÊMIO DE SÃO PAULO 2021 - Race"
},
{
"id": "1083",
"startTime": "2021-11-21T06:00:00.000",
"summary": "FORMULA 1 ROLEX AUSTRALIAN GRAND PRIX 2021 - Race"
},
{
"id": "1085",
"startTime": "2021-12-05T16:00:00.000",
"summary": "FORMULA 1 SAUDI ARABIAN GRAND PRIX 2021 - Race"
},
{
"id": "1084",
"startTime": "2021-12-12T13:00:00.000",
"summary": "FORMULA 1 ETIHAD AIRWAYS ABU DHABI GRAND PRIX 2021 - Race"
}
]
with open("events.json", mode="w", encoding="utf-8") as f:
json.dump(events, f, indent=2, ensure_ascii=False)
From json.dump docs:
If
ensure_asciiis true (the default), the output is guaranteed to have all incoming non-ASCII characters escaped. Ifensure_asciiis false, these characters will be output as-is.
Used encoding="utf-8" in open as the default encoding is platform dependent (whatever locale.getpreferredencoding() returns).
If, s=url['title'] makes s equal to this:
In [48]: s=u'Oscar Winners Best Pictures Box Set \xc2\xa36.49'
Then the problem is
- in the code that defines
url, - or else the content from the web is mal-formed.
If Case 1, we'd need to see the code that defines url.
If Case 2, a quick-and-dirty workaround would be to encode the unicode object s with the raw-unicode-escape codec:
In [49]: print(s)
Oscar Winners Best Pictures Box Set £6.49
In [50]: print(s.encode('raw-unicode-escape'))
Oscar Winners Best Pictures Box Set £6.49
See also this SO question.
Regarding titles like s=u'Star Trek XI £3.99': Again, it would be nice fix the problem before it gets to this stage -- perhaps by looking at how url is defined. But assuming the content from the web is mal-formed, a workaround would be:
In [86]: import re
In [87]: print(re.sub(r'&#x([a-fA-F\d]+);',lambda m: unichr(int(m.group(1),base=16)),s))
Star Trek XI £3.99
A little bit of explanation:
Note that
In [51]: x=u'£'
In [53]: x.encode('utf-8')
Out[53]: '\xc2\xa3'
So the unicode object u'£', encoded with the utf-8 codec, becomes the string object '\xc2\xa3'.
Somehow, url['title'] is getting defined to be the unicode object
u'\xc2\xa3'. (The u makes a big difference!)
Thus we have u'\xc2\xa3' when we desire '\xc2\xa3'.
Encoding the unicode object u'\xc2\xa3' with the raw-unicode-escape codec transforms it to '\xc2\xa3'.
Edit: you have your objects already in unicode. Seems to me there is no reason to actually use enocde/decode at all.
>>> print u'Oscar Winners Best Pictures Box Set \xc2\xa36.49'.replace(u'Â','')
Oscar Winners Best Pictures Box Set £6.49
However it seems to me that something is wrong there. The unicode objects are actually not unicode; see:
>>> print 'Oscar Winners Best Pictures Box Set \xc2\xa36.49'.decode('utf8')
Oscar Winners Best Pictures Box Set £6.49
The repr() you posted should not be unicode object. That's why I was asking where are you getting the data, there is something wrong.
Decode the string to Unicode. Assuming it's UTF-8-encoded:
str.decode("utf-8")Call the
replacemethod and be sure to pass it a Unicode string as its first argument:str.decode("utf-8").replace(u"\u2022", "*")Encode back to UTF-8, if needed:
str.decode("utf-8").replace(u"\u2022", "*").encode("utf-8")
(Fortunately, Python 3 puts a stop to this mess. Step 3 should really only be performed just prior to I/O. Also, mind you that calling a string str shadows the built-in type str.)
Encode string as unicode.
>>> special = u"\u2022"
>>> abc = u'ABC•def'
>>> abc.replace(special,'X')
u'ABCXdef'