python replace special characters with ascii

Replace special characters with ASCII equivalent

stackoverflow.com › questions › 3194516 › replace-special-characters-with-ascii-equivalent

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import unicodedata
text = u'Cześć'
print unicodedata.normalize('NFD', text).encode('ascii', 'ignore')

Answer from nosklo on Stack Overflow

Stack Overflow

stackoverflow.com › questions › 3194516 › replace-special-characters-with-ascii-equivalent

python - Replace special characters with ASCII equivalent - Stack Overflow

Top answer

1 of 6

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import unicodedata
text = u'Cześć'
print unicodedata.normalize('NFD', text).encode('ascii', 'ignore')

2 of 6

The package unidecode worked best for me:

from unidecode import unidecode
text = "Björn, Łukasz and Σωκράτης."
print(unidecode(text))
# ==> Bjorn, Lukasz and Sokrates.

You might need to install the package:

pip install unidecode

The above solution is easier and more robust than encoding (and decoding) the output of unicodedata.normalize(), as suggested by other answers.

# This doesn't work as expected:
ret = unicodedata.normalize('NFKD', text).encode('ascii', 'ignore')
print(ret)
# ==> b'Bjorn, ukasz and .'
# Besides not supporting all characters, the returned value is a
# bytes object in python3. To yield a str type:
ret = ret.decode("utf8") # (not required in python2)

How to replace unicode characters by ascii characters in Python (perl script given)?

Comprehensive character replacement module in python for non-unicode and non-ascii for HTML

Python convert unicode to ASCII

ASCII conversion in Python

python - How to change back to ascii code

More results from stackoverflow.com

YouTube

youtube.com › hey delphi

PYTHON : Replace special characters with ASCII equivalent - YouTube

01:13

PYTHON : Replace special characters with ASCII equivalentTo Access My Live Chat Page, On Google, Search for "hows tech developer connect"As promised, I'm goi...

Published: May 5, 2023

Views: 21

Discussions

How to replace unicode characters by ascii characters in Python (perl script given)? - Stack Overflow

def make_ascii(string): return ...).encode('ascii','ignore'); ... Find the answer to your question by asking. Ask question ... See similar questions with these tags. ... 0 Python: How to remove the majority of special unicode chars but leave accents and mutated vowels intact? 0 Replace the special characters with the near ... More on stackoverflow.com

stackoverflow.com

python - Turn special characters into ascii-like characters or someting else without losing readability - Stack Overflow

Trying to format data from ics calendar file to any outpu such as json or even python print(). Looking for good ways to replace special characters without losing readability and having ascii-like characters. Examples below. More on stackoverflow.com

stackoverflow.com

May 6, 2021

python how to convert ascii codes to original characters - Stack Overflow

Where are trying to fix it using python. We loop through our data character by character fix the corrupted values. and get an ascii # code. let say I want to replace ascii code 226 with ascii code 146, which is a special quote "’": More on stackoverflow.com

stackoverflow.com

string - Replace special characters in python - Stack Overflow

UnicodeEncodeError: 'ascii' codec can't encode characters in position 100-101: ordinal not in range(128) 2011-01-16T14:33:26.867Z+00:00 ... You should post repr() of the string that this fails for and the particular line (i.e. does it fail on the decode?). 2011-01-16T14:36:43.467Z+00:00 ... Find the answer to your question by asking. Ask question ... See similar questions with ... More on stackoverflow.com

stackoverflow.com

Top answer

1 of 5

For converting to ASCII you might want to try ASCII, Dammit or this recipe, which boils down to:

>>> title = u"Klüft skräms inför på fédéral électoral große"
>>> import unicodedata
>>> unicodedata.normalize('NFKD', title).encode('ascii','ignore')
'Kluft skrams infor pa federal electoral groe'

2 of 5

Use the fileinput module to loop over standard input or a list of files,
decode the lines you read from UTF-8 to unicode objects
then map any unicode characters you desire with the translate method

translit.py would look like this:

#!/usr/bin/env python2.6
# -*- coding: utf-8 -*-

import fileinput

table = {
          0xe4: u'ae',
          ord(u'ö'): u'oe',
          ord(u'ü'): u'ue',
          ord(u'ß'): None,
        }

for line in fileinput.input():
    s = line.decode('utf8')
    print s.translate(table),

And you could use it like this:

$ cat utf8.txt 
sömé täßt
sömé täßt
sömé täßt

$ ./translit.py utf8.txt 
soemé taet
soemé taet
soemé taet

Update:

In case you are using python 3 strings are by default unicode and you dont' need to encode it if it contains non-ASCII characters or even a non-Latin characters. So the solution will look as follow:

line = 'Verhältnismäßigkeit, Möglichkeit'

table = {
         ord('ä'): 'ae',
         ord('ö'): 'oe',
         ord('ü'): 'ue',
         ord('ß'): 'ss',
       }

line.translate(table)

>>> 'Verhaeltnismaessigkeit, Moeglichkeit'

py4u

py4u.org › blog › python-replace-typographical-quotes-dashes-etc-with-their-ascii-counterparts

How to Replace Typographical Quotes, Dashes, and Special Characters with ASCII Counterparts in Python (While Preserving Umlauts and Non-ASCII Text)

Replacing typographical characters with ASCII counterparts while preserving non-ASCII text like umlauts is a critical task for ensuring text compatibility across systems. By using Python’s str.translate() with a custom mapping, you can achieve this efficiently and selectively.

Find elsewhere

Google Bing Mojeek

Scaler

scaler.com › home › topics › remove special characters from string python

Remove Special Characters From String Python - Scaler Topics

January 6, 2024 - The output for the above example will be, The maketrans() method saves the characters as their respective ASCII values. For example, the ASCII value of h and b is 104 and 98 respectively. Let us see how to use this mapping table created using the maketrans() function to remove specific special ...

Python Pool

pythonpool.com › home › tutorials › remove unicode characters in python: ascii, symbols, and normalization

Remove Unicode Characters in Python: ASCII, Symbols, and Normalization

July 13, 2026 - Replacement output is usually a temporary diagnostic format, not the final cleaned value. Encoding with ASCII and an error policy can remove characters that are not representable in ASCII.

Stack Overflow

stackoverflow.com › questions › 67425612 › turn-special-characters-into-ascii-like-characters-or-someting-else-without-losi

python - Turn special characters into ascii-like characters or someting else without losing readability - Stack Overflow

Top answer

1 of 2

The data for the .ics file should not be decoded, but passed directly to .from_ical. Use res.content instead. Then Calendar generates the data decoded correctly as UTF-8 (probably part of the .ICS spec) and print can print Unicode strings correctly. For the JSON, write with utf8 encoding and ensure_ascii=False as @JosefZ recommended to see it correctly as well:

import requests
import json
from icalendar import Calendar

url = 'http://www.formula1.com/calendar/Formula_1_Official_Calendar.ics'
res = requests.get(url)
calendar = Calendar.from_ical(res.content)
events = [
    {
        'id': event['UID'].split('@')[-1].strip(),
        'startTime': event['DTSTART'].dt.strftime('%Y-%m-%dT%H:%M:%S.%f')[:-3],
        'summary': event['SUMMARY']
    } for event in calendar.walk('VEVENT') if str(event['UID']).split('@')[0].startswith('Race')]

for event in events:
    print(event['summary'])

with open('events.json', 'w', encoding='utf8') as f:
    json.dump(events, f, ensure_ascii=False, indent=2)

print Output:

FORMULA 1 GULF AIR BAHRAIN GRAND PRIX 2021 - Race
FORMULA 1 PIRELLI GRAN PREMIO DEL MADE IN ITALY E DELL'EMILIA ROMAGNA 2021 - Race
FORMULA 1 HEINEKEN GRANDE PRÉMIO DE PORTUGAL 2021 - Race
FORMULA 1 ARAMCO GRAN PREMIO DE ESPAÑA 2021 - Race
FORMULA 1 GRAND PRIX DE MONACO 2021 - Race
FORMULA 1 AZERBAIJAN GRAND PRIX 2021 - Race
FORMULA 1 HEINEKEN GRAND PRIX DU CANADA 2021 - Race
FORMULA 1 EMIRATES GRAND PRIX DE FRANCE 2021 - Race
FORMULA 1 MYWORLD GROSSER PREIS VON ÖSTERREICH 2021 - Race
FORMULA 1 PIRELLI BRITISH GRAND PRIX 2021 - Race
FORMULA 1 MAGYAR NAGYDÍJ 2021 - Race
FORMULA 1 ROLEX BELGIAN GRAND PRIX 2021 - Race
FORMULA 1 HEINEKEN DUTCH GRAND PRIX 2021 - Race
FORMULA 1 HEINEKEN GRAN PREMIO D’ITALIA 2021 - Race
FORMULA 1 VTB RUSSIAN GRAND PRIX 2021 - Race
FORMULA 1 SINGAPORE AIRLINES SINGAPORE GRAND PRIX 2021 - Race
FORMULA 1 JAPANESE GRAND PRIX 2021 - Race
FORMULA 1 ARAMCO UNITED STATES GRAND PRIX 2021 - Race
FORMULA 1 GRAN PREMIO DE LA CIUDAD DE MÉXICO 2021 - Race
FORMULA 1 HEINEKEN GRANDE PRÊMIO DE SÃO PAULO 2021 - Race
FORMULA 1 ROLEX AUSTRALIAN GRAND PRIX 2021 - Race
FORMULA 1 SAUDI ARABIAN GRAND PRIX 2021 - Race
FORMULA 1 ETIHAD AIRWAYS ABU DHABI GRAND PRIX 2021 - Race

events.json:

[
  {
    "id": "1064",
    "startTime": "2021-03-28T16:00:00.000",
    "summary": "FORMULA 1 GULF AIR BAHRAIN GRAND PRIX 2021 - Race"
  },
  {
    "id": "1065",
    "startTime": "2021-04-18T14:00:00.000",
    "summary": "FORMULA 1 PIRELLI GRAN PREMIO DEL MADE IN ITALY E DELL'EMILIA ROMAGNA 2021 - Race"
  },
  {
    "id": "1066",
    "startTime": "2021-05-02T15:00:00.000",
    "summary": "FORMULA 1 HEINEKEN GRANDE PRÉMIO DE PORTUGAL 2021 - Race"
  },
  {
    "id": "1086",
    "startTime": "2021-05-09T14:00:00.000",
    "summary": "FORMULA 1 ARAMCO GRAN PREMIO DE ESPAÑA 2021 - Race"
  },
  {
    "id": "1067",
    "startTime": "2021-05-23T14:00:00.000",
    "summary": "FORMULA 1 GRAND PRIX DE MONACO 2021 - Race"
  },
  {
    "id": "1068",
    "startTime": "2021-06-06T13:00:00.000",
    "summary": "FORMULA 1 AZERBAIJAN GRAND PRIX 2021 - Race"
  },
  {
    "id": "1069",
    "startTime": "2021-06-13T19:00:00.000",
    "summary": "FORMULA 1 HEINEKEN GRAND PRIX DU CANADA 2021 - Race"
  },
  {
    "id": "1070",
    "startTime": "2021-06-27T14:00:00.000",
    "summary": "FORMULA 1 EMIRATES GRAND PRIX DE FRANCE 2021 - Race"
  },
  {
    "id": "1071",
    "startTime": "2021-07-04T14:00:00.000",
    "summary": "FORMULA 1 MYWORLD GROSSER PREIS VON ÖSTERREICH 2021 - Race"
  },
  {
    "id": "1072",
    "startTime": "2021-07-18T15:00:00.000",
    "summary": "FORMULA 1 PIRELLI BRITISH GRAND PRIX 2021 - Race"
  },
  {
    "id": "1073",
    "startTime": "2021-08-01T14:00:00.000",
    "summary": "FORMULA 1 MAGYAR NAGYDÍJ 2021 - Race"
  },
  {
    "id": "1074",
    "startTime": "2021-08-29T14:00:00.000",
    "summary": "FORMULA 1 ROLEX BELGIAN GRAND PRIX 2021 - Race"
  },
  {
    "id": "1075",
    "startTime": "2021-09-05T14:00:00.000",
    "summary": "FORMULA 1 HEINEKEN DUTCH GRAND PRIX 2021 - Race"
  },
  {
    "id": "1076",
    "startTime": "2021-09-12T14:00:00.000",
    "summary": "FORMULA 1 HEINEKEN GRAN PREMIO D’ITALIA 2021 - Race"
  },
  {
    "id": "1077",
    "startTime": "2021-09-26T13:00:00.000",
    "summary": "FORMULA 1 VTB RUSSIAN GRAND PRIX 2021 - Race"
  },
  {
    "id": "1078",
    "startTime": "2021-10-03T13:00:00.000",
    "summary": "FORMULA 1 SINGAPORE AIRLINES SINGAPORE GRAND PRIX 2021 - Race"
  },
  {
    "id": "1079",
    "startTime": "2021-10-10T06:00:00.000",
    "summary": "FORMULA 1 JAPANESE GRAND PRIX 2021 - Race"
  },
  {
    "id": "1080",
    "startTime": "2021-10-24T20:00:00.000",
    "summary": "FORMULA 1 ARAMCO UNITED STATES GRAND PRIX 2021 - Race"
  },
  {
    "id": "1081",
    "startTime": "2021-10-31T19:00:00.000",
    "summary": "FORMULA 1 GRAN PREMIO DE LA CIUDAD DE MÉXICO 2021 - Race"
  },
  {
    "id": "1082",
    "startTime": "2021-11-07T17:00:00.000",
    "summary": "FORMULA 1 HEINEKEN GRANDE PRÊMIO DE SÃO PAULO 2021 - Race"
  },
  {
    "id": "1083",
    "startTime": "2021-11-21T06:00:00.000",
    "summary": "FORMULA 1 ROLEX AUSTRALIAN GRAND PRIX 2021 - Race"
  },
  {
    "id": "1085",
    "startTime": "2021-12-05T16:00:00.000",
    "summary": "FORMULA 1 SAUDI ARABIAN GRAND PRIX 2021 - Race"
  },
  {
    "id": "1084",
    "startTime": "2021-12-12T13:00:00.000",
    "summary": "FORMULA 1 ETIHAD AIRWAYS ABU DHABI GRAND PRIX 2021 - Race"
  }
]

2 of 2

with open("events.json", mode="w", encoding="utf-8") as f:
    json.dump(events, f, indent=2, ensure_ascii=False)

From json.dump docs:

If ensure_ascii is true (the default), the output is guaranteed to have all incoming non-ASCII characters escaped. If ensure_ascii is false, these characters will be output as-is.

Used encoding="utf-8" in open as the default encoding is platform dependent (whatever locale.getpreferredencoding() returns).

GitHub

gist.github.com › tushortz › 9fbde5d023c0a0204333267840b592f9

Function to replace some annoying characters · GitHub

LATIN_1_CHARS = ( ('\xe2\x80\x99', "'"), ('\xc3\xa9', 'e'), ('\xe2\x80\x90', '-'), ('\xe2\x80\x91', '-'), ('\xe2\x80\x92', '-'), ('\xe2\x80\x93', '-'), ('\xe2\x80\x94', '-'), ('\xe2\x80\x94', '-'), ('\xe2\x80\x98', "'"), ('\xe2\x80\x9b', "'"), ('\xe2\x80\x9c', '"'), ('\xe2\x80\x9c', '"'), ('\xe2\x80\x9d', '"'), ('\xe2\x80\x9e', '"'), ('\xe2\x80\x9f', '"'), ('\xe2\x80\xa6', '...'), ('\xe2\x80\xb2', "'"), ('\xe2\x80\xb3', "'"), ('\xe2\x80\xb4', "'"), ('\xe2\x80\xb5', "'"), ('\xe2\x80\xb6', "'"), ('\xe2\x80\xb7', "'"), ('\xe2\x81\xba', "+"), ('\xe2\x81\xbb', "-"), ('\xe2\x81\xbc', "="), ('\xe2\x81\xbd', "("), ('\xe2\x81\xbe', ")") ) def clean_latin1(data): try: return data.encode('utf-8') except UnicodeDecodeError: data = data.decode('iso-8859-1') for _hex, _char in LATIN_1_CHARS: data = data.replace(_hex, _char) return data.encode('utf8')

Stack Overflow

stackoverflow.com › questions › 25559617 › python-how-to-convert-ascii-codes-to-original-characters › 25559873

python how to convert ascii codes to original characters - Stack Overflow

Top answer

1 of 2

Instead of the chr function you will want the unichr function:

 >>> c = u'â'
 >>> print(ord(c))
 226
 >>> unichr(226)
 u'\xe2'
 >>> print(unichr(226))
 â

See https://docs.python.org/2/howto/unicode.html

2 of 2

You probably want the chr function. Or is your problem that you don't have the right codepage?

Python Guides

pythonguides.com › remove-unicode-characters-in-python

How to Remove Unicode Characters in Python

September 3, 2025 - I quickly realized that removing ... methods I use. ... One of the simplest ways I remove unwanted Unicode characters is by encoding the string into ASCII and then decoding it back....

Stack Overflow

stackoverflow.com › questions › 4705793 › replace-special-characters-in-python › 4705898

string - Replace special characters in python - Stack Overflow

Top answer

1 of 2

If, s=url['title'] makes s equal to this:

In [48]: s=u'Oscar Winners Best Pictures Box Set \xc2\xa36.49'

Then the problem is

in the code that defines url,
or else the content from the web is mal-formed.

If Case 1, we'd need to see the code that defines url.

If Case 2, a quick-and-dirty workaround would be to encode the unicode object s with the raw-unicode-escape codec:

In [49]: print(s)
Oscar Winners Best Pictures Box Set Â£6.49

In [50]: print(s.encode('raw-unicode-escape'))
Oscar Winners Best Pictures Box Set £6.49

Top answer

1 of 4

I would define a dictionary of special characters (that I want to map) then I use translate method.

line = 'Ich möchte die Qualität des Produkts überprüfen, bevor ich es kaufe.'

special_char_map = {ord('ä'):'ae', ord('ü'):'ue', ord('ö'):'oe', ord('ß'):'ss'}
print(line.translate(special_char_map))

you will get the following result:

Ich moechte die Qualitaet des Produkts ueberpruefen, bevor ich es kaufe.

2 of 4

I think it's easiest and clearer to do it on a more straightforward way, using directly the unicode representation os 'ü' better than unichr(252).

>>> s = u'über'
>>> s.replace(u'ü', 'ue')
u'ueber'

There's no need to use repr, as this will print the 'Python representation' of the string, you just need to present the readable string.

You will need also to include the following line at the beggining of the .py file, in case it's not already present, to tell the encoding of the file

#-*- coding: UTF-8 -*-

Added: Of course, the coding declared must be the same as the encoding of the file. Please check that as can be some problems (I had problems with Eclipse on Windows, for example, as it writes by default the files as cp1252. Also it should be the same encoding of the system, which could be utf-8, or latin-1 or others.

Also, don't use str as the definition of a variable, as it is part of the Python library. You could have problems later.

(I am trying on Python 2.6, I think in Python 2.3 the result is the same)