See unicodedata.normalize

title = u"Klüft skräms inför på fédéral électoral große"
import unicodedata
unicodedata.normalize('NFKD', title).encode('ascii', 'ignore')
'Kluft skrams infor pa federal electoral groe'
Answer from Sorantis on Stack Overflow
🌐
Python documentation
docs.python.org › 3 › howto › unicode.html
Unicode HOWTO — Python 3.14.7 documentation
To summarize the previous section: a Unicode string is a sequence of code points, which are numbers from 0 through 0x10FFFF (1,114,111 decimal). This sequence of code points needs to be represented in memory as a set of code units, and code units are then mapped to 8-bit bytes.
Discussions

Explain it like I'm five: Python and Unicode?
There are two types of strings in python: byte strings and unicode strings. Each element in a byte string is a byte. There are only 256 possible bytes. Each element in a unicode string is a character (also called a unicode code point). There are a little over a million characters defined in unicode. Meaning each element/character in a unicode string can be one of those million characters. Byte strings are useful because you can write them to files, transmit them over the network, etc. Unicode strings are useful because you can store pretty much any character that exists. So people usually like to manipulate unicode strings in their programs. But how do you convert a unicode string to a byte string? You encode it. An encoding is a representation of a unicode string. It defines a byte or byte sequence for every* unicode code point; essentially a translation table. For every unicode code point, there is a byte or sequence of bytes. There's more to it than that, but those are the essential bits you need to know. What this means when you're writing a program is that you want to manipulate unicode strings throughout, and when you want to output a string (to a file, or over the network), you encode it. When you read in a byte string from external sources, you decode it. Does that make sense? *some encodings may not support every unicode character; they may only support some subset of unicode. UTF-8 is nice because it supports everything. It defines a sequence of bytes for every unicode character. More on reddit.com
🌐 r/Python
60
106
June 12, 2013
python - What is a unicode string? - Stack Overflow
What exactly is a unicode string? What's the difference between a regular string and unicode string? What is utf-8? I'm trying to learn Python right now, and I keep hearing this buzzword. What ... More on stackoverflow.com
🌐 stackoverflow.com
How to convert a string to utf-8 in Python - Stack Overflow
NOTE: The string passed from the web is already UTF-8 encoded, I just want to make Python to treat it as UTF-8 not ASCII. ... Save this answer. ... Show activity on this post. >>> plain_string = "Hi!" >>> unicode_string = u"Hi!" >>> type(plain_string), type(unicode_string) (, More on stackoverflow.com
🌐 stackoverflow.com
Replacing literal '\u****' in string with corresponding Unicode character
Maybe you have to do this: s = 'blah\\x2Ddude' s.encode().decode('unicode-escape') print(s) 'blah-dude' There's going to be some encoding or decoding that'll make it prettier. The unicode-escape codec can transform embedded Unicode escapes. The string needs to be a byte string, however, hence the .encode() first. (from google) More on reddit.com
🌐 r/learnpython
4
8
March 13, 2020
🌐
B-List
b-list.org › weblog › 2017 › sep › 05 › how-python-does-unicode
How Python does Unicode
September 5, 2017 - To create a str in Python 2, you can use the str() built-in, or string-literal syntax, like so: my_string = 'This is my string.'. To create an instance of unicode, you can use the unicode() built-in, or prefix a string literal with a u, like so: my_unicode = u'This is my Unicode string.'.
🌐
GeeksforGeeks
geeksforgeeks.org › python › python-convert-string-to-unicode-characters
Python - Convert String to unicode characters - GeeksforGeeks
July 15, 2025 - These Unicode values are appended to the unicode list, resulting in [104, 101, 108, 108, 111] for the string "hello".
🌐
Akamai
akamai.com › cloud › guides › how-to-use-unicode-in-python3
Using Unicode in Python 3 | Linode Docs
March 20, 2023 - For example, the Unicode bumblebee emoji is encoded in a three-byte format possessing the Unicode code point U+1F41D. To assign this emoji character to a variable using its escape sequence, pad it out to 0001F41D. After the character is assigned to a string, it can be printed out using the Python print function.
🌐
Reddit
reddit.com › r/python › explain it like i'm five: python and unicode?
r/Python on Reddit: Explain it like I'm five: Python and Unicode?
June 12, 2013 -

I am seriously confused. And whenever I think I got it, I see some - in my opinion - inconsistent behavior. Can it be consistently explained or is it more art than science?

When do I have to encode/decode("UTF-8")? What does it do exactly? Whats so special about unicode("abc"), or is it identical to u"abc"?

Why, if I'm using a HTML-encoding of UTF8, a python-script with encoding-UTF-8 and a UTF-8 capable shell and have them all interact, do I have to randomly start adding the above functions until stuff accidentally doesn't break anymore? :)

My problem is that while I can code quite well, I have no formal computer science education and don't tend to think in bytes.

Top answer
1 of 5
83
There are two types of strings in python: byte strings and unicode strings. Each element in a byte string is a byte. There are only 256 possible bytes. Each element in a unicode string is a character (also called a unicode code point). There are a little over a million characters defined in unicode. Meaning each element/character in a unicode string can be one of those million characters. Byte strings are useful because you can write them to files, transmit them over the network, etc. Unicode strings are useful because you can store pretty much any character that exists. So people usually like to manipulate unicode strings in their programs. But how do you convert a unicode string to a byte string? You encode it. An encoding is a representation of a unicode string. It defines a byte or byte sequence for every* unicode code point; essentially a translation table. For every unicode code point, there is a byte or sequence of bytes. There's more to it than that, but those are the essential bits you need to know. What this means when you're writing a program is that you want to manipulate unicode strings throughout, and when you want to output a string (to a file, or over the network), you encode it. When you read in a byte string from external sources, you decode it. Does that make sense? *some encodings may not support every unicode character; they may only support some subset of unicode. UTF-8 is nice because it supports everything. It defines a sequence of bytes for every unicode character.
2 of 5
22
To answer your specific questions: when you encode("UTF-8") you are converting a unicode string to a byte string. It should be called on unicode strings. When you decode("UTF-8") you are converting a byte string to a unicode string. It should be called on byte strings. unicode("abc") is the same as u"abc": they both create a unicode string with three characters. Most of the confusion comes from the fact that python 2 plays fast and loose with unicode strings. It will try and convert between them for you when you mix them together, which yields unexpected results. Python 3 has much more sane behavior: it forces you to encode or decode explicitly to convert between the two. Basically what you need to do to avoid most problems and confusion is to do your encoding/decoding at the input/output boundaries of your program. Decode as soon as you get a byte string from external sources, use unicode strings throughout the program, and encode it just before it leaves.
Find elsewhere
🌐
O'Reilly
oreilly.com › library › view › python-cookbook › 0596001673 › ch03s18.html
Converting Between Unicode and Plain Strings - Python Cookbook [Book]
July 19, 2002 - Unicode strings can be encoded in plain strings in a variety of ways, according to whichever encoding you choose: # Convert Unicode to plain Python string: "encode" unicodestring = u"Hello world" utf8string = unicodestring.encode("utf-8") asciistring = unicodestring.encode("ascii") isostring = unicodestring.encode("ISO-8859-1") utf16string = unicodestring.encode("utf-16") # Convert plain Python string to Unicode: "decode" plainstring1 = unicode(utf8string, "utf-8") plainstring2 = unicode(asciistring, "ascii") plainstring3 = unicode(isostring, "ISO-8859-1") plainstring4 = unicode(utf16string, "utf-16") assert plainstring1==plainstring2==plainstring3==plainstring4
Authors: Alex MartelliDavid Ascher
Published: 2002
Pages: 608
🌐
Note.nkmk.me
note.nkmk.me › home › python
Convert Between Unicode Code Point and Character: chr, ord | note.nkmk.me
January 31, 2024 - To convert an integer to a hexadecimal string, use the built-in hex() function. s = hex(i) print(s) # 0x41 print(type(s)) # <class 'str'> ... The built-in format() function can be used for more detailed formatting, such as zero-padding and including ...
🌐
DigitalOcean
digitalocean.com › community › tutorials › how-to-work-with-unicode-in-python
How To Work with Unicode in Python | DigitalOcean
The tutorial will cover the basics of Unicode in Python and how Python interprets Unicode characters. It covers the concepts of unicodedata and how to use th…
🌐
Python
docs.python.org › 3 › c-api › unicode.html
Unicode Objects and Codecs — Python 3.14.7 documentation
Return a void pointer to the raw Unicode buffer. unicode has to be a Unicode object in the “canonical” representation (not checked). Added in version 3.3. void PyUnicode_WRITE(int kind, void *data, Py_ssize_t index, Py_UCS4 value)¶ · Write the code point value to the given zero-based index in a string.
🌐
GeeksforGeeks
geeksforgeeks.org › python › convert-unicode-string-to-a-string-in-python
Convert Unicode String to a Byte String in Python - GeeksforGeeks
July 23, 2025 - In this example, the Unicode string is encoded into a byte string using UTF-16 encoding, resulting in a sequence of bytes that represents the mixed-language string. The byte string is then printed to demonstrate the UTF-16 encoded representation of the Unicode characters.
🌐
AskPython
askpython.com › python › string › converting-unicode-strings-to-regular-strings
Converting Unicode Strings to Regular Strings in Python - AskPython
March 30, 2023 - Although most Unicode and ASCII encoding and decoding happen behind the scenes, it’s essential to understand the mechanisms and rules for converting characters to their Unicode counterparts. In this tutorial, we’ve demonstrated how to convert Unicode strings to regular strings in Python with ease.
Top answer
1 of 2
61

Update: Python 3

In Python 3, Unicode strings are the default. The type str is a collection of Unicode code points, and the type bytes is used for representing collections of 8-bit integers (often interpreted as ASCII characters).

Here is the code from the question, updated for Python 3:

>>> my_str = 'A unicode \u018e string \xf1' # no need for "u" prefix
# the escape sequence "\u" denotes a Unicode code point (in hex)
>>> my_str
'A unicode Ǝ string ñ'
# the Unicode code points U+018E and U+00F1 were displayed
# as their corresponding glyphs
>>> my_bytes = my_str.encode('utf-8') # convert to a bytes object
>>> my_bytes
b'A unicode \xc6\x8e string \xc3\xb1'
# the "b" prefix means a bytes literal
# the escape sequence "\x" denotes a byte using its hex value
# the code points U+018E and U+00F1 were encoded as 2-byte sequences
>>> my_str2 = my_bytes.decode('utf-8') # convert back to str
>>> my_str2 == my_str
True

Working with files:

>>> f = open('foo.txt', 'r') # text mode (Unicode)
>>> # the platform's default encoding (e.g. UTF-8) is used to decode the file
>>> # to set a specific encoding, use open('foo.txt', 'r', encoding="...")
>>> for line in f:
>>>     # here line is a str object

>>> f = open('foo.txt', 'rb') # "b" means binary mode (bytes)
>>> for line in f:
>>>     # here line is a bytes object

Historical answer: Python 2

In Python 2, the str type was a collection of 8-bit characters (like Python 3's bytes type). The English alphabet can be represented using these 8-bit characters, but symbols such as Ω, и, ±, and ♠ cannot.

Unicode is a standard for working with a wide range of characters. Each symbol has a code point (a number), and these code points can be encoded (converted to a sequence of bytes) using a variety of encodings.

UTF-8 is one such encoding. The low code points are encoded using a single byte, and higher code points are encoded as sequences of bytes.

To allow working with Unicode characters, Python 2 has a unicode type which is a collection of Unicode code points (like Python 3's str type). The line ustring = u'A unicode \u018e string \xf1' creates a Unicode string with 20 characters.

When the Python interpreter displays the value of ustring, it escapes two of the characters (Ǝ and ñ) because they are not in the standard printable range.

The line s = unistring.encode('utf-8') encodes the Unicode string using UTF-8. This converts each code point to the appropriate byte or sequence of bytes. The result is a collection of bytes, which is returned as a str. The size of s is 22 bytes, because two of the characters have high code points and are encoded as a sequence of two bytes rather than a single byte.

When the Python interpreter displays the value of s, it escapes four bytes that are not in the printable range (\xc6, \x8e, \xc3, and \xb1). The two pairs of bytes are not treated as single characters like before because s is of type str, not unicode.

The line t = unicode(s, 'utf-8') does the opposite of encode(). It reconstructs the original code points by looking at the bytes of s and parsing byte sequences. The result is a Unicode string.

The call to codecs.open() specifies utf-8 as the encoding, which tells Python to interpret the contents of the file (a collection of bytes) as a Unicode string that has been encoded using UTF-8.

2 of 2
-5

Python supports the string type and the unicode type. A string is a sequence of chars while a unicode is a sequence of "pointers". The unicode is an in-memory representation of the sequence and every symbol on it is not a char but a number (in hex format) intended to select a char in a map. So a unicode var does not have encoding because it does not contain chars.

🌐
CodeSignal
codesignal.com › learn › courses › string-manipulation-for-python-coders › lessons › navigating-the-universe-of-python-unicode-encoding-and-decoding-strings-explained
Unicode, Encoding, and Decoding Strings Explained
str2 = b.decode('UTF-16') # Decoding the string print("Decoded string: ", str2) # Prints: 'Hello from Mars!' Unicode supports multiple scripts, thus allowing Python to efficiently handle non-English characters. This feature can be particularly beneficial when communicating with astronauts from various countries onboard the Mars mission.
🌐
r12a
r12a.github.io › app-conversion
Unicode code converter
Note, however, that this (unusually convoluted) text represents two characters using just a code point number – therefore, for this particular example, you should select Treat bare numbers as Hex code points (or decimal) to convert those numbers as well as the other escapes.
🌐
GeeksforGeeks
geeksforgeeks.org › python › convert-a-string-to-utf-8-in-python
Convert a String to Utf-8 in Python - GeeksforGeeks
July 23, 2025 - Unicode Transformation Format 8 (UTF-8) is a widely used character encoding that represents each character in a string using variable-length byte sequences. In Python, converting a string to UTF-8 is a common task, and there are several simple methods to achieve this.
🌐
Medium
medium.com › data-science › byte-string-unicode-string-raw-string-a-guide-to-all-strings-in-python-684c4c4960ba
Byte string, Unicode string, Raw string — A Guide to all strings in Python | by Guangyuan(Frank) Li | TDS Archive | Medium
November 23, 2022 - With the basic concepts understood, let’s cover some practical coding tips in Python. In Python3, the default string is called Unicode string (u string), you can understand them as human-readable characters. As explained above, you can encode them to the byte string (b string), and the byte string can be decoded back to the Unicode string.